We launched an AI feature to 10k users — here's what broke

Postmortem-style writeup of launching an LLM-backed feature. The model wasn't the problem — rate limiting, streaming backpressure, and prompt injection were. Sharing fixes and the monitoring we wish we'd had on day one.

1. The Setup

We built a smart autonomous triage system for our developer workspace. In staging with 50 internal users, everything felt instantaneous and bulletproof. Latencies were ~300ms to first token and memory usage was flat.

2. What Exploded at 10k Users

1. API Rate Limiting: Our upstream providers hard-capped our concurrency during peak US hours. Users were greeted with cascading 429s.
2. Streaming Backpressure: When client mobile connections fluctuated, our Server-Sent Events (SSE) channels backed up, consuming node buffers and inflating memory by 400%.
3. Prompt Injection In The Wild: Real users submitted system-override test prompts within the first 2 hours of launch.

// The throttling fix we implemented
export async function createSafeStream(reader: ReadableStreamDefaultReader) {
  const backpressureTimeout = 15000;
  // Dynamic chunk buffer with client ping backpressure
  return new ReadableStream({
    async pull(controller) {
      const { done, value } = await reader.read();
      if (done) {
        controller.close();
        return;
      }
      controller.enqueue(value);
    }
  });
}

3. Key Solutions & Recommendations

• Enforce strict token-bucket rate limiting before queries reach the model provider.
• Implement explicit backpressure drains on server socket connections.
• Put prompt guardrails in place prior to launch.