Zero-Latency Multi-Provider Fallback Routing Strategies with API100

When building AI agent applications, upstream rate limits and provider outages can completely stall user workflows. We built a resilient router with OpenRouter and API100 that provides automatic hedged requests and streaming failovers.

Hedged Request Strategy

If upstream provider A does not return the first chunk within 400ms, a secondary stream is immediately triggered to provider B. Whichever responds first claims the client SSE pipe while the other is cancelled.

export async function streamWithFallback(prompt: string) {
  const controller = new AbortController();
  const primaryPromise = fetch('/api/v1/chat/completions', { ... });
  const timeoutPromise = new Promise((_, reject) => 
    setTimeout(() => reject(new Error('SLO_BREACH')), 450)
  );

  return Promise.race([primaryPromise, timeoutPromise])
    .catch(() => triggerSecondaryProvider(prompt, controller.signal));
}

Our availability increased from 99.1% to 99.98% over the past 30 days.