How we setup zero-downtime multi-provider failover routing with latency scoring

Provider outages and rate-limits happen daily. We implemented a dynamic sliding-window health score that automatically reroutes LLM inference calls between Anthropic, OpenAI, DeepSeek, and Google without dropping WebSocket client streams.

export async function routeInference(prompt: string) {
  const providers = await getHealthyProvidersByScore();
  for (const provider of providers) {
    try {
      return await provider.stream(prompt);
    } catch (err) {
      recordFailure(provider.id, err);
    }
  }
}