On this page

06. How do you handle rate-limiting or large payloads when consuming AI services?

📄 Noteshard#ai#rate-limiting#performance#scalability

Short answer

For rate limits, respect the provider's 429 responses and Retry-After headers with exponential backoff and jitter, queue requests with a concurrency limit on your backend, enforce your own per-user quotas, and debounce or disable UI triggers on the client. For large payloads, shrink what you send (chunk documents, summarize or trim history, use embeddings/RAG instead of pasting whole files), stream responses back, and move long jobs to background workers with polling or webhooks.

Explanation

An AI API is like a busy restaurant kitchen with a fixed number of chefs. If you send 500 orders at once, most get turned away (429). You need a host at the door (queue), a rule on how many orders per table (quota), and smaller orders (trimmed prompts) so each one is cooked faster.

Rate limiting

WhereTechnique
ClientDebounce inputs, disable Send while streaming, cancel stale requests
Your backendPer-user / per-IP limits (token bucket in Redis), concurrency queue
Provider callsRetry 429/503 with backoff + jitter, honor Retry-After, cap attempts
Cost controlTrack token usage per user, cache identical prompts, choose smaller models for simple tasks

Provider limits are usually both requests per minute and tokens per minute, so a few huge prompts can hit the limit as easily as many small ones.

Large payloads

  • Input: chunk large documents, embed them, and retrieve only relevant chunks (RAG). Trim or summarize old chat history to fit the context window.
  • Uploads: send files to object storage (pre-signed URL) and pass a reference to the backend, rather than base64 in JSON.
  • Output: stream tokens so users see progress; set max_tokens to bound cost.
  • Long jobs (transcription, batch analysis): return a job ID immediately, process in a worker queue, notify via polling, SSE or webhook.
  • Compression: gzip/brotli on your own API responses.

Example

Backend retry with backoff:

async function callWithRetry(fn: () => Promise<Response>, maxAttempts = 4) {
  for (let attempt = 0; ; attempt++) {
    const res = await fn();
    if (res.status !== 429 && res.status < 500) return res;
    if (attempt + 1 >= maxAttempts) return res;

    const retryAfter = Number(res.headers.get("retry-after"));
    const delay = retryAfter > 0
      ? retryAfter * 1000
      : Math.min(1000 * 2 ** attempt, 16_000) + Math.random() * 250; // jitter
    await new Promise((r) => setTimeout(r, delay));
  }
}

Client side: one request at a time, and polling for long jobs:

import { useEffect, useState } from "react";

export function useJobStatus(jobId) {
  const [job, setJob] = useState({ status: "queued" });

  useEffect(() => {
    if (!jobId) return;
    let timer;
    let cancelled = false;

    async function poll() {
      const res = await fetch(`/api/jobs/${jobId}`);
      const data = await res.json();
      if (cancelled) return;
      setJob(data);
      if (data.status === "queued" || data.status === "running") timer = setTimeout(poll, 2000);
    }
    poll();

    return () => { cancelled = true; clearTimeout(timer); };
  }, [jobId]);

  return job;
}

Common mistakes

  • Retrying immediately in a tight loop, which makes the 429 storm worse.
  • Retrying non-idempotent or 4xx errors (bad request, context too long) that will never succeed.
  • Sending the entire chat history every time until it exceeds the context window.
  • Holding an HTTP request open for minutes on a serverless function with a 10-60 second timeout.

Follow-up questions

  • How do you show rate limits to users? Return a clear error code from your API and show "You've hit today's limit, resets at X" instead of a generic failure.
  • How do you estimate token size before sending? Use the provider's tokenizer library (e.g. tiktoken) on the server to count and trim.
  • Should you cache AI responses? For deterministic, repeated prompts (like summarizing the same document), yes, keyed by a hash of model + prompt.

Quick recap

  • Backoff with jitter, honor Retry-After, cap retries.
  • Queue and quota on your backend; debounce on the client.
  • Shrink inputs: chunking, RAG, trimmed history.
  • Stream outputs; use background jobs for long tasks.