> ## Documentation Index
> Fetch the complete documentation index at: https://docs.0xarchive.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Limits and throughput

> Design 0xArchive jobs around rate limits, concurrency, credits, access-gated routes, pagination, retries, and throughput planning.

Throughput is a client-design problem. A stable job controls concurrency, bounds windows, retries safely, and logs request IDs.

<Steps>
  <Step title="Classify route cost">
    Identify whether the route is a shallow market-state call, historical list, L3/L4 depth call, replay stream, or data-quality aggregate.
  </Step>

  <Step title="Limit workers">
    Keep concurrent workers below the account envelope and adjust by response behavior.
  </Step>

  <Step title="Page historical windows">
    Use cursors and bounded time ranges instead of one oversized request.
  </Step>

  <Step title="Back off on 429">
    Apply capped backoff with jitter and avoid retry storms.
  </Step>
</Steps>

<CardGroup cols={2}>
  <Card title="Rate limits" icon="gauge" href="/rate-limits">
    Account controls and client behavior.
  </Card>

  <Card title="Errors" icon="triangle-alert" href="/errors">
    Retry and fail-fast rules.
  </Card>
</CardGroup>

## Worker Model

Use a bounded worker pool. Each worker should own one route family, one symbol or small symbol set, and one time window at a time. That structure makes it possible to retry a failed page without duplicating the whole job or losing which request ID belongs to which output.

## Backoff Model

Use capped exponential backoff with jitter for `429`, transient `5xx`, and network timeouts. Do not retry validation errors, missing auth, unsupported route families, or access-gated requests without changing the input. A fast fail is better than an expensive loop that repeats the same invalid request.

## Measurement

Track request count, returned row count, cursor count, latency, retry count, and request IDs. If a job moves from development to production, add a data-quality preflight and a limit dashboard before increasing symbol coverage.

## Throughput Plan Checklist

Create this checklist before running broad history pulls, replay jobs, generated scripts, or high-concurrency clients.

| Field        | Capture                                                                         |
| ------------ | ------------------------------------------------------------------------------- |
| Route family | Hyperliquid core, Spot, HIP-3, HIP-4, Lighter, data quality, or export workflow |
| Workload     | Symbol set, data family, time window, cursor behavior, and expected row count   |
| Concurrency  | Worker count, queue depth, per-route limits, and account envelope               |
| Budget       | Request budget, credit budget, retry budget, and stop condition                 |
| Backoff      | Which statuses retry, maximum attempts, jitter behavior, and cool-down window   |
| Logging      | Request IDs, cursor IDs, output path, row counts, latency, and skipped windows  |
| Freshness    | Data-quality preflight, tolerated lag, and what happens when freshness fails    |

This checklist should be visible before a coding agent, cron job, notebook, or backend worker runs a wide loop. If a row is unknown, keep the job narrow until the missing limit behavior is understood.

## Evaluation Fit

Throughput is usually a sign that the first workflow is working and the next bottleneck is operational discipline. Before buying more capacity, confirm that the job batches by venue family, uses cursors, preserves request IDs, backs off on 429s, and checks data quality before writing downstream artifacts.

For exports, throughput questions often belong in [Data catalog](/data-catalog), [Export schemas](/export-schemas), and [Enterprise delivery](/enterprise-delivery), not in a wider REST loop.

## Agent Safety

When a coding agent writes a data puller, ask it to show concurrency, retry, and pagination choices before running the job. Do not let an agent generate an unbounded loop across symbols or time windows without rate behavior and stop conditions in the script.

Small limits bugs become expensive quickly. Make the stop condition and worker count visible in code, logs, and job output so later failures can be traced.