Reliability and data gaps

Outcome

Classify one bounded data window as complete, stale, partial, interrupted, skipped, or failed, with a recorded reason and next action.

Prerequisites

Choose one venue family, symbol, data family, and bounded window.
Define what the downstream job does when a quality check fails.
Set a finite retry and recovery budget.
Prepare an output record that can hold quality responses and request IDs.

Inputs

Field	Example
Venue family	Hyperliquid core
Symbol	`BTC`
Data family	Trades
Window	`from=1773273600000&to=1773360000000`
Quality routes	`/v1/data-quality/status` and `/v1/data-quality/coverage/hyperliquid/BTC`
Completion condition	A named quality state and client action are saved beside the output

Steps

Check the service

curl -sS "https://api.0xarchive.io/v1/data-quality/status" \
  -H "X-API-Key: $OXARCHIVE_API_KEY"

Record the resource-specific status and active-incident fields.

Check the selected window

curl -sS "https://api.0xarchive.io/v1/data-quality/coverage/hyperliquid/BTC?from=1773273600000&to=1773360000000" \
  -H "X-API-Key: $OXARCHIVE_API_KEY"

Keep the coverage response for the same family, symbol, and data family as the downstream request.

Assign one state

Mark the output complete only when the requested data and quality checks meet the job policy. Otherwise mark stale, partial, interrupted, skipped, or failed with a reason.

Apply the state

Continue, narrow, delay, retry, rebuild, or stop according to the recorded state. Never hide a quality failure by changing the label to complete.

Expected state

The output has a quality state, the exact family and window, the quality responses, the data request metadata, and the client action. A missing window is not converted into synthetic records.

Verification

Use coverage for the selected market and window, freshness for the selected data family, and incidents for interruptions. A service-level status is a useful preflight but does not decide whether a particular symbol and schema window is usable. Keep the quality request identifier when the route or client exposes one.

Failure and recovery

Signal	Action
Missing coverage	Skip or narrow the window; do not add synthetic rows
Stale freshness	Delay, mark stale, or stop the downstream job
Incident overlap	Record the incident and mark the affected output
Replay gap	Restart from the documented checkpoint or stop reconstruction
`429`	Honor `Retry-After` when present, otherwise back off with jitter and lower concurrency
`5xx` or timeout	Retry within the finite budget, then check status and incidents

For local order-book state, discard unsafe state and resync from a checkpoint before applying more ordered diffs. For REST history, rerun only the smallest window that contains the issue.

Saved run metadata

{
  "run": "btc-trades-quality-state-001",
  "venue_family": "hyperliquid_core",
  "symbol": "BTC",
  "data_family": "trades",
  "window": {
    "from": 1773273600000,
    "to": 1773360000000
  },
  "status_route": "/v1/data-quality/status",
  "coverage_route": "/v1/data-quality/coverage/hyperliquid/BTC",
  "data_route": "/v1/hyperliquid/trades/BTC",
  "request_ids": ["00000000-0000-0000-0000-000000000000"],
  "quality_state": "complete | stale | partial | interrupted | skipped | failed",
  "action": "record continue, narrow, delay, retry, rebuild, or stop",
  "output": "./artifacts/btc-trades-quality-state-001.json"
}

The zero UUID is a shape placeholder only. Replace it with each returned UUID or client request handle.

Next task

Use Monitor freshness before use for lag thresholds, Historical market data for bounded retrieval, or Point-In-Time Backtesting for a research manifest.

​Outcome

​Prerequisites

​Inputs

​Steps

​Expected state

​Verification

​Failure and recovery

​Saved run metadata

​Next task