Reliability and data gaps

Reliable market-data jobs do not assume every source is continuous. They check the market, window, and data family before using results.

Pattern

Failure Behavior

Signal	Client behavior
Missing coverage	Do not fill with synthetic data
Stale freshness	Delay, narrow the window, or mark output as stale
Incident overlap	Store incident metadata with the job
Replay gap	Rebuild from a safe snapshot or stop the reconstruction
5xx or timeout	Retry with jitter, then check status and incidents

Reliability Contract

Define the output status for every job. A result can be complete, stale, partial, skipped, retried, or failed. Do not leave the status implicit. That matters for research notebooks, generated dashboards, downstream agents, and backtests because they may otherwise treat partial market data as clean.

Gap Handling

When a gap appears, stop using the affected local state until it is rebuilt. For REST history, rerun the smallest window that contains the issue. For replay, restart from a safe checkpoint or snapshot. For local books, discard unsafe state and resync before applying more diffs.

Human And Agent Use

Humans should see the gap state in logs or UI. Coding agents should be instructed not to interpolate missing records unless the user explicitly asks for a transformation. The default behavior is to preserve truth, mark the output, and point to the data-quality route that justified the decision.

Before You Commit To A Data Window

When evaluating a historical data source, verify how coverage, freshness, incidents, replay gaps, and request IDs are represented before the data touches research notebooks, dashboards, alerts, or models. Those answers define whether the data is usable, not just available.

Quality Status Per Dataset

Every durable dataset should carry a quality status: clean, stale, partial, interrupted, or skipped. Store the data-quality response and any request IDs that produced that status alongside the output. That makes later charts, models, and reports auditable without re-running the job.

​Pattern

​Failure Behavior

​Reliability Contract

​Gap Handling

​Human And Agent Use

​Before You Commit To A Data Window

​Quality Status Per Dataset