Market Data Replay and Tick Databases: Reconstructing the Book from PCAP and Explaining the Backtest-to-Live Gap

Closing the Backtest-to-Live Gap Through High-Fidelity Market-Data Replay

A backtest that looks excellent on historical data can still lose money the moment it goes live. The gap is rarely caused by a single dramatic bug. It usually arises from a collection of small differences between the data the research system saw and the data, timing and constraints the live system actually experiences. Closing that gap begins with high-fidelity market-data capture, faithful order-book reconstruction, and disciplined replay.

PCAP-level captures and properly engineered tick databases give quant and engineering teams the raw material to reproduce exchange behavior with far higher fidelity than aggregated bars or even conventional tick feeds. The remaining differences must then be measured, attributed and controlled. Accurate market-data replay from PCAP, combined with deterministic order-book reconstruction and explicit modeling of live constraints, is the practical foundation for understanding and shrinking the backtest-to-live performance gap.

Why Aggregated or Normalized Data Is Not Enough

Bar data and even many commercial tick feeds discard the precise sequence, packet timing and microstructure that drive short-horizon strategies. Queue position, flickering liquidity, auction imbalance messages and the exact inter-arrival times of updates all disappear or are approximated. A strategy that appears robust on such data may be reacting to artifacts that never exist in the live feed, or missing signals that only appear in the raw stream.

PCAP captures taken at the exchange or at a co-located point of presence preserve the original packets, sequence numbers and hardware timestamps. When those captures are complete and correctly timestamped, they become the authoritative record of what the market actually published.

Reconstructing the Order Book from PCAP

Reconstruction is the process of turning a stream of incremental messages, adds, modifies, cancels, executions, into a coherent, queryable order book at every moment. The engine must:

• Apply messages in the exact order they were sequenced by the exchange.
• Handle snapshot-plus-incremental recovery after gaps or late joins.
• Maintain correct size-at-price and, where the protocol supplies it, order-level identity.
• Detect and report sequence gaps, packet loss and feed arbitrage between dual channels.

Errors in reconstruction, dropped messages, mis-ordered updates, incorrect handling of implied or auction states, produce a synthetic book that diverges from the book a live feed handler would have built. Any strategy researched against the defective book inherits that divergence.

Deterministic reconstruction, validated by byte-identical or state-identical replay against known captures, is therefore a prerequisite for trustworthy research.

Tick Databases: Storage and Access Patterns

A tick database stores the reconstructed or raw events so that researchers can request arbitrary intervals, instruments and depth levels without re-parsing multi-terabyte PCAPs every time. Design choices include:

• Columnar versus row-oriented layouts for analytical scans.
• Compression that preserves nanosecond timestamps and sequence integrity.
• Indexing by instrument, date and sequence or hardware time.
• Support for both event-by-event iteration and efficient "book as of time T" queries.

The database must also retain enough provenance, capture site, feed handler version, any normalization applied, so that later investigators can understand exactly what the research system consumed.

Market Data Replay as a Research and Acceptance Tool

Replay engines feed historical packets or reconstructed events into the same strategy and risk code that will run live, preserving original timing or applying controlled scaling. High-quality replay supports:

• Strategy research under realistic micro-structure.
• Regression testing of feed handlers and book engines.
• Measurement of residual latency distributions under recorded storms.
• Comparison of live versus replayed decision traces on the same days.

When the live system and the replay system share the same core libraries for book construction, risk and order logic, many classes of discrepancy disappear by construction.

Sources of the Backtest-to-Live Gap

Even with perfect historical data, live results diverge for identifiable reasons:

• Data differences: Live feeds may carry different latency, dual-site arbitration outcomes, or momentary gaps that the historical capture did not experience in the same way. Corporate actions, symbol changes or venue rule updates can also create mismatches if the research data were adjusted and the live path was not.
• Timing and latency modeling: Backtests often assume instantaneous or fixed-latency decision and order paths. Live systems experience variable residual latency, queueing under load, and exchange-side processing delays. Queue-position models that ignore these effects systematically overstate fill probabilities.
• Operational constraints: Live accounts enforce position limits, margin, message-rate caps and pre-trade risk checks that the research environment may only approximate. Kill switches, seat-level throttles and colocation or network failures appear only in production.
• Code and configuration drift: Parameters, feature flags or library versions that differ between the research build and the production build silently change behavior. Timestamp time-zone errors, off-by-one sizing logic and unapplied configuration are classic examples.
• Market impact and adverse selection: Most backtests treat the firm's own orders as negligible. In live trading, especially at larger size, the orders themselves move the book and attract informed flow.

Shrinking the Gap

The practical response is systematic comparison. On every live trading day the firm records the identical market-data stream (or a PCAP of it), the decisions the live system made, and the resulting fills. The same day is then replayed through the research path. Differences in book state, signals, orders and P&L are attributed to one of the categories above and either fixed or explicitly modeled.

Over time the research environment absorbs the same risk checks, latency distributions and data-cleaning rules that production uses. The remaining gap becomes smaller, measurable and understood rather than mysterious.

ShengliSoft and High-Fidelity Market Data Paths

Accurate replay begins with accurate capture. FPGA-based feed handlers that parse and normalize exchange feeds with deterministic low latency also produce clean, high-rate streams suitable for both live trading and historical recording. ShengliSoft's Extreme Feed Handler processes Chinese equity and futures market data in hardware, delivering full-depth books and high message throughput with stable residual latency.

Captures taken downstream of such a path, or raw PCAPs of the exchange multicast itself, give research teams the fidelity needed for trustworthy book reconstruction and replay. The same residual-latency discipline that benefits live trading benefits the research loop that tries to stay faithful to live conditions.

Governance of Research Data

Tick databases and PCAP archives should be treated as production assets: versioned, checksummed, access-controlled and accompanied by clear documentation of capture location, timestamp source and any transformations applied. Periodic audits that re-reconstruct known intervals and compare against live books keep the research foundation honest.

Faithful Capture, Deterministic Reconstruction and Explicit Live Constraints Turn Replay into a Reliable Bridge

Market-data replay and tick databases close the most dangerous gap in quantitative trading: the distance between the idealized world of research and the noisy, constrained world of production. When order books are reconstructed from complete PCAP or equivalent high-fidelity feeds, when replay exercises the same logic that will run live, and when every remaining difference is measured and attributed, the backtest-to-live gap shrinks from a source of surprise into a managed engineering quantity.

Teams that invest in this discipline spend less time explaining losses and more time improving the signals and infrastructure that actually survive contact with the market.

FAQ

1. Why is PCAP preferred for high-fidelity research?

It preserves original packet order, sequence numbers and hardware timestamps, allowing reconstruction of the exact market-data stream the exchange published.

2. What does order-book reconstruction require?

Correct sequencing of incremental messages, proper handling of snapshots and gaps, accurate maintenance of size and order identity, and validation that the resulting book matches what a live feed handler would have produced.

3. What belongs in a tick database?

Raw or reconstructed events with nanosecond timestamps, sequence information, instrument identifiers and provenance metadata, stored so that both sequential replay and point-in-time book queries are efficient.

4. What are the most common causes of backtest-to-live divergence?

Data differences, incomplete latency and queue modeling, live risk and capacity constraints, code or configuration drift, and unmodeled market impact.

5. How can teams detect divergence early?

By replaying every live trading day through the research path and systematically comparing book state, decisions, orders and outcomes.

6. Does hardware feed handling help research as well as trading?

Yes. Deterministic, high-rate feed handlers produce clean streams that are ideal for both live use and high-fidelity historical capture.

7. How does ShengliSoft infrastructure fit this workflow?

FPGA feed handlers such as the Extreme Feed Handler deliver low-latency, high-throughput market data that can be captured and later replayed, supporting accurate book reconstruction for Chinese market instruments.

8. What governance practices protect research data quality?

Version control, checksums, documented capture conditions, restricted access, and periodic re-validation of reconstructed books against known live intervals.

Disclaimer

This content is for informational purposes only and does not constitute investment advice. Financial markets and technology investments carry risk. Please conduct your own research before making investment or business decisions.