When Markets Break: Lessons from Running Trading Systems Under Stress
There’s a kind of pressure that only shows up when things actually go wrong.
Your strategy backtests beautifully. The simulated environment runs smoothly. The first few months after going live are uneventful. Then one night, BTC drops 15% in ten minutes. The order book gets flooded with a wave of cancellations and new orders simultaneously. Market message volume blows past 800,000 messages per minute — and your system starts behaving strangely.
Signals are late. Stop-losses don’t trigger. Cancel orders go out, but you don’t know if they actually made it.
This isn’t a problem with your strategy logic. It’s your data pipeline, quietly becoming a different system under pressure.
Latency Isn’t a Number — It’s a Curve
Many engineers default to average latency when evaluating their systems. In trading infrastructure, that habit is expensive.
Average latency tells you how fast you are under normal conditions. It tells you nothing about how slow you get when things go wrong. And in trading, the worst moments are exactly when you need your system performing at its best.
The right approach is to look at the percentile distribution. P50 tells you typical-case speed — signals arriving a few milliseconds ahead of the competition, orders sitting higher in the queue. That timing edge compounds over thousands of interactions a day and eventually shows up in PnL. P99 and P99.9 reflect resilience under higher load. And P99.99 — the slowest one in every ten thousand transmissions — is often a very good predictor of how your system will behave during the next black swan event.
Line them up and you’re looking at a curve, not a point. A system with a P50 of 1.6ms but a P99.99 of 1992ms isn’t degrading linearly — it’s collapsing under specific conditions. Your average will never tell you that.
One thing worth stating upfront: the cost of chasing tail latency rises exponentially. High-frequency and market-making strategies live in a world where every millisecond of P99.99 has direct PnL implications — heavy infrastructure investment is justified. For mid-to-low frequency strategies where signals play out over minutes or hours, the priority shifts to P99 stability and ensuring risk controls fire correctly under stress, not squeezing every microsecond out of the tail. Knowing which category your strategy falls into determines where your investment should go.
Jitter: The Damage That’s Harder to See
Imagine two data sources. Source A delivers data with 5ms latency every single time. Source B averages 5ms too — but sometimes it’s 1ms, sometimes it’s 13ms.
By average latency, they look identical. To your strategy, they’re completely different systems.
That unpredictability is called jitter, typically measured by the interquartile range (IQR — the spread between P25 and P75). A small IQR means the system behaves like a precision clock. A large IQR means it behaves like a random number generator.
The way jitter damages strategies is subtle. Most quantitative strategies are event-driven — “when X happens, do Y.” That logic rests on a critical assumption: the timing relationship between events is predictable. When jitter is high, the same market event can arrive at different delays on different occasions. Signals that should fire together start drifting apart. The strategy runs in a timing environment it was never designed to handle.
For market makers, it’s even more direct. Your quotes are only valid if you know where the market actually is. In a high-jitter environment, you never know whether the data you’re acting on is a 1ms-old snapshot or a 13ms-old one. Under volatility, that uncertainty translates directly into a higher probability of getting picked off — informed counterparties trading against your stale quotes while you absorb the adverse selection loss.
To compensate for an unstable data environment, strategies often need to add error buffers and dynamic recalibration logic. These aren’t free: they force the strategy to become more conservative. Signal thresholds get raised. Real alpha gets filtered out as noise.
The value of a low-jitter system isn’t just speed — it’s determinism. Determinism lets a strategy run the way it was designed to run, closing the gap between theoretical and actual performance. In real-world testing, a well-designed low-latency data source can maintain an extremely tight IQR, while public feeds in the same timeframe are often several times wider. That gap is invisible in calm markets. When volatility picks up, that’s when it starts charging you.
Under Pressure, Your System Becomes a Different System
System behavior under stress is often not a scaled-up version of normal behavior — it’s qualitatively different.
Physics has a concept called a phase change — water at 100°C doesn’t just get hotter, it becomes something else entirely. Trading system degradation under stress often follows the same pattern: not twice as slow, but suddenly operating in a completely different mode.
Latency degradation under stress typically comes from several compounding sources. When message volume spikes, buffers start filling up, TCP congestion control kicks in, and RTT climbs sharply. Shared public network infrastructure has no mechanism to prioritize your trading traffic. Even after data packets arrive at your machine, they still need to queue for CPU time to be decoded — local queue buildup adds another layer of invisible latency on top of network latency. And most dangerously: reconnection and resync events. When extreme conditions trigger a connection reset, those low-frequency code paths typically have no performance optimization. When triggered under peak load, data interruptions can last seconds or longer.
In stress testing, when market message volume surged to 800,000+ messages per minute, public feed P99.99 latency spiked to nearly 2000ms. At the same moment, infrastructure architecturally designed to absorb that kind of load maintained P99.99 at 170ms — a 12x gap, occurring precisely when accurate data matters most.
This isn’t a sales pitch — it’s the result of engineering choices. When a system was never designed for tail-end pressure, its collapse during a stress event was written into the architecture from the start.
Proper stress testing shouldn’t just run at 2x normal load. It should deliberately manufacture the conditions that trigger phase changes — force buffer overflow, trigger reconnection, simulate out-of-order message delivery — and observe what actually happens. P99.99’s true character only reveals itself in tests like these.
Four Ways Latency Steals Your PnL
Slippage accumulation: Acting on stale market data means a gap between your expected fill price and your actual fill. Each instance looks small, but at high-frequency volumes across hundreds to thousands of trades per day, it’s a continuous, quiet bleed. And the market’s worst days — when losses are largest — are also when slippage costs are highest.
Adverse selection risk: Your resting orders get hit by faster, better-informed counterparties after the market has already moved against you. The higher your latency, the greater the structural probability of this outcome — and it’s amplified during high-volatility periods.
Risk control failure: Stop-losses, position limit triggers, margin call responses — all of these depend entirely on how timely your data is. A risk system making decisions based on 2-second-old data is effectively no risk system at all during extreme moves. Every major market crash event has produced real account liquidations that validated this.
Signal quality degradation: Every signal that depends on market microstructure — order flow imbalance, depth shifts, trade anomalies — loses predictive power as data freshness declines. Stale data doesn’t give you a slightly less accurate picture — it gives you a picture of a market that no longer exists.
The sensitivity across these four channels varies by strategy type. High-frequency and market-making strategies are exposed on all four. Mid-to-low frequency strategies are primarily exposed on the last two — risk control failure and signal degradation. Knowing where your strategy’s exposure actually sits tells you where to direct your infrastructure investment.
One final thing worth holding onto: low-latency data infrastructure won’t turn a bad strategy into a good one. The relationship is multiplicative — good strategy needs good infrastructure to reach its potential, but infrastructure can’t create alpha that isn’t there. Infrastructure investment priority should track the maturity of your strategy. Validate that it actually works first, then determine the right level of infrastructure depth based on how latency-sensitive it really is.
Conclusion
Latency problems are nearly invisible during calm periods. The system appears to be working. PnL looks fine. No obvious alarms. But it’s always there, quietly shaping the quality of every signal, the efficiency of every fill, the actual effectiveness of every risk control mechanism.
When markets truly lose control, it stops being invisible and becomes the main character.
If your team hasn’t seriously looked at your P99.9 latency numbers, doesn’t know how your system actually behaves under stress, or is evaluating data infrastructure choices — this is exactly the kind of problem we’ve spent considerable time testing and solving at QSG. We’d be happy to walk through it with you.
Contact us: [email protected]
Data references: QuanTech Services Group (QSG) — Sytus Feed Stress Test Report
FAQ
What is market data latency and why does it matter?
Market data latency is the delay between a market event occurring and your trading system receiving it. Even small delays can cause your strategy to act on stale prices, leading to slippage, mispriced orders, and risk controls that trigger too late.
Why is average latency a misleading metric?
Average latency describes typical conditions, but trading systems fail at the extremes. A system with a P50 of 1.6ms but a P99.99 of 1992ms will appear healthy under normal conditions — and collapse exactly when you need it most. Tail latency, not the average, predicts your system’s behavior during market stress events.
What is jitter and how does it affect trading strategies?
Jitter is the variability in latency over time, measured by the IQR. High jitter means data arrives unpredictably — sometimes 1ms, sometimes 13ms. This breaks the timing assumptions built into event-driven strategies, forces conservative signal thresholds, and increases adverse selection risk for market makers.
Why does latency get so much worse when markets crash?
During extreme volatility, message volume can spike several times over in a short window, congesting data pipelines. Systems not architecturally designed for this kind of load can see latency jump from milliseconds to seconds. This degradation isn’t linear — it’s a phase change, not a gradual slowdown.
Does latency affect all trading strategies equally?
No. High-frequency and market-making strategies are sensitive across all latency dimensions. Mid-to-low frequency strategies are primarily exposed to risk control failure and signal degradation during extreme events. Understanding where your strategy’s exposure actually sits determines where infrastructure investment should be focused.
Should every team prioritize minimizing tail latency?
Not necessarily. Reducing tail latency has exponentially increasing costs. The right target depends on your strategy type. What every team should do — regardless of strategy — is know their P99 and P99.99 numbers, run genuine stress tests, and understand how their system behaves when conditions deteriorate.
