Latte

Latte is a single-header C++17 telemetry framework built for measurements the act of measuring would normally destroy: scopes are timestamped with raw x86_64 TSC reads into per-thread, lock-free ring buffers, so sampling costs tens of cycles, never blocks another thread, and every statistic is deferred to a cold path after the run.

Latte exists because std::chrono stopped being an answer: I was timing order flow with events in the hundreds of nanoseconds, and a single now() call runs ~150 cycles, two per measurement, on paths sometimes cheaper than one. At that scale the tool's overhead is the signal. Every alternative hashed strings or bumped shared counters on the hot path, so measuring thread A could slow thread B. So I built Latte.

01How it works

Hot path only writes; cold path does all the thinking once, after sampling stops. Aggregation, sorting, statistics, and outlier filtering all live on the cold side.

Reading the clock without the clock lying

Reading the TSC correctly is a spectrum, not one instruction. RDTSC gives no ordering guarantee: instructions around it can reorder, harmless for a millisecond loop, fatal for a 40-cycle snippet. RDTSCP waits for prior instructions to retire: a partial guarantee. Full serialization needs RDTSCP paired with LFENCE, per Intel's whitepaper on benchmarking code execution. Hence three modes:

Mode is chosen per call site, Start and Stop can even differ, and the per-thread stack records which mode captured each timestamp; calibration keys on the (start_mode, stop_mode) pair. A single global mode would either waste serialization everywhere or let reordering smear short measurements.

void ProcessOrder() {
    Latte::Fast::Start(__func__);   // ~30 cycles: one rdtsc + stack push
    // core logic
    Latte::Fast::Stop(__func__);    // ~30 cycles: rdtsc, pop, delta -> ring buffer
}

Scope identity without hashing

Most profilers hash the scope name every sample. Latte keys on the pointer: a string literal like "Physics_Engine" has a fixed address, so every call resolves to the same 64-bit key with zero comparisons. Storage is std::map<const char*, RingBuffer>: O(log N) pointer compares, no strcmp, no hashing.

Corollary: IDs must be pointer-stable. String literals and static const char[] work; a temporary std::string::c_str() or stack buffer silently fragments one scope into several.

Per-thread storage and cache-line discipline

Each thread owns thread_local storage: its own stack, map, ring buffers. First Start(id) allocates that ID's buffer; every later call is allocation- and lock-free, so one thread can't stall another by construction.

Two unrelated variables on the same 64-byte line ping-pong between cores on every write, the false-sharing failure mode from Drepper's "What Every Programmer Should Know About Memory". Latte's ring buffers are alignas(64) and Structure-of-Arrays internally (timestamps contiguous, separate from metadata) so reading timing data pulls only timing data into L1.

Buffers hold 65,536 samples per ID per thread (1 << BUFFER_PWR, power-of-two mandatory), and writes overwrite unconditionally: head = (head + 1) & BUFFER_MASK, wraparound as a single AND.

Two-phase recording

Start() pushes ID, timestamp, and mode onto a per-thread SoA stack (64 slots, nested scopes, no linear search); Stop() pops, computes the delta, writes it to the ring buffer. That's the whole hot path; profilers that maintain running stats pay for it on every sample instead.

The cold path forks two ways: DumpToStream() walks every thread, runs the outlier filter, computes per-scope stats (mean, median, stddev, skewness, min/max/range), applies self-offset calibration, writes tables to any std::ostream. Snapshot(id) returns the raw, uncleaned std::vector<Cycles> for one ID across threads, for plotting or custom analysis.

Filtering scheduler noise without eating the tail

A thread scheduled off-core mid-measurement turns a 200 ns event into 100 us; a mean is destroyed by one such sample. But a plain IQR filter over raw samples over-corrects, since latency distributions are legitimately right-skewed.

Latte's fix: bucket samples in groups of 1,000, record each bucket's max, and compute the IQR cutoff over those maxima instead of raw samples. Genuine spikes stand out against bucket maxima; the real tail doesn't. Samples above the cutoff are counted (the OUTLIER column) and excluded.

02Trade-offs

03Benchmarks

Setup: AMD Ryzen 5 7600X (6 cores, 4.7 GHz base / 5.3 GHz boost), g++ -O3 -march=native, pinned core, 100,000 iterations per trial, 100 trials, one warm-up batch. Measured in cycles via the TSC itself, so the probe and subject share a clock.

Latte's numbers are Start+Stop pairs (two reads); raw intrinsics were benchmarked as single calls. To compare fairly, the chart doubles __rdtsc and __rdtscp (mean x2, std dev x sqrt(2)). lfence and std::chrono::now stay single-call. Bars sorted by latency.

Each Latte mode lands essentially on top of its doubled raw primitive: Fast at 60.1 cycles vs. two raw __rdtsc at 60.2, the stack push/pop/delta/write costs nothing extra. Mid at 119.8 sits ~4 cycles above two __rdtscp at 115.4. Hard at 175.8 is the only mode with visible framework cost, mostly the four fence/read instructions. A complete Fast measurement costs less than one std::chrono::now() call (153.9). At 5.3 GHz boost (0.213 ns/cycle), that's ~13 ns of self-overhead per region.

Calibrated output against a simulated trading pipeline, self-offset correction applied, IQR pass already counting outliers:

#==============================================================================================================#
| LATTE TELEMETRY [TIME][CAL]                                                                                  |
#==============================================================================================================#
| SELF-OFFSET H[Start] x W[Stop]                                                                               |
|                        F             M             H                                                         |
| F                0.21 ns      10.02 ns      10.02 ns                                                         |
| M                0.21 ns      10.02 ns      10.02 ns                                                         |
| H                0.21 ns      10.02 ns      10.02 ns                                                         |
| PULSE           10.02 ns                                                                                     |
|--------------------------------------------------------------------------------------------------------------|
| COMPONENT             SAMPLES       AVG    MEDIAN   STD DEV    SKEW       MIN       MAX     RANGE    OUTLIER |
|--------------------------------------------------------------------------------------------------------------|
| DP_Build_Total              1   38.07 s   38.07 s   0.00 ns    0.00   38.07 s   38.07 s   0.00 ns         0  |
| DP_StateLoop            65536  31.15 us  30.96 us   1.42 us   12.13  29.32 us  95.10 us  65.78 us         0  |
| Sim_Tick_Total           4997 226.79 ns 220.42 ns  76.04 ns    1.40 110.21 ns 821.55 ns 711.34 ns         3  |
| Sim_OrderFlow            4997  43.62 ns  29.63 ns  36.61 ns    2.02   9.59 ns 380.51 ns 370.91 ns         3  |
| Sim_AskLoop              1216   1.03 us 711.34 ns   1.06 us    2.14   0.00 ns   8.30 us   8.30 us         0  |
| Sim_BidLoop              1247   1.12 us 751.42 ns   4.21 us   32.56   0.00 ns 145.96 us 145.96 us         0  |
| Sim_RiskPnL              5000   6.24 ns   9.59 ns   4.67 ns   -0.57   0.00 ns  19.82 ns  19.82 ns         0  |
#==============================================================================================================#

The SELF-OFFSET table is Start/Stop called back-to-back with nothing between, per mode pairing, subtracted per-sample in calibrated mode. That's why Sim_BidLoop's skew of 32.56 and Sim_Tick_Total's three outliers read as signal instead of instrumentation noise.

04Where it went sideways

Stop(id) doesn't validate its argument against the top of the stack: it always pops the most recent Start(), strictly LIFO. Cross two nested scopes' Start/Stop (easy during a refactor) and nothing crashes; one scope's time silently attributes to the other's ID. Validating every Stop() would cost a hot-path compare for a mistake the type system can't prevent anyway, so matching IDs became a contract, not a suggestion.

DumpToStream() and Snapshot() are unsafe to call while any thread is sampling: not approximate, undefined behavior. The manager's mutex protects only the global list of per-thread storage pointers; the ring buffers and history maps are lock-free and atomic-free on purpose. The fix isn't synchronization, it's sequencing: join workers, or hold a barrier, before reporting.

Both edges are the same trade: checks that would normally live in the library got pushed to the caller, because paying for them once at the boundary beats paying on every sample.

05What I'd do better next time

The hard problem was never reading the clock, Intel documented that fifteen years ago. The real constraint was refusing convenience creep back onto the hot path, one cheap compromise at a time.

Next version: the fixed constants (65,536-sample buffers, 1,000-sample IQR buckets) become compile-time parameters, since the right sizes are workload-dependent. MSVC support means conditionally wrapping the GCC/Clang attributes. An ARM port through CNTVCT_EL0 is the interesting problem: ARM's ordering story differs completely from RDTSC/RDTSCP, so the Fast/Mid/Hard split would need rederiving from ARM's memory model.

06Further reading

• How to Benchmark Code Execution Times on Intel IA-32 and IA-64 Instruction Set Architectures: Gabriele Paoloni, Intel, 2010. The reference on RDTSC/RDTSCP serialization and why naive cycle counting goes wrong; the basis for the Fast/Mid/Hard split.
• What Every Programmer Should Know About Memory: Ulrich Drepper, Red Hat, 2007. Cache-line behavior and false sharing in depth; the background for the alignas(64) and Structure-of-Arrays decisions.
• fior512/Latte: the project: single header, C++17, x86_64, MIT licensed, v0.1.0.