LootX

HMP1 and HMP2 format

Updated November 6, 2026 This page as PDF

Byte-by-byte binary format of the heatmap — header, frames, HMP2 volume codes and price series, with a Python decoder.

GET /v1/heatmap?format=hmp1 or format=hmp2 returns the heatmap not as a table but as a binary: it is many times shorter than a table, streams, and is cacheable. It is easy to render — you can parse it frame by frame without waiting for the end of the body. The request parameters are the same as for the table.

format What’s inside Content-Type
hmp1 volumes as exact integers application/octet-stream
hmp2 volume encoded in 6–9 bits, shorter than HMP1; qbits — mantissa bits (6–9, default 6), layout — 0 frames or 1 series application/x-hmp2

format=hmp1|hmp2 on other routes — 400 bad_format; qbits or layout without format=hmp2 — 400 bad_request. Errors arrive as JSON, as always.

Cache and volume

The header sets the heatmap source, the gateway passes it through as is:

Cache-Control When
no-store window ends in the last minute — live edge
public, max-age=600 window end is no older than 50 hours: archive repair can still fill holes
public, max-age=86400, immutable older

Closed ranges have an ETag. A repeat request with If-None-Match: <ETag> gets 304 Not Modified without a body — it doesn’t consume daily volume. An interrupted download consumes as many bytes as it received.

Little-endian, 48 bytes:

Offset Size Field
0 4 magic: HMP1 or HMP2
4 2 version: 1 for HMP1, 2 for HMP2
6 2 flags
8 8 bucket_size (i64) — cell height in price_scale units
16 1 price_scale (u8)
17 1 qty_scale (u8)
18 1 HMP1: 0; HMP2: qty_bits (6–9)
19 1 HMP1: 0; HMP2: layout (0 — frames, 1 — series)
20 4 time_step_ms (u32)
24 8 from_ms (i64)
32 8 to_ms (i64)
40 4 frame_count (u32) — number of time columns
44 4 stream_id (u32), 0 for REST

flags: bit 0 — values aggregated by max (otherwise by average); bit 1 — response hit the source body limit (16 MB) and is truncated on the right by time. To continue, repeat the request with from = last frame time + time_step_ms.

Parse the response by its header (magic, version, layout), not by your request.

Cell price and volume:

TEXT
price = idx * bucket_size / 10^price_scale     lower bound of cell [price, price + step)
qty   = q / 10^qty_scale                       in coins
step  = bucket_size / 10^price_scale

Frames — HMP1 and HMP2 with layout=0

After the header — frame_count frames in sequence:

TEXT
varint  ts_delta_ms    from previous frame's ts; for the first — from from_ms
u8      kind           1 = KEY (full state), 0 = DELTA (changes)
zigzag  best_bid_d     difference from previous frame's best_bid; for the first — from 0
zigzag  best_ask_d     same for best_ask
varint  bid_count
  bid_count times:
    zigzag  idx_delta  from previous cell's idx; for the first — from 0
    varint  qty        ABSOLUTE cell volume (in HMP2 — code); 0 = level disappears
varint  ask_count
  ask_count times:       same

varint — unsigned LEB128. zigzag — LEB128 over (n << 1) ^ (n >> 63), decoded as (u >> 1) ^ -(u & 1).

Maintain two dictionaries idx → qty, for bids and asks:

  • KEY — dictionaries are replaced by the frame contents;
  • DELTA — applied on top: qty > 0 — assign, qty == 0 — remove the key.

The first frame is always KEY, then KEY repeats every 512 frames. The ts, best_bid, best_ask counters live within a single response. The dictionary state after a frame is one heatmap column.

Python
import struct

def uvarint(b, p):
    out = shift = 0
    while True:
        x = b[p]; p += 1
        out |= (x & 0x7F) << shift
        if x < 0x80:
            return out, p
        shift += 7

def svarint(b, p):
    u, p = uvarint(b, p)
    return (u >> 1) ^ -(u & 1), p

def columns(b):
    """HMP1 -> [(ts, {price: qty} bids, {price: qty} asks)]."""
    assert b[:4] == b"HMP1"
    bucket, ps, qs = struct.unpack_from("<qBB", b, 8)
    step_ms, from_ms, _to, frames = struct.unpack_from("<IqqI", b, 20)
    if frames == 0:
        return []
    bid, ask, out, p, ts = {}, {}, [], 48, from_ms
    for _ in range(frames):
        d, p = uvarint(b, p); ts += d
        kind = b[p]; p += 1
        _, p = svarint(b, p); _, p = svarint(b, p)        # best_bid_d, best_ask_d
        for book in (bid, ask):
            n, p = uvarint(b, p)
            cells, idx = [], 0
            for _ in range(n):
                di, p = svarint(b, p); idx += di
                q, p = uvarint(b, p)
                cells.append((idx, q))
            if kind == 1:
                book.clear()
                book.update((i, q) for i, q in cells if q > 0)
            else:
                for i, q in cells:
                    if q: book[i] = q
                    else: book.pop(i, None)
        price = lambda i: i * bucket / 10**ps
        out.append((ts,
                    {price(i): q / 10**qs for i, q in sorted(bid.items())},
                    {price(i): q / 10**qs for i, q in sorted(ask.items())}))
    assert p == len(b), "bytes remain after frames"
    return out

The format=json table is exactly this parse: one row per non-zero cell of each column.

HMP2: volume as code

Frames with layout=0 — byte-for-byte as in HMP1, but instead of varint qty there is varint code:

  • code = 0 — no level (in DELTA — “level disappears”);
  • otherwise code — cell volume in coins in a format with m = qty_bits mantissa bits.

DELTA compares codes: a cell whose code didn’t change is not included in DELTA, even if the raw volume shifted slightly.

Decoding (d = 23 − m): code = 0 → 0, otherwise v = from_bits(((code + (64 << m)) << d) as u32) as float32 — volume in coins, qty_scale is not applied to it. With m = 9 the error is at most 2⁻¹⁰, range from 2⁻⁶³ to 2⁶⁵ coins.

Python
import struct

def hmp2_qty(code, m):
    if code == 0:
        return 0.0
    bits = (code + (64 << m)) << (23 - m)
    return struct.unpack("<f", struct.pack("<I", bits))[0]

Vectors to verify your implementation:

Volume Code m=9 Back Code m=6 Back
1e-8 18607 9.997166e-9 2326 1.0011718e-8
0.001 27148 0.0009994507 3394 0.0010070801
0.5 31744 0.5 3968 0.5
1 32256 1 4032 1
1023 37375 1023 4672 1024
12345.678 39172 12352 4896 12288

HMP2, layout=1: price series

Same header with byte 19 = 1. Body — four streams in sequence, without kind and frame counters:

  1. Time: frame_count times varint ts_delta, zigzag best_bid_d, zigzag best_ask_d — as in a frame.
  2. Series: for bids, then asks — varint R, then R times zigzag idx_delta (first — from 0, then — from the previous idx of this side’s series, strictly increasing) and varint run_count.
  3. Lengths: all bid series, then asks, for each series varint (len − 1). Sum of lengths per series = frame_count.
  4. Codes: in the same order per series zigzag(code − prev), prev = 0 at the start of each series. code = 0 — no level in these columns.

A series without a single non-zero code is not written. A series column cannot be read separately from the whole body — for streaming rendering use layout=0.