HMP1 and HMP2 format
Byte-by-byte binary format of the heatmap — header, frames, HMP2 volume codes and price series, with a Python decoder.
GET /v1/heatmap?format=hmp1 or format=hmp2 returns the heatmap not as a table but as a binary: it is many times shorter than a table, streams, and is cacheable. It is easy to render — you can parse it frame by frame without waiting for the end of the body. The request parameters are the same as for the table.
format |
What’s inside | Content-Type |
|---|---|---|
hmp1 |
volumes as exact integers | application/octet-stream |
hmp2 |
volume encoded in 6–9 bits, shorter than HMP1; qbits — mantissa bits (6–9, default 6), layout — 0 frames or 1 series |
application/x-hmp2 |
format=hmp1|hmp2 on other routes — 400 bad_format; qbits or layout without format=hmp2 — 400 bad_request. Errors arrive as JSON, as always.
Cache and volume
The header sets the heatmap source, the gateway passes it through as is:
Cache-Control |
When |
|---|---|
no-store |
window ends in the last minute — live edge |
public, max-age=600 |
window end is no older than 50 hours: archive repair can still fill holes |
public, max-age=86400, immutable |
older |
Closed ranges have an ETag. A repeat request with If-None-Match: <ETag> gets 304 Not Modified without a body — it doesn’t consume daily volume. An interrupted download consumes as many bytes as it received.
Header
Little-endian, 48 bytes:
| Offset | Size | Field |
|---|---|---|
| 0 | 4 | magic: HMP1 or HMP2 |
| 4 | 2 | version: 1 for HMP1, 2 for HMP2 |
| 6 | 2 | flags |
| 8 | 8 | bucket_size (i64) — cell height in price_scale units |
| 16 | 1 | price_scale (u8) |
| 17 | 1 | qty_scale (u8) |
| 18 | 1 | HMP1: 0; HMP2: qty_bits (6–9) |
| 19 | 1 | HMP1: 0; HMP2: layout (0 — frames, 1 — series) |
| 20 | 4 | time_step_ms (u32) |
| 24 | 8 | from_ms (i64) |
| 32 | 8 | to_ms (i64) |
| 40 | 4 | frame_count (u32) — number of time columns |
| 44 | 4 | stream_id (u32), 0 for REST |
flags: bit 0 — values aggregated by max (otherwise by average); bit 1 — response hit the source body limit (16 MB) and is truncated on the right by time. To continue, repeat the request with from = last frame time + time_step_ms.
Parse the response by its header (magic, version, layout), not by your request.
Cell price and volume:
price = idx * bucket_size / 10^price_scale lower bound of cell [price, price + step)
qty = q / 10^qty_scale in coins
step = bucket_size / 10^price_scaleFrames — HMP1 and HMP2 with layout=0
After the header — frame_count frames in sequence:
varint ts_delta_ms from previous frame's ts; for the first — from from_ms
u8 kind 1 = KEY (full state), 0 = DELTA (changes)
zigzag best_bid_d difference from previous frame's best_bid; for the first — from 0
zigzag best_ask_d same for best_ask
varint bid_count
bid_count times:
zigzag idx_delta from previous cell's idx; for the first — from 0
varint qty ABSOLUTE cell volume (in HMP2 — code); 0 = level disappears
varint ask_count
ask_count times: samevarint — unsigned LEB128. zigzag — LEB128 over (n << 1) ^ (n >> 63), decoded as (u >> 1) ^ -(u & 1).
Maintain two dictionaries idx → qty, for bids and asks:
- KEY — dictionaries are replaced by the frame contents;
- DELTA — applied on top:
qty > 0— assign,qty == 0— remove the key.
The first frame is always KEY, then KEY repeats every 512 frames. The ts, best_bid, best_ask counters live within a single response. The dictionary state after a frame is one heatmap column.
import struct
def uvarint(b, p):
out = shift = 0
while True:
x = b[p]; p += 1
out |= (x & 0x7F) << shift
if x < 0x80:
return out, p
shift += 7
def svarint(b, p):
u, p = uvarint(b, p)
return (u >> 1) ^ -(u & 1), p
def columns(b):
"""HMP1 -> [(ts, {price: qty} bids, {price: qty} asks)]."""
assert b[:4] == b"HMP1"
bucket, ps, qs = struct.unpack_from("<qBB", b, 8)
step_ms, from_ms, _to, frames = struct.unpack_from("<IqqI", b, 20)
if frames == 0:
return []
bid, ask, out, p, ts = {}, {}, [], 48, from_ms
for _ in range(frames):
d, p = uvarint(b, p); ts += d
kind = b[p]; p += 1
_, p = svarint(b, p); _, p = svarint(b, p) # best_bid_d, best_ask_d
for book in (bid, ask):
n, p = uvarint(b, p)
cells, idx = [], 0
for _ in range(n):
di, p = svarint(b, p); idx += di
q, p = uvarint(b, p)
cells.append((idx, q))
if kind == 1:
book.clear()
book.update((i, q) for i, q in cells if q > 0)
else:
for i, q in cells:
if q: book[i] = q
else: book.pop(i, None)
price = lambda i: i * bucket / 10**ps
out.append((ts,
{price(i): q / 10**qs for i, q in sorted(bid.items())},
{price(i): q / 10**qs for i, q in sorted(ask.items())}))
assert p == len(b), "bytes remain after frames"
return outThe format=json table is exactly this parse: one row per non-zero cell of each column.
HMP2: volume as code
Frames with layout=0 — byte-for-byte as in HMP1, but instead of varint qty there is varint code:
code = 0— no level (in DELTA — “level disappears”);- otherwise
code— cell volume in coins in a format withm = qty_bitsmantissa bits.
DELTA compares codes: a cell whose code didn’t change is not included in DELTA, even if the raw volume shifted slightly.
Decoding (d = 23 − m): code = 0 → 0, otherwise v = from_bits(((code + (64 << m)) << d) as u32) as float32 — volume in coins, qty_scale is not applied to it. With m = 9 the error is at most 2⁻¹⁰, range from 2⁻⁶³ to 2⁶⁵ coins.
import struct
def hmp2_qty(code, m):
if code == 0:
return 0.0
bits = (code + (64 << m)) << (23 - m)
return struct.unpack("<f", struct.pack("<I", bits))[0]Vectors to verify your implementation:
| Volume | Code m=9 | Back | Code m=6 | Back |
|---|---|---|---|---|
| 1e-8 | 18607 | 9.997166e-9 | 2326 | 1.0011718e-8 |
| 0.001 | 27148 | 0.0009994507 | 3394 | 0.0010070801 |
| 0.5 | 31744 | 0.5 | 3968 | 0.5 |
| 1 | 32256 | 1 | 4032 | 1 |
| 1023 | 37375 | 1023 | 4672 | 1024 |
| 12345.678 | 39172 | 12352 | 4896 | 12288 |
HMP2, layout=1: price series
Same header with byte 19 = 1. Body — four streams in sequence, without kind and frame counters:
- Time:
frame_counttimesvarint ts_delta,zigzag best_bid_d,zigzag best_ask_d— as in a frame. - Series: for bids, then asks —
varint R, then R timeszigzag idx_delta(first — from 0, then — from the previous idx of this side’s series, strictly increasing) andvarint run_count. - Lengths: all bid series, then asks, for each series
varint (len − 1). Sum of lengths per series =frame_count. - Codes: in the same order per series
zigzag(code − prev),prev = 0at the start of each series.code = 0— no level in these columns.
A series without a single non-zero code is not written. A series column cannot be read separately from the whole body — for streaming rendering use layout=0.