go-iex: Reading an Exchange's Wire Protocol
IEX is a stock exchange that is most famous for introducing a “speed bump” to deter high-frequency trading (Read about them in Flash Boys). But they also made their exchange data available to a degree that was unheard of by the more traditional players like NYSE and NASDAQ. IEX published both a JSON REST API for current quotes and a daily archive of raw packet captures, free, no credentials required.
I wrote go-iex to consume both. The REST client is more-or-less standard fare. The packet captures were the interesting part as I hand’t worked with them before.
Two kinds of market data
Market data is usually described by how deep into the order book it goes. Level 1 is the top of the book: the best bid, the best offer, and the sizes available at each, plus a trade report whenever something actually executes. Level 2 is the depth of the book: every price level with resting interest, so you can see that there are 300 shares resting at one price level and 1,200 more a penny higher.
IEX published one feed of each. TOPS carried top-of-book quotes and trade reports; DEEP carried the aggregated depth. A third endpoint, HIST, listed the archived capture files for a given trading day.
What is actually in the file
A packet capture is a recording of network traffic — the format tcpdump writes and Wireshark reads. IEX’s daily archive is a capture of the multicast feed as it was disseminated, which means reading it requires walking down through several layers of packaging before any market data appears.
At the outside is the pcap file format itself, a small global header followed by per-packet records. Inside each record is an Ethernet frame; inside that an IP packet; inside that a UDP datagram. The datagram’s payload is one IEX-TP segment. The segment contains some number of messages, and those are the quotes and trades.
Peeling those layers by hand would be tedious. Google’s gopacket already decodes everything from the file header down to the UDP payload, so the library’s job starts where gopacket’s ends. PcapScanner sits on top of a PacketDataSource interface whose entire contract is “give me the next payload,” with implementations for pcap files, pcap-ng files, and — added later — a live net.PacketConn for reading the multicast feed directly. The scanner does not know or care which one it has.
Two small conveniences: The archives are gzipped, so the reader peeks at the first two bytes and transparently wraps the stream in a gzip.Reader if it sees the magic bytes (0x1f 0x8b). The two capture formats are likewise distinguished by sniffing the magic number: pcap-ng announces itself with 0x0A0D0D0A.
The transport layer
IEX-TP is the envelope both TOPS and DEEP travel in, and it is a nice small piece of protocol design worth reading even if you never touch market data.
The segment header is exactly 40 bytes, all little-endian:
version | reserved | message protocol ID | channel ID | session ID
payload length | message count | stream offset
first message sequence number | send time
The last four fields are the interesting ones. A multicast feed can drop packets, so every segment carries both a byte offset into the logical stream and the sequence number of its first message. A receiver that misses a segment knows precisely what it missed and how much. Because a session ID identifies one day’s stream, a message is globally identifiable by session plus sequence number. And send time is nanoseconds since the epoch, which is a pointed contrast with the millisecond timestamps in the JSON API — in HFT, a millisecond is an eternity.
The payload is a run of length-prefixed messages: a two-byte length, then that many bytes of message, repeated message count times. The prefix is what makes the whole thing extensible, because it means a decoder can skip a message it does not understand without losing its place in the stream.
The message protocol ID field says which higher-level protocol the messages belong to — TOPS 1.5 is 0x8002, TOPS 1.6 is 0x8003, etc. I handled that with a registry: the iextp package knows how to parse segments and nothing else, and the tops and deep packages call RegisterProtocol from their init functions. Decoding a segment means looking up the protocol ID and handing each message body to whatever registered for it. Adding a protocol version does not require touching the transport code, and a program that only cares about TOPS can avoid linking DEEP by not importing it.
The messages themselves
Inside a segment, the first byte of each message is its type, and the types are ASCII mnemonics: Q for a quote update, T for a trade report, B for a trade break, H for a trading status, D for a security directory entry, X for an official price. This is a small kindness. It means a hex dump of a captured packet is partly readable by eye, which matters more than it should at three in the morning when you are trying to work out why your decoder has gone off the rails.
Prices are fixed-point: a signed 8-byte integer with four implied decimal places, so a price of one dollar fifty travels as 15000. Parsing one is a division:
// ParseFloat parses an IEX 8-byte signed integer, with 4 implied
// decimal point, into a float64.
func ParseFloat(buf []byte) float64 {
n := int64(binary.LittleEndian.Uint64(buf))
return float64(n) / 10000
}
Floating point is the wrong representation for money, and the right thing for a serious application would be to keep the integer. I converted anyway, because the alternative is a decimal type in every struct field and a library nobody wants to use. It is a defensible trade for research code and an indefensible one for anything that places orders, and it is worth being clear about which you are writing.
The spec includes one rule that shaped the decoder more than any other:
IEX reserves the right to grow the message length without notice, but only by adding additional data to the end of the message, so decoders should handle messages that grow beyond the expected length.
So every Unmarshal checks that the buffer is at least long enough and then indexes fixed offsets, rather than requiring an exact length or consuming to the end. Unknown message types decode into an UnsupportedMessage that keeps the raw bytes instead of failing. A decoder written the obvious strict way works perfectly until the morning the exchange ships a protocol revision, and then it stops working for everyone at once.
Making it into data
A stream of typed messages is still not what most quants want. pcap2json dumps everything as JSON for exploration, and pcap2csv does the thing most people actually came for: consolidate trade reports into minute bars.
$ pcap2csv < input.pcap > output.csv
symbol,time,open,high,low,close,volume
AAPL,2017-07-10T14:33:00Z,148.9100,149.0000,148.9100,148.9800,2527
AMZN,2017-07-10T14:33:00Z,364.8900,364.9600,364.8900,364.9600,1486
The consolidator groups trades by symbol, sorts by timestamp, and folds them into an OHLCV bar. It is the least sophisticated code in the repository and it is what turned a protocol exercise into something I actually used.
Update: the API is gone, the format is not
The free JSON API at api.iextrading.com that this library wrapped was folded into IEX Cloud in 2019, and IEX Cloud itself was retired on 31 August 2024 after IEX Group decided to refocus on the exchange business.
IEX continues to publish TOPS and DEEP captures for free download on a T+1 basis — as of March 2026 the archive is more than 17 TB across roughly 5,000 files — in the same IEX-TP format, because a wire protocol with a version field in its header does not need to break. The pcap scanner, the segment decoder, and pcap2csv will still read a file downloaded today. Pretty good free market data for a retail trader!
