BLOB: on-chain chunked binary storage
Binary files stored on chain in chunks: layout, rules and REST routes.
BLOB stores arbitrary binary data entirely on chain, split into chunks of one OP_RETURN output each, and addressed by the SHA-256 of the reassembled bytes. The node collects the chunks, reassembles a blob once every chunk is present and contiguous, checks that its SHA-256 equals the announced blob_id, and then serves the bytes over REST with the right Content-Type. It was written as the storage backend for poker NFT metadata (JSON, SVG, WebP), but nothing in the code is poker-specific. The value attached to each chunk output is unspendable (OP_RETURN) and is summed as "burn".
Status in this node #
- Built-in module
blob(classCBlobModule), always registered:src/modules/builtin_modules.cpp:34. There is no switch to turn it off. - Registered after eMPP, so the eMPP segments of a block are already stored when BLOB reads them (
src/modules/builtin_modules.cpp:28-31). - Database:
<datadir>/modules/blob.sqlite(src/modules/blob/blob_module.cpp:301, directory fromsrc/modules/module_registry.cpp:196). - Indexing only: BLOB changes no consensus or relay rule. The policy limit that makes multi-kilobyte chunks relayable is the node-wide
MAX_OP_RETURN_RELAY = 8005bytes of scriptPubKey (src/script/standard.h:85, overridable with-datacarriersize). - LOKAD id:
"BLOB"=42 4c 4f 42(src/modules/blob/blob_module.cpp:31).
Envelope #
Two transports are accepted, both decoded on every connected block (ExtractChunksFromBlock, src/modules/blob/blob_module.cpp:229-294):
| Transport | Script shape | Decoder | source column |
|---|---|---|---|
| Native | OP_RETURN <push "BLOB"> <push blob_id> <push index> <push total> <push type> <push payload> | DecodeBlobOutput, blob_module.cpp:135 | native |
| eMPP | OP_RETURN OP_RESERVED <push "BLOB" + flat bytes> [more segments] | eMPP module, then DecodeBlobPayloadFlat, blob_module.cpp:208 | empp |
Any output of any transaction (coinbase included) is scanned. For eMPP, see eMPP; BLOB receives each segment's bytes after the 4-byte LOKAD id. If the same (txid, vout) was already taken by the native decoder, the eMPP row is skipped (blob_module.cpp:273); the two shapes cannot actually overlap because native requires a push right after OP_RETURN and eMPP requires OP_RESERVED (0x50).
Byte layout #
Native chunk (one output = one chunk) #
Exactly six data pushes after OP_RETURN (0x6a). All integers are big-endian.
| Offset (in script) | Size | Type | Endianness | Field | Notes |
|---|---|---|---|---|---|
| 0 | 1 | opcode | n/a | OP_RETURN | 0x6a |
| 1 | 1 | opcode | n/a | push length | 0x04 |
| 2 | 4 | bytes | n/a | LOKAD id | 42 4c 4f 42 ("BLOB") |
| 6 | 1 | opcode | n/a | push length | 0x20 |
| 7 | 32 | bytes | n/a | blob_id | SHA-256 of the full reassembled buffer, stored as lowercase hex in byte order |
| 39 | 1 | opcode | n/a | push length | 0x04 |
| 40 | 4 | uint32 | big-endian | chunk_index | 0-based |
| 44 | 1 | opcode | n/a | push length | 0x04 |
| 45 | 4 | uint32 | big-endian | total_chunks | at least 1 |
| 49 | 1 | opcode | n/a | push length | 0x01 |
| 50 | 1 | uint8 | n/a | content_type_id | see the table below |
| 51 | 1, 2, 3 or 5 | opcode(s) | little-endian length | payload push | direct push (1 to 75 bytes), PUSHDATA1, PUSHDATA2 or PUSHDATA4 |
| next | N (at least 1) | bytes | n/a | payload | this chunk's slice of the blob |
The header (offsets 0 to 50) is 51 bytes, so with the default 8005-byte relay ceiling a chunk can carry up to 8005 - 51 - 3 = 7951 bytes of payload (PUSHDATA2 prefix). The -datacarriersize arithmetic is policy only.
eMPP segment body ("flat" layout) #
The segment push contains "BLOB" followed by these bytes (offsets are relative to the end of the LOKAD id):
| Offset | Size | Type | Endianness | Field | Notes |
|---|---|---|---|---|---|
| 0 | 32 | bytes | n/a | blob_id | SHA-256 of the full buffer |
| 32 | 4 | uint32 | big-endian | chunk_index | 0-based |
| 36 | 4 | uint32 | big-endian | total_chunks | at least 1 |
| 40 | 1 | uint8 | n/a | content_type_id | |
| 41 | N (at least 1) | bytes | n/a | payload | runs to the end of the segment |
There are no inner length prefixes: the eMPP push length delimits the payload.
Content types #
The MIME type is chosen only when serving (ContentTypeName, src/modules/blob/blob_module.cpp:35-47); every byte value is accepted when indexing.
content_type_id | MIME type served |
|---|---|
| 0x01 | application/json |
| 0x02 | image/svg+xml |
| 0x03 | image/webp |
| 0x04 | application/json+gzip (clients decompress themselves; no Content-Encoding is sent) |
| 0x05 | image/png |
| 0x06 | image/jpeg |
| 0x07 | text/plain; charset=utf-8 |
| 0x08 | application/octet-stream |
| any other | application/octet-stream |
Versioning #
There is no version field. A layout change would need a new LOKAD id or a new content type value.
Validation rules #
Native chunk (DecodeBlobOutput and its push reader):
- The script is at least 6 bytes long and its first byte is
OP_RETURN(src/modules/blob/blob_module.cpp:136-139). - Every element after
OP_RETURNmust be a data push. Direct pushes 0x01 to 0x4b are taken as-is;OP_PUSHDATA1must carry a length of at least 0x4c,OP_PUSHDATA2at least 0x100,OP_PUSHDATA4at least 0x10000 (blob_module.cpp:78-123). Any other opcode, includingOP_0,OP_1NEGATEandOP_1toOP_16, rejects the output (blob_module.cpp:124). A push running past the end of the script rejects it. - More than 8 elements rejects the output (
blob_module.cpp:148); exactly 6 are required (blob_module.cpp:152). - Element 0 is exactly the 4 bytes
"BLOB"(blob_module.cpp:154-157). - Sizes:
blob_id32 bytes,chunk_index4,total_chunks4,content_type_id1, payload non-empty (blob_module.cpp:158-162). total_chunksis not 0 andchunk_index < total_chunks(blob_module.cpp:173-175).
eMPP chunk (DecodeBlobPayloadFlat), after the eMPP envelope itself has decoded (eMPP validation):
- The body is at least 42 bytes (
blob_module.cpp:210). total_chunksis not 0 andchunk_index < total_chunks(blob_module.cpp:220-222).- The payload after offset 41 is non-empty (
blob_module.cpp:225).
Reassembly (TryCompleteBlob, run for every blob touched by a block):
- A chunk is stored only if no chunk with the same
(blob_id, chunk_index)exists:INSERT OR IGNORE, first one wins (blob_module.cpp:536-540; primary key atblob_module.cpp:322). - While fewer rows than
total_chunksexist, the blob stays pending (blob_module.cpp:430-447). - Once enough rows exist, they are read in
chunk_indexorder and must be exactly0, 1, ..., total_chunks - 1(blob_module.cpp:450-476); otherwise nothing is written. sha256(concatenation) == blob_idsetscomplete = 1and stores the buffer; a mismatch setscorrupt = 1and stores no buffer (blob_module.cpp:478-515).
Nothing checks the output value: zero-value chunks are indexed. Nothing checks that all chunks of a blob agree on total_chunks or content_type_id (see Limits).
Indexing #
Database blob.sqlite (CreateSchema, src/modules/blob/blob_module.cpp:308-377):
blob_chunks, one row per accepted chunk:
| Column | Type | Notes |
|---|---|---|
blob_id | TEXT NOT NULL | 64 lowercase hex characters |
chunk_index | INTEGER NOT NULL | |
total_chunks | INTEGER NOT NULL | as announced by this chunk |
content_type | INTEGER NOT NULL | content_type_id |
txid | TEXT NOT NULL | |
vout | INTEGER NOT NULL | |
block_height | INTEGER NOT NULL | |
block_time | INTEGER NOT NULL | block header time |
sats | INTEGER NOT NULL | burn credited to this chunk: the value of the output (native), or for eMPP the value of the whole hosting output (EmppRow::output_sats) on the first chunk of that output that gets stored, 0 on the other chunks of the same output (12.4.6: one output burns its value once, see eMPP) |
payload | BLOB NOT NULL | chunk bytes |
source | TEXT NOT NULL DEFAULT 'native' | native or empp; added by an idempotent migration on older databases (blob_module.cpp:363-376) |
Primary key (blob_id, chunk_index); index idx_bc_height on block_height DESC.
blobs, one row per blob seen:
| Column | Type | Notes |
|---|---|---|
blob_id | TEXT PRIMARY KEY | |
total_chunks | INTEGER NOT NULL | |
content_type | INTEGER NOT NULL | |
byte_size | INTEGER NOT NULL DEFAULT 0 | size of the reassembled buffer |
complete | INTEGER NOT NULL DEFAULT 0 | 1 when the hash matched |
corrupt | INTEGER NOT NULL DEFAULT 0 | 1 when the hash did not match |
reassembled | BLOB | the bytes, only when complete |
first_seen_height | INTEGER NOT NULL | lowest chunk height |
complete_height | INTEGER | highest chunk height, set at completion |
total_burn_sats | INTEGER NOT NULL DEFAULT 0 | sum of sats over the chunks (so a blob carried by one output counts that output once) |
Indexes idx_blobs_height (first_seen_height DESC) and idx_blobs_complete (complete, first_seen_height DESC). Index idx_bc_output (txid, vout) on blob_chunks finds the chunks of one output. blob_meta(key, value) holds last_indexed_height and data_version (2 since 12.4.6).
Block flow:
- Connect: decode, then in one transaction insert the chunks (value credited once per output, see
sats), runTryCompleteBlobonce per touched blob, setlast_indexed_height(ConnectBlock,ApplyChunksLocked). TryCompleteBlobrewrites the wholeblobsrow from the chunks present: no chunk, the row is deleted; fewer chunks thantotal_chunks, a pending row (complete = 0,corrupt = 0, no bytes, nocomplete_height,first_seen_height= lowest chunk height); all chunks, the reassembly and hash check below.- Disconnect: in one transaction,
DELETE FROM blob_chunks WHERE block_height = h, thenTryCompleteBlobfor every blob that lost a chunk (back to pending, or gone when no chunk is left),last_indexed_height = h - 1(DisconnectBlock). Connecting N blocks then disconnecting K equals connecting NāK (blob_module_tests/connect_disconnect_symmetry_all_depths). Before 12.4.6 theblobsrow was not touched: a blob completed by the disconnected block stayed complete and served. - Startup reconciliation: BLOB does not override
IndexedTipHeight/ResetForReindex(src/modules/blob/blob_module.h:38-76), so the registry classifies it as "no tip state" and never replays or wipes it (src/modules/module_registry.cpp:70-75). Blocks connected while the module was not running are only picked up with a rescan. - Mempool: not indexed. A chunk appears only once its transaction is in a block.
- Rescan (
rescanmodule blob, a background job;RescanRange): reads each block from disk, decodes it outside any lock, then re-inserts its chunks (duplicates ignored, so it is idempotent) undercs_mainand the module mutex after checking that the block is still on the active chain (otherwise the rescan stops, stateaborted).last_indexed_heightonly moves forward, and only if the range starts at or below it + 1. A range above the tip does nothing. - Upgrade (12.4.6).
blob_chunkswas right in older databases except the value repeated on every chunk of a multi-chunk eMPP output;blobscould hold rows of disconnected blocks and corrupt rows created by a REST read of an unknown id. AtInit, a database withoutdata_version2 is repaired in place in one transaction: the value is kept on the first stored chunk of each output (lowest rowid, chunks are stored in segment order) and set to 0 on the others, then everyblobsrow is rebuilt from its chunks withTryCompleteBlob(rows without chunks deleted). Log:BLOB: indexed data version 1 < 2: repairing chunk burns and blob rows. Test:blob_module_tests/upgrade_from_v1_repairs_in_place.
REST API #
All routes are registered through AddModuleRoute with no response-cache TTL (src/api/api_server.cpp:345-348; routes at src/modules/blob/blob_module.cpp:640-654). JSON errors use the common envelope {"error": code, "message": text, "status": n}.
| Method | Path | Answer |
|---|---|---|
| GET | /api/v1/blob/recent?limit=N | {"blobs":[...]}, newest first_seen_height first; limit default 25, clamped to 1..200 |
| GET | /api/v1/blob/<sha256> | the raw bytes |
| GET | /api/v1/blob/<sha256>/info | JSON metadata |
| GET | /api/v1/blob/stats | totals |
| GET, POST | /api/v1/blob/rescan | 403 rescan_moved; use the authenticated RPC rescanmodule blob [start] [end] (see module rescans) |
/blob/recent row (blob_module.cpp:694-713):
{"blobId":"<64 hex>","totalChunks":1,"contentType":"application/json","contentTypeId":1,
"byteSize":7,"complete":true,"corrupt":false,"firstSeenHeight":100,"completeHeight":100,
"totalBurnSats":0,"source":"native"}completeHeight is present only when set; source is native, empp or mixed (both transports in one blob).
/blob/<sha256> (blob_module.cpp:722-827):
- The id must be exactly 64 characters of lowercase hex, otherwise 400
bad_blob_id. - Unknown id: the handler first tries to roll up existing chunks (
TryCompleteBlob), then answers 404not_found. With no chunk at all nothing is stored (before 12.4.6 the roll-up stored acorruptrow, the SHA-256 of nothing, for any id asked for:/blob/statscounted it). - Corrupt: 404
corrupt. Not complete: 404incomplete. - Complete: 200 with
Content-Typefrom the table above,Cache-Control: public, max-age=31536000, immutable,X-Blob-Id,X-Blob-Total-Burn-Sats,Access-Control-Allow-Origin: *.
/blob/<sha256>/info (blob_module.cpp:771-797): blobId, totalChunks, contentType, contentTypeId, byteSize, complete, corrupt, firstSeenHeight, completeHeight (if set), totalBurnSats, and chunksPresent when not complete.
/blob/stats (blob_module.cpp:829-866):
{"completeBlobs":0,"completeBytes":0,"totalBurnSats":0,"pendingBlobs":0,"corruptBlobs":0,"blobs":0,"chunks":0,"lastIndexedHeight":-1}totalBurnSats counts complete blobs only. blobs (every row of blobs) and chunks (rows of blob_chunks) were added in 12.4.6.
/blob/rescan: answers 403 rescan_moved to GET and POST since 12.4.6 and runs nothing. Rescans go through rescanmodule blob [start] [end] on the authenticated JSON-RPC.
Dashboard #
Tab "BLOB" (data-tab="blob", hidden when the module is not loaded), sub-tabs Hub and Recent (share/www/dashboard/js/dashboard.js:2765-2811). The Hub shows stat cards from /blob/stats and the 10 newest blobs from /blob/recent; Recent shows 50. Each blob id links to /api/v1/blob/<id>.
The Hub cards read blobs, chunks and completeBytes of /blob/stats ("Blobs", "Chunks", "Bytes Stored"), plus totalBurnSats and lastIndexedHeight. Before 12.4.6 they read fields the API did not return and always showed 0.
Limits and edge cases #
- First chunk wins per
(blob_id, chunk_index). Anyone who knows ablob_id(it is public as soon as one chunk is broadcast) can publish a different chunk at the same index first; the blob then reassembles to the wrong hash and is marked corrupt. The genuine chunk is ignored, and there is no retry with alternative chunks. - Conflicting
total_chunksorcontent_type_idacross chunks of one blob:TryCompleteBlobtakes those columns from an arbitrary row of an aggregate query (blob_module.cpp:412-415). The SHA-256 check still protects the bytes, but the served MIME type can be either value. - If chunks disagree on
total_chunks, the blob can stay pending forever (rows are present but not contiguous for the chosen total). /blob/<id>can write (roll-up of a blob whose chunks are present) from a GET; an unknown id writes nothing (12.4.6). Since 12.4.6 every handler runs holdingm_db_mutex, so that write cannot interleave withConnectBlockor another request.- Size: no limit of its own beyond relay policy and block size. Every chunk is stored, plus a second copy of the bytes for complete blobs.
/api/v1/moduleslistsblobwith empty version, db path and schema 0: the module does not overrideInfo().
Differences from upstream #
BLOB is a Lotus-only protocol; there is no upstream or eCash specification. Differences from this repository's older doc/blob.md:
- That file says ids from 0x80 are reserved; the code has no such rule.
- It says the decoder enforces "minimum-length opcodes". The code enforces the minimum PUSHDATA form, but rejects
OP_0andOP_1toOP_16entirely. - It says a disconnected block makes a completed blob pending again; the code does since 12.4.6 (see Indexing).
- It mentions
-blobminburnperbyteand-DENABLE_BLOB_MODULEas future options; neither exists.
Worked example #
No mainnet BLOB transaction could be fetched: on 2026-11-14 the public node answered 503 node_warming_up ("Rebuilding module indices from genesis") to every /api/v1/blob/* request. The example below is built by hand from the code and is not an on-chain transaction. There is no BLOB unit or functional test in the tree to borrow it from.
Blob: the 7 bytes {"a":1}, content type JSON, one chunk. sha256("{\"a\":1}") = 015abd7f5cc57a2dd94b7590f04ad8084273905ee33ec5cebeae62276a97f862.
Native output script (59 bytes):
6a 04 424c4f42
20 015abd7f5cc57a2dd94b7590f04ad8084273905ee33ec5cebeae62276a97f862
04 00000000
04 00000001
01 01
07 7b2261223a317d| Bytes | Meaning |
|---|---|
6a | OP_RETURN |
04 424c4f42 | push 4: "BLOB" |
20 015a...f862 | push 32: blob_id |
04 00000000 | push 4: chunk_index = 0 |
04 00000001 | push 4: total_chunks = 1 |
01 01 | push 1: content_type_id = 1 (application/json) |
07 7b2261223a317d | push 7: payload {"a":1} |
Once mined, the block's ConnectBlock inserts one blob_chunks row, TryCompleteBlob finds 1 of 1 chunks, the hash matches, and GET /api/v1/blob/015abd7f...f862 returns {"a":1} with Content-Type: application/json.
The same chunk over eMPP (52-byte segment):
6a 50 34 424c4f42 015abd7f5cc57a2dd94b7590f04ad8084273905ee33ec5cebeae62276a97f862 00000000 00000001 01 7b2261223a317d6a OP_RETURN, 50 OP_RESERVED, 34 push 52 bytes: "BLOB", then blob_id, index 0, total 1, type 1 and the payload. It is stored with source = 'empp'.
Source files #
src/modules/blob/blob_module.h,src/modules/blob/blob_module.cpp: decoder, schema, reassembly, REST.src/modules/empp/empp_module.cpp(SegmentsForBlock),src/modules/empp/empp_decoder.cpp: eMPP transport.src/modules/builtin_modules.cpp,src/modules/module_registry.cpp: registration order, reconciliation.src/script/standard.h:MAX_OP_RETURN_RELAY.share/www/dashboard/js/dashboard.js: BLOB tab.src/test/empp_decoder_tests.cpp: uses "BLOB" as a LOKAD id in eMPP envelope tests only.- Older notes:
doc/blob.md(includes the client-sideblobStore()helper of the poker app),doc/lotus-poker-nft-v1.md(lotus-blob://URIs).
