13 — Sharded distribution
Weft
Deterministic 4 MiB chunking, a signed binary Merkle manifest, and parallel fetch with per-chunk verification. Weft is what lets MARS-anchored artifacts and Sigil-committed datasets cross the P2P fabric without trusting any single peer. Records the DHT serves: /weft/v1/manifest, /weft/v1/chunk, /weft/v1/availability.
A signed Merkle manifest describes the chunks. Parallel fetch assembles the payload while each piece is verified against its address.
- 01Read the signed manifest
- 02Fetch the chunks in parallel
- 03Verify the assembled payload
Deterministic 4 MiB chunking — tunable from 1 byte to 64 MiB.
BLAKE3 chunk IDs + a deterministic binary Merkle tree.
Signed manifests on the weave.weft.manifest.v1 schema.
Parallel fetch with per-chunk verification — bad bytes never reach the app.
DHT records: /weft/v1/manifest/<root>, /weft/v1/chunk/<id>, /weft/v1/availability/<root>.
Pairs with Sigil + MARS — anchored artifacts ship over the P2P fabric.
A handful of lines.
use weft::{Node, ManifestBuilder, FetchConfig};
let node = Node::open(".weave/weft").await?;
// Shred a multi-GB dataset, sign the manifest, publish to the DHT.
let manifest = ManifestBuilder::new()
.chunk_size(4 * 1024 * 1024)
.from_stream(tokio::fs::File::open(&path).await?)
.await?
.sign(&keypair);
let root = node.publish(manifest).await?;
// Anywhere on the network — parallel fetch, per-chunk verify.
let bytes = node.fetch(&root, FetchConfig::parallel(8)).await?;