Store and serve large media with Hyperblobs

Store large files and media as blobs on a Hypercore, replicate them over Hyperswarm, and serve them to a UI over local HTTP with range requests — the Pear/Bare logic, no UI required.

This guide focuses on the Pear/Bare logic. It shows hyperblobs and hypercore-blob-server on their own — no Electron, no UI. For the same blob plumbing wired into full desktop apps, see Stream stored video in a peer-to-peer app, Stream a live camera in a peer-to-peer app, and Back up photos in a peer-to-peer app. The blob logic is identical across all three; only the UI and the frame/file source change. To share a whole folder of files over Hyperdrive instead, see Share files in a peer-to-peer app.

A Hypercore is an append-only log of small blocks — great for messages, awkward for a 200 MB video. Hyperblobs solves that: it chunks arbitrarily large binary data across a Hypercore and hands back a small blob id that addresses it. hypercore-blob-server then serves any blob over local HTTP (127.0.0.1) so a <video> or <img> tag — or any HTTP client — can stream it with range requests, fetching only the bytes it needs.

This is purely Pear-end logic: it runs in a Bare worker (or any Bare/Node process) and never touches a UI.

How addressing works

Writing a blob returns an id describing where the bytes live in the core:

{ byteOffset, blockOffset, blockLength, byteLength }

Combine that id with the blobs core key and you have everything a peer needs to fetch the blob — directly as bytes, or through a blob-server link.

Add the dependencies

npm install corestore hyperswarm hyperblobs hypercore-blob-server hypercore-id-encoding

Write and seed a blob

The writer:

opens a Corestore namespace and wraps it with Hyperblobs (L12–L13)
announces the blobs core on Hyperswarm so readers can find it (L16)
streams a local file through createWriteStream() (L18–L24)
and prints the { key, ...id } payload other peers need (L27–L28):

writer-app/index.js

import Corestore from 'corestore'
import Hyperswarm from 'hyperswarm'
import Hyperblobs from 'hyperblobs'
import idEnc from 'hypercore-id-encoding'
import fs from 'bare-fs'
import process from 'bare-process'

const store = new Corestore('./writer-store')
const swarm = new Hyperswarm()
swarm.on('connection', (conn) => store.replicate(conn))

const blobs = new Hyperblobs(store.get({ name: 'blobs' }))
await blobs.ready()

// Announce the blobs core so readers can discover and replicate it.
swarm.join(blobs.core.discoveryKey, { server: true, client: false })

// Store a local file as a blob.
const ws = blobs.createWriteStream()
fs.createReadStream('./clip.mp4').pipe(ws)
await new Promise((resolve, reject) => {
  ws.on('error', reject)
  ws.on('close', resolve)
})

// Share this with readers — the core key plus the blob id.
const blob = { key: idEnc.normalize(blobs.core.key), ...ws.id }
console.log('blob:', JSON.stringify(blob))

// Keep seeding until interrupted, then tear down cleanly.
process.once('SIGINT', async () => {
  await blobs.close()
  await swarm.destroy()
  await store.close()
  process.exit(0)
})

Read or serve the blob on another peer

The reader:

parses that JSON blob descriptor from the command line (L9)
opens the remote core by key and replicates it (L15–L17)
Option A pulls the raw bytes with blobs.get() and the id fields (L20–L27)
Option B — what desktop apps use — starts hypercore-blob-server and prints a local link the renderer can pass to <video> or <img> (L30–L33)

Option B is what desktop apps use: the worker hands the link to the renderer, and the browser streams the media straight from hypercore-blob-server, requesting byte ranges as the user scrubs.

reader-app/index.js

import Corestore from 'corestore'
import Hyperswarm from 'hyperswarm'
import Hyperblobs from 'hyperblobs'
import BlobServer from 'hypercore-blob-server'
import idEnc from 'hypercore-id-encoding'
import process from 'bare-process'

// { key, byteOffset, blockOffset, blockLength, byteLength }
const blob = JSON.parse(process.argv[2])

const store = new Corestore('./reader-store')
const swarm = new Hyperswarm()
swarm.on('connection', (conn) => store.replicate(conn))

const core = store.get({ key: idEnc.decode(blob.key) })
await core.ready()
swarm.join(core.discoveryKey, { client: true, server: false })

// Option A — read the raw bytes.
const blobs = new Hyperblobs(core)
const bytes = await blobs.get({
  byteOffset: blob.byteOffset,
  blockOffset: blob.blockOffset,
  blockLength: blob.blockLength,
  byteLength: blob.byteLength
})
console.log('read bytes:', bytes.byteLength)

// Option B — serve over local HTTP so a <video>/<img> can stream it with range requests.
const server = new BlobServer(store.session())
await server.listen()
const link = server.getLink(blob.key, { blob, type: 'video/mp4' })
console.log('stream from:', link)

// Close the server and blobs before the swarm and store on exit.
process.once('SIGINT', async () => {
  await server.close()
  await blobs.close()
  await swarm.destroy()
  await store.close()
  process.exit(0)
})

Tear it down

Each app cleans up on Ctrl+C: the SIGINT handler at the end of every file closes the blobs (and, in the reader, the HTTP server) before the swarm and store, so in-flight reads and the local server shut down before the underlying core does.