docs: ARCHITECTURE.md — north star + honest SoC grade
Captures tether as a P2P app backbone, not just a clipboard: - three-tier model (Mesh / Sync / App) with the inter-tier contracts; Mesh+Sync both live in tethercore, App is the only thing outside - transport-agnostic ladder (LAN/RTC/SSE → QUIC → BT/LoRa) behind one trait - two-tier crypto (room key + durable device keypairs), rotation, private discovery (OPRF/PSI lookup, ZKP ownership) — marked target - supernode model: no server, just nodes volunteering signaling/relay/ DHT/discovery; homelab is supernode #1 - §5a: honest separation-of-concerns grade against the actual code — Transport + MessageHandler boundaries are clean; the Mesh/Sync seam is aspirational (Engine holds both), transports mutate sync state, caps are stringly-typed. The MeshEvent refactor is the prerequisite for the Iroh spike — same work. - Iroh + Automerge recommendation; roadmap (ship / next chapter / horizon) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
332
ARCHITECTURE.md
Normal file
332
ARCHITECTURE.md
Normal file
@@ -0,0 +1,332 @@
|
||||
# tether — architecture & north star
|
||||
|
||||
> **What this is.** tether began as a private Universal Clipboard. The clipboard
|
||||
> is the *proof*, not the *point*. The point is the thing underneath it: a
|
||||
> **resilient, transport-agnostic P2P substrate** that any private cross-platform
|
||||
> app can be built on. The clipboard is app #1.
|
||||
|
||||
This document is the durable record of where tether is and where it's going, so
|
||||
the vision survives outside any one conversation. It is a map, not a spec —
|
||||
where it describes things that don't exist yet, they are marked **(target)**.
|
||||
|
||||
---
|
||||
|
||||
## 1. The thesis
|
||||
|
||||
> As long as nodes can come up, find each other, prove who they are, and pass
|
||||
> data — securely, over whatever medium is available — everything else is an
|
||||
> app-layer concern.
|
||||
|
||||
That one sentence is the whole bet. It implies a clean separation into three
|
||||
tiers. Each tier only ever talks to the one below it through a narrow contract,
|
||||
so any tier can be swapped without disturbing the others.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ APP clipboard · (next app) · (your app) │ ← per-app
|
||||
│ reads/writes named state; sends/receives msgs │ code
|
||||
├═══════════════════════════════════════════════════════════════┤
|
||||
│ SYNC CRDT state · message/data passing · presence │ ┐
|
||||
│ turns "bytes between peers" into shared state │ │
|
||||
├─────────────────────────────────────────────────────────────┤ │ tethercore
|
||||
│ MESH identity · transports · discovery · │ │ (one crate,
|
||||
│ (substrate) secure channel · reachability/relay │ │ UniFFI'd to
|
||||
│ promises: "named, authenticated peers you can │ │ every client)
|
||||
│ reach and send bytes to, over any medium" │ ┘
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Tiers vs. crates.** The three tiers are a *contract* boundary, not a packaging
|
||||
one. **Mesh and Sync both live in `tethercore`** — sync is app-*agnostic* (every
|
||||
app reuses dedup, ordering, CRDT merge, delivery, presence), so it belongs in the
|
||||
shared core precisely *because* it's shared, not pushed up into each app. The
|
||||
only thing outside the crate is the App tier. The double line above (`═`) is the
|
||||
crate boundary; the single line (`─`) inside it is the internal Mesh/Sync seam.
|
||||
|
||||
**⚠ Reality check (the seam is aspirational).** As of `33d0fb7` that internal
|
||||
`─` line is a goal, **not implemented**. See §5a for the honest state-of-the-code
|
||||
assessment. The boundaries that *are* real and clean today: the `Transport` trait
|
||||
(Mesh's downward edge) and the `MessageHandler` trait (the App edge). The
|
||||
Mesh/Sync line between them does not yet exist as a boundary — `Engine` holds
|
||||
both — and closing that gap is the prerequisite for the Iroh spike (§8).
|
||||
|
||||
The contracts between tiers:
|
||||
|
||||
- **Mesh → Sync:** "Here is a set of authenticated, named peers. You can send
|
||||
bytes to any of them and receive bytes from them. I don't care what the bytes
|
||||
mean. I'll tell you who's present and reachable; I hide whether that's LAN,
|
||||
WebRTC, QUIC, Bluetooth, or LoRa."
|
||||
- **Sync → App:** "Here is replicated state and a message stream, scoped to an
|
||||
identity. Read it, write it, subscribe to it. I handle conflict resolution,
|
||||
ordering, dedup, and delivery; I hide which peer a change came from."
|
||||
|
||||
Everything tether-specific and clever lives in **Mesh**. Everything reusable
|
||||
across future apps lives in **Sync**. The clipboard is a thin **App**.
|
||||
|
||||
### The one seam that's easy to get wrong
|
||||
|
||||
"Find each other *locally*" is the easy half — mDNS / link-local broadcast on a
|
||||
shared L2 is basically solved. The irreducible hard part of the Mesh — the part
|
||||
that must **not** leak up into the app — is everything that happens when two
|
||||
nodes are *not* on the same network:
|
||||
|
||||
- **Non-local discovery** — peers on different networks finding each other →
|
||||
rendezvous, then DHT.
|
||||
- **NAT traversal** — hole-punching with a relay fallback → supernodes.
|
||||
- **Identity & secure channel** — knowing *who* a peer is and that the link is
|
||||
private → durable keypairs + an authenticated handshake.
|
||||
|
||||
These five — identity, transport, discovery, secure channel, reachability — are
|
||||
the Mesh. Get them right once and the transport medium becomes a detail.
|
||||
|
||||
---
|
||||
|
||||
## 2. Transport-agnostic by construction
|
||||
|
||||
The Mesh sees every medium through one `Transport` trait. The upper tiers never
|
||||
know which one carried a byte. Today's ladder, with room to grow downward:
|
||||
|
||||
| Transport | Status | Reach | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| **LAN** (mDNS + length-prefixed TCP) | shipped | same L2 | zero-infra, fastest, works server-down |
|
||||
| **RTC** (WebRTC data channel) | shipped | cross-network | STUN/TURN via supernode; the web-reachable transport |
|
||||
| **SSE relay** | shipped | cross-network | signaling + last-resort carry of (E2E) ciphertext |
|
||||
| **QUIC** | target | cross-network | likely via Iroh; better than hand-rolled RTC |
|
||||
| **Bluetooth LE** | someday | proximity, no IP | "bitchat-style" — works with no network at all |
|
||||
| **LoRa / other** | someday | long-range, low-bw | same trait, tiny MTU; sync layer must tolerate it |
|
||||
|
||||
The reason BT/LoRa are *plausible later* rather than *fantasy* is precisely the
|
||||
trait boundary: a transport only has to (a) discover peers on its medium and
|
||||
(b) move framed bytes between them. The sync layer already tolerates
|
||||
multi-transport delivery (it dedups across transports today), which is the same
|
||||
discipline a high-latency / low-MTU medium demands.
|
||||
|
||||
---
|
||||
|
||||
## 3. Identity & crypto (the two-tier model)
|
||||
|
||||
Identity is part of the Mesh, not the app. The model is two layers:
|
||||
|
||||
1. **Room key — shared symmetric secret.** Everyone in a room/group can decrypt
|
||||
group content. Today: `key = sha256(account)`, ChaCha20-Poly1305, payload
|
||||
sealed before *any* transport touches it (see `core/src/crypto.rs`). The
|
||||
server and any LAN sniffer see only ciphertext.
|
||||
- **Known weakness (documented, honest):** the account is an email — low
|
||||
entropy. This stops a passive eavesdropper, not someone who knows the
|
||||
account. **(target)** Replace the secret with a high-entropy source
|
||||
(Sign-in-with-Apple `sub`, or a passphrase) fed into the same
|
||||
`from_secret`. No protocol change — just a better secret.
|
||||
|
||||
2. **Device keypairs — durable per-device identity. (target)** Each device has a
|
||||
long-lived X25519 keypair. Peer-to-peer links run an authenticated handshake
|
||||
(Noise via `snow`, or adopt libsignal/MLS rather than hand-roll) so each pair
|
||||
has forward-secret session keys. The room key is then distributed by
|
||||
*sealing it to each device's public key* (sealed-box) instead of being
|
||||
derived from a guessable string.
|
||||
|
||||
**Rotation & unlinkability (target).** New shared key → new room. New device
|
||||
identity → new public key, re-handshake. This buys forward secrecy and
|
||||
unlinkability, but creates a discovery tension (if your identity keeps changing,
|
||||
how does someone find you?). Resolved by a **stable handle + directory
|
||||
indirection**: a durable, user-facing handle maps — *privately* — to whatever
|
||||
the current key/room is.
|
||||
|
||||
**Private discovery (target).** Looking someone up by email/phone/handle without
|
||||
revealing the lookup table:
|
||||
- **OPRF / PSI** for the *lookup* ("do we both know this contact?" without
|
||||
either side learning the other's full set).
|
||||
- **ZKP** for *ownership proofs* ("I control the key behind this handle"), not
|
||||
for the lookup itself.
|
||||
|
||||
These are v2+ research arcs. Don't hand-roll the primitives — stand on vetted
|
||||
libraries.
|
||||
|
||||
---
|
||||
|
||||
## 4. The supernode model (there is no "server")
|
||||
|
||||
There is no server/client split. There are only **nodes**. Some nodes — by
|
||||
virtue of public reachability + uptime + bandwidth — volunteer extra roles.
|
||||
Those are **supernodes** (Skype/ultrapeer/libp2p-relay model). The current Go
|
||||
"server" *is* a supernode; the architecture just generalizes it from *the one*
|
||||
to *one of many, peer-selected.*
|
||||
|
||||
A supernode offers any subset of these — **none of which touch plaintext**:
|
||||
|
||||
- **Signaling / rendezvous** — peer-relayed SDP/ICE exchange (today: the SSE bus).
|
||||
- **TURN relay** — forwards **E2E ciphertext** when hole-punching fails; blind to payload.
|
||||
- **DHT bootstrap + capacity** — stable entry points; hold more of the keyspace.
|
||||
- **Discovery hints** — same-ASN / same-egress-IP co-location hints (the node
|
||||
that can see public IPs). Strong CGNAT false-positive caveat; mostly useful to
|
||||
bias toward the LAN transport and to avoid TURN.
|
||||
- **Store-and-forward mailbox (target)** — hold sealed messages for offline peers.
|
||||
|
||||
**Promotion** = reachable + up + has bandwidth. Auto-detected (a node that finds
|
||||
it's publicly reachable volunteers) or designated (flag the homelab). Patrick's
|
||||
homelab is **supernode #1**; the hosted relay is #2.
|
||||
|
||||
**Why "many, peer-selected" matters:** a single coordinator is a server with
|
||||
extra steps. Several supernodes, cross-checked, are what resist the real failure
|
||||
modes — **eclipse attacks** (one node controlling your view of the network) and
|
||||
**traffic analysis** (metadata is visible even though content is E2E).
|
||||
|
||||
**Two honest constraints:**
|
||||
- **Bootstrap is unavoidable.** You must *know* one supernode to start — a
|
||||
hardcoded list or a DNS name. Even BitTorrent's DHT ships bootstrap nodes.
|
||||
"Decentralized" never meant "zero fixed entry point."
|
||||
- **Incentives at public scale.** For Patrick's own devices, he runs the
|
||||
supernodes. For a *public* backbone, *who runs them* is a real economics
|
||||
question (Skype conscripted user machines — controversial; BitTorrent/Veilid
|
||||
lean on altruism). Decide deliberately, don't default into it.
|
||||
|
||||
---
|
||||
|
||||
## 5. Current state (what actually exists)
|
||||
|
||||
At commit `33d0fb7`. Verified on real hardware: bidirectional cross-network
|
||||
clipboard sync (iPhone-on-cellular ⇄ Mac), file transfer (RTC + LAN),
|
||||
prefer-direct (relay saw only presence), and E2E (relay carried only ciphertext).
|
||||
|
||||
- **`core/`** — `tethercore` Rust crate, the Mesh + a thin Sync. UniFFI-exported
|
||||
to Swift/Kotlin. `Engine` multiplexes LAN + RTC + SSE, dedups across them,
|
||||
prefer-direct (suppress relay once all present peers are directly reachable),
|
||||
delivery receipts (hash, not content), presence, nearby (mDNS), file transfer,
|
||||
E2E (`crypto.rs`). Account-derived room: `room = sha256(account)[..4]`.
|
||||
- **`agent/`** — headless desktop node; installs as a launchd/systemd service;
|
||||
clipboard loop; saves received files.
|
||||
- **`ios/`** — SwiftUI iPhone + Mac apps over the same core (`TetherKit`).
|
||||
- **`android/`** — scaffold; needs NDK build wired up.
|
||||
- **`server/`** — Go supernode: pion signaling + integrated TURN +
|
||||
`/api/turn-cred` (HMAC ephemeral creds) + web client. **⚠ The deployed binary
|
||||
is ahead of this source** (integrated TURN, advanced web client, built from an
|
||||
uncommitted tree ~2026-05-21). Do **not** redeploy `server/` as-is — it would
|
||||
regress TURN. Recovering that source is an open task.
|
||||
|
||||
What's **Mesh** today: identity (weak), LAN/RTC/SSE transports, mDNS + relay
|
||||
discovery, E2E channel, presence/reachability. What's **Sync** today: message
|
||||
passing, dedup, receipts, file frames — but *no replicated state* (no CRDT). What's
|
||||
**App**: the clipboard UIs.
|
||||
|
||||
### 5a. Separation-of-concerns: honest grade
|
||||
|
||||
Graded against the code (`core/src/lib.rs`), not the diagram.
|
||||
|
||||
**Clean and real:**
|
||||
- **`Transport` trait** (`lib.rs:270`) — the strongest boundary. SSE/RTC/LAN each
|
||||
implement it; adding QUIC/BT is "implement the trait." Mesh's downward edge,
|
||||
done right.
|
||||
- **`MessageHandler` trait** (`lib.rs:59`) — the App edge. The clipboard reaches
|
||||
core only through `on_message`/`on_status`/`on_file`; apps never touch engine
|
||||
internals.
|
||||
- **`crypto.rs`** — isolated; one seal point at the engine boundary
|
||||
(`lib.rs:382`) before any transport.
|
||||
|
||||
**Not yet real (the gaps to fix before building on this):**
|
||||
1. **No Mesh/Sync seam.** `Engine` (`lib.rs:291`) holds *both* transport
|
||||
orchestration and dedup/receipts/presence/transfers. Swapping the Mesh (e.g.
|
||||
Iroh) today means rewriting `Engine`, not replacing a module.
|
||||
2. **Transports mutate Sync state — dependency arrow points the wrong way.**
|
||||
`RtcTransport::new(cfg, direct, present, transfers)` (`lib.rs:339`) and
|
||||
`LanTransport::new(cfg, discovered, direct, transfers)` (`lib.rs:345`) hand
|
||||
transports the engine's state maps, which they write directly via shared
|
||||
`Arc<Mutex>`. A transport should move bytes, not own presence/progress. This
|
||||
is the biggest smell and the reason the Mesh isn't swappable.
|
||||
3. **Stringly-typed capabilities.** `send_file` gates carriers with
|
||||
`t.name() == "rtc" || t.name() == "lan"` (`lib.rs:386`); `Transport::send_file`
|
||||
is a default-empty method only 2 of 3 honor. Capability should be typed
|
||||
(a `DirectTransport` sub-trait or a `caps()` flag), not a display-string match.
|
||||
4. **`Engine` is a 745-line god object trending up** — every feature bolted on a
|
||||
field + method. Coherent now; watch it.
|
||||
|
||||
**The one move that fixes the most — `MeshEvent`.** Invert #2: transports *emit*
|
||||
events (`PeerUp`/`PeerDown`/`Frame`/`Presence`/`FileChunk`) and the engine owns
|
||||
all state and reacts. That single change creates the real Mesh/Sync boundary
|
||||
(§1), makes Iroh a drop-in event source, and removes the `Arc<Mutex>`-passing
|
||||
coupling. The refactor and the Iroh spike are therefore the *same work* — do this
|
||||
first.
|
||||
|
||||
---
|
||||
|
||||
## 6. The big gaps (clipboard → backbone)
|
||||
|
||||
The hard networking kernel exists. To become a general substrate:
|
||||
|
||||
1. **State sync, not just messages.** Apps need replicated *state* (a doc, list,
|
||||
board), conflict-free across peers → **CRDTs (Automerge / Yjs).** This is the
|
||||
biggest single addition and the heart of the Sync tier. The clipboard is
|
||||
stateless message-passing; a backbone is replicated state.
|
||||
2. **An app API / SDK.** A clean surface: declare data, subscribe, send. The
|
||||
thing a developer building app #2 actually touches.
|
||||
3. **Web target = WASM.** The Rust core compiles to WASM, but the browser
|
||||
sandbox limits web peers to WebRTC / WebSocket / WebTransport (no raw
|
||||
UDP/TCP/mDNS — same wall as iOS's multicast entitlement). Web is a
|
||||
first-class but transport-limited peer.
|
||||
4. **Capabilities / permissions (target).** Multi-user needs object-capabilities
|
||||
(UCAN), not ACLs.
|
||||
5. **Persistence / offline-first (target).** Local store + sync-on-reconnect
|
||||
(append-only log or CRDT doc store).
|
||||
|
||||
---
|
||||
|
||||
## 7. Don't reinvent the substrate
|
||||
|
||||
This space has serious, aligned prior art. The moat is **product/DX/UX** (the
|
||||
"AirDrop-grade just works" + the app model) — **not** the transport/DHT
|
||||
plumbing, which is being commoditized. Stand on vetted infra:
|
||||
|
||||
- **Iroh** (n0, Rust) — QUIC P2P + relay hole-punching + discovery
|
||||
(DNS/pkarr-DHT) + `iroh-docs`/`iroh-blobs` for sync. Possibly ~70% of the Mesh
|
||||
already, in Rust. Its **relay nodes = our supernodes**; its bootstrap/DHT = our
|
||||
rendezvous. **Evaluate first.**
|
||||
- **libp2p** — the components (relay-v2 + DCUtR + Kademlia) if you want them à la carte.
|
||||
- **Veilid** (cDc, Rust) — privacy-first P2P app framework, DHT + E2E. Aligned
|
||||
with the privacy/DHT angle specifically.
|
||||
- **Automerge / Yjs** — the CRDT Sync layer. **UCAN** — capability auth.
|
||||
**Willow / Earthstar** — capability-scoped P2P data.
|
||||
|
||||
**Strategic recommendation:** adopt **Iroh (Mesh) + Automerge (Sync)**, build
|
||||
identity/E2E/discovery-UX + the app API on top, keep the clipboard as the
|
||||
flagship app that proves the DX. Run the homelab as a self-hosted Iroh relay +
|
||||
bootstrap node — that's supernode #1 — and the substrate hands you peer-relay +
|
||||
DHT + multi-supernode selection for free, instead of a year of hand-rolling NAT
|
||||
and crypto and getting it subtly wrong.
|
||||
|
||||
---
|
||||
|
||||
## 8. Roadmap
|
||||
|
||||
**Near-term (ship the product):**
|
||||
- [ ] Coordinated rebuild + reinstall of all clients so E2E actually activates
|
||||
(running apps still carry the old plaintext core).
|
||||
- [ ] Swap the room secret to a high-entropy source (SiwA `sub` / passphrase).
|
||||
- [ ] Recover the uncommitted deployed `server/` source (don't redeploy until then).
|
||||
- [ ] iOS multicast entitlement (Nearby/LAN are dormant on iOS).
|
||||
- [ ] Android NDK build; clipboard-image auto-send; file-size cap.
|
||||
|
||||
**Next chapter (start the backbone, clean):**
|
||||
- [ ] **Extract the Mesh/Sync seam via `MeshEvent`** (§5a) — transports emit
|
||||
events, engine owns state. Prerequisite for everything below; fixes the
|
||||
transport→sync coupling and the stringly-typed capability check.
|
||||
- [ ] Spike **Iroh** as the Mesh — homelab as self-hosted relay/bootstrap
|
||||
(supernode #1). Compare against the current hand-rolled RTC/LAN/SSE.
|
||||
(Drops in as a `MeshEvent` source once the seam exists.)
|
||||
- [ ] Spike **Automerge** as the Sync tier; re-express the clipboard as a tiny
|
||||
app over replicated state to validate the App API.
|
||||
- [ ] Durable device keypairs + authenticated handshake (Noise/`snow`, or
|
||||
libsignal/MLS); seal the room key to device pubkeys.
|
||||
|
||||
**Horizon (the real vision — stateless P2P app backbone):**
|
||||
- [ ] DHT + multi-supernode peer-relay (mostly *adopted* via Iroh/libp2p, not built).
|
||||
- [ ] Rotatable rooms/keys/identities + stable-handle directory indirection.
|
||||
- [ ] Private discovery (OPRF/PSI lookup; ZKP ownership proofs).
|
||||
- [ ] Capability model (UCAN); offline mailbox; web (WASM) peer.
|
||||
|
||||
---
|
||||
|
||||
## 9. One-line summary
|
||||
|
||||
> **Peers with durable keypairs · E2E content · pluggable transports
|
||||
> (LAN/RTC/QUIC/BT/LoRa) · supernodes volunteering signaling/relay/DHT/discovery
|
||||
> · CRDT state sync · clipboard as app #1.** Built on Iroh + Automerge,
|
||||
> differentiated on the "just works" DX. The network is the product; the
|
||||
> clipboard is the proof.
|
||||
Reference in New Issue
Block a user