Willow Confidential Sync

Status: Proposal (as of 21.11.2025)

The Willow data model specifies how to arrange data, but it does not prescribe how peers should synchronise data. In this document, we specify one possible way for performing synchronisation: the Willow Confidential Sync protocol. This document assumes familiarity with the Willow data model.

Introduction

An ornamental drawing of two Willow data stores. Each store is a three-dimensional space, alluding to the path/time/subspace visualisations in the data model specification. Within each data store, a box-shaped area is highlighted. Between these highlighted areas flows a bidirectional stream of documents. Alfie, Betty, and Dalton lounge around the drawing. Confidential Sync aims to be appropriate for a variety of networking settings, particularly those of peer-to-peer systems where the replicating parties might not necessarily trust each other. Quite a bit of engineering went into the Confidential Sync protocol to satisfy the following requirements:

Incremental sync: peers avoid redundant data transfer by detecting regions of common data with relatively sparse communication.
Partial sync: peers synchronise only those regions of data they both care about, at sub-namespace granularity.
Access control: conformant peers only hand out data if the request authorises its access.
Private area overlap detection: peers can discover common interests without disclosing any non-shared information to each other.
Resource control: peers communicate (and enforce) their computational resource limits so as not to overload each other.
Transport independence: peers can communicate over arbitrary reliable, ordered, byte-oriented channels, whether tcp, quic, or unix pipe.
General efficiency: peers can make use of efficient implementation techniques, and the overall bandwidth consumption stays low.

Confidential Sync provides a shared vocabulary for peers to communicate with, but nothing more. It cannot and does not force peers to use it efficiently or to use the most efficient data structures internally. That is a feature! Implementations can start out with inefficient but simple implementation choices and later replace those with better-scaling ones. Throughout that evolution, the implementations stay compatible with any other implementation, regardless of its degree of sophistication.

Concepts

Data synchronisation for Willow needs to solve a number of sub-problems, which we summarise in this section.

Private Interest Overlap Detection

Confidential Sync lets two peers determine which namespaces and Areas therein they share an interest in, without leaking any data that only one of them wishes to synchronise. We explain the underlying private interest overlap detection protocol here. That protocol also covers read access control.

Partial Synchronisation

To synchronise data, peers specify any number of AreasOfInterestNote that peers need abide to the max_count and max_size limits of the AreasOfInterest only on a best-effort basis. Imagine Betty has just transmitted her 100 newest Entries to Alfie, only to then receive an even newer Entry from Gemma. Betty should forward that Entry to Alfie, despite that putting her total number of transmissions above the limit of 100. per namespace. The non-empty intersections of AreasOfInterest from both peers contain the Entries to synchronise.

Confidential Sync synchronises these intersections via 3d range-based set reconciliation, a technique we explain in detail here.

Post-Reconciliation Forwarding

After performing set reconciliation, peers might receive new Entries that fall into their shared AreasOfInterest. Hence, Confidential Sync allows peers to transmit Entries unsolicitedly.

Payload Transmission

When a peer sends an Entry, it can choose whether to immediately transmit the corresponding Payload as well. Peers exchange preferences11These preferences are not binding. The number of intersections between the peers’ AreasOfInterest can be quadratic in the number of AreasOfInterest, and we do not want to mandate keeping a quadratic amount of state. for eager or lazy Payload transmission based on payload_lengths for each intersection. These preferences are expressive enough to implement the plumtree algorithm (Leitao et al., 2007).

Peers can further explicitly request the Payloads of arbitrary Entries (that they are allowed to access).

Verified Payload Streaming

If transmission of a Payload is cut short (say, because the internet connection drops), peers should be able to work with the data they had received so far. But this can only be done safely if they can verify that the data is indeed a prefix of the expected Payload. To enable this, Confidential Sync expects PayloadDigest to be the digest of a Merkle-tree-based hash function.

Resource Limits

Multiplexing and management of shared state require peers to inform each other of their resource limits, lest one peer overload the other. We use a protocol-agnostic solution based on logical channels and resource handles that we describe here.

Parameters

See Willow’25 for a default recommendation of parameters.Confidential Sync is generic over specific cryptographic primitives. In order to use it, one must first specify a full suite of instantiations of the parameters of the core Willow data model, as well as the following parameters:

Access control and private interest overlap detection require a type ReadCapability of read capabilities,We recommend the Meadowcap McCapabilities with an access mode of read as the type of ReadCapabilities. a type Receiver of receivers, and a type EnumerationCapability of enumeration capabilitiesWe recommend the meadowcap-compatible McEnumerationCapabilities as the type of EnumerationCapabilities. whose receivers are of type Receiver. We require a hash function hash_interests to hash salted PrivateInterests to bytestrings of the fixed width interest_hash_length (the h function from the private area overlap detection sub-spec). The handshake and encryption of the communication channel are out of scope of Confidential Sync,We recommend a handshake and subsequent encryption scheme here. but the ini_pk and res_pk must be of type Receiver.

3d range-based set reconciliation requires a type Fingerprint of Fingerprints (i.e., of hashes of LengthyAuthorisedEntries), and a hash function hash_lengthy_authorised_entries from finite sets of LengthyAuthorisedEntries to Fingerprint.

We specify the requirements for verified streaming in a rather abstract way: there must be a function22A function in the mathematical sense, you would not actually implement the transformation in this form. transform_payload that maps a Payload into a sequence of up to $2^{64} - 1$ bytestrings, called the transformed Chunks. The number and lengths of the Chunks must be fully determined by the length of the Payload, the actual values of the bytes of the Payload must not affect the number and length of the Chunk.To give an example of how this construction maps to verifiable streaming: transform_payload could map a Payload to a Bab baseline verifiable stream. Peers exchange concatenations of Chunks instead of actual payloads, and communicate offsets in that data in terms of Chunk offsets.

Finally, we require a NamespaceId default_namespace_id, a SubspaceId default_subspace_id, a PayloadDigest default_payload_digest, and an AuthorisationToken default_authorisation_token which authorises default_entry(default_namespace_id, default_subspace_id, default_payload_digest).

Protocol

The protocol is message-based. To break symmetry, we refer to the peer that initiated the synchronisation session as Alfie, and the other peer as Betty.

Peers might receive invalid messages, both syntactically (i.e., invalid encodings) and semantically (i.e., logically inconsistent messages). In both cases, the peer to detect this behaviour must abort the sync session. We indicate such situations by writing that something “is an error”. Any message that refers to a fully freed resource handle is an error. More generally, whenever we state that a message must fulfil some criteria, but a peer receives a message that does not fulfil these criteria, that is an error.

Confidential Sync is a purely message-based protocol, built on top of LCMUX. Both peers act as an LCMUX client and an LCMUX server simultaneously. There are several kinds of messages, which the peers create, encode as byte strings, and then transmit via LCMUX SendChannelFrames and SendGlobalFrames.

The messages make use of the following resource handles:

1
The different resource handles employed by the Confidential Sync.
2
enum HandleType {
3
 
Resource handle for the hash-boolean pairs transmitted during private interest overlap detection.
4
 
OverlapHandle
5
 
Resource handle for ReadCapabilities that certify access to some Entries.
6
 
ReadCapabilityHandle
7
 
Resource handle for explicitly requesting (parts of) Payloads beyond what is exchanged automatically.
8
 
PayloadRequestHandle
9
}

The messages are divided across the following logical channels:

1
The different logical channels employed by the Confidential Sync.
2
enum LogicalChannel {
3
 
Logical channel for performing 3d range-based set reconciliation. Channel id:  $0 .$ 
4
 
ReconciliationChannel
5
 
Logical channel for transmitting Entries and Payloads outside of 3d range-based set reconciliation. Channel id:  $1 .$ 
6
 
DataChannel
7
 
Logical channel for controlling the binding of new OverlapHandles. Channel id:  $2 .$ 
8
 
OverlapChannel
9
 
Logical channel for controlling the binding of new CapabilityHandles. Channel id:  $3 .$ 
10
 
CapabilityChannel
11
 
Logical channel for controlling the binding of new PayloadRequestHandles. Channel id:  $4 .$ 
12
 
PayloadRequestChannel
13
}

Messages

We now define the different kinds of messages.

Private Interest Overlap

In private interest overlap detection, the two peers privately determine which PrivateInterests they share, and prove read access for those interests.

`PioBindHash`

The PioBindHash messages let peers bind hashes of PrivateInterests for later reference during private interest overlap detection, as explained here.

1
Bind data to an OverlapHandle for performing private interest overlap detection.
2
struct PioBindHash {
3
 
The result of applying hash_interests to a PrivateInterest.
4
 
hash: [U8; interest_hash_length],
5
 
Whether the peer is directly interested in the hashed PrivateInterest, or whether it is merely a relaxation.
6
 
actually_interested: Bool,
7
}

PioBindHash messages use the OverlapChannel.

`PioAnnounceOverlap`

The PioAnnounceOverlap messages let peers send overlap announcements.

1
Send an overlap announcement, including its announcement authentication and an optional enumeration capability.
2
struct PioAnnounceOverlap {
3
 
The OverlapHandle (bound by the sender of this message) which is part of the overlap. If there are two handles available, use the one that was bound with actually_interested == true.
4
 
sender_handle: U64,
5
 
The OverlapHandle (bound by the receiver of this message) which is part of the overlap. If there are two handles available, use the one that was bound with actually_interested == true.
6
 
receiver_handle: U64,
7
 
The announcement authentication for this overlap announcement.
8
 
authentication: [U8; interest_hash_length],
9
 
The enumeration capability if this overlap announcement is for an awkward pair, or none otherwise.
10
 
enumeration_capability: EnumerationCapability | none ,
11
}

PioAnnounceOverlap messages are global messages.

`PioBindReadCapability`

The PioBindReadCapability messages let peers transmit read capabilities for overlaps between PrivateInterests. These messages simultaneously declare AreasOfInterest for reconciliation, using the granted namespace and granted area of the bound read capability and a max_count and max_size which are explicitly specified in the message. Further, these messages declare a maximum size for eager payload transmissions within the AreaOfInterest: when an included Entry’s payload_length is strictly greater than the indicated size, its Payload may only be transmitted when explicitly requested.

1
Bind a read capability for an overlap between two PrivateInterests. Additionally, this message specifies an AreaOfInterest which the sender wants to sync.
2
struct PioBindReadCapability {
3
 
The OverlapHandle (bound by the sender of this message) which is part of the overlap. If there are two handles available, use the one that was bound with actually_interested == true.
4
 
sender_handle: U64,
5
 
The OverlapHandle (bound by the receiver of this message) which is part of the overlap. If there are two handles available, use the one that was bound with actually_interested == true.
6
 
receiver_handle: U64,
7
 
The ReadCapability to bind. Its granted namespace must be the (shared) namespace_id of the two PrivateInterests. Its granted area must be included in the less specific of the two PrivateInterests.
8
 
capability: ReadCapability,
9
 
The max_count of the AreaOfInterest that the sender wants to sync.
10
 
max_count: U64,
11
 
The max_size of the AreaOfInterest that the sender wants to sync.
12
 
max_size: U64,
13
 
When the receiver of this message eagerly transmits Entries for the AreaOfInterest defined by this message, it must not include the Payload of Entries whose payload_length is strictly greater than two to the power of the max_payload_power. We call the resulting number the sender’s maximum payload size for this AreaOfInterest.
14
 
max_payload_power: U8,
15
}

It is an error when Alfie sends a capability whose receiver is not ini_pk. Likewise, it is an error when Betty sends a capability whose receiver is not res_pk.

To avoid duplicate 3d range-based set reconciliation sessions for the same Areas, only Alfie should react to sending or receiving PioBindReadCapability messages by initiating set reconciliation. Betty should never initiate reconciliation — unless she considers the redundant bandwidth consumption of duplicate reconciliation less of an issue than having to wait for Alfie to initiate reconciliation.

PioBindReadCapability messages use the CapabilityChannel.

Reconciliation

We employ 3d range-based set reconciliation to synchronise the data of the peers. Our ReconciliationSendFingerprint messages serve to transmit 3dRangeFingerprints; they run on the dedicated ReconciliationChannel to help bounded-memory implementations of range-based set reconciliation gauge how much buffer space is still available.

For 3dRangeEntrySets transmission we need four different message type, in order to keep things streaming, interleavable, and adaptable to store mutation that happens concurrently to syncing. ReconciliationAnnounceEntries messages announce a 3dRange whose Entries will be transmitted. For each Entry in the 3dRange, the peer must then transmit its data. This begins with a ReconciliationSendEntry message to transmit the Entry without its payload, followed by many (possibly zero) ReconciliationSendPayload messages to transmit consecutive chunks of the payload, and terminated by exactly one ReconciliationTerminatePayload message for the Entry. Peers neither preannounceThis is a feature; otherwise syncing concurrently to data insertion (possibly from other sync sessions) would be a nightmare to implement. how many Entries they will send in each 3dRange, nor how many Payload bytes they will transmit in total per Entry.

There is a second concern spanning multiple of the reconciliation messages: peers should know when to proceed from 3d Range-Based Set Reconciliation to eager forwarding of new Entries. Roughly speaking, Confidential Sync supports this by annotating ReconciliationSendFingerprint and ReconciliationAnnounceEntries messages which an id for the original “root” 3dRange which the two peers are actually syching. Peers can track a counter per root 3dRange, incrementing for ReconciliationSendFingerprint messages and decrementing for ReconciliationAnnounceEntries. Once the counter reaches zero, a peer knows that all Entries in the root 3dRange have been synchronised.

To this end, we (implicitly) assign an id to certain messages through the following mechanism: each peer tracks two numbers, alfies_range_counter and betties_range_counter, initialised to $1$ and $2$ respectively. ReconciliationAnnounceEntries and ReconciliationSendFingerprint messages include an optional root_id. Setting that value to none indicates that the message is a root message — this should be done when peers initiate reconciliation for a 3dRange because they detected an overlap of interests, but not when they send a message in response to another reconciliation message. The33Both peers assign the same values to the same messages. root message id of that message is the current value of alfies_range_counter (if Alfie sent the message) or of betties_range_counter (if Betty sent the message). After sending or receiving such a message and assigning the root message id, the corresponding range counter is increased by two.

When a peer responds to a reconciliation message with one or more reciprocal reconciliation messages, these responses should set their root_id to the root message id they pertain to. With this information, peers can track easily whether any root range has been fully reconciled yet or not.A malicious peer might set root_id fields to incorrect values. Implementations should ensure that the only impact of such malicious behavior is incomplete syncing with that particular peer.

For ease of presentation, we bundle the root_id and other fields shared between ReconciliationSendFingerprint and ReconciliationAnnounceEntries messages into a struct:

1
The data shared between ReconciliationSendFingerprint and ReconciliationAnnounceEntries messages.
2
struct RangeInfo {
3
 
Indicates the root message id of the prior root message this message refers to (when set to a non-zero U64), or indicates that this message is a fresh root message itself (when set to none).
4
 
root_id: U64 | none ,
5
 
The 3dRange the message pertains to.
6
 
range: 3dRange,
7
 
A ReadCapabilityHandle bound by the sender of this message. The granted area of the corresponding read capability must fully contain the range.
8
 
sender_handle: U64,
9
 
A ReadCapabilityHandle bound by the receiver of this message. The granted area of the corresponding read capability must fully contain the range.
10
 
receiver_handle: U64,
11
}

Finally, to map transmitted Chunks of Payloads to their Entries, each peer maintains a piece of state, an AuthorisedEntry called its reconciliation_current_entry. It is initialised to the pair of default_entry(default_namespace_id, default_subspace_id, default_payload_digest) and the default_authorisation_token. Upon receiving a ReconciliationSendEntry message, a peer sets its reconciliation_current_entry to the received entry.

`ReconciliationSendFingerprint`

The ReconciliationSendFingerprint messages let peers transmit 3dRangeFingerprints for range-based set reconciliation.

1
Send a Fingerprint as part of 3d range-based set reconciliation.
2
struct ReconciliationSendFingerprint {
3
 
The RangeInfo for this message.
4
 
info: RangeInfo,
5
 
The Fingerprint of all LengthyAuthorisedEntries the peer has in  info.range.
6
 
fingerprint: Fingerprint,
7
}

ReconciliationSendFingerprint messages use the ReconciliationChannel.

`ReconciliationAnnounceEntries`

The ReconciliationAnnounceEntries messages let peers begin transmission of their LengthyAuthorisedEntries in a 3dRange for range-based set reconciliation.

1
Prepare transmission of the LengthyAuthorisedEntries a peer has in a 3dRange as part of 3d range-based set reconciliation.
2
struct ReconciliationAnnounceEntries {
3
 
The RangeInfo for this message.
4
 
info: RangeInfo,
5
 
Must be true if and only if the the sender has zero Entries in  info.range.
6
 
is_empty: Bool,
7
 
A boolean flag to indicate whether the sender wishes to receive a ReconciliationAnnounceEntries message for the same 3dRange in return.
8
 
want_response: Bool,
9
 
Whether the sender promises to send the Entries in  info.range sorted ascendingly by subspace_id , using paths (sorted lexicographically) as the tiebreaker.
10
 
will_sort: Bool,
11
}

Actual transmission of the announced LengthyAuthorisedEntries in info.range happens via ReconciliationSendEntry messages.

Promising (and then fulfilling) sorted transmission via the will_sort enables the receiver to determine which of its own Entries it can omit from a reply in constant space. For unsorted Entries, receivers that cannot allocate a linear amount of memory have to resort to possibly redundant Entry transmissions to uphold the correctness of 3d range-based set reconciliation.

ReconciliationAnnounceEntries messages use the DataChannel.

`ReconciliationSendEntry`

The ReconciliationSendEntry messages let peers transmit LengthyAuthorisedEntries for range-based set reconciliation. Receiving a ReconciliationSendEntry sets the receiver’s reconciliation_current_entry.

1
Send a LengthyAuthorisedEntry as part of 3d range-based set reconciliation.
2
struct ReconciliationSendEntry {
3
 
The LengthyAuthorisedEntry itself.
4
 
entry: LengthyAuthorisedEntry,
5
 
The index of the first (transformed) Payload Chunk that will be transmitted for entry. If you will not transmit any Chunks, this can be set arbitrarily (zero is a good choice).
6
 
offset: U64,
7
}

ReconciliationSendEntry messages use the DataChannel.

`ReconciliationSendPayload`

The ReconciliationSendPayload messages let peers transmit (successive parts of) the concatenation of the transformed Payloads of the receiver’s reconciliation_current_entry immediately during range-based set reconciliation. The sender can freely decide how many (including zero) Chunks to eargerly transmit.

1
Send some Chunks as part of 3d range-based set reconciliation.
2
struct ReconciliationSendPayload {
3
 
The number of transmitted Chunks.
4
 
amount: U64,
5
 
The bytes to transmit, the concatenation of the Chunks obtained by applying transform_payload to the Payload of the receiver’s reconciliation_current_entry, starting at the offset of the corresponding ReconciliationSendEntry message plus the number of Chunks for the current Entry that were already transmitted by prior ReconciliationSendPayload messages.
6
 
bytes: &[U8],
7
}

ReconciliationSendPayload messages use the DataChannel.

`ReconciliationTerminatePayload`

The ReconciliationTerminatePayload messages let peers indicate that they will not send more payload bytes for the current Entry as part of set reconciliation. The messages further indicate whether more LengthyAuthorisedEntries will follow for the current 3dRange.

1
Signal the end of the currentPayload transmission as part of 3d range-based set reconciliation, and indicate whether another LengthyAuthorisedEntry transmission will follow for the current 3dRange.
2
struct ReconciliationTerminatePayload {
3
 
Setting this to true indicates that no further ReconciliationSendEntry message will be sent as part of reconciling the current 3dRange. This
4
 
is_final: Bool,
5
}

ReconciliationTerminatePayload messages use the DataChannel.

Data

After 3d Range-Based Set Reconciliation, peers can forward new Entries and their (transformed) Payloads to each other.

To map transmitted Chunks of Payloads to their Entries, each peer maintains a piece of state, an AuthorisedEntry called its data_current_entry. It is initialised to the pair of default_entry(default_namespace_id, default_subspace_id, default_payload_digest) and the default_authorisation_token. Upon receiving a DataSendEntry message, a peer sets its data_current_entry to the received entry.

`DataSendEntry`

The DataSendEntry messages let peers send AuthorisedEntries outside of set reconciliation. Receiving a DataSendEntry sets the receiver’s data_current_entry.

1
Transmit an AuthorisedEntry and set the receiver’s data_current_entry.
2
struct DataSendEntry {
3
 
The AuthorisedEntry to transmit.
4
 
entry: AuthorisedEntry,
5
 
The index of the first (transformed) Payload Chunk that will be transmitted for entry. Can be set arbitrarily if no Chunks will be transmitted, should be set to  $0$  in that case.
6
 
offset: U64,
7
}

DataSendEntry messages use the DataChannel.

`DataSendPayload`

The DataSendPayload messages let peers transmit (successive parts of) the concatenation of the transformed Payloads of the receiver’s data_current_entry.

1
Send some Chunks of the receiver’s data_current_entry.
2
struct DataSendPayload {
3
 
The number of transmitted Chunks.
4
 
amount: U64,
5
 
The bytes to transmit, the concatenation of the Chunks obtained by applying transform_payload to the Payload of the receiver’s data_current_entryEntry, starting at the offset of the corresponding DataSendEntry message plus the number of Chunks for the current Entry that were already transmitted by prior DataSendPayload messages.
6
 
bytes: &[U8],
7
}

DataSendPayload messages use the DataChannel.

`DataSetEagerness`

The DataSetEagerness messages let peers politely ask44DataSetEagerness messages are not binding, they merely present an optimisation opportunity. the other peer whether to eagerly include the Payloads of Entries which it pushes in some overlap of two read capabilities, or whether to omit the Payloads. This allows peers to implement the Plumtree algorithm (Leitao et al., 2007).

1
Express eagerness preferences for the Payload transmissions in the overlaps of the granted areas of two ReadCapabilities.
2
struct DataSetEagerness {
3
 
A ReadCapabilityHandle bound by the sender of this message. This message pertains to the granted area of the corresponding read capability.
4
 
sender_handle: U64,
5
 
A ReadCapabilityHandle bound by the receiver of this message. This message pertains to the granted area of the corresponding read capability.
6
 
receiver_handle: U64,
7
 
Whether the receiver should eagerly include Payloads when it pushes Entries from the overlap of the granted areas of the ReadCapability corresponding to sender_handle and receiver_handle.
8
 
set_eager: Bool,
9
}

DataSetEagerness messages are global messages.

PayloadRequest

The following messages allow peers to explicitly request Payloads, and to reply to such requests.

`PayloadRequestBindRequest`

The PayloadRequestBindRequest messages let peers issue requests for specific (parts of) Payloads, binding a PayloadRequestHandle in the process (which is used to map responses to requests).

1
Bind a request for (parts of) a Payload.
2
struct PayloadRequestBindRequest {
3
 
The namespace_id of the Entry whose Payload to request.
4
 
namespace_id: NamespaceId,
5
 
The subspace_id of the Entry whose Payload to request.
6
 
subspace_id: SubspaceId,
7
 
The path of the Entry whose Payload to request.
8
 
path: Path,
9
 
The payload_digest of the Entry whose Payload to request.
10
 
payload_digest: PayloadDigest,
11
 
A ReadCapabilityHandle bound by the sender of this message. The granted area of the corresponding read capability must contain the namespace_id, subspace_id, and path.
12
 
sender_handle: U64,
13
 
A ReadCapabilityHandle bound by the receiver of this message. The granted area of the corresponding read capability must contain the namespace_id, subspace_id, and path.
14
 
receiver_handle: U64,
15
}

When receiving a PayloadRequestBindRequest message, a peer must not reply with the corresponding Payload if the Entry’s timestamp does not fall within both the read capability corresponding to sender_handle and that corresponding to receiver_handle.

We will add the ability to query for arbitrary slices (indexed in transformed chunks, not bytes) in the future; this will be part of ongoing grant work.PayloadRequestBindRequest messages use the PayloadRequestChannel.

`PayloadRequestSendResponse`

The PayloadRequestSendResponse messages let peers transmit (successive parts of) the concatenation of the transformed Payloads of requested Entries.

1
Send some Chunks of a requested Entry.
2
struct PayloadRequestSendResponse {
3
 
The PayloadRequestHandle of the request this is responding to.
4
 
handle: U64,
5
 
The number of transmitted Chunks.
6
 
amount: U64,
7
 
The bytes to transmit, the concatenation of the Chunks obtained by applying transform_payload to the Payload of the requestedEntry, starting at the requested offset plus the number of Chunks for the same request that were already transmitted by prior PayloadRequestSendResponse messages.
8
 
bytes: &[U8],
9
}

PayloadRequestSendResponse messages use the DataChannel.

ResourceHandle

The ResourceHandleFree messages let peers indicate they wish to free a resource handle.

1
Indicate that the sender wants to free a resource handle.
2
struct ResourceHandleFree {
3
 
The type of resource handle to free.
4
 
handle_type: HandleType,
5
 
Whether the resource handle to free was bound by the sender (true) or the receiver (false) of this message.
6
 
mine: Bool,
7
 
The numeric id of the resource handle to free.
8
 
handle_id: U64,
9
 
The sender’s reference count for the resource handle to free.
10
 
reference_count: U64,
11
}

ResourceHandleFree messages are global messages.

Encodings

We now describe how to encode the various messages of Confidential Sync. When a peer receives bytes it cannot decode, this is an error.

Parameters

To be able to encode messages, we require certain properties from the protocol parameters:

When using McCapability as the type of ReadCapabilities, you can use EncodeMcCapabilityRelativePrivateInterest.A relative encoding relation EncodeReadCapability encoding ReadCapabilities relative to PersonalPrivateInterests. Note that the information in the PersonalPrivateInterest must not appear in the relative codes, to protect against active eavesdroppers.
When using McEnumerationCapability as the type of EnumerationCapabilities, you can use EncodeMcEnumerationCapabilityRelativePrivateInterest.A relative encoding relation EncodeEnumerationCapability encoding EnumerationCapabilities val relative to the pair of the granted namespace and the receiver of val.enumeration_capability. Note that thegranted namespace must not appear in the relative codes, to protect against active eavesdroppers.
Used indirectly when encoding Entries, Areas, and 3dRanges.An encoding function for SubspaceId.
The total order makes 3dRanges meaningful, the least element and successors ensure that every Area can be expressed as an equivalent 3dRange.A total order on SubspaceId with least element default_subspace_id, in which for every non-maximal SubspaceId s there exists a successor t such that s is less than t and no other SubspaceId is greater than s and less than t.
When using MeadowcapAuthorisationToken as the type of AuthorisationTokens, you can use EncodeMeadowcapAuthorisationTokenRelative.A relative encoding relation EncodeAuthorisationToken encoding AuthorisationTokens relative to pairs of an AuthorisedEntry (the previously transmitted AuthorisedEntry) and an Entry (the Entry currently being transmitted).
An encoding relation EncodeFingerprint for Fingerprint.

We can now define the encodings for all messages, to the be transmitted via LCMUX SendChannelFrames and SendGlobalFrames.

Private Interest Overlap

PioBindHash

We define an encoding relation EncodePioBindHash for PioBindHash. The codes in EncodePioBindHash for any PioBindHash val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0	`1` iff val.actually_interested
1 – 7	Bits can be set arbitrarily.
The raw bytes of val.hash.

PioBindHash messages use the OverlapChannel, so they are transmitted via SendChannelFrames with channel set to $2 .$

PioAnnounceOverlap

We define an encoding relation EncodePioAnnounceOverlap for PioAnnounceOverlap. Let val be any PioAnnounceOverlap.

If val.enumeration_capability is not none, let pair denote the pair of the granted namespace and the receiver of val.enumeration_capability.

Then the codes in EncodePioAnnounceOverlap for val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0, 1	The bitstring `00`.
2	`1` iff val.enumeration_capability is not none
3, 4	A tag of width $2$ for val.sender_handle.
5 – 7	A tag of width $3$ for val.receiver_handle.
The corresponding compact U64 encoding for bits 3, 4.
The corresponding compact U64 encoding for bits 5 – 7.
The raw bytes of val.authentication.
If val.enumeration_capability is not none: any relative code in EncodeEnumerationCapability for val.enumeration_capability, relative to pair, otherwise: the empty string.

PioAnnounceOverlap messages are global messages, so they are transmitted via SendGlobalFrames.

PioBindReadCapability

We define an encoding relation EncodePioBindReadCapability for PioBindReadCapability. Let val be any PioBindReadCapability.

Let ppi denote the PersonalPrivateInterest whose

user_key is
- ini_pk if the sender of val is Alfie,
- or res_pk if the sender of val is Betty, and whose
private_interest is the PrivateInterest whose hash is bound to val.sender_handle.

Then the codes in EncodePioBindReadCapability for val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0, 1	A tag of width $2$ for val.sender_handle.
2, 3	A tag of width $2$ for val.receiver_handle.
4, 5	A tag of width $2$ for val.max_count + 1 modulo $2^{64}$ .
6, 7	A tag of width $2$ for val.max_size + 1 modulo $2^{64}$ .
The corresponding compact U64 encoding for bits 0, 1.
The corresponding compact U64 encoding for bits 2, 3.
The corresponding compact U64 encoding for bits 4, 5.
The corresponding compact U64 encoding for bits 6, 7.
val.max_payload_power.
Any relative code in EncodeReadCapability for val.capability, relative to ppi.

PioBindReadCapability messages use the CapabilityChannel, so they are transmitted via SendChannelFrames with channel set to $3 .$

Reconciliation

To efficiently encode 3dRanges, each peer maintains two pieces of state, a 3dRange called its previously_received_fingerprint_3drange, and a 3dRange called its previously_received_itemset_3drange. Both are initialised to default_3d_range(default_subspace_id). Upon receiving a ReconciliationSendFingerprint or ReconciliationAnnounceEntries message, a peer sets its previously_received_fingerprint_3drange or previously_received_itemset_3drange respectively to the message’s 3dRange.

ReconciliationSendFingerprint

In the following, a root_id of none is encoded as if it was the U64 0.

We define an encoding relation EncodeReconciliationSendFingerprint for ReconciliationSendFingerprint. The codes in EncodeReconciliationSendFingerprint for any ReconciliationSendFingerprint val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 3	A tag of width $4$ for val.info.root_id.
4, 5	A tag of width $2$ for val.info.sender_handle.
6, 7	A tag of width $2$ for val.info.receiver_handle.
The corresponding compact U64 encoding for bits 0 – 3.
The corresponding compact U64 encoding for bits 4, 5.
The corresponding compact U64 encoding for bits 6, 7.
Any relative code in Encode3dRangeRelative3dRange for val.info.range, relative to previously_received_fingerprint_3drange.
Any code in EncodeFingerprint for val.fingerprint.

ReconciliationSendFingerprint messages use the ReconciliationChannel, so they are transmitted via SendChannelFrames with channel set to $0 .$

ReconciliationAnnounceEntries

In the following, a root_id of none is encoded as if it was the U64 0.

We define an encoding relation EncodeReconciliationReconciliationAnnounceEntries for ReconciliationAnnounceEntries. The codes in EncodeReconciliationReconciliationAnnounceEntries for any ReconciliationAnnounceEntries val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 2	The bitstring `000`.
3	`1` iff val.is_empty
4	`1` iff val.want_response
5	`1` iff val.will_sort
6, 7	A tag of width $2$ for val.info.root_id.
8 – 11	A tag of width $4$ for val.info.sender_handle.
12 – 15	A tag of width $4$ for val.info.receiver_handle.
The corresponding compact U64 encoding for bits 6, 7.
The corresponding compact U64 encoding for bits 8 – 11.
The corresponding compact U64 encoding for bits 12 – 15.
Any relative code in Encode3dRangeRelative3dRange for val.info.range, relative to previously_received_itemset_3drange.
Any code in EncodeFingerprint for val.fingerprint.

ReconciliationAnnounceEntries messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

ReconciliationSendEntry

We define an encoding relation EncodeReconciliationSendEntry for ReconciliationSendEntry. Let val be any ReconciliationSendEntry.

Let relative_to_entry be an arbitrary Bool.

Then the codes in EncodeReconciliationSendEntry for val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 2	The bitstring `001`.
3	`1` iff relative_to_entry
4, 5	A tag of width $2$ for val.offset.
6, 7	A tag of width $2$ for val.entry.available.
The corresponding compact U64 encoding for bits 4, 5.
The corresponding compact U64 encoding for bits 6, 7.
If `relative_to_entry`: any relative code in EncodeEntryRelativeEntry for the Entry of val.entry, relative to reconciliation_current_entry, otherwise: any relative code in EncodeEntryInNamespace3dRange for the Entry of val.entry, relative to the pair of previously_received_itemset_3drange and the granted namespace of the info of the preceding ReconciliationAnnounceEntries message.
Any relative code in EncodeAuthorisationToken for the AuthorisationToken of val.entry, relative to reconciliation_current_entry and val.entry.

ReconciliationSendEntry messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

ReconciliationSendPayload

We define an encoding relation EncodeReconciliationSendPayload for ReconciliationSendPayload. The codes in EncodeReconciliationSendPayload for any ReconciliationSendPayload val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 2	The bitstring `010`.
3 – 7	A tag of width $5$ for val.amount.
The corresponding compact U64 encoding for bits 3 – 7.
The raw bytes of val.bytes.

ReconciliationSendPayload messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

ReconciliationTerminatePayload

We define an encoding relation EncodeReconciliationTerminatePayload for ReconciliationTerminatePayload. The codes in EncodeReconciliationTerminatePayload for any ReconciliationTerminatePayload val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 2	The bitstring `011`.
3	`1` iff val.is_final
4 – 7	Bits can be set arbitrarily.

ReconciliationTerminatePayload messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

Data

DataSendEntry

We define an encoding relation EncodeDataSendEntry for DataSendEntry. The codes in EncodeDataSendEntry for any DataSendEntry val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 2	The bitstring `100`.
3 – 7	A tag of width $5$ for val.offset.
The corresponding compact U64 encoding for bits 3 – 7.
Any relative code in EncodeEntryRelativeEntry for the Entry of val.entry, relative to data_current_entry.
Any relative code in EncodeAuthorisationToken for the AuthorisationToken of val.entry, relative to data_current_entry and val.entry.

DataSendEntry messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

DataSendPayload

We define an encoding relation EncodeDataSendPayload for DataSendPayload. The codes in EncodeDataSendPayload for any DataSendPayload val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 2	The bitstring `101`.
3 – 7	A tag of width $5$ for val.amount.
The corresponding compact U64 encoding for bits 3 – 7.
The raw bytes of val.bytes.

DataSendPayload messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

DataSetEagerness

We define an encoding relation EncodeDataSetEagerness for DataSetEagerness. The codes in EncodeDataSetEagerness for any DataSetEagerness val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0, 1	The bitstring `01`.
2	`1` iff val.set_eager
3, 4	A tag of width $2$ for val.sender_handle.
5 – 7	A tag of width $3$ for val.receiver_handle.
The corresponding compact U64 encoding for bits 3, 4.
The corresponding compact U64 encoding for bits 5 – 7.

DataSetEagerness messages are global messages, so they are transmitted via SendGlobalFrames.

PayloadRequest

PayloadRequestBindRequest

We define an encoding relation EncodePayloadRequestBindRequest for PayloadRequestBindRequest. The codes in EncodePayloadRequestBindRequest for any PayloadRequestBindRequest val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0 – 3	A tag of width $4$ for val.sender_handle.
4 – 7	A tag of width $4$ for val.receiver_handle.
The corresponding compact U64 encoding for bits 0 – 3.
The corresponding compact U64 encoding for bits 4 – 7.
The code in encode_namespace_id for val.namespace_id.
The code in encode_subspace_id for val.subspace_id.
Any code in EncodePath for val.path.
The code in encode_payload_digest for val.payload_digest.

PayloadRequestBindRequest messages use the PayloadRequestChannel, so they are transmitted via SendChannelFrames with channel set to $4 .$

PayloadRequestSendResponse

We define an encoding relation EncodePayloadRequestSendResponse for DataSendPayload. The codes in EncodePayloadRequestSendResponse for any DataSendPayload val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0, 1	The bitstring `11`.
2, 3	A tag of width $2$ for val.handle.
4 – 7	A tag of width $4$ for val.amount.
The corresponding compact U64 encoding for bits 2, 3.
The corresponding compact U64 encoding for bits 4 – 7.
The raw bytes of val.bytes.

PayloadRequestSendResponse messages use the DataChannel, so they are transmitted via SendChannelFrames with channel set to $1 .$

ResourceHandle

We define an encoding relation EncodeResourceHandleFree for ResourceHandleFree. The codes in EncodeResourceHandleFree for any ResourceHandleFree val are the bytestrings that are concatenations of the following form:

Bits	Big-Endian Bitfield
0	The bitstring `1`.
1, 2	If `val.handle_type == OverlapHandle`: the bitstring `00`. If `val.handle_type == ReadCapabilityHandle`: the bitstring `01`. If `val.handle_type == PayloadRequestHandle`: the bitstring `00`.
3	`1` iff `val.mine`
4, 5	A tag of width $2$ for val.handle_id.
6, 7	A tag of width $2$ for val.reference_count.
The corresponding compact U64 encoding for bits 4, 5.
The corresponding compact U64 encoding for bits 6, 7.

ResourceHandleFree messages are global messages, so they are transmitted via SendGlobalFrames.

A Confidential Sync emblem: A stylised drawing of a grapevine next to hand-lettered typewriter style rendition of 'Confidential Sync'.

References

Leitao, Joao ; Pereira, Jose ; Rodrigues, Luis: Epidemic broadcast trees. In: 2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007) : IEEE, 2007, pp. 301–310