Willow Transfer Protocol
Status: Sketch (as of 29.01.2026)
The Willow data model specifies how to arrange data, but it does not prescribe how peers can exchange it. The Willow Transfer Protocol is a fairly simple protocol with HTTP-inspiredWhile the semantics are reminiscent of HTTP, the WTP and HTTP wire encodings are completely unrelated. GET and PUT-like requests to allow a client to access and write Willow data.
This document assumes familiarity with the Willow data model.
Introduction
The WTP is an asymmetric protocol in which a client proactively sends requests to a purely reactive server, who replies with a single response to each request. The client implicitly trusts the server to be allowed to know about all the metadata of its requests, such as NamespaceIds, SubspaceIds, Paths, or AuthorisationTokens. The server, however, does not need to trust the client at all — the client must prove its access rights for all its requests.
The client can request individual Entries and optionally their Payloads, it can also request data by AreaOfInterest or 3dRange, it can request metadata for a purely client-driven range-based set reconciliation, and it can send new Entries and their Payloads to the server. The server does not need to do anything but to reply to incoming requests — and it can always reply that it refused to do what was requested. The server needs to maintain only a small, constant amount of state per session (typically eclipsed by the state required for maintaining the network connection itself).
Whereas the Confidential Sync protocol assembles some sophisticated techniques to allow for high confidentiality between completely untrusted peers, supports bidirectional eager forwarding of novel information, and multiplexes several independent data streams, the WTP makes simplifying trust assumptions, has an unidirectionally flow of initiative, and places responsibility for avoiding head-of-line blocking on the client. In exchange, it goes easy on the computational resources of the server, and it is actually enjoyable and straightforward to implement.
Partial implementations of the WTP can meaningfully interact with fully-featured ones. The client need not be able to process replies to types of requests it never makes. The server can refuse to process any incoming request, and it can communicate when the reason is due to unimplemented features. This allows the client to adapt its behaviour and to refrain from issuing requests the server does not support.
The WTP runs over any reliable, ordered, bidirectional, byte-oriented communication channel. If the client plans on working with non-public data, the communication channel must be confidential and it must be impossible for an active attacker to inpersonate the trusted server. If the client only works with public data, the channel is allowed to be non-confidential. Eavesdroppers might then learn about the client’s interests; it is up to the client to gauge whether that is acceptable.
Parameters
See Willow’25 for a default recommendation of parameters.The WTP is generic over specific cryptographic primitives. In order to use it, one must first specify a full suite of instantiations of the parameters of the core Willow data model. The hash_payload function must be a member of the Bab family of hash functions. Additionally, the WTP requires the following parameters:
A type ReadCapability of read capabilities,We recommend the Meadowcap McCapabilities with an access mode of read as the type of ReadCapabilities. a type Receiver of receivers, and the type Signature of signatures issued by the Receiver.
A type Fingerprint of Fingerprints (i.e., of hashes of LengthyAuthorisedEntries), and a hash function hash_lengthy_authorised_entries from finite sets of LengthyAuthorisedEntries to Fingerprint.
Protocol
The WTP is message-based. The first message sent by each peer is a dedicated setup message. After having sent its setup message, the client can send any number of request messages, in arbitrary order. The server replies to each request message with exactly one response message.
Peers might receive invalid messages, both syntactically (i.e., invalid encodings) and semantically (i.e., logically inconsistent messages). In both cases, the peer to detect this behaviour must abort the sync session. We indicate such situations by writing that something “is an error”. Whenever we state that a message must fulfil some criteria, but a peer receives a message that does not fulfil these criteria, that is an error.
We define all messages as purely logical data types first, with the actual wire encoding defined last.
Setup Messages
We start the message descriptions with the setup messages. The ServerSetupMessage contains a challenge, a 128-bit random number for which the client must provide a valid signature in its own setup message. The ServerSetupMessage also contains some (non-binding) information about which WTP features the server implements. For each feature, the server supplies an Availability:
The server does not want to supply any information about this feature. Reasons might include reducing the surface for fingerprinting particular implementations or deployments, or perhaps the implementer did not want to spend time on providing accurate information. The feature is available. The client should assume that requests which rely on this feature will get a satisfactory reply.}The ServerSetupMessage consists of the challenge plus the Availability information for various features:
The first message sent by the server. The various Availability serve as optimisation hints for the client, but they are not binding: the server might claim that a feature is Available, yet later reply to a request that the feature is unsupported. Such behaviour is obviously not ideal, but not expressly forbidden. Whether the server supports properly responding to RequestGet messages. Whether the server supports serving raw, byte-indexed Payloads slices in its responses to RequestGet messages (as opposed to Bab-based authenticated slices). Whether the server supports serving Bab authenticated Payloads slices in its responses to RequestGet messages (as opposed to raw, byte-indexed slices). Whether the server supports serving Bab authenticated Payloads slices in its responses to RequestGet messages with a k other than 1. Whether the server supports storing raw, byte-indexed Payload slices of RequestPut messages (as opposed to Bab-based authenticated slices). Whether the server supports storing Bab authenticated Payload slices of RequestPut messages (as opposed to raw, byte-indexed slices). Whether the server supports storing Bab authenticated Payload slices of RequestPut messages with a k other than 1.}The first message sent by the client consists of a Receiver, and a signature with that Receiver over the previously received challenge issued by the server. This Receiver must be the receiver of every ReadCapability which the client will supply over the course of this WTP session. This Receiver is one of the few pieces of state the server must maintain for the duration of the WTP session.
The first message sent by the client. If the underlying signature scheme is secure, a valid ClientSetupMessage can only be sent after the client has received the ServerSetupMessage message. A signature issued by the read_capability_receiver over the challenge of the previously received ServerSetupMessage.}Requests and Responses
Now, we describe the requests the client can make, and the corresponding responses with which the server can reply.
The server can reply to requests in arbitrary order. To map responses to requests, each request is implictly assigned an unsigned 64-bit RequestId: the first11The ClientSetupMessage does not count. request sent by the client has RequestId zero, the next has RequestId one, and so on. When reaching22This will never happen in practice. is a pretty large number. instead continue with RequestId zero again.
There are three different kinds of requests, and three different kinds of request-specific responses. The responses each consist of the RequestId of the request to which they reply, followed by a (request-specific) status code, followed by the (request-specific) response data — if the status code indicated success.
RequestGet
The first kind of request allows to request a specific contiguous slice of the Payload of a specific AuthorisedEntry. The requested slice might be empty, which amounts to requesting only the AuthorisedEntry itself.
Request a contiguous, possibly empty subslice of the Payload of a specific AuthorisedEntry. The namespace_id of the requested AuthorisedEntry. The subspace_id of the requested AuthorisedEntry. The path of the requested AuthorisedEntry. A ReadCapability whose receiver must be the read_capability_receiver of the ClientSetupMessage, whose granted namespace must be the requested namespace_id, and whose granted area must be able to include Entries of the requested subspace_id and path. Optionally the expected payload_digest of the requested AuthorisedEntry. If this is not none and the server has an Entry of the correct namespace_id, subspace_id and path, but its payload_digest does not match this value, then the server replies with a non-successful status code and no additional data. If this is true, the response to this request omits the requested AuthorisedEntry, transmitting only the Payload slice. Should only be used with an actual payload_digest, to ensure that the client does not accidentally receive a Payload slice from an unexpected Entry. Specifies a minimum timestamp for any AuthorisedEntry the server might reply with. If this is not none and the server stores a matching AuthorisedEntry, but its timestamp is less than this minimum_timestamp, the server responds with the too_old status code instead of supplying the AuthorisedEntry (and/or a part of its Payload).
If this is not none, and the payload_digest is also not none, the semantics of the payload_digest change: it does not need to match exactly any more, instead it factors into the newer-than relation: if the server has an appropriate AuthorisedEntry of timestamp exactly minimum_timestamp, it responds with the too_old status code only if the payload_digest of the candidate AuthorisedEntry is strictly less than the payload_digest.
Finally, this field must be none if skip_entry is true. There is no status code for indicating a mismatch because the message encoding ensures this.
The start offset (zero-indexed, inclusive) of the requested Payload slice. The length of the requested Payload slice. Note that requesting a very large slice might result in a very large reply, which could take a very long time to transmit, blocking off possibly smaller replies to other requests. Hence, server interested in large (slices of) Payloads might want to issue multiple requests for smaller subslices. If this is none, then the slice_from and slice_length fields are number of bytes, and the response simply contains (a prefix of) the Payload slice as raw bytes. If PartialVerification is given, however, slice_from and slice_length are numbers of Bab chunks. The response then contains not a raw subslice of the Payload, but part of a Bab verifiable stream. The details are described on the PartialVerification struct.}Options for controlling the Bab-based, verifiable transmission of a Payload slice. Bab-based Payload slice transmission uses Bab’s k-grouped light slice streaming, with the value of k specified in these options. The left_skip for the slice stream. The right_skip for the slice stream.}ResponseGet
The response to a RequestGet message starts with the RequestId of the RequestGet message, followed by one of the following status codes:
The different status codes in a response to a RequestGet message. If multiple codes would apply, the one listed earliest takes precedence. The request could be processed, and the server stored an appropriate AuthorisedEntry. The response contains it, unless skip_entry was true. The request was not processed. The feature_get of the ServerSetupMessage gives more information. The receiver of the read_capability was not the read_capability_receiver in the client’s ClientSetupMessage.
The server processed the reqest, but did not have an AuthorisedEntry of matching namespace_id, subspace_id and path. The server processed the reqest, did have an AuthorisedEntry of matching namespace_id, subspace_id and path, but the request had a non-none minimum_timestamp, and the timestamp of the matching AuthorisedEntry was strictly less than the minimum_timestamp (or, if the request’s payload_digest is not none, this status code is also sent if the timestamp of the matching AuthorisedEntry is equal to the minimum_timestamp but its payload_digest is strictly less than the request’s payload_digest). Exactly the same as too_old, but the server is also kindly asking the client to bring it up to speed with some RequestPut messages. The server processed the reqest, did have an AuthorisedEntry of matching namespace_id, subspace_id and path, but its payload_digest did not match the expected payload_digest specified in the request. The server processed the reqest, did have an AuthorisedEntry of matching namespace_id, subspace_id and path, but its timestamp did not fall into the TimeRange of the granted area of the read_capability.
}Every ResponseGetStatusCode but yay marks the end of the response. If the ResponseGetStatusCode is yay, the response continues with the requested AuthorisedEntry (skipped if the request had set skip_entry to true), followed by a secondary status code to indicate whether the requested Payload slice can be served:
The different status codes indicating whether a Payload slice could be served in response to a RequestGet message. If multiple codes would apply, the one listed earliest takes precedence. The request could be processed, and a prefix of the requested Payload slice is part of this response. The request for a Payload slice was not processed. The feature_get_payload of the ServerSetupMessage gives more information. The request for a Payload slice was not processed, because the partial_verification was none. The feature_get_raw_slices of the ServerSetupMessage gives more information. The request for a Payload slice was not processed, because the partial_verification was not none. The feature_get_authenticated_slices of the ServerSetupMessage gives more information. The request for a Payload slice was not processed, because the partial_verification set k to a value other than 1. The feature_get_fancy_k of the ServerSetupMessage gives more information.}Every ResponseGetPayloadStatusCode but yay marks the end of the response. If the ResponseGetPayloadStatusCode is yay, the response continues with the requested Payload slice, and some accompanying metadata.
More specifically, the response does not have to contain the complete requested slice, but merely a prefix of it. The response first indicates how much of the requested slice is actually part of the response. If the partial_verification of the request was none, the response simply states the number of Payload bytes it contains, starting at the requested slice_from. If the partial_verification of the request was not none, then the response indicates the length of the response slice, measured in Bab chunks. In both cases, the indicated slice length must not exceed the originally requested slice_length.
The slice data itself consists of raw Payload bytes if the partial_verification of the request was none, and of the k-grouped light verifiable slice stream over the indicated number of chunks otherwise, omitting the verification metadata indicated by the left_skip and right_skip of the request. Note that the right_skip indicates which metadata to skip based on the originally requested slice, not based on the prefix with which the server responds.
If the slice_from exceeds payload_length of the addressed AuthorisedEntry, the message must be treated as if slice_length was zero.
Finally, if the response sends a strict prefix of the requested slice instead of the full slice, it contains a boolean to indicate whether the client should issue a new request for the remaining part (for example, if the server simply did not want to send a single, comically large response), or not (for example, if the server has no Payload bytes beyond the cutoff point).
Bringing it all together:
Responds to a RequestGet message. The RequestId of the request to which this responds. The status code for this response. The AuthorisedEntry the the request requested. If the status_code is not yay, or if the request set skip_entry to true, then and only then must this be none. The status code for the Payload part of this response. Must be none if and only if the status_code field is not yay. The information concerning the requested Payload. Must be none if and only if status_code field is not yay or payload_status_code field is not yay.}Information concerning the requested Payload. The length of the prefix of the requested slice contained in this response.
If the partial_verification of the request was none, this is simply the number of bytes in the response slice.
Otherwise, this is the number of Bab chunks this response provides. Note that the actual data it transmits consists of more than just those chunks; it also includes the verification data of the requested k-grouped light slice stream. The length of the actually transmitted slice_data in bytes can (and must) be deterministically computed from the number of chunks and the index of the first included chunk.
The slice data, either as raw bytes or as a Bab stream. The length of this is given by (or can be derived from) slice_length. Whether the client should issue more requests for this Payload in order to obtain the data missing from this response. This must be false if the response slice_length is equal to the requested slice_length.}RequestGetMany
The second kind of request allows to request many LengthyAuthorisedEntries in the same namespace_id simultaneously, either by AreaOfInterest or by 3dRange.
Crucially, there are two optimisations. First, the request may carry a Fingerprint. If the set of requested LengthyAuthorisedEntries hashes to exactly that Fingerprint, the server does not need to respond with its LengthyAuthorisedEntries at all. And second, if the number of LengthyAuthorisedEntries matching the query would exceed a threshold specified in the request, then the server replies with some summary data (Fingerprint, number of matching LengthyAuthorisedEntries, and/or their summed Payload lengths) instead of transmitting all the matching LengthyAuthorisedEntries.
Taken together, these optimisations allow for an entirely optional, client-request-driven range-based set reconciliation. And, less ambitiously, for pagination.
Request multiple LengthyAuthorisedEntries at once, while optionally giving some optimisation conditions for omitting the LengthyAuthorisedEntries in favour of compact metadata. The requested LengthyAuthorisedEntries within the namespace_id. A ReadCapability whose receiver must be the read_capability_receiver of the ClientSetupMessage, and which must include all potential Entries which could possibly be included in the query in the namespace_id. Under some circumstances, the server might reply to this request with compact metadata instead of actual LengthyAuthorisedEntries. When this flag is true, the client wants this metadata to include the Fingerprint over all matched LengthyAuthorisedEntries. Under some circumstances, the server might reply to this request with compact metadata instead of actual LengthyAuthorisedEntries. When this flag is true, the client wants this metadata to include the number of all matched LengthyAuthorisedEntries. Under some circumstances, the server might reply to this request with compact metadata instead of actual LengthyAuthorisedEntries. When this flag is true, the client wants this metadata to include the sum of the payload_lengths of all matched LengthyAuthorisedEntries. Optionally the expected Fingerprint of all LengthyAuthorisedEntries the server has in the query in the namespace_id. If this is not none and the matching LengthyAuthorisedEntries of the server have exactly this Fingerprint, the server can omit from its response all the LengthyAuthorisedEntries. If this is true, fingerprint is not none, but the server opts out of Fingerprint computation, then the server must respond with metadata instead of the actual matching LengthyAuthorisedEntries.
Must be false if fingerprint is none (the encoding ensures this).
If this is nonzero, and the number of matching LengthyAuthorisedEntries is greater than or equal to this value, the server should respond with metadata instead of the actual matching LengthyAuthorisedEntries. If this is true, and the number of matching LengthyAuthorisedEntries is greater than or equal to the threshold_count or the server does not want to compute the number of matches, then the server must respond with metadata instead of the actual matching LengthyAuthorisedEntries.
Must be false if threshold_count is zero (the encoding ensures this).
If this is nonzero, and the sum of the payload_lengths of the matching LengthyAuthorisedEntries is greater than or equal to this value, the server should respond with metadata instead of the actual matching LengthyAuthorisedEntries. If this is true, and the sum of the payload_lengths of the matching LengthyAuthorisedEntries is greater than or equal to the threshold_size or the server does not want to compute the number of matches, then the server must respond with metadata instead of the actual matching LengthyAuthorisedEntries.
Must be false if threshold_size is zero (the encoding ensures this).
The order in which the client would like to receive the LengthyAuthorisedEntries.
The response LengthyAuthorisedEntries must be sorted ascendingly by novelty first, using the subspace_id as tiebreaker, and path as secondary tiebreaker. The response LengthyAuthorisedEntries must be sorted ascendingly by subspace_id first, using the path as tiebreaker, and timestamp as secondary tiebreaker. The response LengthyAuthorisedEntries must be sorted either ascendingly by novelty first, using the subspace_id as tiebreaker, and path as secondary tiebreaker, or ascendingly by subspace_id first, using the path as tiebreaker, and timestamp as secondary tiebreaker. The response LengthyAuthorisedEntries need not be sorted. ,}ResponseGetMany
alj: TODO
RequestPut
The third and final kind of request lets the client push a contiguous slice of the Payload of an AuthorisedEntry to the server. The Payload slice might be empty, which amounts to pushing only the AuthorisedEntry itself.
The AuthorisedEntry to push to the server. The start offset (zero-indexed, inclusive) of the transmitted Payload slice. The length of the transmitted Payload slice. If this is none, then the slice_from and slice_length fields are number of bytes, and transmitted Payload slice is simply a sequence of raw bytes. If PartialVerification is given, however, slice_from and slice_length are numbers of Bab chunks. Then, the transmitted Payload slice is not a raw subslice of the Payload, but part of a Bab verifiable stream. The details are described on the PartialVerification struct. The slice data, either as raw bytes or as a Bab stream. The length of this is given by (or can be derived from) slice_length.}ResponsePut
The response to a RequestPut message starts with the RequestId of the RequestPut message, followed by one of the following status codes:
The different status codes in a response to a RequestPut message. If multiple codes would apply, the one listed earliest takes precedence. The request could be processed, and the server was interested in the AuthorisedEntry it received. The request was not processed. The feature_put of the ServerSetupMessage gives more information. The request could be processed, but the server was not interested in the AuthorisedEntry it received. The request could be processed, and server would have been interested in the AuthorisedEntry it received — if it had not stored that data already. The request could be processed, but the received AuthorisedEntry had to be deleted immediately due to prefix pruning. The AuthorisationToken of the entry was invalid.
}Every ResponsePutStatusCode but yay marks the end of the response. If the ResponsePutStatusCode is yay, the response continues with a secondary status code to indicate whether the requested Payload slice can be served:
The different status codes indicating whether a Payload slice was ingested in response to a RequestPut message. If multiple codes would apply, the one listed earliest takes precedence. The Payload slice was discarded. The feature_put_payload of the ServerSetupMessage gives more information. The Payload slice was discarded, because the partial_verification was none. The feature_put_raw_slices of the ServerSetupMessage gives more information. The Payload slice was discarded, because the partial_verification was not none. The feature_put_authenticated_slices of the ServerSetupMessage gives more information. The Payload slice was discarded, because the partial_verification set k to a value other than 1. The feature_put_fancy_k of the ServerSetupMessage gives more information. The Payload slice was discarded, because the server wishes to store only a single, contiguous prefix of the Payload, but the transmitted slice starts outsidethe prefix the server has so far. The response contains the offset starting from which the server would like to receive a new RequestPut message.}Every ResponsePutPayloadStatusCode but want_earlier_slice marks the end of the response. If the ResponsePutStatusCode is want_earlier_slice, the response continues with a U64 indicating the slice start (in a unit of raw bytes or Bab chunks, depending on the partial_verification of the request) starting from which the server would like to receive the Payload.
Bringing it all together:
Responds to a RequestPut message. The RequestId of the request to which this responds. The status code for this response. The status code regarding whether the Payload slice was stored. Must be none if and only if the status_code field is not yay. The slice start (in a unit of raw bytes or Bab chunks, depending on the partial_verification of the request) starting from which the server would like to receive the Payload again. Must be none if and only if payload_status_code field is not want_earlier_slice.}Encodings
alj: TODO