CBOR Sequences: Streaming Independent Data Items
Introduction: Beyond Single Items
Previous chapters have explored the encoding of individual Concise Binary Object Representation (CBOR) data items, covering fundamental types like integers, strings, booleans, and null, as well as structured types like arrays and maps. We examined how definite-length and indefinite-length encodings work, and how semantic tags (Major Type 6) extend the basic data model. The focus thus far has been on representing self-contained, individual data structures, analogous to a single JSON document or a distinct object in memory.
However, many real-world applications involve data that doesn't naturally fit into a single, monolithic structure. Consider scenarios like generating log entries over time, receiving a continuous stream of sensor measurements, or exchanging a series of independent commands and responses between systems. While it's possible to wrap such sequences within a top-level CBOR array (Major Type 4), perhaps even an indefinite-length one, this approach can be inefficient or semantically awkward. It forces a collection structure onto items that might be fundamentally independent, and it requires either knowing the total count upfront (for definite-length arrays) or managing start and end markers (for indefinite-length arrays).
To address these scenarios more directly and efficiently, the IETF defined CBOR Sequences in (RFC-8742). A CBOR Sequence provides a way to represent a stream or series of distinct CBOR data items without enclosing them in an overarching container like an array.
Formally, a CBOR Sequence is defined recursively as a sequence of bytes that is either:
- An empty (zero-length) sequence of bytes.
- The sequence of bytes representing a single, well-formed encoded CBOR data item (as defined in (RFC-8949)), immediately followed by another CBOR Sequence.
In essence, a CBOR Sequence is generated by simply concatenating the binary encodings of zero or more individual CBOR data items. This concatenation is the core mechanism. Crucially, there are no explicit delimiters, framing bytes, or termination markers inserted between the constituent CBOR items within the sequence itself. This minimalist design is possible because standard CBOR data items are inherently self-delimiting; the initial byte(s) of any CBOR item contain information about its type and length (or value), allowing a parser to determine exactly where that item ends and the next one begins. This contrasts sharply with formats like JSON Text Sequences ((RFC-7464)), which require explicit markers (like the ASCII Record Separator character followed by a newline) between JSON texts because JSON values themselves are not always self-delimiting when concatenated.
The official media type associated with this format is application/cbor-seq
. Additionally, the structured syntax suffix +cbor-seq
has been registered, allowing other media types to indicate that their underlying structure is a CBOR Sequence, analogous to how +cbor
signifies a base of a single CBOR item.
This definition highlights the fundamental simplicity of CBOR Sequences – mere concatenation. The absence of sequence-level headers, item counts, or termination markers is a deliberate design choice rooted in CBOR's self-describing nature. Since each CBOR item encodes its own type and length, a parser can theoretically determine item boundaries without extra framing. Adding such framing would introduce overhead, running counter to CBOR's goal of conciseness, particularly for applications streaming many small items. However, this simplicity places the responsibility of determining the overall end of the sequence entirely on mechanisms external to the sequence format itself, such as the end of a file or the closure of a network connection. This characteristic has significant implications for how sequences are transported and how errors are handled, which will be explored later in this chapter.
Why CBOR Sequences? Technical Motivations
The design of CBOR Sequences stems from specific technical requirements where traditional single-item encodings or array structures fall short. The primary motivations include enabling efficient streaming, facilitating incremental processing, offering potential efficiency gains over arrays, and leveraging the simplicity of concatenation.
-
Streaming Data: CBOR Sequences are particularly well-suited for scenarios where data is generated or consumed continuously over time, and the total volume or number of items is not known when the process begins. Common examples include streaming application logs, transmitting time-series sensor data from IoT devices, or handling real-time event feeds. In such cases, appending a new, independently encoded CBOR item to the end of an existing sequence is straightforward. This contrasts with definite-length CBOR arrays, which require the element count to be specified upfront, and indefinite-length arrays, which, while streamable, still represent a single logical collection that must eventually be terminated by a specific 'break' byte (
0xFF
). Sequences allow indefinite extension without modifying previously transmitted data. -
Incremental Processing: A key advantage is that sequences allow both producers and consumers to operate on discrete items one at a time. A producer can fully encode and transmit a single CBOR item. A consumer can receive the bytes for that item, decode it completely, and process it before the next item even begins to arrive. This model avoids the need for complex streaming parsers or encoders that must handle partially received or generated structures (like elements within a large array). This simplification is especially valuable for resource-constrained environments, such as IoT devices, where memory limitations might make buffering large, monolithic arrays impractical.
-
Efficiency Compared to Arrays: When representing a list or sequence of items, CBOR arrays (Major Type 4) introduce a small amount of overhead. A definite-length array requires an initial byte (or potentially more for very large counts) to encode the number of elements it contains. An indefinite-length array requires an initial byte indicating the indefinite type (
0x9F
for arrays,0xBF
for maps) and must be terminated by a final0xFF
break byte. For a sequence containing N items, using a CBOR array introduces 1 to 9 bytes of structural overhead (for the count or start/end markers). In contrast, a CBOR Sequence adds zero bytes of overhead beyond the concatenated bytes of the items themselves. While this overhead is often negligible, it can become significant when dealing with a very large number of very small items, a common pattern in sensor data or event streams. -
Simplicity of Concatenation: The definition itself highlights this: generating a CBOR Sequence is achieved simply by concatenating the byte representations of individually encoded CBOR items. Furthermore, concatenating two valid CBOR Sequences always results in another valid CBOR Sequence. This property can simplify certain data aggregation pipelines or forwarding proxies where streams of CBOR items from different sources need to be merged.
These motivations reveal a fundamental design choice: CBOR Sequences prioritize the representation of a flow of independent items over a structured collection. CBOR arrays and maps (Major Types 4 and 5) represent semantically coherent, single data items within the CBOR data model; they possess a defined structure and element/pair count. Sequences, lacking this inherent enclosing structure, are better suited for streams where items might be processed individually and may not form a single logical entity. The decision between using an array or a sequence often hinges on whether the data is conceptually viewed as "one large object containing parts" or "many small, sequential objects". If the application requires an explicit marker for the end of the sequence within the data stream itself, (RFC-8742) suggests that encoding the items within a CBOR array might be the more appropriate representation. Choosing sequences implies a shift away from processing a single, potentially large, structure towards processing a series of smaller, independent units.
Encoding: Simple Concatenation
The mechanism for encoding a CBOR Sequence is remarkably straightforward: individually encode each constituent data item according to the standard CBOR encoding rules defined in (RFC-8949) (and detailed in a preceding chapter), and then simply concatenate the resulting byte strings in the desired order. No additional bytes, delimiters, or framing information are introduced between the encoded items as part of the sequence format itself.
Let's illustrate this with hexadecimal examples:
-
Example 1: Simple Sequence
Consider encoding the sequence of values: 1, then "foo", then true.
- The integer
1
(Unsigned Integer, Major Type 0, Additional Information 1) encodes as a single byte:01
. - The text string
"foo"
(Text String, Major Type 3, Additional Information 3 indicates length 3) encodes as the header byte63
followed by the 3 UTF-8 bytes for "foo" (66 6f 6f
):63 666f6f
. - The boolean
true
(Simple Value, Major Type 7, Additional Information 21) encodes as a single byte:f5
. The resulting CBOR Sequence is the direct concatenation:01 63 666f6f f5
.
- The integer
-
Example 2: Sequence Containing Structured Items
Consider encoding the sequence:
[10, false]
, then{"a": -1}
.- The array
[10, false]
(Array, Major Type 4, Additional Information 2 indicates 2 elements) encodes as:82
(header) followed by0a
(encoding for 10) andf4
(encoding for false):82 0a f4
. - The map
{"a": -1}
(Map, Major Type 5, Additional Information 1 indicates 1 pair) encodes as:a1
(header) followed by61 61
(encoding for key "a") and20
(encoding for value -1):a1 61 61 20
. The resulting CBOR Sequence is the concatenation:82 0a f4 a1 61 61 20
.
- The array
-
Example 3: Empty Sequence
An empty CBOR Sequence, containing zero items, is represented by an empty (zero-length) sequence of bytes.
It is instructive to contrast the CBOR Sequence encoding with CBOR array encodings for the same logical list of items. Taking the data from Example 1 (1
, "foo"
, true
):
- As a Definite-Length Array
[1, "foo", true]
: Encoded as83 01 63 666f6f f5
. The initial byte83
signifies Major Type 4 (Array) with Additional Information 3 (three elements follow). - As an Indefinite-Length Array
[1, "foo", true]
: Encoded as9f 01 63 666f6f f5 ff
. This starts with9f
(Major Type 4, AI 31 - indefinite-length array), includes the encoded elements, and ends with theff
(Major Type 7, AI 31 - break code) marker.
The following table summarizes these differences visually, including a comparison to JSON:
Data Model | Representation | Hexadecimal Encoding | Overhead Bytes | Framing Mechanism |
---|---|---|---|---|
[1, "foo", true] | JSON | [31, 2c, 22 66 6f 6f 22, 2c, 74 72 75 65] (ASCII) | Variable | Text delimiters ([ , , , ] ) |
[1, "foo", true] | CBOR Definite Array | 83 01 63 666f6f f5 | 1 (83 ) | Initial byte (Type 4 + Count 3) |
[1, "foo", true] | CBOR Indefinite Array | 9f 01 63 666f6f f5 ff | 2 (9f , ff ) | Start marker (9f ) + Break (ff ) |
1, "foo", true | CBOR Sequence | 01 63 666f6f f5 | 0 | None (Self-delimiting items) |
This comparison clearly shows that CBOR Sequences eliminate the structural overhead associated with arrays by relying entirely on the self-delimiting nature of the constituent CBOR items. This byte-level difference underscores the efficiency motivation, especially for streams of numerous small items.
Decoding: Reading Item by Item
Decoding a CBOR Sequence involves processing the input byte stream iteratively or recursively, extracting one complete CBOR data item at a time until the stream is exhausted. The fundamental process is:
- Check for End: Determine if the input stream or remaining buffer is empty. If it is, the sequence decoding is complete.
- Decode Item: Attempt to decode a single, complete CBOR data item starting from the current position in the stream/buffer. This requires the decoder to correctly interpret the initial byte(s) to understand the item's major type, additional information, and any subsequent length or value bytes, thereby determining the total number of bytes constituting this single item.
- Yield and Consume: If decoding the item is successful, yield or otherwise process the resulting data model value. Advance the position in the stream/buffer by the number of bytes consumed by the decoded item.
- Repeat: Go back to Step 1 with the remainder of the stream/buffer.
The self-delimiting property of standard CBOR items is the cornerstone of this process; the decoder must be able to precisely identify the boundaries of each item based solely on the CBOR encoding rules.
Handling Stream Boundaries and Errors:
- Normal Termination: Successful decoding concludes when the input stream is fully consumed exactly after a complete CBOR item has been decoded.
- Truncation: If the input stream ends unexpectedly while decoding an item (i.e., the header indicates more bytes are needed than are available), this signifies truncation. A decoder designed for streaming data might pause at this point, waiting for more bytes to arrive before declaring an error. For file-based decoding, this typically indicates an incomplete file.
- Malformed Item: If the bytes encountered do not form a well-formed CBOR data item (e.g., invalid major type/additional information combination, inconsistent length information), the decoder loses its ability to determine where the erroneous item ends. Because there are no explicit delimiters between items in a sequence, the decoder cannot reliably find the beginning of the next potential item. Consequently, a single malformed item usually prevents the decoding of the remainder of the sequence. While sophisticated error recovery might be attempted in some implementations, it is not guaranteed by the specification.
- Missing Items: The CBOR Sequence format itself provides no way to detect if expected items are simply absent from the end of the sequence. If the stream terminates cleanly after the last item that was actually present, the decoder will report success. Detecting missing items requires application-level logic, such as checking expected counts, using timeouts in network protocols, or implementing specific acknowledgement mechanisms.
The fragility in the face of malformed items is a direct consequence of the design choice to omit explicit delimiters for the sake of conciseness. Unlike newline-delimited formats like NDJSON, where a parser can often resynchronize by searching for the next newline character even after encountering invalid JSON, a CBOR Sequence parser relies entirely on the internal integrity of each item to navigate the byte stream. If an item's structure is compromised, the parser effectively becomes lost. This implies that applications relying on CBOR Sequences should prioritize robust data validation before or during sequence generation, or they must be prepared for the possibility that transmission errors affecting a single item could render a large portion of a sequence unusable. For applications demanding higher resilience against such errors, incorporating additional framing or error-checking mechanisms at a higher protocol layer might be necessary.
CBOR Sequences in Diagnostic Notation
The current draft for CBOR Extended Diagnostic Notation (EDN) proposes a way to represent CBOR Sequences in a human-readable format using <<
and >>
as delimiters, with the items separated by commas:
<< item1, item2, item3, ... >>
The sequence from the example given above would be represented as:
<< 1, "foo", true >>
If you enter this into the CBOR Playground and convert it to the serialized hexadecimal representation, you'll see that it converts the sequence to a CBOR byte string:
46 # bytes(6)
0163666F6FF5
If we manually parse this out, we can see that the first byte 0x46
indicates a byte string of length 6, followed by the bytes for the integer 1
, the string "foo"
, and the boolean true
with no other delimiters or framing:
46 # bytes(6)
01 # unsigned(1)
63666F6F # "foo"
F5 # true
The fact that the byte string header 0x46
is included might be confusing, as it implies that the sequence is a single item.
When we convert the serialized sequence back into diagnostic notation, we just get the byte string representation, as we would expect:
h'0163666F6FF5'
CBOR arrays begin with a header that specifies the array's fixed length, and indefinite-length arrays begin with the indefinite array item and end with the break item. But sequences do not themselves have delimiters or other framing.
The EDN notation is just a way to represent the sequence in a human-readable format, but it does not change the underlying encoding of the sequence itself. Sequences serialized this way are therefor not self-identifying. A CBOR decoder could be instructed to decode a byte string as a sequence, but the fact that it is a sequence cannot be determined by inspecting the byte string itself.
Practical Use Cases
The characteristics of CBOR Sequences make them suitable for a variety of applications, particularly those involving streams or sequences of independent data units.
-
Streaming Applications: This is a primary motivator for CBOR Sequences.
- Logs, Metrics, and Events: Systems generating continuous streams of structured log entries, performance metrics, or discrete events can encode each entry/event as an individual CBOR item and concatenate them into a sequence for transmission or storage. The low overhead is advantageous, and the independence of items aligns well with the nature of logs or events.
- Sensor Data Feeds: IoT devices often transmit time-series data from sensors. Using CBOR Sequences allows sending each reading or batch of readings as a separate item, benefiting from CBOR's general compactness and avoiding the per-sequence overhead of arrays, which can be significant for frequent, small readings.
-
Record Sequences (Binary NDJSON Equivalent): CBOR Sequences provide a binary alternative to text-based formats like Newline Delimited JSON (NDJSON) or JSON Lines (JSONL). They can be used for efficiently transferring large datasets as a sequence of records, such as rows from a database query or batches of results from an API, allowing for incremental processing on the receiving end. The key advantage over NDJSON is the potential for smaller size and faster parsing due to the binary encoding and native support for binary data types without base64 encoding.
-
Message Delimitation in Protocols: In network protocols built on persistent connections (like TCP or WebSockets), where multiple distinct messages need to be exchanged, each message can be encoded as a single CBOR data item. A CBOR Sequence can represent the stream of these messages. For example, a sequence of Remote Procedure Call (RPC) requests or responses, or a stream of server-sent events, could be structured this way. However, a critical caveat applies: the CBOR Sequence format itself does not provide message framing over stream transports like raw TCP. The protocol implementation must rely on the transport layer (e.g., WebSocket message boundaries) or add its own framing mechanism (like length prefixing) to allow the receiver to distinguish individual messages within the stream.
-
Sequential File Formats: Large datasets can be stored in files as a CBOR Sequence, allowing applications to read or write the file incrementally, processing one CBOR item at a time without loading the entire file into memory. This approach might be combined with proposed mechanisms for adding CBOR magic numbers or tags at the beginning of the file or sequence for identification purposes.
Observing these use cases reveals a common underlying pattern: they often rely on an external mechanism to determine the boundaries of the sequence or the items within it, especially in streaming contexts. CBOR Sequence defines the content format (concatenated self-delimiting items) but not the framing for transport or storage. Streaming logs over TCP relies on the connection lifecycle or application-level protocols; WebSocket usage relies on WebSocket framing; file storage relies on the end-of-file marker. Therefore, while CBOR Sequences offer an efficient encoding for sequential data, engineers must consciously address how these sequences (or the individual items within them) are delimited and detected within the specific transport or storage context being used. Relying solely on the CBOR Sequence format definition without considering this framing aspect can lead to implementation pitfalls.
Comparison with Alternatives
Choosing the right data representation format involves understanding the trade-offs. CBOR Sequences should be compared against other relevant CBOR structures (arrays) and common streaming formats (like NDJSON).
-
CBOR Sequences vs. CBOR Arrays (Definite and Indefinite):
- Structure: A fundamental difference lies in the data model. A CBOR Sequence represents multiple, independent top-level data items concatenated together. A CBOR Array (Major Type 4) is always a single top-level data item, whose content is an ordered list of elements.1
- Framing: Sequences have no internal framing; boundaries are implicit based on item self-delimitation. Definite-length arrays encode the element count in their header. Indefinite-length arrays use a specific start byte (
0x9F
) and require a terminating0xFF
break byte.1 - Overhead: Sequences introduce zero framing overhead. Definite arrays add 1-9 bytes for the count. Indefinite arrays add exactly 2 bytes (
0x9F
+0xFF
). - Processing Model: Sequences naturally lend themselves to item-by-item streaming and processing. While indefinite-length arrays also allow streaming of their elements, they are still conceptually processed as a single array unit that is only complete upon encountering the
0xFF
marker. Definite-length arrays typically imply processing as a whole unit once all elements are received. - Use Case Alignment: Sequences are ideal for streams of independent items, especially when the total count is unknown or potentially unbounded, and minimizing overhead is paramount. Arrays are better suited when the data represents a single, semantically coherent list, the structure of the collection itself is significant, and an explicit end marker (for indefinite) or count (for definite) is desirable within the CBOR structure itself.
- Error Handling: As discussed, a malformed item in a sequence can prevent decoding the rest of the stream. Errors within an array element might be more contained, although recovery, especially in indefinite arrays, can still be challenging.
-
CBOR Sequences vs. NDJSON / JSON Lines:
- Encoding: Sequences use CBOR's binary encoding, which is typically more compact and faster to parse than NDJSON's text-based JSON encoding for each line.
- Delimitation: Sequences rely on the self-delimiting nature of CBOR items. NDJSON uses an explicit newline character (
\n
) after each JSON text, making it line-oriented. - Efficiency: CBOR Sequences generally offer better performance in terms of size and processing speed due to their binary nature.
- Error Handling: The explicit newline delimiter in NDJSON often makes it easier for parsers to skip over a malformed JSON line and attempt to process the next one. CBOR Sequences lack this explicit delimiter, making recovery from malformed items harder.
- Binary Data: CBOR has native support for byte strings (Major Type 2), allowing efficient embedding of binary data. NDJSON requires binary data to be encoded within JSON strings, typically using Base64, which adds significant size overhead (around 33%) and processing complexity.
- Human Readability: NDJSON, being text-based, is human-readable with standard text tools. CBOR Sequences require specialized tools for inspection.
The following table summarizes the key characteristics of these alternatives:
Feature | CBOR Sequence | CBOR Indefinite Array | NDJSON / JSON Lines |
---|---|---|---|
Encoding | Binary | Binary | Text (JSON per line) |
Structure | Concatenated items (multi-root) | Single array item (single root) | Concatenated lines (multi-root) |
Item Delimitation | Implicit (self-delimiting) | Explicit (0xFF break marker) | Explicit (newline \n ) |
Overhead | None (beyond items) | 2 bytes (0x9F /0xBF + 0xFF ) | 1 byte per item (\n ) |
Processing | Item-by-item (native streaming) | Element-by-element streaming | Line-by-line (native streaming) |
Readability | Low (requires tools) | Low (requires tools) | High (text editor-friendly) |
Binary Data | Native (byte strings) | Native (byte strings) | Requires Base64 encoding |
Error Recovery | Difficult (malformed item breaks stream) | Difficult (malformed element) | Easier (can skip bad lines) |
Standard | RFC 8742 | RFC 8949 | Informal spec (ndjson.org) |
This comparison highlights that CBOR Sequences offer a high-performance, low-overhead binary format for streaming independent items, trading off some error recovery robustness and human readability compared to text-based alternatives like NDJSON, and differing structurally from CBOR arrays.
Practical Advice for Engineers
When considering or implementing CBOR Sequences, several practical aspects require attention to ensure correct and robust behavior.
-
Guidance: When to Choose Sequences Over Arrays:
- Favor Sequences when:
- The data naturally represents a stream of independent records, events, or messages.
- The total number of items is unknown upfront or potentially unbounded.
- Minimizing encoding overhead is a primary concern (e.g., for high-frequency, small items).
- Incremental, item-by-item processing is the desired model for both producer and consumer.
- The simplicity of direct concatenation aligns with the application logic (e.g., merging streams).
- Favor Arrays (Definite or Indefinite) when:
- The data represents a single, semantically coherent collection or list.
- The overall structure of the collection itself is meaningful.
- The total count of items is known (definite) or will eventually be known (indefinite).
- An explicit end marker within the CBOR data stream itself is required (indefinite arrays provide
0xFF
). - Compatibility with systems expecting a single top-level CBOR item is necessary.
- Favor Sequences when:
-
Transport Layer Considerations (Framing):
- A critical point often overlooked is that CBOR Sequence format does not inherently solve message framing over stream-based transports like TCP. TCP provides a reliable byte stream, but it does not preserve message boundaries. Sending a raw CBOR Sequence (concatenated items) over TCP means the receiver might receive partial items, multiple items, or parts of multiple items in a single
read()
call. The receiver cannot reliably identify item boundaries just by looking at TCP packet arrivals. - To handle this, a framing mechanism must be implemented above TCP but below or as part of the application logic utilizing CBOR Sequences:
- Length Prefixing: Before sending each CBOR item, transmit its length as a fixed-size integer (e.g., 4 bytes network order) or a variable-length integer (like Protobuf varints). The receiver first reads the length, then reads exactly that many bytes to get the complete CBOR item. This is a common pattern but reintroduces framing overhead.
- Delimiter-Based Framing: Use a specific byte sequence (chosen carefully to avoid collision with valid CBOR data) to mark the end of one CBOR item and the start of the next. This is generally less robust and less common than length prefixing.
- Higher-Level Protocols: Utilize protocols that provide built-in message framing. WebSockets, for instance, delivers data in discrete messages; each WebSocket message could contain exactly one CBOR item from the sequence.15 HTTP/2 streams or QUIC streams also offer framing capabilities.
- Self-Contained Items: If each item in the sequence is itself a complex structure like a COSE object 11, it might be possible (though potentially complex) to parse partially received data to determine if a complete object has arrived. This relies heavily on the internal structure of each item.
- Connection Lifecycle: For simple request-response or single-shot transfers, closing the TCP connection can signal the end of the sequence or item. This is inefficient for continuous streams.
- The essential takeaway is that the application or protocol designer must explicitly choose and implement a framing strategy when using CBOR Sequences over raw stream transports.
- A critical point often overlooked is that CBOR Sequence format does not inherently solve message framing over stream-based transports like TCP. TCP provides a reliable byte stream, but it does not preserve message boundaries. Sending a raw CBOR Sequence (concatenated items) over TCP means the receiver might receive partial items, multiple items, or parts of multiple items in a single
-
Implementation Notes & Common Patterns:
- Library Support: Check if the CBOR library being used offers specific support for sequences. For example, the Go
fxamacker/cbor
library providesUnmarshalFirst
andDiagnoseFirst
functions that decode only the first item from a byte slice and return the remaining bytes, facilitating iterative processing of a sequence. StandardUnmarshal
functions in many libraries might error if trailing bytes exist after the first item, as per RFC-8949 for single items. - Buffering: When reading from a network stream or file, employ appropriate buffering. Read data chunks into a buffer, attempt to decode one or more complete CBOR items from the buffer, consume the bytes corresponding to successfully decoded items, and handle cases where an item might be split across buffer boundaries (requiring more data to be read).
- Generators/Iterators: In programming languages offering generator functions or iterators (like Python, JavaScript, C#), these constructs provide an idiomatic way to implement a CBOR Sequence decoder. The decoder function can yield each successfully decoded item one at a time, naturally supporting the incremental processing model.
- Library Support: Check if the CBOR library being used offers specific support for sequences. For example, the Go
-
Potential Pitfalls:
- Framing Neglect: The most common pitfall is assuming CBOR Sequences provide message framing over TCP or similar streams. Always implement explicit framing.
- Error Handling Brittleness: Underestimating the consequence that a single malformed item can halt the processing of the rest of the sequence. Implement input validation or accept the potential for data loss on errors.
- Security Gaps: Remember that CBOR Sequences themselves offer no cryptographic protection. If integrity, authenticity, or confidentiality are required, each item (or potentially batches of items) must be individually protected using mechanisms like COSE. Securing the sequential relationship (preventing reordering, deletion, or insertion) often requires additional application-level mechanisms like sequence numbers or chained signatures.
- Resource Exhaustion: While sequences facilitate streaming, a naive decoder implementation that buffers all decoded items in memory before processing can still lead to memory exhaustion. Ensure that processing keeps pace with decoding in a truly incremental fashion.
- Ambiguity/Compatibility: Ensure both communicating parties understand that a CBOR Sequence is being used. Employing the
application/cbor-seq
media type or the+cbor-seq
structured syntax suffix in protocols that support content types (like HTTP, CoAP) can help avoid ambiguity.
Understanding these points requires recognizing the layered nature of communication protocols. CBOR Sequence (RFC 8742) operates at the data representation layer, defining how to encode the content of a stream. Framing mechanisms (length prefixing, WebSocket messages) operate at the transport or session layer to define message boundaries. Security mechanisms like COSE operate at the application layer to protect the content. File system metadata or magic numbers provide context at the storage layer. Engineers must address requirements at each relevant layer; expecting the CBOR Sequence format alone to solve framing or security problems will lead to incomplete or flawed implementations.
Conclusion: Sequences in the CBOR Ecosystem
CBOR Sequences should be viewed as a specific tool within the broader CBOR toolkit, complementing definite-length items, indefinite-length items (including arrays and maps), and semantic tags. They are the appropriate choice when the primary goal is to stream or serialize a sequence of independent CBOR items with minimal overhead, particularly when the total count is unknown. When data represents a single logical collection, or when explicit framing within the CBOR structure itself is desired, CBOR arrays remain the more suitable option.
Looking ahead, understanding CBOR Sequences provides context for related concepts:
- CDDL (Concise Data Definition Language): While CDDL (RFC-8610) is primarily used to define the structure of single CBOR data items, conventions exist to describe the expected content of items within a sequence. This often involves defining an array structure in CDDL and adding explanatory text stating that the elements represent the items in a sequence, or using the
.cborseq
control operator for sequences embedded within byte strings.6 - Deterministic Encoding: Rules for deterministic encoding, such as Core Deterministic Encoding defined in (RFC-8949), apply to each individual CBOR item within a sequence if a canonical byte representation is required for those items.41 The sequence structure (concatenation) is itself inherently deterministic.
By understanding the mechanics, motivations, and practical considerations of CBOR Sequences, engineers can effectively leverage this format for efficient data streaming and serialization in appropriate contexts.