HIP-1056: Block Streams #1056
HIP-1056: Block Streams #1056
Conversation
✅ Deploy Preview for hedera-hips ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| retain on disk, without breaking the cryptographic integrity of the block stream. Different block node operators in | ||
| different legal jurisdictions can make different decisions about what data to retain. Or, block node operators may | ||
| subset the stream to retain minimal data and minimize storage costs. | ||
| - **Errata Handling**: Errata refers to software bugs where a node executed the transaction correctly but did not record |
There was a problem hiding this comment.
Alternative option would be the given block node to be temporarily excluded from the network of block nodes and it's block stream invalidated. Then, upon fixing the bug, the block node operator could again join the network, sync the state and continue it's normal work.
There was a problem hiding this comment.
Errata are needed when a bug on the consensus node produces incorrect output. This section refers to the necessity to change the recorded block chain. I believe this has been necessary a small number of times since genesis on mainnet.
Block Nodes are not a network; each node is completely independent. Those details will be specified in a different HIP.
| ``` | ||
|
|
||
| ```protobuf | ||
| message CryptoTransferOutput { |
There was a problem hiding this comment.
The crypto create might result in auto account creation. Shouldn't we also add a record for it for those cases?
There was a problem hiding this comment.
The account creation will be present in StateChanges, no additional output is needed.
There was a problem hiding this comment.
Need to call out parent child considerations more explicitly. Might be done but noting here as a reminder to come back in case there's room for improvement
Signed-off-by: Nana Essilfie-Conduah <[email protected]>
Signed-off-by: Nana Essilfie-Conduah <[email protected]>
HIP/hip-1056.md
Outdated
| The Block Stream plans to ensure visibilty of all modified entities by externalizing the full entity state in the stream. | ||
| This is to solve the issue of partial data in record streams where a client may miss the externalization of state data and would | ||
| therefore possess a partial state for entities with few easy options to retrieve the missing data. | ||
| By externalizing the full entity state in tehe block stream a client can get the complete entity details on any update and won't have |
There was a problem hiding this comment.
Typo:
| By externalizing the full entity state in tehe block stream a client can get the complete entity details on any update and won't have | |
| By externalizing the full entity state in the block stream a client can get the complete entity details on any update and won't have |
HIP/hip-1056.md
Outdated
| to employ upsert logic. An emergent feature when externalizing the full entity state of modified entities is that entities who's | ||
| properties are modified multiple times would be externalized in the stream multiple times. |
There was a problem hiding this comment.
Possible rewording:
| to employ upsert logic. An emergent feature when externalizing the full entity state of modified entities is that entities who's | |
| properties are modified multiple times would be externalized in the stream multiple times. | |
| to employ "upsert" logic. An emergent feature when externalizing the full entity | |
| state of modified entities is that when entity properties are modified multiple | |
| times those entities may be externalized in the stream multiple times. |
| for every entity which has undesired impacts on state size. | ||
| Another inspired approach is that on the first observation of an entity in a block would cause the full entity to be extenralized, | ||
| after that only the delta changes are externalized for the remainder of the block. | ||
| In this way any consumer of a block would have the complete state by applying the diffs onto the initial state. |
There was a problem hiding this comment.
Missing reasoning:
| In this way any consumer of a block would have the complete state by applying the diffs onto the initial state. | |
| In this way any consumer of a block would have the complete state by applying | |
| the diffs onto the initial state. This shares the downside of requiring | |
| additional temporary storage, and also requires the creation of a "difference" | |
| form for every entity potentially stored in state, and could, paradoxically, | |
| _increase_ the total size of blocks with very many changes to small entities. |
| oneof new_value { | ||
| /** | ||
| * A change to the block info singleton. | ||
| */ | ||
| proto.BlockInfo block_info_value = 1; | ||
|
|
||
| /** | ||
| * A change to the congestion level starts singleton. | ||
| */ | ||
| proto.CongestionLevelStarts congestion_level_starts_value = 2; | ||
|
|
||
| /** | ||
| * A change to the Entity Identifier singleton. | ||
| */ | ||
| google.protobuf.UInt64Value entity_number_value = 3; | ||
|
|
||
| /** | ||
| * A change to the exchange rates singleton. | ||
| */ | ||
| proto.ExchangeRateSet exchange_rate_set_value = 4; | ||
|
|
||
| /** | ||
| * A change to the network staking rewards singleton. | ||
| */ | ||
| proto.NetworkStakingRewards network_staking_rewards_value = 5; | ||
|
|
||
| /** | ||
| * A change to a raw byte array singleton. | ||
| */ | ||
| google.protobuf.BytesValue bytes_value = 6; | ||
|
|
||
| /** | ||
| * A change to a raw string singleton. | ||
| */ | ||
| google.protobuf.StringValue string_value = 7; | ||
|
|
||
| /** | ||
| * A change to the running hashes singleton. | ||
| */ | ||
| proto.RunningHashes running_hashes_value = 8; | ||
|
|
||
| /** | ||
| * A change to the throttle usage snapshots singleton. | ||
| * <p> | ||
| * Throttle usage snapshots SHALL be updated for _every transaction_ | ||
| * to reflect the amount used for each tps throttle and | ||
| * for the gas throttle. | ||
| */ | ||
| proto.ThrottleUsageSnapshots throttle_usage_snapshots_value = 9; | ||
|
|
||
| /** | ||
| * A change to a raw `Timestamp` singleton.<br/> | ||
| * An example of a raw `Timestamp` singleton is the | ||
| * "network freeze time" singleton state, which, if set, stores | ||
| * the time for the next scheduled freeze. | ||
| */ | ||
| proto.Timestamp timestamp_value = 10; | ||
|
|
||
| /** | ||
| * A change to the block stream status singleton. | ||
| */ | ||
| com.hedera.hapi.node.state.blockstream.BlockStreamInfo block_stream_info_value = 11; | ||
|
|
||
| /** | ||
| * A change to the platform state singleton. | ||
| */ | ||
| com.hedera.hapi.platform.state.PlatformState platform_state_value = 12; | ||
|
|
||
| /** | ||
| * A change to the roster state singleton. | ||
| */ | ||
| com.hedera.hapi.node.state.roster.RosterState roster_state_value = 13; | ||
| } |
There was a problem hiding this comment.
Lots of tab characters for indentation here.
Might be worthwhile to re-copy the block from the protobufs repo.
Pro Tip: set your tab width in github (Settings|Appearance) to 12 and tabs will stand out like a road flare on a cloudy night.
HIP/hip-1056.md
Outdated
|
|
||
| ```protobuf | ||
| message CallContractOutput { | ||
| repeated proto.TransactionSidecarRecord sidecars = 1; |
There was a problem hiding this comment.
| repeated proto.TransactionSidecarRecord sidecars = 1; | |
| /** | |
| * A list of additional outputs | |
| */ | |
| repeated proto.TransactionSidecarRecord sidecars = 1; |
|
|
||
| ```protobuf | ||
| message CreateContractOutput { | ||
| repeated proto.TransactionSidecarRecord sidecars = 1; |
There was a problem hiding this comment.
| repeated proto.TransactionSidecarRecord sidecars = 1; | |
| /** | |
| * A list of additional outputs | |
| */ | |
| repeated proto.TransactionSidecarRecord sidecars = 1; |
HIP/hip-1056.md
Outdated
|
|
||
| With the record stream design nodes had to independently upload record streams and their signatures files. | ||
| These were stored in varied public cloud buckets and anyone in the world could download the files | ||
| e.g. Mirror Nodes do this today to support queries. |
There was a problem hiding this comment.
formatting nit
| e.g. Mirror Nodes do this today to support queries. | |
| (e.g. Mirror Nodes do this today to support queries.) |
HIP/hip-1056.md
Outdated
| - **Signature Files**: Each node currently produces a v6 record stream with a corresponding signature file for | ||
| verification. Mirror nodes must download a strong minority (1/3 of consensus weight) worth of these signature files, | ||
| along with the record file, to verify v6 records. This frequent access to cloud storage (GCS/S3) is costly. To reduce | ||
| this expense and complexity, blocks include a TSS signature from the majority weight of consensus nodes. |
There was a problem hiding this comment.
Today the plan is to make the threshold a majority, but it could change in the future. There are various levels of guarantees that can be made by thresholds of 1/3, >1/2, and >2/3. I suggest:
| this expense and complexity, blocks include a TSS signature from the majority weight of consensus nodes. | |
| this expense and complexity, blocks include a TSS signature from a threshold (minimum of 1/3) weight of consensus nodes. |
There was a problem hiding this comment.
If it is important for the diagram to match the verbage, it should include the other stream files that are being consolidated: sidecar files and signature files
HIP/hip-1056.md
Outdated
| simplify node software by allowing nodes to reconnect with a block node, instead of another consensus node. This | ||
| is a significant and important enhancement to reconnect that is needed for permissionless nodes. | ||
| - **Query Endpoints:** As Block Nodes or other consumers of the Block Stream can maintain a live state in sync with the | ||
| consensus nodes, they can answer any queries against state. They can also produce cryptograph state proofs to prove |
There was a problem hiding this comment.
| consensus nodes, they can answer any queries against state. They can also produce cryptograph state proofs to prove | |
| consensus nodes, they can answer any queries against state. They can also produce cryptographic state proofs to prove |
HIP/hip-1056.md
Outdated
| ``` | ||
| NOTE: Update: Add missing Round header after Block Header to image | ||
| ``` |
There was a problem hiding this comment.
This can be removed since the diagram has the Round Header
There was a problem hiding this comment.
The label for the Input Block Items Merkle Tree is labeled "Output"
There was a problem hiding this comment.
A few comments:
- Where is the Round Header included?
- It might be worth stating explicitly that the order in which the items appear in the block merkle tree are not the same as the order in which they are streamed. This is implied in the description that says the items are organized into items known pre-execution and those known after, but better to state it explicitly IMO.
- Maybe this comes later, but if not, I think it's worth explaining why the hash of the state merkle tree from the beginning of the block is included in the tree rather than the hash that results from all the stuff included in that block.
HIP/hip-1056.md
Outdated
| Each block within the block stream is a self-contained entity, including every event and all transactions that were | ||
| part of the block, the state changes as a result of those transactions, and a | ||
| _network signature_ (using a BLS threshold signature scheme we call TSS) that proves the block was signed by a | ||
| threshold of the network. |
There was a problem hiding this comment.
| threshold of the network. | |
| subset of nodes whose stake accounts for more than 1/3 of the network's consensus weight. |
HIP/hip-1056.md
Outdated
| The block stream has a separate `block stream merkle tree` which is a merkle tree of all the items in the block stream | ||
| over the entire history of the stream. Of course, older blocks can be represented by their hash rather than the actual | ||
| full contents, so it is possible to have a very large block stream merkle tree *in concept* that is still very efficient | ||
| to use. |
There was a problem hiding this comment.
| The block stream has a separate `block stream merkle tree` which is a merkle tree of all the items in the block stream | |
| over the entire history of the stream. Of course, older blocks can be represented by their hash rather than the actual | |
| full contents, so it is possible to have a very large block stream merkle tree *in concept* that is still very efficient | |
| to use. | |
| At each block boundary, the state merkle tree is a depth two *subtree* of a `block merkle tree`, which is a binary merkle tree whose four subtrees at depth two also include the block's input items, the block's output items, and the merkle tree of the previous block. (Of course hash computation for the block `N` merkle tree reuses the hash already computed for the block `N-1` merkle tree.) |
HIP/hip-1056.md
Outdated
| Block streams are built out of Items. Those items can be delivered in a continuous stream containing one or more blocks. | ||
| Or they can be delivered as a single block using the `Block` message type container. If a single block is delivered as | ||
| a file it expected to be either a raw protobuf `Block` message with file name of `<BLOCK_NUMBER>.blk` e.g. | ||
| `0000000000000000001.blk` . With total of 19 digits to allow for 2^63 blocks. Or a compressed file e.g. |
There was a problem hiding this comment.
Currently Block file names are 36-digit strings padded with leading zeroes. FileBlockItemWriter
|
Should this HIP at all mention/reference the behaviors around consensus node to block node communication? Will this information be covered in the block node HIP? |
HIP/hip-1056.md
Outdated
| Record files are streams in a blockchain, with the hash of previous file included in the next record file, ensuring | ||
| immutability. Additionally, each consensus node on the Hedera mainnet signs a hash of the record files as they are |
There was a problem hiding this comment.
| Record files are streams in a blockchain, with the hash of previous file included in the next record file, ensuring | |
| immutability. Additionally, each consensus node on the Hedera mainnet signs a hash of the record files as they are | |
| Record file contents are summarized in a chain of hashes, with each file's contents containing the hash of the previous file; so that the hash of the current file depends on the contents of the entire record stream. Additionally, each consensus node on the Hedera mainnet signs the hash of each record file as it is |
There was a problem hiding this comment.
...containing the hash of the previous file...
| can authenticate transactional data without the cost and complexity of downloading multiple signature files for | ||
| verification. | ||
| 3. As a verifier for a network, I want a single, self-contained stream of data with aggregated signature signed by a | ||
| threshold of stake weights, so that I can cost-effectively confirm that the stream represents the consensus output of |
There was a problem hiding this comment.
We should really decide what terms we are going to use. Some places use stake weights and others simply use weight. Either is fine as long as it is used consistently. Inconsistent use will contribute to confusion.
HIP/hip-1056.md
Outdated
| The Block Stream will be defined by the continuous transmission of Block information defined as follows. | ||
| - Each `Block` represents all inputs and outputs of the consensus network for zero or more whole `Rounds`. The maximum | ||
| number of rounds per block will be configurable and start at 1 round per block. It could be increased later if | ||
| necessary for performance reasons or if round become very frequent and small. |
There was a problem hiding this comment.
| necessary for performance reasons or if round become very frequent and small. | |
| necessary for performance reasons or if rounds become very frequent and small. |
| number of rounds per block will be configurable and start at 1 round per block. It could be increased later if | ||
| necessary for performance reasons or if round become very frequent and small. | ||
| - A block *may* be zero rounds because the last block prior to a network "freeze" must communicate the state hash | ||
| (via a block proof) of the state immediately prior to network shutdown. This is only possible with an extra empty |
There was a problem hiding this comment.
I'm confused by the (via a block proof) part here. In normal cases, the state hash is part of the block merkle tree and the block proof has an aggregated signature of the block merkle tree root (although now I realize that this has not been explicitly state in the HIP thus far, and it probably should be). Is this section saying that in a block with no rounds, the proof is simply on the hash of the state root and not a merkle tree root hash that also includes previous block root hash and null padding?
There was a problem hiding this comment.
A block with no rounds is the same as any other block except for the lack of any content between block header and block proof.
The bullet point is describing a specific case where a block with no rounds may
be required, and describing how that "empty" block resolves a sequence problem.
The wording is intended to explain that, because block proof contains the state hash at the beginning of the block (end of the prior block), the state hash at end-of-block is communicated in the block stream as a field in the block proof for the following block.
There was a problem hiding this comment.
the state hash at end-of-block is communicated in the block stream as a field in the block proof
Aha! I had not gotten to the BlockProof proto yet.
HIP/hip-1056.md
Outdated
| - A `Round` is a sequence of zero or more `Events` that have reached consensus at the same time. Round length in real | ||
| world time depends on internet latency and event creation rate. In Hedera mainnet today it is around 1 round per | ||
| second. In the future this could be shorter to lower latency, but would never make sense to be less than ping time. | ||
| - An `Event` is a sequence of zero or more `EventTransactions` submitted by a single node. |
There was a problem hiding this comment.
A round is more than just its events, and an event is more than just its transactions. Perhaps it is simplest to say "contains" instead of "is" so we don't have to explain the metadata that comes with each.
| These concepts are core to the Hashgraph consensus mechanism and have direct impacts on the structure of data in the | ||
| streams. | ||
|
|
||
| Block streams are built out of Items. Those items can be delivered in a continuous stream containing one or more blocks. |
There was a problem hiding this comment.
I don't understand. If a block is made up of items, how can those items be streamed such that they contain one or more blocks?
There was a problem hiding this comment.
The stream contains one or more blocks, which are made up of many items (often thousands).
Agree that this wording is a little awkward at expressing that.
HIP/hip-1056.md
Outdated
|
|
||
| Block streams are built out of Items. Those items can be delivered in a continuous stream containing one or more blocks. | ||
| Or they can be delivered as a single block using the `Block` message type container. If a single block is delivered as | ||
| a file it expected to be either a raw protobuf `Block` message with file name of `<BLOCK_NUMBER>.blk` e.g. |
There was a problem hiding this comment.
| a file it expected to be either a raw protobuf `Block` message with file name of `<BLOCK_NUMBER>.blk` e.g. | |
| a file it must be either a raw protobuf `Block` message with file name of `<BLOCK_NUMBER>.blk` e.g. |
HIP/hip-1056.md
Outdated
| Block streams are built out of Items. Those items can be delivered in a continuous stream containing one or more blocks. | ||
| Or they can be delivered as a single block using the `Block` message type container. If a single block is delivered as | ||
| a file it expected to be either a raw protobuf `Block` message with file name of `<BLOCK_NUMBER>.blk` e.g. | ||
| `0000000000000000001.blk` . With total of 19 digits to allow for 2^63 blocks. Or a compressed file e.g. |
HIP/hip-1056.md
Outdated
| } | ||
| ``` | ||
|
|
||
| The order of block items in a block stream is important in some cases it designates useful positions in the stream that |
There was a problem hiding this comment.
| The order of block items in a block stream is important in some cases it designates useful positions in the stream that | |
| The order of block items in a block stream is important and in some cases it designates useful positions in the stream that |
| first transaction. Block items for one block can also be packaged in a `Block` message so they can be stored in a | ||
| protobuf file and served as a single entity if desired. | ||
|
|
||
|  |
There was a problem hiding this comment.
There are some cases where multiple changes made to a singleton state type over the course of a round will be consolidated into a single change set for that singleton type for the round instead. How do those fit into the stream and block merkle tree? Do they need to be placed somewhere specific both in the stream and in the tree?
HIP/hip-1056.md
Outdated
| #### BlockHeader | ||
|
|
||
| ### BlockHeader |
There was a problem hiding this comment.
Duplicate Blockheader section titles
This is currently documented in the block node codebase, and should be included, to the extent appropriate, in the forthcoming "Block Node" HIP. That HIP will cover all of the Block Node APIs, not just the publishBlockStream API used by consensus nodes. |
| 🚨 **Open Detail:** How do we define the `first_transaction_consensus_time` of a block with no transactions? Can it be | ||
| first event consensus time if no transactions? what about empty block with no events? |
There was a problem hiding this comment.
Some details and edge cases:
- For "normal" rounds that contain events and transactions, the round timestamp is equal to the timestamp of the last transaction in the round
- For empty rounds (no events), the round timestamp is equal to the timestamp of the last transaction in the previous round plus 1,000 nanos, rounded up to the nearest 1,000 nanos. Is there is no previous round, then the round timestamp is the median of the judge created times.
To me, #1 makes no sense at all. An event's consensus timestamp is equal to the timestamp of the FIRST transaction it contains. If the round follow the same rules, the round's timestamp would be equal to the timestamp of the first event and transaction it contains. As far as I'm aware, there is no reason we cannot change the round timestamp to work this way if we want. If we did that, we could just put the round timestamp in this protobuf instead of the first transaction timestamp and call it a day.
If we can't do that or don't want to do that, then we have some other options:
- Keep the transaction timestamp but make it nullable. If there are no transactions in the round, it is null.
- Keep the timestamp and make it always populated (with the one exception of an empty block sent after the freeze round), but for rounds with no transactions, it will be the round timestamp
- Add a second field for the round timestamp which will always be populated (with the one exception of an empty block sent after the freeze round).
None of these sound good because there are exceptions and edge cases that need to be explained.
| #### EventHeader | ||
|
|
||
| The event header depicts the beginning of an event. All items after this and before the next `EventHeader` or | ||
| `BlockProof` are part of this event or results of execution of this event’s contents. It is designed so that it is easy |
There was a problem hiding this comment.
Wouldn't a third "event end" marker would be the non-transactional state changes?
There was a problem hiding this comment.
There is no mechanism for a reader to differentiate non-transactional state changes from transactional state changes. To a reader of the Block Stream, those are just additional state changes in the event.
Is it important to differentiate those in some way? We might be missing another header if that's the case...
HIP/hip-1056.md
Outdated
|
|
||
| The event header depicts the beginning of an event. All items after this and before the next `EventHeader` or | ||
| `BlockProof` are part of this event or results of execution of this event’s contents. It is designed so that it is easy | ||
| to reconstruct `GossipEvent`’s as needed for hashgraph reconstruction from the contents in the block stream. To make |
There was a problem hiding this comment.
| to reconstruct `GossipEvent`’s as needed for hashgraph reconstruction from the contents in the block stream. To make | |
| to reconstruct `GossipEvent`s as needed for hashgraph reconstruction from the contents in the block stream. To make |
HIP/hip-1056.md
Outdated
| of user and system transactions, and matches the structure of Events within the hashgraph algorithm. A state signature | ||
| system transaction is a non-user transaction internally created to carry out a state change in conformance with network | ||
| hashgraph logic. This transaction does not count towards network TPS and can only be issued internally via the consensus | ||
| platform. |
There was a problem hiding this comment.
Let's future proof this a bit
| platform. | |
| node software. |
| The block stream supports complete reconstruction of the consensus event stream in a simple and secure manner. Any | ||
| entity wishing to reconstruct the event stream may begin with the `EventHeader` and append each following | ||
| `EventTransaction` *in order* as it appears in the Block Stream. The event ends when the next `EventHeader` or | ||
| `Block Proof` is encountered in the stream. |
There was a problem hiding this comment.
or non-transactional state changes?
Signed-off-by: Nana Essilfie-Conduah <[email protected]>
…ick and Joseph Signed-off-by: Nana Essilfie-Conduah <[email protected]>
Abstract
This HIP introduces a new output data format for consensus nodes, called Block Streams, that replaces the existing
event streams, record streams, state files, and signature files with one single stream.
Each block within the block stream is a self-contained entity, including every event and all transactions that were
part of the block, the state changes as a result of those transactions, and a
network signature (using a BLS threshold signature scheme we call TSS) that proves the block was signed by a
threshold of the network.
By including state changes, downstream services can seamlessly maintain and verify state alongside consensus nodes. This
enhancement fosters greater transparency and trust within the Hedera network. Any downstream service can independently
rebuild and verify the state of consensus nodes at the time of the block, verified by the TSS network signature. Using
this state, they can provide additional services such as state proofs, state snapshots, and more.
With the event information within blocks, downstream services can be reconstructed and the hashgraph algorithm replayed,
permitting anyone to verify the correct execution of the consensus algorithm and removes the need for the extra event
stream. In doing so Hedera users gain comprehensive visibility into network activities through an easily consumable
format that can be delivered with low latency.
A key design criteria is for the block stream to be easily consumed by any programming language, and with minimal
complexity or dependencies. For example, state data can be utilized for basic queries without having to reconstruct a
merkle tree.
Block streams are an upgrade to the existing RecordStream V6. Block streams restructure and aggregate the multiple
record types in record streams, including
EventStream,RecordStream, and Hedera state data to produce a singleunified stream of
items.The key enhancements offered by block streams include:
single cohesive data stream.
consensus and state changes.
consensus nodes are clearly specified, paving the way for future implementations and greater client diversity.
With the adoption of block streams, data output by Hedera consensus nodes will be consolidated into a single, verifiable
chronicle of the network, thereby strengthening the integrity and transparency of the Hedera ecosystem.