Feature: note plaintext size generalization #2

PaulLaux · 2024-04-30T17:06:55Z

(updated description 20.08.2024)
This PR continues the discussion from: zcash/librustzcash#746. Some things have changed, and we need an updated review to continue.

In order to support note encryption for ZSA, we suggest extending the current zcash_note_encryption implementation. Currently, the COMPACT_NOTE_SIZE is a constant; however, we need to support variable note sizes to include the AssetId field for ZSA notes.

Current state in `zcash_note_encryption`:

/// The size of a compact note.
pub const COMPACT_NOTE_SIZE: usize = 1 + // version
    11 + // diversifier
    8  + // value
    32; // rseed (or rcm prior to ZIP 212)
/// The size of [`NotePlaintextBytes`].
pub const NOTE_PLAINTEXT_SIZE: usize = COMPACT_NOTE_SIZE + 512;

and

pub const ENC_CIPHERTEXT_SIZE: usize = NOTE_PLAINTEXT_SIZE + AEAD_TAG_SIZE;

Proposed changes:

We suggest converting these constants into new abstract types within the Domain trait: NotePlaintextBytes, NoteCiphertextBytes, CompactNotePlaintextBytes, and CompactNoteCiphertextBytes. These types would then be implemented in the orchard and sapling-crypto crates.

After the discussion of the first version of this PR, the following methods are also proposed to be added to the Domain trait to safely convert byte slices into these new types: parse_note_plaintext_bytes, parse_note_ciphertext_bytes, and parse_compact_note_plaintext_bytes.

Updated `Domain` trait:

pub trait Domain {
    type EphemeralSecretKey: ConstantTimeEq;
    type EphemeralPublicKey;
    type PreparedEphemeralPublicKey;
    type SharedSecret;
    type SymmetricKey: AsRef<[u8]>;
    type Note;
    type Recipient;
    type DiversifiedTransmissionKey;
    type IncomingViewingKey;
    type OutgoingViewingKey;
    type ValueCommitment;
    type ExtractedCommitment;
    type ExtractedCommitmentBytes: Eq + for<'a> From<&'a Self::ExtractedCommitment>;
    type Memo;

    // Types for variable note size handling:
    type NotePlaintextBytes: NoteBytes;
    type NoteCiphertextBytes: NoteBytes;
    type CompactNotePlaintextBytes: NoteBytes;
    type CompactNoteCiphertextBytes: NoteBytes;

    // New parsing methods for safe conversions:
    fn parse_note_plaintext_bytes(plaintext: &[u8]) -> Option<Self::NotePlaintextBytes>;
    fn parse_note_ciphertext_bytes(output: &[u8], tag: [u8; AEAD_TAG_SIZE]) -> Option<Self::NoteCiphertextBytes>;
    fn parse_compact_note_plaintext_bytes(plaintext: &[u8]) -> Option<Self::CompactNotePlaintextBytes>;

Here, NoteBytes is a helper trait designed to simplify and unify the definition and implementation of these new associated types.

Additionally, constants will be removed from function signatures since they are not known at compilation time. For example:

pub fn try_note_decryption<D: Domain, Output: ShieldedOutput<D, ENC_CIPHERTEXT_SIZE>>(...)

will be replaced with:

pub fn try_note_decryption<D: Domain, Output: ShieldedOutput<D>>(...)

Implementation:

We have provided our initial implementation, complemented by the appropriate changes in the orchard and sapling-crypto crates. See the following modules for details:

https://github.com/QED-it/orchard/blob/zsa1/src/note_encryption/domain.rs
https://github.com/QED-it/sapling-crypto/blob/zsa1/src/note_encryption.rs

Additionally, we made several minor updates in the zcash_primitives crate of QED-it's fork of the librustzcash repository to align it with the described changes in zcash_note_encryption, orchard, and sapling-crypto crates:

https://github.com/QED-it/librustzcash/tree/zsa1/zcash_primitives

* Copy updated src folder from librustzcash/zsa1 * Cherry-pick the latest update of zcash_note_encryption from zcash/librustzcash (it was missed in QED-it/librustzcash) * Upgrade Rust version * Upgrade Rust version to 1.65 * Add --features=alloc to command line for build-nodefault jon in ci.yml * Downgrade Rust version back to 1.61.0 --------- Co-authored-by: Dmitry Demin <[email protected]>

Co-authored-by: Dmitry Demin <[email protected]>

PaulLaux · 2024-04-30T17:09:51Z

Regarding the compile time vs runtime discussion, we currently set the array size during compile time in the implementation of the trait: https://github.com/QED-it/orchard/blob/a9af64c146bef38d7ac957983bc78c8a3f54f76e/src/note_encryption/note_encryption_zsa.rs#L12

impl OrchardDomain for OrchardZSA {
    const COMPACT_NOTE_SIZE: usize = COMPACT_NOTE_SIZE_ZSA;

    type NotePlaintextBytes = NoteBytes<{ Self::NOTE_PLAINTEXT_SIZE }>;
    type NoteCiphertextBytes = NoteBytes<{ Self::ENC_CIPHERTEXT_SIZE }>;
    type CompactNotePlaintextBytes = NoteBytes<{ Self::COMPACT_NOTE_SIZE }>;
    type CompactNoteCiphertextBytes = NoteBytes<{ Self::COMPACT_NOTE_SIZE }>;
...

…on (#6) * Update CHANGELOG.md with changes for note plaintext size generalization * Improve the description in CHANGELOG.md --------- Co-authored-by: Dmitry Demin <[email protected]>

Co-authored-by: Dmitry Demin <[email protected]>

str4d · 2024-06-25T12:02:12Z

src/lib.rs

+    type NotePlaintextBytes: AsMut<[u8]> + for<'a> From<&'a [u8]>;
+    type NoteCiphertextBytes: AsMut<[u8]> + for<'a> From<(&'a [u8], &'a [u8])>;
+    type CompactNotePlaintextBytes: AsMut<[u8]> + for<'a> From<&'a [u8]>;
+    type CompactNoteCiphertextBytes: AsRef<[u8]>;


It is incorrect to have From bounds on these types, as the conversions are necessarily fallible (the slices can be any length, and the types cannot) and this would force panics in the public API (not just in the internal trait API that we control both sides of). Additionally, NoteCiphertextBytes should not take the tag as a slice (we know its length).

Replace all of these with trait methods that "parse" the slices to get these, e.g. fn parse_note_plaintext_bytes(plaintext: &[u8]) -> Option<Self::NotePlaintextBytes>.

Done: added three new methods to Domain trait to parse note bytes.

fn parse_note_plaintext_bytes(plaintext: &[u8]) -> Option<Self::NotePlaintextBytes> fn parse_note_ciphertext_bytes(output: &[u8], tag: [u8; AEAD_TAG_SIZE]) -> Option<Self::NoteCiphertextBytes> fn parse_compact_note_plaintext_bytes(plaintext: &[u8]) -> Option<Self::CompactNotePlaintextBytes>

str4d

Reviewed b8bd2a1.

str4d · 2024-06-25T12:42:34Z

src/lib.rs

+    /// Exposes the note ciphertext of the output. Returns `None` if the output is compact.
+    fn enc_ciphertext(&self) -> Option<D::NoteCiphertextBytes>;


This no longer "exposes" the output, and instead forcibly clones it, which is undesirable in usages of ShieldedOutput outside of zcash_note_encryption. We should instead return a reference that the caller can clone if desired (by adding a Clone bound on `Domain::NoteCiphertextBytes):

Suggested change

/// Exposes the note ciphertext of the output. Returns `None` if the output is compact.

fn enc_ciphertext(&self) -> Option<D::NoteCiphertextBytes>;

/// Exposes the note ciphertext of the output.

///

/// Returns `None` if the output is only compact.

fn enc_ciphertext(&self) -> Option<&D::NoteCiphertextBytes>;

Also, document this change to the trait in the changelog.

We tried to change the code to return references but encountered a few technical challenges due to how NoteCiphertextBytes and CompactNoteCiphertextBytes are implemented in both the orchard and sapling-crypto crates. These types are defined and implemented as newtypes over fixed-size byte arrays:

type NoteCiphertextBytes = NoteBytesData<{ ENC_CIPHERTEXT_SIZE }>; type CompactNoteCiphertextBytes = NoteBytesData<{ COMPACT_NOTE_SIZE }>; pub struct NoteBytesData<const N: usize>(pub [u8; N]);

The issue is that these newtypes wrap arrays that are not directly stored as such in the structs implementing ShieldedOutput. Therefore, we cannot return a reference to them directly because they are constructed on-the-fly from these arrays.

A possible workaround is to change NoteCiphertextBytes and CompactNoteCiphertextBytes type implementations in orchard and sapling-crypto to wrap references instead of arrays, but we would need to adjust the Domain trait to allow lifetimes and define associated types as generic with lifetimes using Rust's Generic Associated Types (GAT):

trait Domain { type NoteCiphertextBytes<'a>; type CompactNoteCiphertextBytes<'a>; }

This approach would involve additional changes, impacting all Domain and ShieldedOutput implementations and adding complexity to the codebase. If using references is crucial, we can proceed with this strategy though.

After discussion in the ZSA meeting, we will make my suggested change. If an action is non-compact, we know it is possible to get a reference to the appropriate type, and the data is large enough that we don't want to force clones.

Done: now enc_ciphertext returns Option of a reference.

str4d · 2024-06-25T12:43:28Z

src/lib.rs

+    /// Exposes the compact note ciphertext of the output.
+    fn enc_ciphertext_compact(&self) -> D::CompactNoteCiphertextBytes;


Likewise, we should avoid forcibly cloning here, and instead have a Clone bound on Domain::CompactNoteCiphertextBytes:

Suggested change

/// Exposes the compact note ciphertext of the output.

fn enc_ciphertext_compact(&self) -> D::CompactNoteCiphertextBytes;

/// Exposes the compact view of the output's note ciphertext.

fn enc_ciphertext_compact(&self) -> &D::CompactNoteCiphertextBytes;

Also, document this change to the trait in the changelog.

After discussion in the ZSA meeting, we'll leave this method as-is; the issue is that for non-compact actions, we can't get a reference to a fixed-length sub-array. The alternative would be to have a separate CompactShieldedOutput trait (which is effectively what we had in the past), but we can consider that refactor later.

Got it: left enc_ciphertext_compact as-is.

str4d · 2024-06-25T12:48:12Z

src/lib.rs

-        output[NOTE_PLAINTEXT_SIZE..].copy_from_slice(&tag);
-
-        output
+        D::NoteCiphertextBytes::from((output, tag.as_ref()))


Once the Froms are replaced, this becomes

Suggested change

D::NoteCiphertextBytes::from((output, tag.as_ref()))

D::parse_note_ciphertext_bytes(output, tag).expect("output is correct length")

the unwrap here being fine because we know that output is derived from D::NotePlaintextBytes, which we can assume is the correct size relative to D::NoteCiphertextBytes.

Non-blocking: I have ideas on how to make this infallible in a type-safe way, but I'll look at that refactor after this PR lands.

Done (added and used parse.. method).

str4d · 2024-06-25T12:52:26Z

src/lib.rs

+    fn extract_memo(
+        &self,
+        plaintext: &Self::NotePlaintextBytes,
+    ) -> (Self::CompactNotePlaintextBytes, Self::Memo);


Hmm, I dislike this change, in part because we want to move away from the notion of compact memo representations in future NUs. But looking at how it is used in this crate, I think it's fine for now if we rename it:

Suggested change

fn extract_memo(

&self,

plaintext: &Self::NotePlaintextBytes,

) -> (Self::CompactNotePlaintextBytes, Self::Memo);

fn split_plaintext_at_memo(

&self,

plaintext: &Self::NotePlaintextBytes,

) -> (Self::CompactNotePlaintextBytes, Self::Memo);

Done: renamed to split_plaintext_at_memo.

str4d · 2024-06-25T12:55:05Z

src/lib.rs

    ) -> Option<(Self::Note, Self::Recipient)>;

-    /// Extracts the memo field from the given note plaintext.
+    /// Splits the memo field from the given note plaintext.


Paired with the method rename:

Suggested change

/// Splits the memo field from the given note plaintext.

/// Splits the given note plaintext into the compact part (containing the note) and

/// the memo field.

str4d · 2024-06-25T12:56:16Z

src/lib.rs

-/// The size of a compact note.
-pub const COMPACT_NOTE_SIZE: usize = 1 + // version
-    11 + // diversifier
-    8  + // value
-    32; // rseed (or rcm prior to ZIP 212)
-/// The size of [`NotePlaintextBytes`].
-pub const NOTE_PLAINTEXT_SIZE: usize = COMPACT_NOTE_SIZE + 512;


Either document these removals in the changelog, or (my preference) undo these removals (the constants are still relevant in downstream crates) and alter their docstrings to say that these are pre-ZSA.