Illustrate multiblock encoder/decoder API

[Gozala]

Problem Statement

#35 removes registries which simplifies things for the encoder which had to communicate which encoder and hashing algorithm to use by code or a name and now it does just by reference. It does however pushes some complexity onto decoder which needs a table of decoders (and in some instances hashers) in order to be able to decode dags encoded with multiple codecs.

Overview of changes here.

Changes in this pull request are based on #35 and attempt to introduce solution for above described problem, that is, make decoding dags that use multiple codecs as convenient and simple as it was with registries. Conceptually it does so by introducing or combinator allowing one to combine several multi(thing)s and turn them into one multi(thing) that can encode / decode in either of the combined things.

Base decoders / encoders

1. Have a method encoder.baseEncode(bytes) that encode bytes with underlying base encoding (without multibase prefix) and corresponding decoder.baseDecode(string) that decodes string (which isn't multibase prefixed) with an underlying base.

import { base32 } from 'multiformats/bases/base32'

base32.baseEncode(Buffer.from('hello'))
// => 'nbswy3dp'
Buffer.from(base32.baseDecode('nbswy3dp')).toString()
// 'hello'

2. They also have method encoder.encode(bytes) that encode bytes with underlying base encoding and return it with multibase prefix. Corresponding decoder.decode(multibase) checks the base and if supported decodes it otherwise throws an exception:

base32.encode(Buffer.from('hello'))
// => 'bnbswy3dp'
Buffer.from(base32.decode('bnbswy3dp')).toString()
// => 'hello world'

import { base58btc } from 'multiformats/bases/base58'

base58btc.decode('bnbswy3dp')
// => Error: base58btc expects input starting with z and can not decode "bnbswy3dp"

3. Base decoders can be composed in order to be able to decode multibase in any of the base encoding of it's composition composition:

const baseDecoder = base32.decoder.or(base58btc.decoder)

baseDecoder.decode(base32.encode(Buffer.from('hello')))
// => [ 104, 101, 108, 108, 111 ]
baseDecoder.decode(base58btc.encode(Buffer.from('hello')))
// => [ 104, 101, 108, 108, 111 ]

import { base58btc } from 'multiformats/bases/base64'

baseDecoder.decode(base64.encode(Buffer.from('hello')))
// => Error: Unable to decode multibase string maGVsbG8, only inputs prefixed with b,z are supported

Primary idea is that if you need base decoder that handles multiple bases, you can create a composition and not worry about which one you're dealing with.

Codec

Same idea as above had been expanded to codecs. Just like baseEncode / baseDecode codecs have now encodeBlock / decodeBlock that encode / decode via concrete codec. In addition there is now encode({ code, value }) => {code, bytes} and decode({ code, bytes }) => {code, value}. Just like baseEncode / baseDecode operated on strings and encode / decode operated on multibase tagged strings, encodeBlock / decodeBlock operate on raw value <-> bytes and encode/decode operate on code tagged value <-> bytes hence { code, value } <-> { code, bytes }

import json from 'multiformats/codecs/json'

b1 = json.encodeBlock({ hello: 'world' })
json.decodeBlock(b1)
// => { hello: 'world' }

import raw from 'multiformats/codecs/raw'

json.decode({ code: json.code, bytes: b1 })
// => { code: 512, value: { hello: 'world' } }
json.decode(raw.encode({ code: raw.code, value: Buffer.from('hello') }))
// => Error: Codec (code: 85) is not supported, only supports: 512

Similar to base decoders codecs can also be composed:

const c = json.or(raw)

c.decode(json.encode({ code: json.code, value: { hello: 'world' } }))
// => { code: 512, value: { hello: 'world' } }
c.decode(raw.encode({ code: raw.code, value: Buffer.from('hello') }))
// => { code: 85, value: Uint8Array(5) [ 104, 101, 108, 108, 111 ] }

And encoders compose as well

c.encode({ code: json.code, value: { hello: 'world' } })
// => { code: 512, bytes: ... }
c.encode({ code: raw.code, value: Buffer.from('hello') })
// => { code: 85, value: Uint8Array(5) [ 104, 101, 108, 108, 111 ] }

Just like logical or json.or(raw) is also drop-in replacement for json because it's the operand on the left assumes default role:

json.decodeBlock(c.encodeBlock({ hello: "world" }))
// => { hello: "world" }
c.decodeBlock(json.encodeBlock({ hello: "world" }))
// => { hello: 'world' }

c.decodeBlock(raw.encodeBlock(Buffer.from('hello')))
// => SyntaxError: Unexpected token h in JSON at position 0

So what's going on here is that there is a BlockEncode/BlockDecoder pair of interfaces and MulticodecEncoder/MulticodecDecoder pair of interfaces. When codec is created it implements all four which makes codecs and codec compositions swappable. This enables high level APIs like dag API (see blow) to not care how many codecs are used in the dag it just needs a codec (or composition of them) to encode / decode it.

Originally this pull request separated BlockEncode/BlockDecoder from MulticodecEncoder/MulticodecDecoder and you had to build later with former. I think that allowing codecs to compose without having to introduce another layer makes a an API simple yet powerful, but I am also curious to learn how others perceive it.

Hashers

Hashers got similar treatment. There is digestBytes(bytes: Uint8Array): Promise<MultihashDigest<Code>> and digest(input: {code: Code, bytes: Uint8Array }): Promise<MultihashDigest<Code>> which makes them composable:

import { sha256, sha512 } from ''multiformats/hashes/sha2'

await sha256.digestBytes(Buffer.from('hello'))
// => Digest { code: 18, size: 32, digest: Uint8Array(32), bytes: bytes: Uint8Array(34) }

await sha256.digest({ code: sha256.code, bytes: Buffer.from('hello') })
// => Digest { code: 18, size: 32, digest: Uint8Array(32), bytes: bytes: Uint8Array(34) }

await sha512.digest({ code: sha256.code, bytes: Buffer.from('hello') })
// => Error: Unsupported hashing algorithm (code: 18), this hasher only supports: 19

const sha = sha256.or(sha512)

await sha.digest({ code: sha256.code, bytes: Buffer.from('hello') })
// => Digest { code: 18, size: 32, digest: Uint8Array(32), bytes: bytes: Uint8Array(34) }
await sha.digest({ code: sha512.code, bytes: Buffer.from('hello') })
// => Digest { code: 18, size: 64, digest: Uint8Array(64), bytes: bytes: Uint8Array(66) }

await sha.digest({ code: /*MD5*/213, bytes: Buffer.from('hello') })

// => Error: Unsupported hashing algorithm (code: 213), this hasher only supports: 18,19

Dag

Previous block.js was replaced with more flexible dag.js that can be used to create dag encode / decoder with blocks in various encodings:

import { codecs, hashes, dag } from 'multiformats/basics'

const { encoder, decoder } = dag.codec({
  multicodec: codecs.json.or(codecs.raw),
  hasher: hashes.sha256.or(hashes.sha512)
})

// Uses 
var b1 = encoder.encodeBlock({ hello: 'world' })
// b1._cid
// => null

[...b1.links()]
// => []

[...b1.tree()]
// => ['hello']

b1.get('/hello')
// => { value: 'world' }

await b1.cid()
// CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)
b1._cid
// CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)

var b2 = encoder.encode({ code: codecs.raw.code, value: Buffer.from('greetings') })
var b3 = encoder.encode({
  code: codecs.json.code,
  value: { link: await b2.cid() }
})

[...b3.links()]
// => [ ['link', CID(bafkreid52tzpa57ejg2hefjvt2acbqfwzapbqtjmmfcimjdmxd3qzld2oa)] ]
b3.get('/link')
// => { value: CID(bafkreid52tzpa57ejg2hefjvt2acbqfwzapbqtjmmfcimjdmxd3qzld2oa), remaining: '' }

var b4 = decoder.decode({ code: b3.code, bytes: b3.bytes })
var b5 = decoder.decode({ code: b2.code, bytes: b2.bytes })

Just like all the other things dag codecs also implement or interface so that one could take dag codec from one sub-system and compose it with dag codec from another and get a codec that can be used with both:

const sys1 = dag.codec({
  multicodec: codecs.json.or(codecs.raw),
  hasher: hashes.sha256.or(hashes.sha512)
})

import cbor from "@ipld/cbor"
import pb from "@ipld/pb"

const sys2 = dag.codec({
    multicodec: cbor.or(pb)
    hasher: hashes.sha256
})

const { encoder, decoder } = sys1.or(sys2)

[rvagg]

const c = json.or(raw)
c.decodeBlock(raw.encodeBlock(Buffer.from('hello')))
// => SyntaxError: Unexpected token h in JSON at position 0

This is going to get annoying I reckon. Especially when you scale up through the dag.or(..) pattern and lose track of which was the original left-hand-side of the thing.

One way to solve this is to do away with encodeBlock() and decodeBlock() entirely, but then you'd need to be doing { code: json.code, value: ... } on every encode and extracting out your value from the returned object on every decode (annoying if all you want to do is encode with a single codec, which is the dominant pattern). Another way would be to say that these MulticodecEncoder/MulticodecDecoder abstractions don't get the *Block() methods since it's just confusing and once you start composing then you have to be specific. It breaks the drop-in replacement niceness, but I see so much foot-gun potential in this dominant-left pattern when you're so far removed from the setup. I think I'd end up making names like dagJsonPrimary for clarity, which is a bit smelly.

Other than that; I don't really have strong feelings about any of these changes. I mostly just want to try these patterns out to form opinions, I don't like churn but don't see good reasons to object other than that. For CarReader#get() I suppose I still get to return a single thing but it becomes the Block type defined in src/dag/block.js, that seems acceptable.

[rvagg]

I know this discussion is probably breaking your nice inheritance patterns here but this or() composition API really bothers me. Thinking out loud about the usability here:

When you're calling encode() and decode() and passing in the code, you're very likely going to use this pattern that uses json.code and raw.code to get the actual value. So if you're doing that, you have json and raw at hand anyway. Which makes me wonder why I would ever c.encode({ code: json.code, value: { ... } }) when I have json at hand anyway? I'd always opt for json.encodeBlock() wouldn't I unless it was really convenient to get that return value as a pair and I couldn't be bothered forming it myself. It becomes a trade-off between thing = c.encode({ code: json.code, value: { ... }) and thing = { code: json.code, json.encodeBlock({ ... }) }. At that point, I'd be opting for explicit anyway because maybe the chain of custody of c is far enough removed that I don't have 100% confidence that it has what I want.

The same could be said for c.decode() as well I think because you still need the code in an object pair.

Maybe composition for codecs doesn't actually make as much sense since you are dealing with a pair rather than a singular thing like with the multibase and multihash encoders.

Similar problems exist when you get to dag. You have the left-side confusion plus you are probably going to need to have codec around for codec.code so maybe I just want to use codec directly at that point, same with the hasher, and what I really want is something I can call a Block to wrap around it that does predictable and well-defined things. So maybe I should just be able to Block.create(codec, hasher, { data }).

Again, it's on the decode where this stuff matters. I need to have a bank (I'm avoiding calling it a "registry" because that seems to be a burdened term by now) of multibases, multihashes and multicodecs to pick from. If I have CID+Binary then the bank lets me figure out what I have and form it into a Block. This is only place that I can see composition maybe being a help. But you're really getting back to the add() pattern you've been trying to avoid. BlockCreator.add([...mymultihashes, ...mymulticodecs]).decode(cid, binary) -> Block. Maybe trying to make these things symmetrical is a mistake because the operations and choices are often asymmetrical?

[Gozala]

Thanks @rvagg for going through and providing, feedback. It really helps, especially because I had been debating some of the decisions that you've called out here myself. I'll respond to individual points inline below:

const c = json.or(raw)
c.decodeBlock(raw.encodeBlock(Buffer.from('hello')))
// => SyntaxError: Unexpected token h in JSON at position 0

This is going to get annoying I reckon. Especially when you scale up through the dag.or(..) pattern and lose track of which was the original left-hand-side of the thing.

I agree, in fact in the first iteration BlockCodec and MultiblockCodec where separate things, and there was no notion of default codec in the later or way to encode / decode without providing a code. The problem with that approach was that json.or(raw) would return a thing that had a different API from json or raw.

I wonder if the original approach would be less of a footgun here. We could also alleviate json.or(raw) with multicodec(json).or(raw).

@rvagg do you thing that would make more sense ?

When you're calling encode() and decode() and passing in the code, you're very likely going to use this pattern that uses json.code and raw.code to get the actual value. So if you're doing that, you have json and raw at hand anyway. Which makes me wonder why I would ever c.encode({ code: json.code, value: { ... } }) when I have json at hand anyway?
I'd always opt for json.encodeBlock() wouldn't I unless it was really convenient to get that return value as a pair and I couldn't be bothered forming it myself. It becomes a trade-off between thing = c.encode({ code: json.code, value: { ... }) and thing = { code: json.code, json.encodeBlock({ ... }) }. At that point, I'd be opting for explicit anyway because maybe the chain of custody of c is far enough removed that I don't have 100% confidence that it has what I want.

If you do have json and raw codecs I agree there is no real good reason to use composition. In fact that was one of the motivations for changes in the #36 so that it would be more convenient to do this with references than with names for them.

That said this is geared towards for different use cases, e.g. dag.js is next layer in the stack that can create a dag encoder / decoder API without knowing or caring about underlying codecs it will work with. User of that dag API may also not have access to the codec implementations but to the code table instead. In fact this is more or less how shared IPFS worker does things. All the codecs live in the worker thread but main thread does know codes for those so it can draft the dag and hand it off to the worker to encode it.

The same could be said for c.decode() as well I think because you still need the code in an object pair.

Decode especially makes more sense for cases like CarReader where instead of having a registry of codecs to lookup in by a code you'd just be able to pass code and block bytes to the the composed decoder instead.

Similar problems exist when you get to dag. You have the left-side confusion plus you are probably going to need to have codec around for codec.code so maybe I just want to use codec directly at that point, same with the hasher, and what I really want is something I can call a Block to wrap around it that does predictable and well-defined things. So maybe I should just be able to Block.create(codec, hasher, { data }).

I don't think this is true for dag. The way I see it if you care about what it should be encoded in you should provide the code otherwise you use whatever the default happens to be. Note that intent here is not that you compose bunch of things and use them directly (in that case using plain codecs would be more convenient) but rather that you create a composition that you can hand it over to the higher level API so it could encode / decode and hash things across all of those codecs and hashing algorithms. And default will tell that higher level API what the default configuration should be for encoding / decoding / hashing.

It is true that a.or(b) may lead to a mistake, but I do not think it is as big of the problem as you do.

I need to have a bank (I'm avoiding calling it a "registry" because that seems to be a burdened term by now) of multibases, multihashes and multicodecs to pick from. If I have CID+Binary then the bank lets me figure out what I have and form it into a Block. This is only place that I can see composition maybe being a help. But you're really getting back to the add() pattern you've been trying to avoid. BlockCreator.add([...mymultihashes, ...mymulticodecs]).decode(cid, binary) -> Block.

I think it's fine to call it registry (I call it a compostion) and yes intent of all this is a better registry. As I mentioned in the original issue and pull I think registry is needed for higher level APIs I just did not wanted everywhere. There were also bunch of issues that I called out with .add pattern and the compositions address all of those, while hopefully also addressing use cases that original registry was trying to address.

Maybe trying to make these things symmetrical is a mistake because the operations and choices are often asymmetrical?

I think symmetry does comes with a cost and maybe it's taken too far here ? But as I did point out I do have use case (with worker) for encoder composition where simply using codec directly isn't really viable.

I think main problem you're calling out is with defaults. I do think that a.or(b) keeping a's default behavior is not surprising and likely not cause much issues in practice, but I'm also open to getting rid of the defaults entirely.

[rvagg]

I agree, in fact in the first iteration BlockCodec and MultiblockCodec where separate things, and there was no notion of default codec in the later or way to encode / decode without providing a code. The problem with that approach was that json.or(raw) would return a thing that had a different API from json or raw.

Yes, I think separating them might be the better option, and maybe the API challenge can be avoided by not performing the composition action on the BlockCodec but as a separate factory for a MultiblockCodec, so you keep the two concepts separate.

I do think that a.or(b) keeping a's default behavior is not surprising and likely not cause much issues in practice

You've gone to all this trouble to inject explicitness into the interfaces, factories, methods and the way that it's used. Explicitness is good. But this or() then jumps in and removes the explicitness and leaves you in this strange land where you are left holding something that's not as cleanly defined. This is mostly in the category of "code smell" than something I'm going to push too hard on as a purely rational API critique, it just doesn't feel right. But also, its utility is dubious, I'm just not convinced that combining them in this way is very useful. If you're building toward this "dag" thing, what do we lose by not having this MultiblockCodec thing? Maybe I could just give it an array of BlockCodecs?

I think real-world use of these APIs will be most informative. It doesn't feel quite right but that could be biases I'm bringing that aren't helpful in designing this. Mostly I'm suspicious of having so many free floating things that need to be pulled together into one place, but that's kind of inherent in what we're trying to achieve here so it's a matter of how can we do it best?

[Gozala]

Yes, I think separating them might be the better option, and maybe the API challenge can be avoided by not performing the composition action on the BlockCodec but as a separate factory for a MultiblockCodec, so you keep the two concepts separate.

Main downside I see with this approach is that codecs exposed by multiformats aren’t going to be composable, and they’d need to be wrapped e.g.:

import raw from ‘multiformats/codecs/raw’
import json from ‘multiformats/codecs/json’
import cbor from ‘@ipfs/dag-cbor’

const codec = multicodec(raw)
  .or(multicodec(json))
  .or(multicodec(cbor))

In my experience when you have to import another library to compose things it is too tempting to craft solution inline.

That said there could be other options worth considering, maybe just like MulticodecCodec is { MulticodecEncoder, MulticodecDecoder } pair it could turned into something more explicit like
{ encoder: MulticodecEncoder, decoder: MulticodecDecoder, defaultEncoder: BlockEncoder, defaultDecoder: BlockDecoder } that way you will not have:

const c = json.or(raw)
c.decodeBlock(raw.encodeBlock(Buffer.from('hello')))
// => SyntaxError: Unexpected token h in JSON at position 0

Instead you’ll have

const codec = json.or(raw)

codec.defaultDecoder.decode(Buffer.from(‘hello’))
// => SyntaxError: Unexpected token h in JSON at position 0

Not sure if that makes things any more explicit though.

Another option could be to make codec.decodeBlock(bytes) even more similar to parser combinators and instead of making it use leftmost block decoder try each one from left to right. Please note that this would remove notion of default but order of composition will remain. I think it fits better the use case where you just want to decode a block without knowing what codec was used to encode it. And name could probably improved e.g. codec.tryBlockDecode to make it even more explicit.

You've gone to all this trouble to inject explicitness into the interfaces, factories, methods and the way that it's used. Explicitness is good. But this or() then jumps in and removes the explicitness and leaves you in this strange land where you are left holding something that's not as cleanly defined.

This is a fare critique. I think primary motivation for combining BlockCodec with BlockMulticodec was to remove a need to wrap former with later. Which worked out great with multibase codecs and multihash hashers because both multihashes and multibases carry code information so decoder doesn’t need anything extra and can decode it back. Block encoders on the other hand do not include code information and therefor (multicodec) decode has no way of knowing if it’s capable of decoding, at best it can try to decode with a default (current behavior in this pull) or try multiple codecs in some deterministic order (in order to have deterministic behavior).

Given all this I think only viable options are:

• Use (deterministic) default (current implementation).
• Try multiple codecs in (deterministic) order (can be achieved by changing implementation details of or).
• Require code to be provided (I think this is what @rvagg is leaning towards).

But also, its utility is dubious, I'm just not convinced that combining them in this way is very useful. If you're building toward this "dag" thing, what do we lose by not having this MultiblockCodec thing? Maybe I could just give it an array of BlockCodecs?

What you loose is either Dag needs to always be told explicitly which encoder to use, or you need to introduce yet another thing that you need to pass to tell it to use a thing as default e.g.:

const { encoder, decoder } = dag.codec({
  multicodec: codecs.json.or(codecs.raw),
  defaultCodec: codecs.json,

  hasher: hashes.sha256.or(hashes.sha512),
  defaultHasher: hasher
})

Which otherwise was deterministically inferred from the composition. Furthermore you still have a same problem as before, just in the next layer

decoder.decodeBlock(raw.encodeBlock(Buffer.from('hello')))

This also means you have a same problem when you do dagA.or(dagB) in which is the default encoder or a hasher ? With a current design it just left leaning at every layer. And if you want to change a default without knowing what’s the given codec’s default is that’s also easy:

const { encoder, decoder } = dag.codec({
  multicodec: defaultCodec.or(givenMulticodec),
  hasher
})

All this is to say that you can either:

1. Remove all the implicitness in all of the layers.
2. You create deterministic rules about implicit behavior and have it at all layers.

My hope was to strike a balance between two by:

1. Having very explicit APIs where both encode and decode are explicit.
2. Have an implicit API for convenience with deterministic rules about it’s behavior.

That way user is empowered to make a more informed decision between explicitness and connivence based on constraints they’re facing.

Maybe I could just give it an array of BlockCodecs?

Sure Dag could take array of codecs and array of hashers, but then if you want combine two Dag things you need to come up with a way to combine those (maybe array of Dags) but as more layers are added things get messy. I really think typed functional languages have nailed it by having compositional primitives that are more flexible, I am afraid to mention an m word, but having an operator that knows how to unpack underlying things and repack them back (like .or here) is more flexible and robust than passing around arrays of things. As an evidence to support this claim I’d like to point out that most JS tooling with their plugin systems eventually introduce extends field (eslint, babel, typescript etc...) because once you have two complex configurations combining their underlying sub-configurations becomes tedious and error prone given they could evolve to have more thing over time. Compositional operator like or avoids that because evolves with implementation of the component it’s part of. And a common operator across layers creates intuitive way to compose things at every layer.

To summarize my position, I am not opposed to been fully explicit everywhere, but that has tradeoffs and approach so far had been leaning towards convenience. Which is why I think it would be best to allow choice between convenience and explicitness and make choice about which one to use case by case basis. We could improve naming of things to make those tradeoffs more clear to users and we can also switch semantics of or from try first to try all so that this controversial example would do the right thing:

c = json.or(raw)

c.decodeBlock(raw.encodeBlock(Buffer.from('hello'))).toString()
// =>'hello'

[rvagg]

I think you've done really well at exposing our lack of mechanism to identify a block type within the binary (as per your discussion on "multiblock"). But that's unavoidable, at some layer of the stack, anyway (binary formats we have to consume but can't control, storage mechanisms that wouldn't support additional pieces, etc.), so we have to deal with it.

Trying to think of the way I would want to interact with this kind of system, what I see is:

• low-level codecs, multihashes, multibases to do specific things, perhaps I only ever do dag-cbor, sha2-256, base32, and maybe don't even need any of the fancy path resolution stuff, so having these singular primitives around makes more sense. I want contracts around the pieces so they work in predictable ways, and I probably want CID utilities.
• high-level API like js-block that ties CID:Binary together such that the CID gives us the "multiblock" type identifier--that's sort of one of its purposes. I've been quite happy with being able to put all the pieces together to end up with the useful triple of b = Block.encoder({ head: await b1.cid() }, 'dag-cbor'), b = Block.decoder(someBuffer, 'dag-cbor') and b = Block.create(binary, cid). With those things and a well-defined interface around Block I can get things done and I have this tightly coupled CID:Binary pairing that can be so powerful.

So right now, my take away from all of this composition work at this layer is that I can't really see myself reaching for it. I'm either going to go lower or higher. You seem to have usecase that makes the in-between a little more interesting? It might also be the case that I'm not quite seeing the broad utility yet. I really just want to see what the higher portion is going to be and I still feel like you and @mikeal have slightly different visions about what that is so I'll wait and see what the synthesis is.