Create bip-bm02.mediawiki

[PeterSurda]

First proposal.

[justusranvier]

Thanks for proposing this.

This repo probably isn't the best place for a standard like this to live, but I can't think of a better venue right now so we can start here.

The largest concerns I have are the use of msgpack an zlib hard-coded into the protocol.

A strongly-typed system like protobuf seems like a safer option for dealing with arbitrary inputs because it's easier to create a validator which can reject malformed (and possibly malicious) messages.

Here's an example of a protobuf schema that could accomplish this:

syntax = "proto2";

package bitmessage;
option optimize_for = LITE_RUNTIME;

enum CompressionType {
    COMPRESS_NONE = 0;
    COMPRESS_ZLIB = 1;
}

message Object {
    optional uint32 version = 1;
    optional CompressionType compression = 2;
    optional bytes data = 3;
}

message File {
    optional uint32 version = 1;
    optional string name = 2;
    optional string mimetype = 3;
    optional bytes data = 4;
}

message Message {
    optional uint32 version = 1;
    optional string subject = 2;
    optional string body = 3;
    repeated string to = 4;
    repeated string cc = 5;
    repeated string parent = 6;
    optional sint64 timestamp = 7;
    repeated File attachment = 8;
}

The Object message holds a serialized child protobuf, and stores the type of compression used as an emum value. This field can be extended to support more types of compression in the future if desired.

All messages have a version field which can be used for validation.

The fields are specified as optional to make the schema forward compatible with proto3, validation can be provided by a separate library. (See the opentxs-proto repository here for an example).

The message field has to and cc fields for informational purposes, since frequently the recipients of an email would like to see this information if it exists.

[PeterSurda]

Hello and thanks for the feedback.

I don't really care where the BIPs are stored as long as all the devs know about it. For improved communication we also now have https://bitmessage.slack.com (repurposed from the original plan to have a BM-specific nonprofit) and a #bitmessage chan on freenode. Feel free to use it (I may have to approve slack registrations but I have no problem with that).

I did evaluate and reject protobuf, originally, but now that I think about it more, maybe I was too hasty. I still haven't made any of this code live so I can still change my position.

I worry that To and Cc inside the message itself may cause problems. Also, the encryption format should be adjusted in the future to support multi-recipient messages, but let's take it one step at a time.

[PeterSurda]

So I'm looking at this in a bit more detail. If you declare everything as optional, what's the point of using protobuf for validation then? You have to write a wrapper with defaults around it anyway.

And why have the compression type in the protobuf object, if the recipient needs to signal to the sender what he supports anyway? Currently he'd either have to use a bitfield or an address version to do that.

[justusranvier]

So I'm looking at this in a bit more detail. If you declare everything as optional, what's the point of using protobuf for validation then? You have to write a wrapper with defaults around it anyway.

Yes, you have to write validation functions, but protobuf has still done much of the heavy lifting for you.in terms of making sure the data you just deserialized is well-formed according to the schema. Your validation functions mostly call the has_foo() methods to make sure the fields you expect are present, along with higher level semantic criteria which wouldn't be in the schema anyway.

See here for examples:

https://github.com/Open-Transactions/opentxs-proto/blob/pre-1.0/src/verify/hdpath/HDPath_1.cpp
https://github.com/Open-Transactions/opentxs-proto/blob/pre-1.0/src/verify/contactitem/ContactItem_1.cpp

And why have the compression type in the protobuf object, if the recipient needs to signal to the sender what he supports anyway? Currently he'd either have to use a bitfield or an address version to do that.

Address versions can work perfectly for this.

The rule can be that a version 1 address only accept version 1 Object messages, and version 1 Object messages are only allowed to use COMPRESS_ZLIB.

At some future point, we could define a version 2 Object message which has more than one allowed values for compression. Clients would signal their willingness to accept version 2 Object messages with the address version.

[PeterSurda]

Yes, you have to write validation functions, but protobuf has still done much of the heavy lifting for you.in terms of making sure the data you just deserialized is well-formed according to the schema. Your validation functions mostly call the has_foo() methods to make sure the fields you expect are present, along with higher level semantic criteria which wouldn't be in the schema anyway.

I don't see that being "heavy lifting", but "light lifting" without much benefit. You also still have to add a UTF-8 validator at least and probably others that will pop up once I start implementing it. And what if someone wants to use a new element that is not a part of the schema, why should that be treated as invalid? Something like email headers. And just like with email, it's probably better if the standard client just ignores it instead.

Address versions can work perfectly for this.

That is what I also thought originally, but I opted for a bitfield in the end, because using an address requires you to create a new address in order to signal you support it. When using a bitfield, sending a new pubkey object is sufficient. Having a new address is a burden and creates usability problems. Also, it's not necessary for the wire protocol because that does not need to change.

The rule can be that a version 1 address only accept version 1 Object messages, and version 1 Object messages are only allowed to use COMPRESS_ZLIB.

We're at address version 4 now already, and historically a bump only occurred when the wire protocol changed (and I believe that was a correct approach). The willingness to return an ack on the other hand is signalled by bitfield. PyBitmessage didn't actually implement the ability to turns this off until recently, but now that it does, you don't need to create a new address in order to use this feature.