Maybe reconsider Protobuf

[Timmmm]

I have done very similar marshalling between Rust and Flutter (and Typescript) myself and also initially went with Protobuf. However it turned out to be a bad choice for a few reasons:

1. You have to add ordinals to all the fields. This is for versioning reasons but is totally unnecessary for this use case because we can always guarantee that the frontend and backend are using exactly the same versions. It's quite tedious.
2. All fields are optional. Again this is to handle versioning, but is totally unnecessary for the same reasons, and it makes working with the types extremely tedious because so many things become Option<T>.

I ended up writing my own serialisation format. Even had a VSCode extension for it. It was pretty sweet. Unfortunately for a company I used to work for so I don't have access to it anymore.

The schema was a subset of Rust to allow struct definitions, and some very minor modifications to allow function calls with streaming input and output arguments. Serialisation used Bincode.

Maybe writing your own entire schema format and RPC system is a bit much but I would still consider finding a format that has required fields and doesn't require field ordinals.

FlatBuffers looks far more suitable and already supports Dart and Rust. I haven't used it though.

[temeddix]

I agree on the points you have listed. Actually there's more places where FlatBuffers excels Protobuf:

• It doesn't involve memory copy when serializing/deserializing, which is beneficial for large messages.
• You can access only the field you are interested in, without deserializing over and over again.

So the thing is, we will reconsider Protobuf and perhaps switch to FlatBuffers, if the support is equally enough on both sides of Dart and Rust. We will need to do some experiments before this change.

Bincode doesn't produced Dart code, so it looks like it would be hard to use here.

Thank you for starting this discussion :)

[Timmmm]

Yeah I wrote a parser for Bincode in Dart (and also Typescript). It's actually not that difficult. (But probably not something you want to do.)

[temeddix]

Looks like FlatBuffers also has its own tedious part when writing code:

extern crate flatbuffers;

use person_generated::MyGame::Example::Person;
use flatbuffers::{FlatBufferBuilder, WIPOffset};

fn main() {
    let mut builder = FlatBufferBuilder::new();

    // Create a Person object
    let name = builder.create_string("John");
    let age = 30;
    let person = Person::create(&mut builder, &Person::new(&mut builder, &name, age));

    // Finish building the buffer
    builder.finish(person, None);

    // Access the FlatBuffer data as bytes
    let buffer = builder.finished_data();

    // Deserialize the data
    let parsed_person = flatbuffers::get_root::<Person>(&buffer);

    // Access the fields
    let parsed_name = parsed_person.name();
    let parsed_age = parsed_person.age();

    println!("Name: {}, Age: {}", parsed_name, parsed_age);
}

[Timmmm]

Yeah that does look tedious! But I believe Serde also supports FlatBuffers. Should be less tedious. What's it like on the Dart side?

[temeddix]

It's at the links below, maybe after we find out the ways to make things clean in both Dart and Rust we can migrate to FlatBuffers :)

[temeddix]

• https://flatbuffers.dev/flatbuffers_guide_use_dart.html
• https://flatbuffers.dev/flatbuffers_guide_use_rust.html
• About zero copy google/flatbuffers#4426
• How to achieve zero-copy for binary blob google/flatbuffers#4927
• https://brunocalza.me/what-zero-copy-serialization-means/

[temeddix]

Non-tedious example code in Rust:

https://github.com/google/flatbuffers/blob/0343396e49d1c0bf4ca1058130efd9585ecb3c8f/samples/sample_binary.rs#L75-L87

// Create the monster using the `Monster::create` helper function. This
// function accepts a `MonsterArgs` struct, which supplies all of the data
// needed to build a `Monster`. To supply empty/default fields, just use the
// Rust built-in `Default::default()` function, as demonstrated below.
let orc = Monster::create(&mut builder, &MonsterArgs{
    pos: Some(&Vec3::new(1.0f32, 2.0f32, 3.0f32)),
    mana: 150,
    hp: 80,
    name: Some(name),
    inventory: Some(inventory),
    color: Color::Red,
    weapons: Some(weapons),
    equipped_type: Equipment::Weapon,
    equipped: Some(axe.as_union_value()),
    //path: Some(path),
    ..Default::default()
});

FlatBuffers does need things like builders for fields with non-fixed size:

let name = builder.create_string("Orc");
let weapons = builder.create_vector(&[sword, axe]);

[temeddix]

Non-tedious example code in Dart:

https://github.com/google/flatbuffers/blob/0343396e49d1c0bf4ca1058130efd9585ecb3c8f/dart/example/example.dart#L86-L100

// Create the builder here so we can use it for both weapons and equipped
// the actual data will only be written to the buffer once.
var axe = my_game.WeaponObjectBuilder(name: 'Axe', damage: 5);

var monsterBuilder = my_game.MonsterObjectBuilder(
  pos: my_game.Vec3ObjectBuilder(x: 1.0, y: 2.0, z: 3.0),
  mana: 150,
  hp: 300,
  name: 'Orc',
  inventory: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
  color: my_game.Color.Red,
  weapons: [my_game.WeaponObjectBuilder(name: 'Sword', damage: 3), axe],
  equippedType: my_game.EquipmentTypeId.Weapon,
  equipped: axe,
);

[temeddix]

https://github.com/cunarist/rust-in-flutter/blob/b4ddd2c352dfdb3d7863dbc42778bc0d2e04e252/example/native/hub/src/sample_functions.rs#L23-L46

I've implemented FlatBuffers in a separate branch in both Dart and Rust, but the cleanest code possible was this. FlatBuffers might be more memory-efficient but the code readability is much lower.

[Timmmm]

I haven't followed exactly what this code is doing, but can't you use Serde derive?

[temeddix]

Not likely, it is not possible to use Serde here, because it doesn't guarantee same schemas in Dart and Rust.

[Timmmm]

Ah I see. Yeah I just generated the Rust Serde code from the schema so it was fine.