refactor: store bytes in vars instead of memory block index

[zshipko]

Fixes an incompatibility identified here: extism/go-pdk#34

Storing the handle makes a lot of sense here, but the PDKs expect for the handles that are passed into var_set and returned from var_get to be transient.

[chrisdickinson]

Hm, I need to double-check the behavior here for workers. I think this would cause a desync between plugins run on a background worker and the host functions on the foreground, since the variable data would no longer be held in allocator memory. But! On the other hand, thinking through this, there might be a bug there already that this PR exposes 😅

[chrisdickinson]

I think there might be something tricky to navigate around here with background workers. I think this would fail in the following case:

createPlugin('path/to/some/plugin', {
  runInWorker: true,
  functions: {
    'a namespace': {
       async hostfunc(context) {
         context.getVariable("foo") // should return BAZ but returns null
         context.setVariable("foo", "bar")
       }
    }
  }
})

Where the pseudocode for the plugin would be something like:

Var.set("foo", "BAZ")
const {hostfunc} = Host.getFunctions();

func();
Var.get("foo") // should return "bar", but instead returns "BAZ"

because the worker and main thread only sync memory held in #blocks. I can see about writing this up as a test if that'd be helpful!

[zshipko]

Ah interesting, yeah I'm not 100% clear on some of the memory details - if you could add that test it would be super helpful!

In the meantime I will poke around and try to understand my changes a little better

[zshipko]

It seems like it may already be a bug like you mentioned, I am using node to run test plugins that contain some code like this:

  plugin.setVar("test_var", "something");

  const c = plugin.getVar("test_var") catch unreachable orelse "";
  if (!std.mem.eql(u8, c, "something")) {
      unreachable;
  }

and it's hitting that unreachable block because setVar is calling free on the value before it returns. I also noticed that the rust-pdk leaks these values, which is why we may not have run into this with plugins written in Rust: extism/rust-pdk#57.

[chrisdickinson]

I believe the js-pdk also leaks these values - it might be worth formalizing that behavior as "ownership of pointers passed to extism host functions are moved to the host"?

[zshipko]

I believe the js-pdk also leaks these values - it might be worth formalizing that behavior as "ownership of pointers passed to extism host functions are moved to the host"?

Yeah I think that makes a lot of sense, I will investigate how Rust/Go handle pointers passed to the host and come up with some fixes.

[chrisdickinson]

I think this is fixed in #80?

[zshipko]

Yep, closing this!