feat: support metadata for genesis txs

[zivkovicmilos]

Description

This PR introduces metadata support for genesis transactions (such as timestamps), in the form of a new Gno genesis state that's easily generate-able.

Shoutout to @clockworkgr for sanity checking the idea, and providing insights that ultimately led to this PR materializing.

BREAKING CHANGE
The GnoGenesisState is now modified, and all functionality that references it (ex. gnogenesis, tx-archive) will need to adapt.

What we wanted to accomplish

The Portal Loop does not save "time" information upon restarting (from block 0). This means that any transaction that resulted in a Realm / Package calling time.Now() will get differing results when these same transactions are "replayed" as part of the loop (in the genesis state).

We wanted to somehow preserve this timestamp information when the transactions (from a previous loop), are executed as part of the genesis building process.

For example:

• Portal Loop chain is on block 100
• tx A results in a call to time.Now(), which returns time T, and saves it somewhere (Realm state)
• the Portal Loop restarts, and uses all the transactions it encountered in the past iteration as genesis txs
• the genesis is generated by executing the transactions
• when tx A is re-executed (this time as part of the genesis block), it should return time T, as with the original execution context

It is worth noting that this functionality is something we want in gnodev, so simple helpers on the Realms to update the timestamps wouldn't work.

What this PR does

I've tried to follow a couple of principles when working on this PR:

• don't break backwards compatibility
• don't modify critical APIs such as the SDK ABCI, but preserve existing, working, functionality in its original form
• don't add another layer to the complexity pancake
• don't implement a solution that looks (and works) like a hack

I'm not a huge fan of execution hooks, so the solution proposed in this PR doesn't rely on any hook mechanism. Not going with the hook approach also significantly decreases the complexity and preserves readability.

The base of this solution is enabling execution context modification, with minimal / no API changes.
Having functionality like this, we can tailor the context during critical segments such as genesis generation, and we're not just limited to timestamps (which is the primary use-case).

We also introduce a new type of AppState, called MetadataGenesisState, where metadata is associated with the transactions. We hide the actual AppState implementation behind an interface, so existing tools and flows don't break, and work as normal.

What this PR doesn't do

There is more work to be done if this PR is merged:

• we need to add support to tx-archive for supporting exporting txs with metadata. Should be straightforward to do
  • the portal loop also needs to be restarted with this new "mode" enabled
• we need to add support to existing gnoland genesis commands to support the new MetadataGenesisState. It is also straightforward, but definitely a bit of work
• if we want support for something like this in gnodev, the export / import code of gnodev also needs to be modified to support the new genesis state type (cc @gfanton)
  • fix(gnodev): timstamps issue on reload #2943

Related PRs and issues:

• feat(gno.land): support setting custom height/timestamp for genesis txs #2751
• RFC: shorter lifespan for testnets and enhancing portal loop #2744

cc @moul @thehowl @jeronimoalbi @ilgooz

Contributors' checklist...

• Added new tests, or not needed, or not feasible
• Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
• Updated the official documentation or not needed
• No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
• Added references to related issues and PRs
• Provided any useful hints for running manual tests
• Added new benchmarks to generated graphs, if any. More info here.

[codecov]

Codecov Report

Attention: Patch coverage is 85.84906% with 15 lines in your changes missing coverage. Please review.

Project coverage is 63.32%. Comparing base (af05780) to head (335048c).

Files with missing lines	Patch %	Lines
gno.land/pkg/integration/testing_node.go	11.11%	8 Missing ⚠️
tm2/pkg/sdk/helpers.go	71.42%	3 Missing and 1 partial ⚠️
contribs/gnodev/pkg/dev/node.go	86.66%	2 Missing ⚠️
gno.land/pkg/gnoland/node_inmemory.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2941   +/-   ##
=======================================
  Coverage   63.32%   63.32%           
=======================================
  Files         548      548           
  Lines       78511    78574   +63     
=======================================
+ Hits        49719    49760   +41     
- Misses      25438    25459   +21     
- Partials     3354     3355    +1     

Flag	Coverage Δ
contribs/gnodev	61.11% <93.33%> (+0.53%)	⬆️
contribs/gnofaucet	14.82% <ø> (ø)
gno.land	67.26% <80.00%> (+0.08%)	⬆️
gnovm	67.88% <ø> (ø)
misc/genstd	79.72% <ø> (ø)
tm2	62.39% <87.09%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gfanton]

Amazing! I've made some additional modifications to gnodev locally, and I can confirm that this PR correctly fixes the timestamp issue. I also managed to get Reload, Reset, Export, and Import working for gnodev. I will open a PR asap after this one.
Good job! 👏 🎉

[gfanton]

I personally like this solution; it's simple and elegant :)
Tested and approved 👏 🚀

[moul]

The implementation seems to be heading in a very good direction. Please check my comments regarding naming and the need to avoid creating unnecessary layers of indirection.

[moul]

preapproving, but check comments, thx.

[zivkovicmilos]

@gfanton @moul

I've tried a few options to change this abstraction, that you've left comments on:

// GnoGenesis defines the gno genesis API,
// adopted by differing genesis state implementations
type GnoGenesis interface {
	// GenesisBalances returns the genesis balances associated
	// with the Gno genesis state
	GenesisBalances() []Balance

	// GenesisTxs returns the genesis transactions associated
	// with the Gno genesis state
	GenesisTxs() []GenesisTx
}

type GenesisTx interface {
	// Tx returns the standard TM2 transaction
	Tx() std.Tx

	// Metadata returns the metadata tied
	// to the tx, if any
	Metadata() *GnoTxMetadata
}

In the end, this is definitely, and by far, the most simple way to make this PR change, without introducing complications, and I detail why below.

TL;DR:

• change is completely backwards compatible, meaning old genesis files work fine (encoding)
• minimal code changes, no loaders or parsers touched
• keeps custom "application" logic (like portal loop) out of the standard gno application logic

I agree with @moul's sentiment that the gnoland application (in gnoland) does not necessarily need to have 2 genesis states, but in the end we don't have a portal-loop application, and this is the reasoning behind having AppState an any type in the SDK -- it's swappable.

Attempt 1: Keeping GnoGenesisState, but making Txs an interface

type GnoGenesisState struct {
	Balances []Balance `json:"balances"`
	Txs      []GenesisTx  `json:"txs"`
}

type GenesisTx interface {
	// Tx returns the standard TM2 transaction
	Tx() std.Tx

	// Metadata returns the metadata tied
	// to the tx, if any
	Metadata() *GnoTxMetadata
}

type TxWithMetadata struct {
	GenesisTx  std.Tx        `json:"tx"`
	TxMetadata GnoTxMetadata `json:"metadata"`
}

func (m TxWithMetadata) Tx() std.Tx {
	return m.GenesisTx
}

func (m TxWithMetadata) Metadata() *GnoTxMetadata {
	return &m.TxMetadata
}

type TxWithoutMetadata struct {
	GenesisTx  std.Tx        `json:"tx"`
}

func (m TxWithMetadata) Tx() std.Tx {
	return m.GenesisTx
}

func (m TxWithMetadata) Metadata() *GnoTxMetadata {
	return nil
}

This change seems simple and straightforward at first, but introduces a lot of porting logic ([]std.Tx -> GenesisTx, which is amplified by the fact that Go doesn't support covariance on slices (a slice of actual implementations of an interface is not a slice of that same interface)).

The porting isn't even the biggest problem, but it's actually about the Amino encoding. Having this change changes how transactions are encoded (they now live under txs -> tx, instead of directly in txs. It breaks backwards compatibility with older genesis.json versions, including the ones for our active testnets.

Attempt 2: Changing the genesis transaction type to include metadata

type GnoGenesisState struct {
	Balances []Balance    `json:"balances"`
	Txs      []MetadataTx `json:"txs"`
}

type MetadataTx struct {
	Tx  std.Tx            `json:"tx"`
	Metadata GnoTxMetadata `json:"metadata"`
}

This approach keeps the GnoGenesisState as a struct, but modifies what the transactions actually are.

The problem with this approach is that changing what the type of Txs is in the GnoGenesisState has a ripple effect on a ton of code that loads and parses std.Tx, that now have to support a custom type. It also breaks backwards compatibility.

Conclusion

I've gone to 100% with all of these approaches to find their shortcomings, and none of them (apart from the one I went with) gives us the benefits we need, which are:

• clear separation
• backwards compatibility
• minimal code changes

[moul]

Just note that when you mention "portal loop," it also includes "gnodev," a tool that isn't "officially official" but is likely to become so. This pattern of local development and staging, where we want to replay in that manner, will probably become common in the Gno ecosystem.

Did you try making:

type TxWithMetadata struct {
	Tx // embedded
	Metadata Metadata
}

I believe this can achieve the backward compatibility you're referring to.

In practice, I don't care much about backward compatibility with previous testnets; test4 will be sunset soon, and test5 can launch after this change.

---

I have a feeling that one of the constraints is your desire to make this compatible with the generic tm2 tx-archive tool: https://github.com/gnolang/tx-archive.

However, the goal of creating a custom type GnoGenesis is to differentiate it from the basic tm2 Genesis. We also have GnoAccount, which extends tm2std.Account, and we will likely introduce other similar types.

We are not concerned about backward compatibility with deprecated testnets. Instead, we prioritize a straightforward genesis system that makes sense for Gno and does not modify the tm2 source code. Ideally a single system, both for gnodev and for chains.

Additionally, I'm considering how accurate "TxMetadata" is. Should we instead back up the block metadata and link each transaction to its block in the backup? All the missing TxMetadata is available in the block, right? It might make sense to ensure our export and restore tools use a single format, specifically the Gno format, which reflects what Gno does on top of tm2.
Edit: TxWithMetadata is appropriate because it refers to extending a transaction with its missing metadata when necessary. Up to you.

[zivkovicmilos]

Just note that when you mention "portal loop," it also includes "gnodev," a tool that isn't "officially official" but is likely to become so. This pattern of local development and staging, where we want to replay in that manner, will probably become common in the Gno ecosystem.

Did you try making:

type TxWithMetadata struct {
	Tx // embedded
	Metadata Metadata
}

I believe this can achieve the backward compatibility you're referring to.

This is another moment where Amino is waiting behind the corner with a baseball bat :)
@thehowl had this exact same issue in his PR #2751, which is: Amino doesn't support struct embedding:

type MyStr struct {
	A string `json:"field_a"`
	B string `json:"field_b"`
}

type myStr2 struct {
	MyStr

	C string `json:"field_c"`
}

func main() {
	s := myStr2{
		MyStr: MyStr{
			A: "hello",
			B: "goodbye",
		},
		C: "salut",
	}

	jsonMarshal, err := json.Marshal(s)
	if err != nil {
		panic(err)
	}

	aminoMarshal, err := amino.MarshalJSON(s)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(jsonMarshal))
	fmt.Println(string(aminoMarshal))
}

The results you'll get are:

{"field_a":"hello","field_b":"goodbye","field_c":"salut"} // Go JSON
{"MyStr":{"field_a":"hello","field_b":"goodbye"},"field_c":"salut"} // Amino JSON

Not to mention that the field needs to be exported in Amino (which wouldn't be a problem for us).

I have a feeling that one of the constraints is your desire to make this compatible with the generic tm2 tx-archive tool:

gnolang/tx-archive.

However, the goal of creating a custom type GnoGenesis is to differentiate it from the basic tm2 Genesis. We also have GnoAccount, which extends tm2std.Account, and we will likely introduce other similar types.

We are not concerned about backward compatibility with deprecated testnets. Instead, we prioritize a straightforward genesis system that makes sense for Gno and does not modify the tm2 source code. Ideally a single system, both for gnodev and for chains.

I've actually not considered how tools will need to adapt to this change, but scoped it only to the monorepo changes that need to be made -- and they're everything apart from minimal if we don't have an abstraction like we do now 🙁

I agree backwards compatibility is not an issue, it's definitely not a dealbreaker for me at this point. The biggest worry I have is bundling super use-case specific logic (portal loop, gnodev) with our core chain / future multichain logic. I don't think having metadata that references past blocks / whatever makes sense in new genesis files that have nothing to do with previous chain histories. The loader / parser code can be changed, albeit it's gonna be a bigger change.

Let me know how you want to move forward @moul @gfanton 🙏

[moul]

Blocks are essential for maintaining a chain history.

A TxMetadata or tx-export format isn't necessary to run a chain; it's merely an intermediary format we chose because it made sense (as an intermediary format). It allows us to create genesis from an export. Since this format is not used by the chain in real-time and does not synchronize the chain, we should design it in a way that we believe is most effective for us and for tools. In my opinion, a tx-export format or TxWithMetadata should either include the missing metadata or link to the block metadata in some way. Let's strive to find the best compromise regarding size and other factors.

When we establish this new format, we should not view it as an exception for gnodev. (2cts, an exception for gnodev would be beneficial for this ecosystem, where local development should excel.) Instead, we should consider this as defining tx-export format v2 or the Gnovm-chains' tx-export format. The Genesis/InitChainer part can certainly utilize this new format rather than relying on TXs without context, which are essentially incomplete.

Regarding struct embedding and Amino, that's unfortunate. Currently, I prefer not to create an interface but to define TxMetadata as the format we want for the tx-exports tool as well, an official new type defined in the sdk/vm. I’m not sure if tx-exports currently points to std.Tx or something else, but it should point to vm.TxWithMetadata. This is not just an exception for gnodev; it’s how Gnoland and gnovm-chains can create a meaningful tx-export. So, we have gnodev, portal loop, genesis creation, and potentially other cases in the future.

[zivkovicmilos]

@moul @gfanton
I've applied the modifications to GnoGenesisState that you've requested:
335048c

Let me know what you think, and if we should go ahead

[moul]

Can we avoid this and patch the genesisTxs file accordingly?

[moul]

You can create a similar helper in the gnoland package.

[moul]

LGTM