Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ON HOLD] Feature: Full declare_native_blueprint_state macro #1318

Closed
wants to merge 30 commits into from

Conversation

dhedey
Copy link
Contributor

@dhedey dhedey commented Aug 9, 2023

PR status update

This full PR has been parked, and instead, will do it in bits (starting with just the core versioned models + schema stuff) - will create a new PR to capture that.

Replacement PR 2023: #1358
Extension PR 2024: #1744

Summary

The goal of this PR is to introduce a more structured / type-safe mechanism for interfacing with our native substates... and to enable easily wrapping them with lazily-upgradable versioned enum wrappers in a type-safe way.

This PR has:

  • (Small bit) Created a versioned! macro to make versioning easy
  • (Bulk) Created a declare_native_blueprint_state macro to create typed native substate models which creates:
    • Data Models for Substate data and their keys
      • "Latest" version / versioned wrapper for content (reading layer 3)
      • (System) Payload (reading at layer 2)
      • For substates (reading at layer 1)
    • Data model for state init
      • Including for system layer
      • And output at kernel layer (for eg flashing)
    • "Node layout" models including Partition enum
    • Typed models for Core API and their mapping
    • Creation of Schema
    • Feature enum and "FeatureInit"

Generic "Versioned" data model

In terms of versioning conventions, I believe we are going to:

  • Version substate contents for all native substates at the application layer
  • Not version function arguments, because we can create v2 methods in future if we need to.

When implementing for X = Schema, I decided the following naming schema fitted best - keeping code looking as neat as possible:

  • VersionedX is the versioned enum type. This will be used for substate contents, or other types which need versioning.
  • X is a type alias created for "latest version of this type"
  • XV1 is the concrete type which should be created, which maps to the type alias.

Questions to be answered before proceeding:

  • As part of this change, do we wish to create macros to define Native State types more easy / clear / type-safe?
  • If so, I'll implement that macro... then I'll work native blueprint-by-blueprint with separate PRs into this branch

Changes for integrators

I'll merge this into the node. This shouldn't have any impact outside of Scrypto and Node.

@dhedey dhedey force-pushed the feature/key-data-model-versioning branch 3 times, most recently from 6a7c239 to 69e29e2 Compare August 10, 2023 12:39
@dhedey dhedey force-pushed the feature/key-data-model-versioning branch from 69e29e2 to 24aaef8 Compare August 10, 2023 16:55
@dhedey dhedey changed the base branch from develop to data-model-versioning-branch-point August 15, 2023 14:16
@github-actions
Copy link

github-actions bot commented Aug 15, 2023

Benchmark for 0f3006c

Click to view benchmark
Test Base PR %
costing::bench_prepare_wasm 67.9±1.20ms 68.3±1.44ms +0.59%
costing::decode_sbor 17.0±0.03µs 17.0±0.03µs 0.00%
costing::deserialize_wasm 1420.2±1.93µs 1432.5±28.81µs +0.87%
costing::instantiate_flash_loan 5.7±0.04ms 5.8±0.04ms +1.75%
costing::instantiate_radiswap 5.5±0.09ms 5.5±0.14ms 0.00%
costing::spin_loop 8.8±0.02ms 8.4±0.02ms -4.55%
costing::validate_sbor_payload 30.2±0.62µs 30.0±0.74µs -0.66%
costing::validate_secp256k1 84.7±0.26µs 84.7±0.10µs 0.00%
costing::validate_wasm 35.6±0.15ms 35.4±0.09ms -0.56%
decimal::add/0 7.2±0.04ns 7.2±0.00ns 0.00%
decimal::add/rust-native 9.5±0.01ns 9.5±0.05ns 0.00%
decimal::add/wasmer 144.4±0.14ns 145.1±1.06ns +0.48%
decimal::add/wasmer-call-native 592.9±2.03ns 591.1±0.33ns -0.30%
decimal::add/wasmi 486.6±1.00ns 481.2±0.75ns -1.11%
decimal::add/wasmi-call-native 6.2±0.07µs 6.1±0.05µs -1.61%
decimal::div/0 154.1±0.14ns 154.8±0.30ns +0.45%
decimal::from_string/0 192.5±0.15ns 192.1±0.13ns -0.21%
decimal::mul/0 110.0±0.32ns 110.4±0.07ns +0.36%
decimal::mul/rust-native 113.1±0.04ns 113.1±0.14ns 0.00%
decimal::mul/wasmer 1698.5±3.53ns 1700.0±13.38ns +0.09%
decimal::mul/wasmer-call-native 708.9±2.39ns 707.1±0.43ns -0.25%
decimal::mul/wasmi 28.9±0.15µs 29.5±0.06µs +2.08%
decimal::mul/wasmi-call-native 6.2±0.04µs 6.2±0.03µs 0.00%
decimal::pow/0 531.2±0.69ns 530.8±0.25ns -0.08%
decimal::pow/rust-native 533.7±0.67ns 533.2±0.32ns -0.09%
decimal::pow/wasmer 7.5±0.01µs 7.5±0.08µs 0.00%
decimal::pow/wasmer-call-native 1172.8±0.66ns 1178.3±1.11ns +0.47%
decimal::pow/wasmi 137.2±0.19µs 138.9±0.38µs +1.24%
decimal::pow/wasmi-call-native 8.4±0.05µs 8.4±0.02µs 0.00%
decimal::root/0 9.8±0.06µs 9.6±0.01µs -2.04%
decimal::sub/0 7.4±0.46ns 7.2±0.00ns -2.70%
decimal::to_string/0 514.1±0.84ns 516.2±0.72ns +0.41%
precise_decimal::add/0 7.8±0.00ns 8.3±0.00ns +6.41%
precise_decimal::add/rust-native 9.6±0.01ns 9.7±0.13ns +1.04%
precise_decimal::add/wasmer 149.8±0.54ns 149.9±0.09ns +0.07%
precise_decimal::add/wasmer-call-native 601.3±0.34ns 602.8±0.65ns +0.25%
precise_decimal::add/wasmi 626.3±1.58ns 591.4±0.61ns -5.57%
precise_decimal::add/wasmi-call-native 6.6±0.04µs 6.6±0.27µs 0.00%
precise_decimal::div/0 280.1±0.35ns 277.0±0.06ns -1.11%
precise_decimal::from_string/0 276.2±1.04ns 274.5±0.13ns -0.62%
precise_decimal::mul/0 305.9±0.66ns 307.7±0.96ns +0.59%
precise_decimal::mul/rust-native 298.9±56.81ns 279.1±0.12ns -6.62%
precise_decimal::mul/wasmer 4.0±0.00µs 4.0±0.01µs 0.00%
precise_decimal::mul/wasmer-call-native 906.3±2.46ns 904.1±1.92ns -0.24%
precise_decimal::mul/wasmi 80.9±0.06µs 79.9±0.18µs -1.24%
precise_decimal::mul/wasmi-call-native 6.9±0.05µs 6.7±0.03µs -2.90%
precise_decimal::pow/0 1664.8±1.08ns 1645.6±2.85ns -1.15%
precise_decimal::pow/rust-native 1298.1±1.28ns 1302.3±3.26ns +0.32%
precise_decimal::pow/wasmer 19.1±0.02µs 19.0±0.03µs -0.52%
precise_decimal::pow/wasmer-call-native 2.1±0.01µs 2.1±0.00µs 0.00%
precise_decimal::pow/wasmi 395.7±11.48µs 386.2±0.23µs -2.40%
precise_decimal::pow/wasmi-call-native 11.4±0.06µs 11.3±0.05µs -0.88%
precise_decimal::root/0 63.6±0.62µs 62.4±0.07µs -1.89%
precise_decimal::sub/0 8.3±0.01ns 8.9±0.00ns +7.23%
precise_decimal::to_string/0 760.1±3.46ns 751.8±0.96ns -1.09%
schema::validate_payload 255.4±0.89µs 305.4±0.71µs +19.58%
transaction::radiswap 5.5±0.07ms 5.5±0.05ms 0.00%
transaction::transfer 2.2±0.01ms 2.3±0.01ms +4.55%
transaction_processing::prepare 2.5±0.00ms 2.6±0.01ms +4.00%
transaction_processing::prepare_and_decompile 6.8±0.03ms 7.0±0.13ms +2.94%
transaction_processing::prepare_and_decompile_and_recompile 24.2±0.27ms 25.0±0.17ms +3.31%
transaction_validation::validate_manifest 46.3±0.05µs 46.4±1.54µs +0.22%
transaction_validation::verify_ecdsa 82.4±0.10µs 82.4±1.03µs 0.00%
transaction_validation::verify_ed25519 51.6±0.06µs 51.6±0.16µs 0.00%

@dhedey dhedey changed the base branch from data-model-versioning-branch-point to develop August 17, 2023 01:39
use sbor::LocalTypeIndex;
use crate::internal_prelude::*;

define_wrapped_hash!(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice macro!

@@ -195,7 +188,7 @@ pub fn to_typed_substate_key(
}
SCHEMAS_PARTITION => {
let key = substate_key.for_map().ok_or_else(|| error("Schema key"))?;
TypedSubstateKey::Schema(TypedSchemaSubstateKey::SchemaKey(
TypedSubstateKey::SchemaModule(TypedSchemaSubstateKey::SchemaKey(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I wouldn't consider this a module. In our model a module is specifically something with logic/methods. In this case the schemas are just there as data for use.

}

/// This is used mostly for flashing
pub fn into_kernel_main_partitions(self, feature_checks: [<$blueprint_ident FeatureChecks>]) -> Result<NodeSubstates, RuntimeError> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is right place for doing kernel mappings. The problem is that this mapping now needs to be maintained in multiple places and enforces the requirement that the mapping is static and may never change in the future, losing flexibility/updatability. The proper mechanism instead would be to use "System Layer" code to do this mapping.

);
}
let mut partitions = package_state_init
.into_kernel_main_partitions(own_features.into())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't be the PackageStateInit which should decide where things are mapped, an example of why this model is off.

@@ -738,139 +589,26 @@ pub fn create_bootstrap_package_partitions(

fn globalize_package<Y>(
package_address_reservation: Option<GlobalAddressReservation>,
package_structure: PackageStructure,
mut package_state_init: PackageStateInit,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting idea!

blueprint_version_auth_configs,
code_vm_type: vm_type_substates,
code_original_code: original_code_substates,
code_instrumented_code: instrumented_code_substates,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice structure!

@@ -1493,16 +1154,16 @@ impl PackageRoyaltyNativeBlueprint {
if royalty_charge.is_non_zero() {
let handle = api.kernel_open_substate(
receiver,
MAIN_BASE_PARTITION,
&PackageField::Royalty.into(),
PackagePartition::Field.as_main_partition(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, would rather not rely on outside system code to specify what the main_partition is.

type Error = ();

fn try_from(offset: PartitionOffset) -> Result<Self, Self::Error> {
// NOTE: This should really be a fixed mapping - but it's hard to do with declarative macros
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is this mapping really shouldn't be done here.

description: "Enables the package royalty substate",
}
},
instance_schema_types: {},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generic_arguments?

@dhedey dhedey added the substate Any change to substates, the state model, or what's stored in the DB label Aug 17, 2023
@dhedey dhedey marked this pull request as ready for review August 17, 2023 15:14
@dhedey dhedey marked this pull request as draft August 21, 2023 01:16
@dhedey dhedey changed the title Feature: Versioned native data models Feature: Full declare_native_blueprint_state macro Aug 21, 2023
@dhedey dhedey changed the title Feature: Full declare_native_blueprint_state macro Feature: Full declare_native_blueprint_state macro Aug 21, 2023
@dhedey dhedey changed the title Feature: Full declare_native_blueprint_state macro [ON HOLD] Feature: Full declare_native_blueprint_state macro Sep 1, 2023
@dhedey
Copy link
Contributor Author

dhedey commented Mar 22, 2024

Closing in favour of:

Replacement PR 2023 - merged: #1358
Extension PR 2024 - draft: #1744

@dhedey dhedey closed this Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
substate Any change to substates, the state model, or what's stored in the DB
Development

Successfully merging this pull request may close these issues.

2 participants