Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean hot swap #658

Closed
wants to merge 72 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
3c17efb
Ban unsafe arithmetic operations
keithtensor May 24, 2024
1a0d09f
Merge pull request #529 from opentensor/ban-unsafe-arithmetic-devnet
sam0x17 Jun 19, 2024
5e9955a
Merge remote-tracking branch 'origin/devnet-ready' into sam-add-freez…
sam0x17 Jun 19, 2024
68ab335
appease clippy
sam0x17 Jun 19, 2024
6000266
Merge pull request #550 from opentensor/sam-add-freeze-layout-devnet
sam0x17 Jun 19, 2024
705bfe7
Merge remote-tracking branch 'origin/devnet-ready' into sam-add-freez…
sam0x17 Jun 21, 2024
925b06c
bump CI
sam0x17 Jun 21, 2024
b5bc72d
Merge remote-tracking branch 'origin/devnet-ready' into fix-chainspec…
sam0x17 Jun 21, 2024
61fa5bd
Merge pull request #555 from opentensor/fix-chainspecs-devnet-companion
sam0x17 Jun 21, 2024
fce53c0
Merge pull request #554 from opentensor/sam-add-freeze-layout-devnet-2
sam0x17 Jun 21, 2024
f339e8e
bump CI
sam0x17 Jun 21, 2024
fae9db0
Merge branch 'devnet-ready' into fix-add-adjust-senate-call-vune-devn…
Jun 21, 2024
9f09d94
chore: clippy
Jun 21, 2024
f34be4c
Merge pull request #557 from opentensor/fix-add-adjust-senate-call-vu…
distributedstatemachine Jun 21, 2024
2b07824
Merge remote-tracking branch 'origin/devnet-ready' into keith-safe-mo…
sam0x17 Jun 21, 2024
a4f9ad7
re-generate Cargo.lock after merge
sam0x17 Jun 21, 2024
74e1beb
fix
sam0x17 Jun 21, 2024
1692d24
Merge pull request #559 from opentensor/keith-safe-mode-devnet-compan…
distributedstatemachine Jun 21, 2024
ced39fd
override default pages to 60k
orriin Jun 23, 2024
f5dff31
update doc
orriin Jun 24, 2024
abe2f3e
Merge branch 'devnet-ready' into devnet_companion_feat/liquid_alpha_2
Jun 24, 2024
abc7798
clippy
Jun 24, 2024
ab6212a
Merge pull request #562 from opentensor/override-default-pages-devnet…
sam0x17 Jun 24, 2024
b1c26b3
feat: clippy
Jun 24, 2024
5c66560
Merge pull request #568 from opentensor/devnet_companion_feat/liquid_…
sam0x17 Jun 24, 2024
2b10e12
refactor: hotkey swap + tests
Jun 17, 2024
7eb29f9
chore: lints
Jun 17, 2024
6ddcfe2
chore: clippy
Jun 17, 2024
45e828d
chore: give 1k on faucet for easier to create networks locally
Jun 17, 2024
c12ec37
Correct the expected weights on the register extrinsic
keithtensor Jun 17, 2024
5eaa088
Update pallets/subtensor/src/swap.rs
distributedstatemachine Jun 17, 2024
969a4f8
chore: updates from using real weights in tests
Jun 17, 2024
6b68263
chore: review comments , make swap cost a constant
Jun 17, 2024
ec27e77
chore: runtime consts
Jun 17, 2024
fc82551
chore: clear prefix for removing old value from stake map
Jun 17, 2024
5057472
chore: pr comments assert keys
Jun 17, 2024
dd9672b
chore: pr comments: remove last tx block
Jun 17, 2024
60019a9
chore: fmt
Jun 17, 2024
2cd90dd
Use concrete numbers for weight expectations
keithtensor Jun 18, 2024
5743ae7
Update pallets/subtensor/src/swap.rs
distributedstatemachine Jun 18, 2024
f358847
chore: add test_swap_hotkey_tx_rate_limit_exceeded
Jun 18, 2024
768e8e4
chore: fmt
Jun 18, 2024
cb078a2
chore: review comments
Jun 18, 2024
814447f
fix: pr comments
Jun 21, 2024
680e4ad
chore: lint
Jun 21, 2024
f7b567a
fix: remove unused function
Jun 21, 2024
78663d8
add requirement for `devnet-companion` label for `devnet-ready` PRs
sam0x17 Jun 19, 2024
76d9439
add testnet-ready labels check
sam0x17 Jun 19, 2024
6bd2e42
update CONTRIBUTING.md with additional info
sam0x17 Jun 19, 2024
ad85c27
update docs
sam0x17 Jun 19, 2024
66c87af
clippy fix
sam0x17 Jun 21, 2024
01dca6a
make clippy happy
Jun 24, 2024
5c46be0
Merge pull request #567 from opentensor/devnet_companion_refactor/hot…
sam0x17 Jun 24, 2024
ecb6221
update spec file usage
orriin Jun 25, 2024
efb18f3
Merge pull request #571 from opentensor/chainspec-file-update
orriin Jun 25, 2024
81ed321
feat: bump spec
Jun 25, 2024
2ea077b
feat: benchmark adjust senate
Jun 25, 2024
2b6d54f
Merge pull request #583 from opentensor/feat/adjust_senate_benchmarks
distributedstatemachine Jun 25, 2024
dc3f178
Make weighted_median iterative
keithtensor Jul 1, 2024
bba97c8
Merge remote-tracking branch 'origin/main' into merge-main-into-devne…
sam0x17 Jul 12, 2024
177c47a
fix merge conflict
sam0x17 Jul 12, 2024
98a1ed2
Merge pull request #629 from opentensor/merge-main-into-devnet-ready-…
sam0x17 Jul 15, 2024
aa5e890
Merge main into devnet ready 7 18 2024 (#650)
camfairchild Jul 19, 2024
ce3e87e
Merge pull request #628 from opentensor/end-recursion-devnet
keithtensor Jul 22, 2024
1c50295
Feat/devnet companion/check metadata hash extension (#649)
camfairchild Jul 22, 2024
07e46e3
initial
Jul 22, 2024
0432b53
add tests
Jul 22, 2024
7f938f7
tests pass
Jul 22, 2024
659b53c
clean the code
Jul 22, 2024
b8cfd6c
fix tests
Jul 22, 2024
6b6460d
clippy and fmt
Jul 22, 2024
d66cb46
Merge branch 'devnet-ready' into clean_hot_swap
unconst Jul 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
485 changes: 277 additions & 208 deletions Cargo.lock

Large diffs are not rendered by default.

139 changes: 70 additions & 69 deletions Cargo.toml

Large diffs are not rendered by default.

6 changes: 5 additions & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,8 @@ lint:
@echo "Running cargo clippy with automatic fixes on potentially dirty code..."
just clippy-fix
@echo "Running cargo clippy..."
just clippy
just clippy

production:
@echo "Running cargo build with metadata-hash generation..."
cargo +{{RUSTV}} build --profile production --features="runtime-benchmarks metadata-hash"
1 change: 1 addition & 0 deletions node/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ sp-io = { workspace = true }
sp-timestamp = { workspace = true }
sp-inherents = { workspace = true }
sp-keyring = { workspace = true }
frame-metadata-hash-extension = { workspace = true }
frame-system = { workspace = true }
pallet-transaction-payment = { workspace = true }
pallet-commitments = { path = "../pallets/commitments" }
Expand Down
2 changes: 2 additions & 0 deletions node/src/benchmarking.rs
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ pub fn create_benchmark_extrinsic(
pallet_transaction_payment::ChargeTransactionPayment::<runtime::Runtime>::from(0),
pallet_subtensor::SubtensorSignedExtension::<runtime::Runtime>::new(),
pallet_commitments::CommitmentsSignedExtension::<runtime::Runtime>::new(),
frame_metadata_hash_extension::CheckMetadataHash::<runtime::Runtime>::new(true),
);

let raw_payload = runtime::SignedPayload::from_raw(
Expand All @@ -152,6 +153,7 @@ pub fn create_benchmark_extrinsic(
(),
(),
(),
None,
),
);
let signature = raw_payload.using_encoded(|e| sender.sign(e));
Expand Down
2 changes: 1 addition & 1 deletion pallets/subtensor/src/benchmarks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ benchmarks! {
// This is a whitelisted caller who can make transaction without weights.
let caller: T::AccountId = whitelisted_caller::<AccountIdOf<T>>();
let caller_origin = <T as frame_system::Config>::RuntimeOrigin::from(RawOrigin::Signed(caller.clone()));
let netuid: u16 = 1;
let netuid: u16 = 0;
let version_key: u64 = 1;
let tempo: u16 = 1;
let modality: u16 = 0;
Expand Down
2 changes: 0 additions & 2 deletions pallets/subtensor/src/epoch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -188,8 +188,6 @@ impl<T: Config> Pallet<T> {
// =================================

// Compute emission scores.

// Compute normalized emission scores. range: I32F32(0, 1)
// Compute normalized emission scores. range: I32F32(0, 1)
let combined_emission: Vec<I32F32> = incentive
.iter()
Expand Down
46 changes: 23 additions & 23 deletions pallets/subtensor/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ mod root;
mod serving;
mod staking;
mod swap;
mod swap_hotkey;
mod uids;
mod utils;
mod weights;
Expand Down Expand Up @@ -272,7 +273,7 @@ pub mod pallet {
}

#[pallet::storage]
pub(super) type SenateRequiredStakePercentage<T> =
pub type SenateRequiredStakePercentage<T> =
StorageValue<_, u64, ValueQuery, DefaultSenateRequiredStakePercentage<T>>;

/// ============================
Expand Down Expand Up @@ -367,7 +368,7 @@ pub mod pallet {
pub type TotalColdkeyStake<T: Config> =
StorageMap<_, Identity, T::AccountId, u64, ValueQuery, DefaultAccountTake<T>>;
#[pallet::storage]
/// MAP (hot, cold) --> stake | Returns a tuple (u64: stakes, u64: block_number)
/// MAP (hot, cold) --> u64, u64) | Returns a tuple (u64: stakes, u64: block_number)
pub type TotalHotkeyColdkeyStakesThisInterval<T: Config> = StorageDoubleMap<
_,
Identity,
Expand Down Expand Up @@ -796,15 +797,15 @@ pub mod pallet {
}

#[pallet::storage] // --- ITEM ( tx_rate_limit )
pub(super) type TxRateLimit<T> = StorageValue<_, u64, ValueQuery, DefaultTxRateLimit<T>>;
pub type TxRateLimit<T> = StorageValue<_, u64, ValueQuery, DefaultTxRateLimit<T>>;
#[pallet::storage] // --- ITEM ( tx_rate_limit )
pub(super) type TxDelegateTakeRateLimit<T> =
pub type TxDelegateTakeRateLimit<T> =
StorageValue<_, u64, ValueQuery, DefaultTxDelegateTakeRateLimit<T>>;
#[pallet::storage] // --- MAP ( key ) --> last_block
pub type LastTxBlock<T: Config> =
StorageMap<_, Identity, T::AccountId, u64, ValueQuery, DefaultLastTxBlock<T>>;
#[pallet::storage] // --- MAP ( key ) --> last_block
pub(super) type LastTxBlockDelegateTake<T: Config> =
pub type LastTxBlockDelegateTake<T: Config> =
StorageMap<_, Identity, T::AccountId, u64, ValueQuery, DefaultLastTxBlock<T>>;

/// Default value for serving rate limit.
Expand Down Expand Up @@ -1084,41 +1085,39 @@ pub mod pallet {
StorageMap<_, Identity, u16, Vec<(T::AccountId, u64, u64)>, OptionQuery>;

#[pallet::storage] // --- DMAP ( netuid ) --> active
pub(super) type Active<T: Config> =
pub type Active<T: Config> =
StorageMap<_, Identity, u16, Vec<bool>, ValueQuery, EmptyBoolVec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> rank
pub(super) type Rank<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
pub type Rank<T: Config> = StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> trust
pub(super) type Trust<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
pub type Trust<T: Config> = StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> consensus
pub(super) type Consensus<T: Config> =
pub type Consensus<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> incentive
pub(super) type Incentive<T: Config> =
pub type Incentive<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> dividends
pub(super) type Dividends<T: Config> =
pub type Dividends<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> emission
pub(super) type Emission<T: Config> =
pub type Emission<T: Config> =
StorageMap<_, Identity, u16, Vec<u64>, ValueQuery, EmptyU64Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> last_update
pub(super) type LastUpdate<T: Config> =
pub type LastUpdate<T: Config> =
StorageMap<_, Identity, u16, Vec<u64>, ValueQuery, EmptyU64Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> validator_trust
pub(super) type ValidatorTrust<T: Config> =
pub type ValidatorTrust<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> pruning_scores
pub(super) type PruningScores<T: Config> =
pub type PruningScores<T: Config> =
StorageMap<_, Identity, u16, Vec<u16>, ValueQuery, EmptyU16Vec<T>>;
#[pallet::storage] // --- DMAP ( netuid ) --> validator_permit
pub(super) type ValidatorPermit<T: Config> =
pub type ValidatorPermit<T: Config> =
StorageMap<_, Identity, u16, Vec<bool>, ValueQuery, EmptyBoolVec<T>>;

#[pallet::storage] // --- DMAP ( netuid, uid ) --> weights
pub(super) type Weights<T: Config> = StorageDoubleMap<
pub type Weights<T: Config> = StorageDoubleMap<
_,
Identity,
u16,
Expand All @@ -1129,7 +1128,7 @@ pub mod pallet {
DefaultWeights<T>,
>;
#[pallet::storage] // --- DMAP ( netuid, uid ) --> bonds
pub(super) type Bonds<T: Config> = StorageDoubleMap<
pub type Bonds<T: Config> = StorageDoubleMap<
_,
Identity,
u16,
Expand Down Expand Up @@ -2045,9 +2044,10 @@ pub mod pallet {

/// Attempt to adjust the senate membership to include a hotkey
#[pallet::call_index(63)]
#[pallet::weight((Weight::from_parts(0, 0)
.saturating_add(T::DbWeight::get().reads(0))
.saturating_add(T::DbWeight::get().writes(0)), DispatchClass::Normal, Pays::Yes))]
#[pallet::weight((Weight::from_parts(50_000_000, 0)
.saturating_add(Weight::from_parts(0, 4632))
.saturating_add(T::DbWeight::get().reads(5))
.saturating_add(T::DbWeight::get().writes(1)), DispatchClass::Normal, Pays::Yes))]
pub fn adjust_senate(origin: OriginFor<T>, hotkey: T::AccountId) -> DispatchResult {
Self::do_adjust_senate(origin, hotkey)
}
Expand Down
85 changes: 37 additions & 48 deletions pallets/subtensor/src/math.rs
Original file line number Diff line number Diff line change
Expand Up @@ -889,59 +889,48 @@ pub fn weighted_median(
score: &[I32F32],
partition_idx: &[usize],
minority: I32F32,
partition_lo: I32F32,
partition_hi: I32F32,
mut partition_lo: I32F32,
mut partition_hi: I32F32,
) -> I32F32 {
let n = partition_idx.len();
if n == 0 {
return I32F32::from_num(0);
}
if n == 1 {
return score[partition_idx[0]];
}
assert!(stake.len() == score.len());
let mid_idx: usize = n.saturating_div(2);
let pivot: I32F32 = score[partition_idx[mid_idx]];
let mut lo_stake: I32F32 = I32F32::from_num(0);
let mut hi_stake: I32F32 = I32F32::from_num(0);
let mut lower: Vec<usize> = vec![];
let mut upper: Vec<usize> = vec![];
for &idx in partition_idx {
if score[idx] == pivot {
continue;
let mut current_partition_idx = partition_idx.to_vec();
while !current_partition_idx.is_empty() {
let n = current_partition_idx.len();
if n == 1 {
return score[current_partition_idx[0]];
}
let mid_idx: usize = n.saturating_div(2);
let pivot: I32F32 = score[current_partition_idx[mid_idx]];
let mut lo_stake: I32F32 = I32F32::from_num(0);
let mut hi_stake: I32F32 = I32F32::from_num(0);
let mut lower: Vec<usize> = vec![];
let mut upper: Vec<usize> = vec![];
for &idx in &current_partition_idx {
if score[idx] == pivot {
continue;
}
if score[idx] < pivot {
lo_stake = lo_stake.saturating_add(stake[idx]);
lower.push(idx);
} else {
hi_stake = hi_stake.saturating_add(stake[idx]);
upper.push(idx);
}
}
if score[idx] < pivot {
lo_stake = lo_stake.saturating_add(stake[idx]);
lower.push(idx);
if partition_lo.saturating_add(lo_stake) <= minority
&& minority < partition_hi.saturating_sub(hi_stake)
{
return pivot;
} else if (minority < partition_lo.saturating_add(lo_stake)) && (!lower.is_empty()) {
current_partition_idx = lower;
partition_hi = partition_lo.saturating_add(lo_stake);
} else if (partition_hi.saturating_sub(hi_stake) <= minority) && (!upper.is_empty()) {
current_partition_idx = upper;
partition_lo = partition_hi.saturating_sub(hi_stake);
} else {
hi_stake = hi_stake.saturating_add(stake[idx]);
upper.push(idx);
return pivot;
}
}
if (partition_lo.saturating_add(lo_stake) <= minority)
&& (minority < partition_hi.saturating_sub(hi_stake))
{
return pivot;
} else if (minority < partition_lo.saturating_add(lo_stake)) && (!lower.is_empty()) {
return weighted_median(
stake,
score,
&lower,
minority,
partition_lo,
partition_lo.saturating_add(lo_stake),
);
} else if (partition_hi.saturating_sub(hi_stake) <= minority) && (!upper.is_empty()) {
return weighted_median(
stake,
score,
&upper,
minority,
partition_hi.saturating_sub(hi_stake),
partition_hi,
);
}
pivot
I32F32::from_num(0)
}

/// Column-wise weighted median, e.g. stake-weighted median scores per server (column) over all validators (rows).
Expand Down
Loading
Loading