Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(oracle): WIP add scores account #407

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
b343951
feat(oracle): WIP add scores account
keyvankhademi Jun 7, 2024
1561e80
feat: use bitset to reduce scores account size
keyvankhademi Jun 10, 2024
b58625c
fix: pre-commit
keyvankhademi Jun 10, 2024
18c6787
fix: add price test
keyvankhademi Jun 13, 2024
32c577e
fix: rename score to cap
keyvankhademi Jun 13, 2024
2ec0004
feat: remove publisher sorting features
keyvankhademi Jun 14, 2024
a02a10b
feat: improve caps account calculations
keyvankhademi Jun 14, 2024
0c98527
feat: rename score to publisher_caps
keyvankhademi Jun 14, 2024
87f61ec
fix: bypass false positive clippy
keyvankhademi Jun 14, 2024
2f6f359
feat: optimize oracle command enum
keyvankhademi Jun 14, 2024
2697cb2
feat: test new way of calculating score
keyvankhademi Jun 14, 2024
d07a256
fix: correct rust doc format
keyvankhademi Jun 17, 2024
98a6a52
fix: clippy
keyvankhademi Jun 17, 2024
4260e2e
feat: add vscode settings
keyvankhademi Jun 17, 2024
6c9d3e6
fix: clippy warning
keyvankhademi Jun 17, 2024
8e00103
feat: change OracleCommand repr from i32 to u32
keyvankhademi Jun 17, 2024
aa1e0a7
feat: add initial tests for publisher caps
keyvankhademi Jun 17, 2024
2372be6
feat: add migration step
keyvankhademi Jun 17, 2024
63352ec
test: try a different implementation for program size
keyvankhademi Jun 17, 2024
aaa12bf
test: try manual swap to decrease program size
keyvankhademi Jun 17, 2024
e2aa645
fix: add price when initializing the caps account
keyvankhademi Jun 17, 2024
2169e8d
fix: rename scores account to cap account
keyvankhademi Jun 17, 2024
de0f7c3
refactor: rename calculate_scores to calculate_caps
keyvankhademi Jun 17, 2024
4396054
refactor: rename symbols to prices in PublisherCapsAccount
keyvankhademi Jun 17, 2024
0563db6
refactor: update publisher permission index to use price instead of s…
keyvankhademi Jun 17, 2024
6c321bf
refactor: update publisher permission index to use price instead of s…
keyvankhademi Jun 17, 2024
9e90897
chore: Update num-traits dependency to version 0.2
keyvankhademi Jun 17, 2024
5c1f750
feat: add initial quickcheck for caps account
keyvankhademi Jun 18, 2024
02f6257
refactor: update upd_price.rs to use send_message_to_message_buffer f…
keyvankhademi Jun 18, 2024
7800412
feat: send caps to message buffer for add publisher
keyvankhademi Jun 19, 2024
9b52a40
fix: comment migration code to test program size
keyvankhademi Jun 19, 2024
a18ec0c
feat: remove init mapping to reduce program size
keyvankhademi Jun 19, 2024
5932402
feat: keep init mapping for tests
keyvankhademi Jun 19, 2024
21c38c8
fix: self ref error
keyvankhademi Jun 19, 2024
413abc3
fix: revert mapping account changes
keyvankhademi Jun 19, 2024
23a1da8
feat: add cfg test flag for init mapping
keyvankhademi Jun 19, 2024
1a11ca4
temporary remove pipeline test
keyvankhademi Jun 19, 2024
1fa4c6a
remove upd_permissions to check size
keyvankhademi Jun 19, 2024
dd453fb
add feature test to be able to compile oracle for test
keyvankhademi Jun 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions program/c/src/oracle/oracle.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,12 @@ extern "C" {
#define PC_PUBKEY_SIZE 32
#define PC_PUBKEY_SIZE_64 (PC_PUBKEY_SIZE/sizeof(uint64_t))
#define PC_MAP_TABLE_SIZE 640
#define PC_MAX_PUBLISHERS 256
#define PC_MAX_SYMBOLS 1024
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what we do if we exceed this numbers?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a Pythnet v2 problem anyway

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The program doesn't allow exceeding these numbers, we will probably face a lot of issues if we decide to increase these numbers. We are hoping to go to pythnet V2 before we need to increase these numbers. From the trasactions limit that we currently have, i don't think this will an issue.
But if you think there is a chance that we exceed these number before pythnet v2, we better address it now.

// This is the number of 64-bit words needed to store the bitset of symbols
#define PC_MAX_SYMBOLS_64 (PC_MAX_SYMBOLS/64)

// Total price component slots available
// Total price component slots available
#define PC_NUM_COMP_PYTHNET 128

// PC_NUM_COMP - number of price components in use
Expand All @@ -49,11 +53,12 @@ extern "C" {
#define PC_STATUS_IGNORED 4

// account types
#define PC_ACCTYPE_MAPPING 1
#define PC_ACCTYPE_PRODUCT 2
#define PC_ACCTYPE_PRICE 3
#define PC_ACCTYPE_TEST 4
#define PC_ACCTYPE_PERMISSIONS 5
#define PC_ACCTYPE_MAPPING 1
#define PC_ACCTYPE_PRODUCT 2
#define PC_ACCTYPE_PRICE 3
#define PC_ACCTYPE_TEST 4
#define PC_ACCTYPE_PERMISSIONS 5
#define PC_ACCTYPE_SCORE 6


// Compute budget requested per price update instruction
Expand Down
2 changes: 2 additions & 0 deletions program/rust/src/accounts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ mod mapping;
mod permission;
mod price;
mod product;
mod score;

// Some types only exist during use as a library.
#[cfg(feature = "strum")]
Expand All @@ -64,6 +65,7 @@ pub use {
update_product_metadata,
ProductAccount,
},
score::PublisherCapsAccount,
};

// PDA seeds for accounts.
Expand Down
188 changes: 188 additions & 0 deletions program/rust/src/accounts/score.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
use {
super::{
AccountHeader,
PythAccount,
},
crate::c_oracle_header::{
PC_ACCTYPE_SCORE,
PC_MAX_PUBLISHERS,
PC_MAX_SYMBOLS,
PC_MAX_SYMBOLS_64,
},
bytemuck::{
Pod,
Zeroable,
},
solana_program::{
program_error::ProgramError,
pubkey::Pubkey,
},
std::cmp::max,
};

#[repr(C)]
#[derive(Copy, Clone)]
/*
* This account is part of Community Integrity Pool (CIP) project.
* It is used to store the caps of the publishers which will be sent
* to the `integrity_pool` program on mainnet to calculate the rewards.
*/
keyvankhademi marked this conversation as resolved.
Show resolved Hide resolved
pub struct PublisherCapsAccount {
pub header: AccountHeader,
pub num_publishers: usize,
pub num_symbols: usize,
// Z is a constant used to normalize the scores
pub z: u32,
// M is a constant showing the target stake per symbol
pub m: u32,

// array[x][y] is a u64 whose bits represent if publisher x publishes symbols 64*y to 64*(y+1) - 1
pub publisher_permissions: [[u64; PC_MAX_SYMBOLS_64 as usize]; PC_MAX_PUBLISHERS as usize],
pub caps: [f64; PC_MAX_PUBLISHERS as usize],
pub publishers: [Pubkey; PC_MAX_PUBLISHERS as usize],
pub symbols: [Pubkey; PC_MAX_SYMBOLS as usize],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually already have something similar to this (a list of all symbols) in the mapping account but the product accounts (and not the price accounts) are listed there

}

impl PublisherCapsAccount {
fn get_publisher_permission(&self, x: usize, y: usize) -> bool {
(self.publisher_permissions[x][y / 64] >> (y % 64)) & 1 == 1
}

fn set_publisher_permission(&mut self, x: usize, y: usize, value: bool) {
if value {
self.publisher_permissions[x][y / 64] |= 1 << (y % 64);
} else {
self.publisher_permissions[x][y / 64] &= !(1 << (y % 64));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_publisher_permission and set_publisher_permission could almost be in their own struct PermissionTable, that we can proptest

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try this at some point to see how much it will change the program size, my current guess is that it will increase it.

}

pub fn add_publisher(
&mut self,
publisher: Pubkey,
price_account: Pubkey,
) -> Result<(), ProgramError> {
let publisher_index = self
.publishers
.iter()
.position(|&x| x == publisher)
.unwrap_or(self.num_publishers);

if publisher_index == self.num_publishers {
if self.num_publishers == PC_MAX_PUBLISHERS as usize {
return Err(ProgramError::AccountDataTooSmall);
}

self.publishers[self.num_publishers] = publisher;
self.num_publishers += 1;
}

let symbol_index = self
.symbols
.iter()
.position(|&x| x == price_account)
.ok_or(ProgramError::InvalidArgument)?;

self.set_publisher_permission(publisher_index, symbol_index, true);
self.calculate_scores()
}

pub fn del_publisher(
&mut self,
publisher: Pubkey,
price_account: Pubkey,
) -> Result<(), ProgramError> {
let publisher_index = self
.publishers
.iter()
.position(|&x| x == publisher)
.ok_or(ProgramError::InvalidArgument)?;

let symbol_index = self
.symbols
.iter()
.position(|&x| x == price_account)
.ok_or(ProgramError::InvalidArgument)?;

self.set_publisher_permission(publisher_index, symbol_index, false);
self.calculate_scores()
}

pub fn add_price(&mut self, symbol: Pubkey) -> Result<(), ProgramError> {
let symbol_index = self
.symbols
.iter()
.position(|&x| x == symbol)
.unwrap_or(self.num_symbols);

if symbol_index == self.num_symbols {
if self.num_symbols == PC_MAX_SYMBOLS as usize {
return Err(ProgramError::AccountDataTooSmall);
}

self.symbols[self.num_symbols] = symbol;
self.num_symbols += 1;
}

Ok(())
}

pub fn del_price(&mut self, symbol: Pubkey) -> Result<(), ProgramError> {
let symbol_index = self
.symbols
.iter()
.position(|&x| x == symbol)
.ok_or(ProgramError::InvalidArgument)?;

// update symbol list
self.symbols[symbol_index] = self.symbols[self.num_symbols - 1];
self.symbols[self.num_symbols - 1] = Pubkey::default();

// update publisher permissions
for i in 0..self.num_publishers {
let value = self.get_publisher_permission(i, self.num_symbols - 1);
self.set_publisher_permission(i, symbol_index, value)
}

self.num_symbols -= 1;
self.calculate_scores()
}

pub fn calculate_scores(&mut self) -> Result<(), ProgramError> {
let symbol_scores: Vec<u32> = self
.symbols
.iter()
.enumerate()
.map(|(j, _)| {
let score = self
.publisher_permissions
.iter()
.fold(0, |score, permissions| score + permissions[j] as u32);
keyvankhademi marked this conversation as resolved.
Show resolved Hide resolved
max(score, self.z)
})
.collect();

for i in 0..self.num_publishers {
self.caps[i] = self
.symbols
.iter()
.enumerate()
.filter(|(j, _)| self.get_publisher_permission(i, *j))
.map(|(j, _)| 1f64 / symbol_scores[j] as f64)
.sum();
}
Ok(())
}
}

impl PythAccount for PublisherCapsAccount {
const ACCOUNT_TYPE: u32 = PC_ACCTYPE_SCORE;
// Calculate the initial size of the account
const INITIAL_SIZE: u32 = 75816;
}

// Unsafe impl because product_list is of size 640 and there's no derived trait for this size
unsafe impl Pod for PublisherCapsAccount {
}

unsafe impl Zeroable for PublisherCapsAccount {
}
18 changes: 14 additions & 4 deletions program/rust/src/processor/add_price.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ use {
accounts::{
PriceAccount,
ProductAccount,
PublisherCapsAccount,
PythAccount,
},
c_oracle_header::{
Expand Down Expand Up @@ -34,6 +35,7 @@ use {
// account[0] funding account [signer writable]
// account[1] product account [signer writable]
// account[2] new price account [signer writable]
// account[3] scores account [signer writable]
pub fn add_price(
program_id: &Pubkey,
accounts: &[AccountInfo],
Expand All @@ -48,10 +50,12 @@ pub fn add_price(
)?;


let (funding_account, product_account, price_account, permissions_account) = match accounts {
[x, y, z, p] => Ok((x, y, z, p)),
_ => Err(OracleError::InvalidNumberOfAccounts),
}?;
let (funding_account, product_account, price_account, permissions_account, scores_account) =
match accounts {
[x, y, z, p] => Ok((x, y, z, p, None)),
[x, y, z, p, s] => Ok((x, y, z, p, Some(s))),
keyvankhademi marked this conversation as resolved.
Show resolved Hide resolved
_ => Err(OracleError::InvalidNumberOfAccounts),
}?;

check_valid_funding_account(funding_account)?;
check_permissioned_funding_account(
Expand Down Expand Up @@ -80,5 +84,11 @@ pub fn add_price(
price_data.min_pub_ = PRICE_ACCOUNT_DEFAULT_MIN_PUB;
product_data.first_price_account = *price_account.key;

if let Some(scores_account) = scores_account {
let mut scores_account =
load_checked::<PublisherCapsAccount>(scores_account, cmd_args.header.version)?;
scores_account.add_price(*price_account.key)?;
}

Ok(())
}
14 changes: 12 additions & 2 deletions program/rust/src/processor/add_publisher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ use {
accounts::{
PriceAccount,
PriceComponent,
PublisherCapsAccount,
PythAccount,
},
c_oracle_header::PC_NUM_COMP,
Expand Down Expand Up @@ -36,6 +37,7 @@ use {
/// Add publisher to symbol account
// account[0] funding account [signer writable]
// account[1] price account [signer writable]
// account[2] scores account [signer writable]
pub fn add_publisher(
program_id: &Pubkey,
accounts: &[AccountInfo],
Expand All @@ -48,8 +50,9 @@ pub fn add_publisher(
ProgramError::InvalidArgument,
)?;

let (funding_account, price_account, permissions_account) = match accounts {
[x, y, p] => Ok((x, y, p)),
let (funding_account, price_account, permissions_account, scores_account) = match accounts {
[x, y, p] => Ok((x, y, p, None)),
[x, y, p, s] => Ok((x, y, p, Some(s))),
_ => Err(OracleError::InvalidNumberOfAccounts),
}?;

Expand Down Expand Up @@ -99,6 +102,13 @@ pub fn add_publisher(
}

price_data.header.size = try_convert::<_, u32>(PriceAccount::INITIAL_SIZE)?;

if let Some(scores_account) = scores_account {
let mut scores_account =
load_checked::<PublisherCapsAccount>(scores_account, cmd_args.header.version)?;
scores_account.add_publisher(cmd_args.publisher, *price_account.key)?;
}

Ok(())
}

Expand Down
18 changes: 14 additions & 4 deletions program/rust/src/processor/del_price.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ use {
accounts::{
PriceAccount,
ProductAccount,
PublisherCapsAccount,
},
deserialize::{
load,
Expand Down Expand Up @@ -30,17 +31,20 @@ use {
// account[0] funding account [signer writable]
// account[1] product account [signer writable]
// account[2] price account [signer writable]
// account[3] scores account [signer writable]
/// Warning: This function is dangerous and will break any programs that depend on the deleted
/// price account!
pub fn del_price(
program_id: &Pubkey,
accounts: &[AccountInfo],
instruction_data: &[u8],
) -> ProgramResult {
let (funding_account, product_account, price_account, permissions_account) = match accounts {
[w, x, y, p] => Ok((w, x, y, p)),
_ => Err(OracleError::InvalidNumberOfAccounts),
}?;
let (funding_account, product_account, price_account, permissions_account, scores_account) =
match accounts {
[x, y, z, p] => Ok((x, y, z, p, None)),
[x, y, z, p, s] => Ok((x, y, z, p, Some(s))),
_ => Err(OracleError::InvalidNumberOfAccounts),
}?;

let cmd_args = load::<CommandHeader>(instruction_data)?;

Expand Down Expand Up @@ -83,5 +87,11 @@ pub fn del_price(
**price_account.lamports.borrow_mut() = 0;
**funding_account.lamports.borrow_mut() += lamports;

if let Some(scores_account) = scores_account {
let mut scores_account =
load_checked::<PublisherCapsAccount>(scores_account, cmd_args.version)?;
scores_account.del_price(*price_account.key)?;
}

Ok(())
}
Loading
Loading