-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] add functions to wallet API #9464
base: master
Are you sure you want to change the base?
[WIP] add functions to wallet API #9464
Conversation
As we can see from the table above, there are quite some things that I think we should not just rush headlong into adding those many things. After all, if we are successful, this API will be very important, will live for a long time, and will be used by many people. If the new resulting interface is not well designed, it will haunt us for years to come. Problem is that there are too many ways how you could design it, it's not "naturally clear" what types and methods there have to be. My feeling is that you could do almost everything in a hundred different ways. To contribute something useful in these quite difficult circumstances I try to come up with some general rules to follow when defining the API that will hopefully guide us towards a good solution. |
Make it complete This rule is pretty self-evident and clear, but I repeat it here nevertheless in the sake of completeness. Goal is to provide a viable successor to the "API" that the public things in Because of this, we have to make sure that the Wallet API is feature complete. We must not lose any wallet functionality that is needed and in actual use. |
Make it small We should try, while respecting the first rule of completeness of course, to make the API small. The smaller it is, the faster will new devs be able to read into it, the easier it will be to find the right methods to use, the less work to document it, etc. Where do we have potential to reduce the number of methods? One way I see is this: Do not include methods from I have mentioned an example of such method in this earlier issue about wallet related methods: There is a variant of Of course use common sense: It may be essential that already the code of the Wallet API filters or sorts something because otherwise too much data must get transferred, or doing things in the client would be much, much slower. |
Be careful about terminology A good API uses a consistent terminology when naming things, and uses terms that are "expressive", hard to misunderstand and hard to confuse. IMHO quite a number of things in This motivated me to propose to ban the term transfer outright exactly because it's so easy to misunderstand and easy to confuse. I detailed my proposal here. Of course many things in the Wallet API already have the names they have, and we must let them stand and are not completely free how to name the new things we add. Still I see quite a degree of freedom left. |
Limit the influence of existing clients We have 2 main clients of Of course, striclty only seen from the point of view of that migration, the more similar the Wallet API is to the "old" I propose "No" as answer. I think having a good API is much more important than trying to optimize towards migration work on the CLI wallet and the wallet RPC server being small. That migration happens once and is then over for all times, whereas the Wallet API will go indefinitely into the future and (hopefully) shape tons of Monero related software that together trump any migration work. In detail that means we don't try to give each method of |
Keep the style consistent I write this down more for the sake of completeness, like the very first rule Make it complete: There is a clash in naming style between the Wallet API and most of the rest of the Monero codebase: The first uses casing to form names like For the things we add we basically have two choices: We keep the naming style consistent with the things that are already there in the Wallet API, or we use the snake casing that is much more widely used in the codebase. Both choices are well possible, and both have advantages as well as disadvantages. (Renaming existing things as a third option and a way to achieve overal naming style consistency is probably not on the table.) We dicussed this in a Seraphis wallet workgroup meeting, and it seemed that more people voted for keeping the style consistent. I myself was also in that camp: To continue with the "non-standard" naming style is IMHO the lesser evil than mixing styles within a single API. |
Now my opinion about one the more important design questions:
As I explained here, neither the Wallet API, nor the RPC interface, nor Woodser's wallet API have such separate calls. They basically work with a single call that get payments / transactions of all types, using a struct that is "encompassing" i.e. is able to store all details of all these types. I hope I got that right, the Wallet API call for this seems to be According to the rules Make it small and Limit the influence of existing clients I propose to continue with this one call and not create direct equivalents of these methods in the Wallet API. Probably the type |
src/wallet/api/wallet2_api.h
Outdated
virtual bool isFrozen(const std::string multisig_sign_data) const = 0; | ||
// QUESTION : Should we add the following function, which returns strings containing the key image for every frozen enote? I think it can be useful | ||
// virtual std::vector<std::string> getAllFrozenEnotes() const = 0; | ||
// TODO : Would this better fit in one of `Subaddress`/`SubaddressAccount` classes? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you give some possible arguments for pushing this down to the account level (if I understand you correctly)? I don't see any right now, but you have probably give this much more thought than I :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quick update:
There is no need to move createOneOffSubaddress
to the account level. Since this is an exotic function, maybe we should add a note/warning that this function should only be used in special cases?
(moneromoo: "It was intended to help people who generated a subaddress far into the random index values.")
e.g. something like:
* brief: createOneOffSubaddress - Create a new account or subaddress for given index, without adding it to known accounts and subaddresses.
* Use `SubaddressAccount::addRow()` or `Subaddress::addRow()` instead, to make sure the wallet adds accounts/subaddresses consecutively and keeps track of accounts and subaddresses.
WalletImpl::addSubaddress
& WalletImpl::addSubaddressAccount
also do not refresh the known accounts/subaddresses, if that's not for a good reason (that I don't see) I would either add refresh at the end of those functions, or at least give the same warning as proposed above.
Seems the GUI only uses the addRow
functions (see here) not the addSubaddress*
ones (see here).
src/wallet/api/wallet2_api.h
Outdated
* brief: getTransfers - get all transfers | ||
* outparam: transfers - | ||
*/ | ||
// virtual void getTransfers(wallet2::transfer_container& transfers) const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here we probably enter a whole region of things that we won't need in this form if my proposal to make a single encompassing "get payments / transactions" methods really flies.
src/wallet/api/wallet2_api.h
Outdated
* brief: getMinRingSize - get the minimum allowed ring size | ||
* return: min ring size | ||
*/ | ||
virtual std::uint64_t getMinRingSize() const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can go. We don't have the possibility to choose ringsize anymore for years, and well, soon we won't have any rings at all. Maximum that I see here is an info "ring size" in the new WalletState
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would require something like this: tobtoht@894adf0. Can PR.
* messages and prompts. When it finally calls this method here "to catch up" so to say we can't use | ||
* incremental update anymore, because with that we might miss some txs altogether. | ||
*/ | ||
virtual void updatePoolState(std::vector<std::tuple<cryptonote::transaction, std::string, bool>> &process_txs, bool refreshed = false, bool try_incremental = false) = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure yet what possibilities of "pool info update" the Wallet API has to offer, or should offer, but I am pretty sure that this method in this form is on too low a level of abstraction, like the following processPoolState
as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is too complex for me right now to fully dive into, maybe someone with more experience in this area can comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I try to follow the general principle of YAGNI, especially when making APIs. Don't add it unless someone asks for it. A use case for processing abstractions of the pool state might pop up in the future at some point, but it also might not. Until then, these functions are internal business logic and shouldn't be exposed to the API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't add them to the API, how would the wallet-rpc & wallet-cli call this function, if the goal is to get rid of all the wallet2 function calls? Sorry if I'm missing something.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I checked, and it's used indeed, so it's not a case of YAGNI like @jeffro256 suspected, and after I saw that @moneromooo-monero fixed an information leak a few years back when using remote daemons where somehow this method and the next for processing the pool state were involved, I think it's best to just add those to the interface and keep them "public".
* return: true if succeeded | ||
* note: sets status error on fail | ||
*/ | ||
virtual bool saveMultisigTx(const PendingTransaction &multisig_ptx, const std::string &filename) const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder whether it would make sense to group all the multisig related methods together in the source, and not e.g. put non-multisig and multisig versions of methods that do similar things together. Thus, "write normal tx to file" and "write multisig transaction to file" would not be next to each other, but the first in the group of "normal" methods and the second in the multisig method group.
My argument: Many wallet apps will probably continue to not offer multisig, and their devs may welcome that all those multisig methods do not "clutter" the API for them.
There also have been thoughts that ideally, "multisig wallet" would be a completely separate wallet type, with its own API, and maybe even with the code in a separate repository. That multisig stuff is so highly special, so different, and so complicated ...
I also ask myself wether we can't make some of such methods "smart" anyway so they decide themselves what there is to do, based on internal knowledge whether a transaction is multisig or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In regards to grouping multisig stuff together, that's already done in another PR (can be seen e.g. here or here). If I remember correctly for this PR I was asked to add everything new in one place, if the "organize functions" get merged this can get based onto that (actually there is still another one for in between in the pipeline I didn't PR yet, the "add comments" found here).
A separate multisig wallet really sounds ideal, not sure how many clients currently use the multisig functionality from the API, the GUI afaik doesn't. But that's out of scope for now I'd say!?
I'll look into if I can come up with such "smart" methods.
src/wallet/api/wallet2_api.h
Outdated
* brief: getAccountTags - get the list of registered account tags | ||
* return: first.Key=(tag's name), first.Value=(tag's label), second[i]=(i-th account's tag) | ||
*/ | ||
virtual const std::pair<std::map<std::string, std::string>, std::vector<std::string>>& getAccountTags() = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems to me, if we "go with the pattern", we have a method somewhere that gives back all existing accounts with info, using an account_info
struct. The tag would be part of that struct, and thus this call here not needed because not "essential".
Or maybe a call giving back all subaddresses, with info about them, and info how they group into accounts - and any tags of course.
Methods like the following setAccountTag
stay of course even if we use this pattern whenever we can, because changing things is something fundamentally different and altogether impossible if there is no method, after all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that tags would fit into SubaddressAccountRow
, but returning the "i-th account's tag" is just one part of this function, it also returns all tag descriptions (used by wallet-cli and wallet-rpc). So we probably need at least a getAccountTagDescriptions()
method anyways, or alternatively we could store the tag description in SubaddressAccountRow
, too.
src/wallet/api/wallet2_api.h
Outdated
* return: daemon adjusted time | ||
* note: sets status error on fail | ||
*/ | ||
virtual std::uint64_t getDaemonAdjustedTime() const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I could see in the code of the CLI wallet that uses this method it's only there to support that super-special case where you could lock a transaction not by giving a blockheight, but a time span, e.g. "10 years". Something nobody ever did in earnest. We anyway removed already all CLI commands to lock.
I am pretty sure we don't need this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should have a list somewhere of public wallet2
methods that do not get a counterpart in the Wallet API, with a short reason per method? Otherwise people in the future may wonder, or worse still, assume an oversight, and get tempted to add it ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A list like the following? Just needs a better place then in the comments of this PR. So far we have:
List of public wallet2
methods that do not get a counterpart in the wallet API
wallet2 method | source | reason |
---|---|---|
bool unset_ring(const std::vector<crypto::key_image> &key_images) |
source | |
bool unset_ring(const crypto::hash &txid) |
source | |
bool find_and_save_rings(bool force = true) |
source | |
bool is_output_blackballed(const std::pair<uint64_t, uint64_t> &output) const |
source | |
bool is_transfer_unlocked(uint64_t unlock_time, uint64_t block_height) |
source | |
uint64_t get_min_ring_size() |
source | Rings will become deprecated soon with FCMPs. |
uint64_t adjust_mixin(uint64_t mixin) |
source | Rings will become deprecated soon with FCMPs. |
bool is_deprecated() const |
source | This information is now part of WalletState . |
std::string get_daemon_address() const |
source | This information is now part of WalletState . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes!
src/wallet/api/wallet2_api.h
Outdated
/// view_tag | ||
crypto::view_tag view_tag; | ||
|
||
// TODO : figure out if we need other members from transfer_details too, these are the ones that are not part of `TransactionInfo`, but my first impression is we could have an overlap so we can find TransactionInfo from EnoteDetails and vice versa. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what you mean with "overlap". I think you have to be very careful to not become victim of the confusion in wallet2
regarding the term transfer. If only people would not use terminology confusingly and inconsistently, things could be much easier :)
If you look at the big picture, we have enotes and we have transactions, and these two are two clearly distinct and different things. They are as different as ships and containers. A ship transports containers, that's it already. And a transaction has some enotes as inputs and some other enotes as outputs. Really simple.
Nobody in their right mind would ever confuse ships and containers, that's why everytime I look at some definition in wallet2.h
, or check a command in the CLI wallet, and it's not immediately clear whether I am looking at ships (transactions) or at containers (enotes), I ask myself, am I really so dumb, or are these definitions and command names simply not as clear as they should, and could, be?
Anyway, enough ranting, back to the questions at hand: Because enotes and transactions are different things, there are different infos to be had about those two. Any "overlap" would immediately rise my suspicion of a confusion somewhere.
Of course you can cross-reference things, and we should pass on such cross-references whenever wallet2
provides them: A) at a transaction, enotes that were the inputs of it, B) at a transaction, enotes that where the outputs of it, C) at an enote, the transaction it arrived with, D) at an enote, if spent, the transaction that it was spent with.
From some earlier investigation I am moderately sure that info of type D) is not available because wallet2
does not store it anywhere. Maybe it would be a good exercise for you to get a clearer overview to confirm (or refute) that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rbrunner and I already had a chat about this, but for completeness I post my conclusion here.
With overlap I meant what we currently have with "tx_id". To stay with the analogy:
The "tx_id" is an unique identifier for each ship (e.g. crypto::hash m_tx_hash;
in payment_details
, for ships that deliver containers into our wallet) and we store this identifier alongside each container that we received (as crypto::hash m_txid;
for transfer_details
). So it's possible to receive multiple containers from the same ship and you can later check which ship delivered which containers and the other way around, looking at the containers, which container got delivered by which ship.
AFAIUI this is the current state of A)-C) (I'm still struggling with cryptonote::transaction_prefix
and all the things that come along, so this part may be the most wrong.)
Scenario | Type | Struct | Cross-Reference |
---|---|---|---|
A) | outgoing transaction | confirmed_transfer_details & unconfirmed_transfer_details |
key_offsets and k_image in std::vector<txin_v> vin; in cryptonote::transaction_prefix m_tx; if the variant txin_v is a txin_to_key |
B) | incoming transaction | payment_details |
m_tx_hash (loop over wallet2::m_transfers and compare txids) |
C) | - | transfer_details |
m_txid |
I came to the following result for scenario D):
TLDR: It's not easily possible.
Long story: It seems the necessary information is very close to be available in three out of four cases:
m_spent
only gets set here
set_spent()
only gets called from 4 different places (here 1, here2, here 3, here 4) which all have access to the tx_id (except #2 wallet2::rescan_spent()
), so it's not as easy as just adding tx_id_out
to the function parameter of set_spent()
and m_tx_id_out
to transfer_details
.
Slightly offtopic, but I assume the k_image
in txin_to_key
is the reason for this 10 year old TODO?
Also AFAIUI in confirmed_transfer_details
& unconfirmed_transfer_details
we also store the key image twice, in m_tx.vin[i]
and m_rings[i].first
.
And in confirmed_transfer_details
we store the unlock time in m_tx.unlock_time
and m_unlock_time
.
After further investigation into cryptonote::transaction_prefix
it looks like tx*_to_script
and tx*_to_scripthash
are not actually used anywhere.
In conclusion so far I tend towards structuring it the following way:
transaction_prefix members |
TransactionInfo replacements |
---|---|
size_t version; |
std::size_t m_tx_version (new) |
uint64_t unlock_time; |
uint64_t m_unlock_time; |
std::vector<txin_v> vin; |
see table below for "txin_v variant" |
std::vector<tx_out> vout; |
see table below for "txout_target_v variant" |
vout[i].amount |
std::vector<Transfer> m_transfers; m_transfers[i].amount |
std::vector<uint8_t> extra; |
QUESTION : Do we really need tx_extra in TransactionInfo? If yes, I'd add: std::string m_tx_extra; (new) |
txin_v variant |
variant members | TransactionInfo replacements |
EnoteDetails replacements |
---|---|---|---|
txin_gen (coinbase) |
size_t height; |
uint64_t m_blockheight |
- |
txin_to_key |
uint64_t amount; |
uint64_t m_amount *¹ |
std::uint64_t m_amount |
- | std::vector<uint64_t> key_offsets; |
- | QUESTION : Couldn't find a good reason why we would need this in EnoteDetails , any other opinion? |
- | crypto::key_image k_image; |
- | std::string m_key_image |
*¹ Actually this is the sum of all ammounts in the vector vin
.
txout_target_v variant |
variant members | TransactionInfo replacements |
EnoteDetails replacements |
---|---|---|---|
txout_to_key (before HF_VERSION_VIEW_TAGS ) |
crypto::public_key key; |
- | std::string m_onetime_address |
txout_to_tagged_key (after HF_VERSION_VIEW_TAGS ) |
crypto::public_key key; |
- | std::string m_onetime_address |
- | crypto::view_tag view_tag; |
- | std::string m_view_tag |
Is it correct that key
in txout_to_key
and txout_to_tagged_key
is an output onetime address?
src/wallet/api/wallet2_api.h
Outdated
*/ | ||
struct EnoteDetails | ||
{ | ||
// QUESTION : If this struct is done, should I create the files "src/wallet/api/enote_details.[h/cpp]", put all the member variables inside the header and only set virtual getter functions in here, like it's done for other structs below? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Difficult question. I would tend to keep the "style" of using getters, to have things consistent, even if we probably agree that the whole programming pattern is overkill.
Maybe something to bring up in the next meeting?
src/wallet/api/wallet2_api.h
Outdated
* note: sets status error on fail | ||
*/ | ||
virtual void rewriteWalletFile(const std::string &wallet_name, const std::string &password) = 0; | ||
// QUESTION : Should we change this function from the current behavior in wallet2, so `wallet_name` is just the name of the new wallet instead of changing the `m_wallet_file` for the current wallet? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand the question. Can you please elaborate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure.
The current behavior of this function in wallet2 (src) aiui:
We provide a parameter wallet_name
and an outparam new_keys_file_name
.
The first call is to prepare_file_names(wallet_name);
, which then calls do_prepare_file_names(wallet_name, m_keys_file, m_wallet_file, m_mms_file)
. Inside do_prepare_file_names()
lays the problem, it changes the m_wallet_file
of the current wallet to the new wallet_name
.
Before going into more detail (I can give more if desired):
As I typed this I just took another look into where this write_watch_only_wallet()
is actually used and saw it's only one place in simplewallet, where it uses m_wallet_file
as the wallet_name
. So even though the current behavior aiui is a mess, it's probably unimportant, and can be circumvented by removing the wallet_name
as parameter from the API function and make it always use the current m_wallet_file
internally. Ideally new_keys_file_name
should not be an outparam, if you want to write a new watch only wallet with a different name than m_wallet_file + "-watchonly.keys"
.
Let me know if things are still unclear.
II think this "interface draft" will soon reach a state where we should consider to try to get feedback about the draft in the form it reached from as many wallet app writers and maintainers as we can reach. Not sure how to best do that, however. |
Do not merge, this is still in development and just PRd for easier discussion.
This is an attempt to make the wallet API "feature complete" and it's part of this proposal.
The approach was to search for every
wallet2
function insimplewallet
andwallet_rpc_server
, if it is used in either of those, but is not implemented in the API I considered it missing and it was added in this PR.I believe
wallet2_api.h
now contains all missing functions, but some are not implemented yet. I'm waiting on feedback on those, if you want to have a look, things I need help with are marked asQUESTION
, you can also comment on things markedTODO
(those may become QUESTIONs if I'm unable(/unsure how) to solve them) or anything else.Also gathered this collection of (mostly minor) issues found during API work.