Feature/wormholescan db migration #1791
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ftocal
force-pushed
the
feature/wormholescan_db_migration
branch
2 times, most recently
from
3aef1fc
to
8d27653
Compare
* fly - add event dispatcher for vaa-pipeline * Add to repository on conflict behaviour to queries * Add observations and vaas metrics to repository * add support to vaa pythnet table * Add new postgres repository for guardian set in common * add common method to exec and scan * add query log * fly - add event dispatacher for governor config message * update migration.sql file * fix inserted flag issue, add metrics and event dispatcher for vaas and governor config * Add comment to pgx cmdTag bug and workaround for this cases * update pipeline vaa event & migration * Add in fly-event-processor support to governor config chain * Add in fly-event-processor governor_status postgres repository * Fix table wh_governor_config_chains * fix governor config chain flow * fly-event-processor - fix governor config flow * Add health for postgres in common * add dummy governor status processor and refactor run and build * Add comment in fly-event-processor * Add support to duplicate vaa flow in the migration * refactor fly * refactor main.go * wip: guardian set repository * fix healthcheck in fly mongo/postgres * fly - remove/add comments to postgres respository * fly - rename env var VAA_PIPELINE_SNS_URL * common - add comment and remove unused code * fly-event-processor - fix comments * fly - renames and add comments * fix postgres guardian set * fix vaa processing --------- Co-authored-by: Fernando Torres <[email protected]>
Co-authored-by: walker-16 <[email protected]>
* Add new database abstraction in common * fly - migrate observations to postgres * fly - migrate vaas and governor status to postgres * add new postreSQL repository to also write to the new db * add postresql_repository.go * undo unwanted renames in pipeline component * undo unwanted renames in pipeline component * add upsetOriginTx * add upsert-target-tx * change constructor and add missing parameter * upsert-origin-tx tested locally * add to fly/migration.sql the creation of wh_operation-_transactions * working insert in postgres for messages coming from pipeline-sqs and from notifications-sqs * add missing config * more changes * more changes * add missing id * add run configuration * add fallback and consider wormchain * small change * move vaaRepository to a separate package in order to fix the import cycle * working for wormchain * remove comment * now working both wormchain and with fallback * remove some comments * fix digest in mongodb * revert wrong resolve conflict from merge * adjust logic for checking if it has to update or not * rename repository to clarify * fix indent * code-review fixes * change to runMode * multiple changes * add unit-test to AlreadyProcessed for sourceTx * more fixes --------- Co-authored-by: Agustin Pazos <[email protected]>
…ory (#1595) Co-authored-by: walker-16 <[email protected]>
add postgres support to parser component
* Add new database abstraction in common * fly - migrate observations to postgres * fly - migrate vaas and governor status to postgres * fly - add event dispatcher for vaa-pipeline * Add to repository on conflict behaviour to queries * Add observations and vaas metrics to repository * add support to vaa pythnet table * Add new postgres repository for guardian set in common * add common method to exec and scan * add query log * fly - add event dispatacher for governor config message * update migration.sql file * fix inserted flag issue, add metrics and event dispatcher for vaas and governor config * Add comment to pgx cmdTag bug and workaround for this cases * update pipeline vaa event & migration * Add in fly-event-processor support to governor config chain * Add in fly-event-processor governor_status postgres repository * Fix table wh_governor_config_chains * fix governor config chain flow * fly-event-processor - fix governor config flow * Add health for postgres in common * add dummy governor status processor and refactor run and build * Add comment in fly-event-processor * migration of pipeline component * finish full test * fix broken merge * add missing implementation * small changes * change condition * add unit-test to pipeline * add missing Close() * fix unit-test --------- Co-authored-by: Agustin Pazos <[email protected]>
* modify postgres db schemma to wormholescan * add db_layer env var and modify parser * add env db and db log enabled to parser deploy
* add db_layer env support in tx-tracker deploy * rename postgresdb schemma to wormholescan * add db_log_enabled and db to tx-tracker deploy
* add deploy to pipeline-postgres component * fix pipeline-postgres deploy config * modify db schemma in pipeline component * fix healthcheck and change env to DB_LAYER
* fly-event-processor improve SELECT queries * fly-event-processor add ctx to postgres repository rallback method * handle on conflict insert governor status
Fix parser and tx-tracker deployment Co-authored-by: walker-16 <[email protected]>
* move logger error to debug in tx-tracker * add vaaID in event that is send to the pipeline from the fly Co-authored-by: ftocal <[email protected]> * fix communication between fly - pipeline and pipeline components Co-authored-by: ftocal <[email protected]> * fix and refactor tx-tracker Co-authored-by: ftocal <[email protected]> * Fix Makefile build for ci/cd Co-authored-by: walker-16 <[email protected]> * Fixes in the communication between fly and pipeline Fixes and improvements in message processing in tx-tracker Co-authored-by: walker-16 <[email protected]> * Add db layer configurable by environment --------- Co-authored-by: ftocal <[email protected]>
Co-authored-by: walker-16 <[email protected]>
Co-authored-by: walker-16 <[email protected]>
Co-authored-by: walker-16 <[email protected]>
* common - add consistency level to repository * fly-event-processor - add consistenct level to repository * fly - add consistency level to repository * parser - add consistency level to repository * fly - change serviceName metrics from postgres db layer mode * fly-event-processor modify duplicated vaa metrics to suppport db layer * fly-event-processor modify governor status metrics to support db layer * parser - add support to db layer in postgres metrics * analytics - add metrics to truck insert in transferPrices * improve analytics, fly-event-processor, parser metrics
* Change vaa_id to message_id column name in wh_attestation_vaa_properties, wh_operation_transactions and wh_operation_prices tables Modify primary key in wh_operation_transactions to support compose protocols Remove 0x prefix from and to address in wh_operation_transactions table Add fee detail as json column in wh_operation_transactions table Fix rpc_response column in wh_operation_transactions table * Fix attestation_vaas_id for target transaction * Normalization of from and to in target transactions for wh_operation_transactions table * add metrics to verify insert in mongo and postgres repository --------- Co-authored-by: Agustin Pazos <[email protected]>
Co-authored-by: walker-16 <[email protected]>
Co-authored-by: ftocal <[email protected]>
Co-authored-by: walker-16 <[email protected]>
* Change to dual db type Change total_token data type in postgres table Co-authored-by: walker-16 <[email protected]> * Change wh_operation_transactions_processed table definition Co-authored-by: walker-16 <[email protected]> --------- Co-authored-by: walker-16 <[email protected]>
* Migrate vaas endpoints Migrate observation endpoints Migrate relays endpoints Co-authored-by: walker-16 <[email protected]> * fix timestamp in vaa endpoint response (#1739) * Migrate governor limit endpoint (#1743) * Migrate governor status endpoints (#1744) * Feature/1660 api migration governor config (#1745) * fly - add to wh_governor_config json field chains * wip --------- Co-authored-by: walker-16 <[email protected]>
* Migrate ready and health endpoints (#1753) * add address endpoint in postgres (#1754) * Migrate scorecards endpoint (#1756) * Normalize aptos address (#1757) * Add payload_type column in wh_attestation_vaa_properties table (#1758) * normalize from and to address for targer txs (#1761) * Feature/migration global tx postgres (#1760) * add support to globalTx endpoint with postgres * add denormalized txHash and address methods by chain * vaa endpoint migration add pyth and fix (#1764) --------- Co-authored-by: walker-16 <[email protected]>
* Migration of endpoint /governor/notional/limit to postgres (#1769) migration /governor/notional/limit to postgres * migrate governor notional limit by chainID * adapt governor-notional-limit-by-chain-id mongo entity * Feature/ws 2 add support wh operation table (#1768) * modify db migration add wh_operation_addresses table * insert destination address in wh_operation_addresses table * modify tx-tracker to support insert source operation address * migrate /governor/notional/available/:chainId to postresql (#1775) * migrate /governor/notional/available/:chainId to postresql * add order by * fix identation * migrate governor-notional-available (#1778) migratae governor-notional-available * Migrate operations and transactions endpoints (#1785) * Migrate operations by id endpoint Co-authored-by: walker-16 <[email protected]> * migrate find transaction by id * Migration of operation search endpoint by txHash, address, source chains, destination chains and app ids * Improve postgres queries for operations endpoint Co-authored-by: walker-16 <[email protected]> * fix transaction by id migration (#1782) * Changes indexes to improve performace Separate find operations in two sql queries Co-authored-by: walker-16 <[email protected]> --------- Co-authored-by: walker-16 <[email protected]> * Migrate governor notional max available (#1780) * migrate governor-notional-max_available * remove unused types * readjust types * adjust types contemplating mongodb * migrate /governor/enqueued-vaas (#1787) * migrate /governor/enqueued-vaas * fix identation * migrate /governor/enqueues_vaas/:chainID * fix identation * migrate governor-vaa endpoint to postgres (#1788) * Migration of transactions search endpoint (#1790) --------- Co-authored-by: Mariano <[email protected]> Co-authored-by: Mariano <[email protected]> Co-authored-by: walker-16 <[email protected]>
Co-authored-by: ftocal <[email protected]>
* Remove unused config map key Co-authored-by: ftocal <[email protected]> * Add postgres config for api deployment Co-authored-by: walker-16 <[email protected]> --------- Co-authored-by: walker-16 <[email protected]>
Co-authored-by: ftocal <[email protected]>
* Fix sql query for operations endpoint * Fix vaa_id column name to message_id in pipeline * Fix id on conflict for wh_operation_transactions insertion * fix wh_governor_config update column chains * normalize tx_hash when fetch tx details fail (#1807) * Add source_event and track_id_event columns to source event tracking (#1808) Add source_event and track_id_event columns to source event tracking in analytics, parser, pipeline and tx-tracker processing Improve column and table names --------- Co-authored-by: Agustin Pazos <[email protected]>
Co-authored-by: walker-16 <[email protected]>
* migrate /v1/governor/available_notional_by_chain * governor * adapt grpc controller * fix db annotations * fix identation * migrate /governor/enqueued_vaas * migrate governor/is_vaa_enqueued * migrate governor/token_list * migrate /guardianset/current * migrate /v1/heartbeats * adjust endpoint /v1/signed_vaa/:chain_id/:emitter/:seq * small change
* postgres migration - fix governor available notional endpoint * fix governor notional by chain endpoint * fix max notional available by chain endpoint migration * fix enqueued vaa endpoint list and filter by chain * fix max notional endpoint
#1830) migration - fill fromChainID with vaa emitterChain if the vaa can not be parsed.
* fix guardian-api get last heartbeats - postgres migration * postgres migration - fix is vaa enqueued and get enqueued vaa endpoint * turn off cache to check postgres queries * turning on cache * fix guardian-api endpoint fix available notional by chain * fix get token list endpoint - migration * fix missing fields in heartbeat endpoint migration
* Add missing error not found check in governor endpoints * fix governor_status enqueued vaa response format
ftocal
force-pushed
the
feature/wormholescan_db_migration
branch
from
e165eea
to
fefbbf4
Compare
Fix from address in pipeline for postgres Co-authored-by: walker-16 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.