Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate from api-generated-TSV ingestion to chainhook-generated-Parquet #550

Open
lgalabru opened this issue Apr 2, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request reliability

Comments

@lgalabru
Copy link
Contributor

lgalabru commented Apr 2, 2024

Implement a new command:

chainhook archiver start --config-path

That would spin up a simple http server (no re-org handling), ignoring bitcoin blocks (I think?), that would just write stacks blocks to parquet - the constraint being minimizing as many error code path as possible.

Replace TSV ingestion with parquet ingestion.

Intention: Decoupling chainhook & the API.
Today, Chainhook downloads TSV file that stacks-blockchain-api is producing.
Ideally: Chainhook ingests from scratch from a stacks-node, builds Parquet file.
For future runs, chainhook would download the Parquet file that chainhook built.

Extra benefit: We would have two independent components indexing the stacks blockchain.

@MicaiahReid MicaiahReid added this to the Production Reliability milestone Apr 8, 2024
@lgalabru lgalabru assigned MicaiahReid and unassigned lgalabru Jun 12, 2024
@MicaiahReid MicaiahReid removed their assignment Jun 18, 2024
@MicaiahReid MicaiahReid added the enhancement New feature or request label Jun 18, 2024
@smcclellan smcclellan removed this from the Production Reliability milestone Jun 28, 2024
@tippenein tippenein self-assigned this Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request reliability
Projects
Status: 📋 Backlog
Development

No branches or pull requests

4 participants