New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle high memory usage in CEMS parquet-to-duckdb conversion #3812

Open

1 task

Tracked by

jdangerx opened this issue Aug 26, 2024 · 0 comments

Open

1 task

Tracked by

Handle high memory usage in CEMS parquet-to-duckdb conversion #3812

jdangerx opened this issue Aug 26, 2024 · 0 comments

Member

jdangerx commented Aug 26, 2024

Overview

Details here: #3741 (comment)

But basically, the CEMS parquet file is pretty big and that's causing a memory issue when trying to convert to DuckDB.

Success Criteria

Hourly CEMS makes it into DuckDB

Next steps

Beta Give feedback

figure out if we can convert the parquet file by chunks
Options

jdangerx mentioned this issue

Load parquet files to a duckdb file #3739

Open

2 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment