Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minio integration: write intermediate data to minio, instead of storing #79

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from
Draft

minio integration: write intermediate data to minio, instead of storing #79

wants to merge 36 commits into from

Conversation

saurav-malani
Copy link

No description provided.

supradeep2819 and others added 30 commits April 17, 2024 11:36
@supradeep2819
changes for activation to support etl
894a6bb
@rajvarkala
Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev
d845bdc
@supradeep2819
Merge pull request #70 from valmi-io/main
e06dc9e 
chore: Back merge main to dev
@gane5hvarma
fix: add missing import to connector lib
5d31e97
@rajvarkala
Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev
ba79b23
@gane5hvarma
feat: add normalization and dbt step into dag (#71)
80cd1fc 
* feat: add normalization and dbt step into dag
@supradeep2819
feature: Added endpoint to retrive last run status
a4999b8
@supradeep2819
feature: Added endpoint for getting last run status
ececb3a
@supradeep2819
Merge pull request #72 from valmi-io/latest_run_status
f82d3b9 
feature: Added endpoint to retrive last run status
@supradeep2819
feat: Handled missed schema for last run status
e06bbc1
@supradeep2819
fix: retriving last run ID in descending order
d758da6
@supradeep2819
fix: retriving last run ID in descending order
a1ce573
@supradeep2819
Merge pull request #73 from valmi-io/latest_run_status
682460e 
fix: retriving last run ID in descending order
@supradeep2819
"feat: Enhance source postgres configuration to provide specific tabl…
1b56fcb 
…e schema
@supradeep2819
Merge pull request #74 from valmi-io/latest_run_status
a909e5c 
"feat: Enhance source postgres configuration to provide specific tabl…
@rajvarkala
Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev
fb0d957
@gane5hvarma
feat: changes needed for destination postgres to work
f69dcd8
@rajvarkala
feat: Adding checkpointing to etl sources and destinations
374bf85
@rajvarkala
version up
b22b4a2
@rajvarkala
feat: for etl, batching source records only on seeing state messages
d307b97
@supradeep2819
feat: generating catalog when query is present postgres config
e61e1a4
@rajvarkala
feat: multiple state messages stored for etl sources
79654bb
@rajvarkala
Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev
dd210cf
@gane5hvarma
Fix finalizer step which got stuck because of transformation (#75)
172195f
@gane5hvarma
feat: rm print statements and update version
bad286b
@supradeep2819
feat: Add endpoint to fetch latest successful sync
dea7d8d
@supradeep2819
feat: renamed timestamp with run_end_at in latest_run_status endpoint
5ff5f39
@supradeep2819
feat: Used schemas instead of dict in sync service
3197240
@supradeep2819
Merge pull request #76 from valmi-io/feat.last_successful_sync
5db346c 
feat: Add endpoint to fetch latest successful sync
@rajvarkala
bug: metrics fix for etl
c3d109e
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Base automatically changed from dev to main June 19, 2024 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@saurav-malani @CLAassistant @supradeep2819 @rajvarkala @gane5hvarma