Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync Concurrency configuration #49925

Open
alonso-cobre opened this issue Dec 18, 2024 · 0 comments
Open

Sync Concurrency configuration #49925

alonso-cobre opened this issue Dec 18, 2024 · 0 comments
Labels
area/platform issues related to the platform community team/platform-move type/bug Something isn't working

Comments

@alonso-cobre
Copy link

Helm Chart Version

1.1.0

What step the error happened?

Other

Relevant information

Im facing some issues for having multiple sync jobs, Im expecting to have 500 sync/jobs to be running every 5 minutes, right now I have 100 ish sync/jobs running fine, but once I started adding more connections (Every connection is scheduled as a cron to run every 5 minutes), jobs runs for longer (normally they take about 1-2 minute, I have Incremental Append from AuroraMySQL to S3), sometimes they remain in a running status for hours without finishing properly, I need to cancel the sync and the next run normally finishes normally, any configuration parameters that I could look into? I am already using MAX_*_WORKERS and WORKLOAD_LAUNCHER_PARALLELISM with values ranging from 100-200, I have 3 workers (3 Kubernetes pods), I also read that TEMPORAL_WORKER_PORTS somewhat affect concurrency, I want to know how you can configure Airbyte and its components to support this type of concurrency? Attaching some relevant logs of failed syncs.

ce0d8ff3-04d8-44c1-a2ed-6d6e86565e20_1747738_0_sync.txt

Thanks.

Relevant log output

2024-12-18 01:32:30 platform > Using job log path: /workspace/1705756/0/logs.log
2024-12-18 01:32:30 platform > Executing worker wrapper. Airbyte version: 1.1.0
2024-12-18 01:32:30 platform > Creating workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync
2024-12-18 01:32:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:34:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:36:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:38:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:40:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:42:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:44:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
2024-12-18 01:46:31 platform > Workload 759ffc02-2e73-4088-bab7-5ffa4ba9da2d_1705756_0_sync is pending
@alonso-cobre alonso-cobre added area/platform issues related to the platform needs-triage type/bug Something isn't working labels Dec 18, 2024
@marcosmarxm marcosmarxm changed the title [source-mysql] Concurrency configuration Sync Concurrency configuration Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/platform issues related to the platform community team/platform-move type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@marcosmarxm @octavia-squidington-iii @alonso-cobre