Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove max_number_of_messages for SQS+S3-based inputs #12101

Open
strawgate opened this issue Dec 13, 2024 · 0 comments
Open

Remove max_number_of_messages for SQS+S3-based inputs #12101

strawgate opened this issue Dec 13, 2024 · 0 comments

Comments

@strawgate
Copy link
Contributor

With elastic/beats#40699 the max_number_of_messages setting is now ignored for SQS inputs as of 8.15. It is still exposed in integration settings. It is still relevant for <8.15.

The new setting is number_of_workers, which the SQS-based input shares with the S3 bucket polling input in integration settings.

Unfortunately, the default value for number_of_workers is just 1. Users using an SQS-based S3 input upgrading to 8.15 agents will find that they've gone from having five workers (or whatever number they had customized it to) for processing objects to just a single worker processing objects.

We should consider one or more of the following:

  1. Using the max_number_of_messages setting for the number_of_workers value if queue_url is present to maintain backwards compatibility while indicating that the setting will be removed in a future version.
  2. Indicating that max_number_of_messages is no longer applicable for 8.15 and to use number_of_workers when using an SQS queue
  3. Bumping the number_of_workers default value to 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants