Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] More observability for stream sources #91

Open
s-vitaliy opened this issue Aug 7, 2024 · 0 comments
Open

[FEATURE] More observability for stream sources #91

s-vitaliy opened this issue Aug 7, 2024 · 0 comments
Assignees
Labels
code/new-feature New feature or request

Comments

@s-vitaliy
Copy link
Contributor

s-vitaliy commented Aug 7, 2024

Description

Recently, we encountered a problem where the SQL streaming source was executing schema update queries on the target SQL table at a frequency equivalent to the change capture interval. Ideally, the stream should only initiate these updates when a schema change is detected. It's important to note, however, that we currently have no evidence linking this issue directly to the change capture interval. Therefore, to gain a better understanding of the issue, we need to improve the observability of our streaming sources.

Possible solution

Add additional metrics describing the following events:

  • A schema change detection event happened.
  • A change capture event happened in the stream

Alternatives

Add logging for those events.

Context

Required for all stream sources in the framework that supports schema change detection (SQL and CDM for now).

@s-vitaliy s-vitaliy added the code/new-feature New feature or request label Aug 7, 2024
@s-vitaliy s-vitaliy self-assigned this Aug 7, 2024
@s-vitaliy s-vitaliy added this to Arcane Aug 7, 2024
@s-vitaliy s-vitaliy moved this to Backlog in Arcane Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code/new-feature New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

1 participant