Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider placing notifications in separate table #1023

Open
keith-turner opened this issue Mar 9, 2018 · 0 comments
Open

Consider placing notifications in separate table #1023

keith-turner opened this issue Mar 9, 2018 · 0 comments

Comments

@keith-turner
Copy link
Contributor

Fluo's notifications are currently placed in a separate locality group. This means when scanning for notifications, user data is not read. Notifications can only be garbage collected with a full major compaction. However full major compactions do not happen often. This means that over time notification delete markers build up. This build up over time makes scanning for notifications more expensive.

Work was done for #457 to drop delete marker for notifications that were inserted and processed before an Accumulo minor compaction happens. The work in #457 also has a chance of dropping delete markers on partial compactions. However if a delete marker makes it to the largest files, it may be there for a long time. The effectiveness of #457 is inversely proportional to the average time it takes to process a notification. As the average time goes up, the effectiveness goes down. If too many delete markers have built up, then the best work around is a full compaction of the table.

If notifications were in a separate table, then the full compaction could be cheap. Also the work done for ACCUMULO-4500 could be leveraged to automatically make smart decisions about when to compact based on the number of notifications vs the number of delete markers.

@keith-turner keith-turner added this to the 1.3.0 milestone Mar 9, 2018
@keith-turner keith-turner removed this from the 1.3.0 milestone Aug 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant