Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: manual compaction #3988

Merged
merged 15 commits into from
May 22, 2024
Merged

Conversation

v0y4g3r
Copy link
Contributor

@v0y4g3r v0y4g3r commented May 20, 2024

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This PR adds support for manually trigger different types of compactions.

The syntax is:

SELECT COMPACT_TABLE("<table_name>", ["<compact_type>"], ["<options>"])

Currently, compact_type supports regular and strict_window.

  • regular: regular compaction like those triggered by flush.
  • strict_window: split SST files strictly by time window.

The <options> argument is type-sepecific compaction options. For strict_window, the option value is the compaction window in seconds.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.

@github-actions github-actions bot added the docs-not-required This change does not impact docs. label May 20, 2024
@v0y4g3r v0y4g3r changed the title Feat/manual compaction feat/manual compaction May 20, 2024
@sunng87
Copy link
Member

sunng87 commented May 20, 2024

We need to add restrictions to this types functions to avoid cross-catalog invocation.

@v0y4g3r v0y4g3r requested a review from sunng87 May 21, 2024 09:14
@github-actions github-actions bot added docs-required This change requires docs update. and removed docs-not-required This change does not impact docs. labels May 21, 2024
@v0y4g3r v0y4g3r changed the title feat/manual compaction feat: manual compaction May 21, 2024
@v0y4g3r v0y4g3r marked this pull request as ready for review May 21, 2024 09:17
@v0y4g3r v0y4g3r requested review from evenyag, waynexia and a team as code owners May 21, 2024 09:17
Copy link

codecov bot commented May 21, 2024

Codecov Report

Attention: Patch coverage is 82.92952% with 155 lines in your changes are missing coverage. Please review.

Project coverage is 85.15%. Comparing base (9800807) to head (40ef3b5).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3988      +/-   ##
==========================================
- Coverage   85.48%   85.15%   -0.33%     
==========================================
  Files         977      980       +3     
  Lines      169462   170113     +651     
==========================================
  Hits       144859   144859              
- Misses      24603    25254     +651     

src/mito2/src/compaction.rs Outdated Show resolved Hide resolved
src/mito2/src/compaction.rs Show resolved Hide resolved
src/session/src/table_name.rs Show resolved Hide resolved
src/mito2/src/compaction/window.rs Show resolved Hide resolved
src/mito2/src/compaction/window.rs Outdated Show resolved Hide resolved
Copy link
Member

@sunng87 sunng87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from the permission perspective

@v0y4g3r v0y4g3r force-pushed the feat/manual-compaction branch 2 times, most recently from 1f60210 to 01830b5 Compare May 22, 2024 06:27
@v0y4g3r v0y4g3r added this pull request to the merge queue May 22, 2024
@v0y4g3r v0y4g3r removed this pull request from the merge queue due to a manual request May 22, 2024
@v0y4g3r v0y4g3r enabled auto-merge May 22, 2024 09:27
@v0y4g3r v0y4g3r added this pull request to the merge queue May 22, 2024
Merged via the queue into GreptimeTeam:main with commit 090b59e May 22, 2024
21 of 28 checks passed
@v0y4g3r v0y4g3r deleted the feat/manual-compaction branch May 22, 2024 09:59
@sunng87
Copy link
Member

sunng87 commented May 24, 2024

By the way, I remember we have an http api the flush tables, is it removed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required This change requires docs update.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Manually trigger major compaction
3 participants