Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Better index cleanup #3809

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

thommyhh
Copy link

@thommyhh thommyhh commented Sep 28, 2023

  • removes cleanup logic from ReIndex task to avoid having an empty index/no search results directly afterwards
  • adds new CleanupIndex task which deletes all documents older than the given time in days

Fixes: #3808

* remove cleanup from `ReIndex` task to avoid having an empty index/no search results directly afterwards
* add new `CleanupIndex` task which deletes all documents older than the given time in days
@dkd-kaehm
Copy link
Collaborator

@thommyhh Thanks for this feature.
That is good idea to have the "older than" feature, but it should be optional in the ReIndex task.
The behavior of ReIndex task must not be changed without the option, because it is a huge breaking change, which must be announced in long term.
The Documents removing by type must stay, because the index can contain documents, which must not be cleaned,

  • because they are not indexed by EXT:solr, so not in index queue
  • or are other indexing types of EXT:solr* addons.

New Scheduler-tasks should be implemented as schedulable commands.

@thommyhh
Copy link
Author

@dkd-kaehm I get your point about the breaking change. Could we at least make the cleanup optional with default enabled in the ReIndex task? I think it is a really bad idea to remove all documents - even by type - when reinitializing the index queue, because it would completely remove any search results for the time being.

I'm fine with the command approach. However, I've no idea how to write tests for that.

Would you be fine with the optional cleanup switch for the ReIndex task? I would than split this into to PRs, one with the optional cleanup and another one with the older than cleanup as a command.

@dkd-kaehm
Copy link
Collaborator

Would you be fine with the optional cleanup switch for the ReIndex task?

Yes, please do this in this PR and other features we'll do in new issues.

Thanks for your understanding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Better index cleanup
2 participants