es7-persistance creates unlimited amount of indices

[astelmashenko]

Describe the bug
After certain amount of time shards problem raised. We checked ES and found out around 2300 shards, names like conductor_task_log_20221004.

Investigating code I see

// class ElasticSearchRestDAOV7
    private void createIndexesTemplates() {
        try {
            initIndexesTemplates();
            updateIndexesNames();
            Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);
        } catch (Exception e) {
            logger.error("Error creating index templates!", e);
        }
    }
//...
    private void updateIndexesNames() {
        logIndexName = updateIndexName(LOG_DOC_TYPE);
        eventIndexName = updateIndexName(EVENT_DOC_TYPE);
        messageIndexName = updateIndexName(MSG_DOC_TYPE);
    }

    private String updateIndexName(String type) {
        String indexName =
                this.indexPrefix + "_" + type + "_" + SIMPLE_DATE_FORMAT.format(new Date());
        try {
            addIndex(indexName);
            return indexName;
        } catch (IOException e) {
            logger.error("Failed to update log index name: {}", indexName, e);
            throw new NonTransientException(e.getMessage(), e);
        }
    }

Where updateIndexesNames creates new index every week.
Can someone explain why does it change names for indices?

*_conductor_task_log*
*_conductor_message*
*_conductor_event*

I'd like to change this behavior, because we reaching limits on shards.

I also checked below methods are not used anywhere:

IndexDAO.getEventExecutions
IndexDAO.getMessages

so it is probably safe to stop indexing them:

conductor.app.eventMessageIndexingEnabled=false
conductor.app.eventExecutionIndexingEnabled=false

However to stop creating new indices we need to change this config:

conductor.elasticsearch.autoIndexManagementEnabled=false

And provision indices for conductor manually.

Details
Conductor version: 3.18.0+
Persistence implementation: Postgres
Queue implementation: Postgres
Lock: Redis

To Reproduce
Steps to reproduce the behavior:
Just use elastic for a while

Expected behavior
Data is cleaned up periodically.

[hallo1144]

I'm also studying house keeping mechanism now
I think it's a good idea to have elasticsearch ILM set on index patterns, though it's not implemented yet in conductor so you have to manually configure it on elasticsearch