[Feature Request]: auto_logrotate: auto split and archive mo-service log

[aronchanisme]

Is there an existing issue for the enhancement?

• I have checked the existing issues.

Enhancement Description

Currently when MO_DEPLOY_MODE set to git or binary, mo-service is started with cmd like this,

cd /data/mo/matrixone/ && GOMEMLIMIT=19090MiB /data/mo/matrixone/mo-service -daemon -debug-http :12345  -launch /data/cus_reg/mo//20240929_110421/matrixone/etc/launch/launch.toml >/data/mo//logs/stdout-20240929_110548.log 2>/data/mo/log_ym/logs/stderr-20240929_110548.log

In a long running system, like PRD / QA / etc, the 2 log files will grow larger and larger as time passes, hence we need a solution allowing user to set auto split and archive mo-service log.

Enhancement Implementation

1. Add more confs

# enum: yes(default)|no , where,
# 'yes': meaning automatically creating timestamp as log file name postfix, e.g. stdout-20240929_110548.log. Everytime when mo-service is started or restarted, 2 new log files will be created and log content will be written.  
# 'no': meaning no postfix will be added to the log file name, e.g. stdout.log. Everytime when mo-service is started or restarted, the log content will be appended to the log files used last time and before.
MO_LOG_POSTFIX="no"  

# enum: daily(default)|size, where,
# 'daily': only takes effect when MO_LOG_POSTFIX is set to no, meaning mo-service log(including stdout.log and stderr.log), will be auto splited and archived on a daily basis
# 'size': only takes effect when MO_LOG_POSTFIX is set to no, meaning mo-service log(including stdout.log and stderr.log), will be auto splited and archived when their sizes exeeced the setting of MO_LOG_MAX_SIZE
MO_LOG_AUTO_SPLIT="daily"

# only takes effect when MO_LOG_AUTO_SPLIT is set to size, where:
# ${MAX_SIZE}: Maximum log size before it will be splitted and archived, e.g. 1024M
# unit: bytes(default), KB (sizek), or MB (sizem)
MO_LOG_MAX_SIZE="1024M"

# only takes effect when MO_LOG_AUTO_SPLIT is set to 'size' or 'daily', where:
# ${NUM}: number of archived log files , e.g. 100
MO_LOG_RESERVE_NUM="100"

2. Examples of usage

Step_1. Set confs
daily mode

mo_ctl set_conf MO_LOG_POSTFIX=no
mo_ctl set_conf MO_LOG_AUTO_SPLIT=daily
mo_ctl set_conf MO_LOG_RESERVE_NUM=100

size mode

mo_ctl set_conf MO_LOG_POSTFIX=no
mo_ctl set_conf MO_LOG_AUTO_SPLIT=size
mo_ctl set_conf MO_LOG_RESERVE_NUM=100

Step_2. Enable auto_logrotate

mo_ctl auto_logrotate enable

Step_3. Check auto_logrotate status

mo_ctl auto_logrotate status
mo_ctl auto_logrotate

Step_4. Disable auto_logrotate

mo_ctl auto_logrotate disable

3. Demo

daily mode

github@test0:/data/cus_reg_bk/mo/log_ym/log_date/mo-backup/logs$ ls -lthr
total 76M 
-rw-r--r-- 1 github github  14M Sep 27 00:00 stdout.log-20240927_000000.gz # <- archived log
-rw-r--r-- 1 github github 2.9M Sep 28 00:00 stdout.log-20240928_000000.gz # <- archived log
-rw-r--r-- 1 github github  18M Sep 29 00:00 stdout.log-20240929_000000.gz # <- archived log
-rw-r--r-- 1 github github  41M Sep 29 11:06 stdout.log  # <- currently using mo-service log 

github@test0:/data/cus_reg_bk/mo/log_ym/log_date/mo-backup/logs$ head -n 10 ./stdout.log
{"level":"INFO","time":"2024/09/29 00:00:01.927764 +0800","name":"log-service","caller":"taskservice/task_service_cron.go:44","msg":"cron task scheduler started or is stopping","service":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf"}
{"level":"INFO","time":"2024/09/29 00:00:01.930107 +0800","caller":"dbutils/runtime.go:127","msg":"VectorPool[trasient-vector-pool][0/512]: FixSizedVec[0/307] VarlenVec[0/205], Reset/Hit/totalStats:[1569/(97941,103186)201127/201332]\nVectorPool[small-vector-pool][0/10240]: FixSizedVec[0/6144] VarlenVec[0/4096], Reset/Hit/totalStats:[4/(2628439,1300176)3928615/3928615]"}
{"level":"INFO","time":"2024/09/29 00:00:01.934896 +0800","name":"log-service","caller":"task/task_scheduler.go:71","msg":"task schedule query tasks","service":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","created":2,"expired":0}
{"level":"INFO","time":"2024/09/29 00:00:01.943604 +0800","name":"log-service","caller":"task/task_scheduler.go:161","msg":"task allocated","service":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","task-id":140465,"task-metadata-id":"ETLMergeTask:6717","task-runner":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf"}
{"level":"WARN","time":"2024/09/29 00:00:01.943807 +0800","name":"cn-service","caller":"frontend/util.go:504","msg":"unexpected protocol closed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","session_info":"connectionId 42501||account sys:f36f0e27-399d-49cd-87be-d12c16658363|goRoutineId 0|migrate-goRoutineId 0|01923321-faec-715e-95ac-132d2fe72868","role":"moadmin","session_id":"01923321-faec-715e-95ac-132d2fe72868","statement_id":"0192395c-4797-7a55-af2d-aa56a8649f20","span":{"trace_id":"01c74c53-6157-b3b9-f104-13e54d81c478","span_id":"5b7a8bbcb1e1a207"}}
{"level":"INFO","time":"2024/09/29 00:00:01.953632 +0800","name":"log-service","caller":"task/task_scheduler.go:161","msg":"task allocated","service":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"7c4dccb4-4d3c-41f8-b482-5251dc7a41bf","task-id":140466,"task-metadata-id":"retention:2240","task-runner":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf"}
{"level":"WARN","time":"2024/09/29 00:00:01.953823 +0800","name":"cn-service","caller":"frontend/util.go:504","msg":"unexpected protocol closed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","session_info":"connectionId 42501||account sys:f36f0e27-399d-49cd-87be-d12c16658363|goRoutineId 0|migrate-goRoutineId 0|01923321-faec-715e-95ac-132d2fe72868","role":"moadmin","session_id":"01923321-faec-715e-95ac-132d2fe72868","statement_id":"0192395c-47a1-7b14-b38f-1fe71e6c16a5","span":{"trace_id":"01c74c53-6157-b3b9-f104-13e54d81c478","span_id":"5b7a8bbcb1e1a207"}}
{"level":"INFO","time":"2024/09/29 00:00:02.586651 +0800","caller":"dbutils/mem.go:107","msg":"HeapInfo:TotalAlloc:1182387MB Sys:1717MB HeapAlloc:1324MB HeapSys:1681MB HeapIdle:324MB HeapReleased:106MB HeapInuse:1357MB NextGC:1504MB NumGC:1745 PauseNs:578214965"}
{"level":"INFO","time":"2024/09/29 00:00:02.588073 +0800","caller":"checkpoint/runner.go:974","msg":"[flushtabletail] mem scan result","pressure":0.0002896767109632492,"totalSize":"303.75KB"}
{"level":"INFO","time":"2024/09/29 00:00:02.588264 +0800","caller":"jobs/flushTableTail.go:297","msg":"[FLUSH-START]","task":"[FT-13550]272398-__mo_index_secondary_0191c099-dd2d-7746-b193-e829a3b6cafc","extra-info":{"a-objs":"55486a3b95b5,","a-tombstones":"b797e41c9990,"},"txn-info":"ctx[0192395C4A1C729A8FC44E634EB8467C][1727539202586789462-0->9223372036854775807-4294967295][Active]: <nil>","aobj-ndv":2}

Additional information

n.a.

[aronchanisme]

Use linux bulit-in tool/cmd logrotate as the solution.
Ref: https://linux.die.net/man/8/logrotate

Example of conf:

1. daily

github@test0:/data/cus_reg_bk/mo/log_ym/log_date/mo-backup/logs$ cat /etc/logrotate.d/mo-service 
/data/cus_reg_bk/mo/log_ym/log_date/mo-backup/logs/*.log {
daily
compress
rotate 100
copytruncate
missingok
notifempty
dateext
dateformat -%Y%m%d_%H%M%S
}

2. size

/data/cus_reg_bk/mo/log_ym/log_date/mo-backup/logs/*.log {
size 1024M
compress
rotate 100
copytruncate
missingok
notifempty
dateext
dateformat -%Y%m%d_%H%M%S