RUNNING, SUBMITTED status not found in trace file

[typekey]

Description

I'm trying to use trace to track a pipeline run. I found that the pipeline running status NEW, SUBMIT, and RUNNING are not recorded in the trace file. However, it is stated in the nextflow documentation that these states can be recorded.
In my many tests, the running status NEW, SUBMITTED, and RUNNING will only appear in the log file, indicating that the pipeline can be tracked normally.
But in the trace file I can only find the completed status, such as CACHED, COMPLETED, FAILED.
What can I do to keep track of all the statuses including NEW, SUBMITTED, RUNNING, CACHED, COMPLETED, FAILED etc.
Thank you for your help.

Command Line

nextflow -log ./log/test.log run main.nf

config

trace {
    enabled = true
    overwrite = true
    raw = true
    file = "./track/trace.txt"
    fields = 'task_id,name,status,exit,realtime,%cpu,rss'
}

trace

task_id	name	status	exit	realtime	%cpu	rss
2	TRIM_GALORE_PE (2)	CACHED	0	104908	170.2	421302272
28	TRIM_GALORE_PE (28)	CACHED	0	128724	174.7	420839424
101	HISAT3N_MAPPING_GENOME_PE (11)	COMPLETED	0	1940950	1070.5	5408940032

log

[bentsherman]

Hi @typekey , the trace file is meant to be viewed after a run, so you would never see tasks in the trace file with a status like RUNNING or SUBMITTED.

If you want to monitor the task status while the pipeline is running, you can use something like Seqera Platform.

[typekey]

Thank you for your reply. If I simply want to monitor the status of a running task without needing the web panel, what should I do? Could you provide some guidance?

[bentsherman]

The stdout of the nextflow run is the only other way that I can think of. If you run with -ansi-log false then it will log each task submission separately rather than showing the process list with percentages, but it doesn't log when a task completed so it's not a perfect solution. This recent blog post shows an upcoming feature in the nf-core CLI which displays the log in a more user-friendly way.

If you want to monitor the status of individual tasks through the terminal, you could build a tool that consumes the nextflow log and produces a task table with individual task statuses. The log will contain all of the necessary information for it but you'll need to preprocess the logs to produce a more readable task table

[typekey]

Thank you for the detailed approach. It seems like using the new version + the log processing pipeline is a good way.