Only grouped metrics are displayed in the WebUI

[alexx-km]

Describe the bug

Normally, all metrics are displayed in a grouped plot and each individual metric is displayed in an individual plot. For old runs in my self-hosted setup, the individual plots are no longer displayed. The data must be available because the grouped plot seems to be correct.

Bug:

To reproduce

I'm not quite sure how to reproduce this bug. I had to change the permissions on some files in the database because it was originally set up for the wrong user. This may have had an impact on the problem.

Expected behaviour

Metrics should be displayed in grouped but also in individual plots:

Environment

• Server type: self hosted
• ClearML SDK Version: 1.16.2
• ClearML Server Version: WebApp: 1.15.1-478 • Server: 1.15.1-478 • API: 2.29
• Python Version: 3.10
• OS: Windows \ Linux

[ainoam]

@alexx-km Please let us understand a little bit better what's going on:

• Are there specific tasks that consistently show both grouped plots and individual plots, and others that consistently only show the grouped plots?
• When you get both grouped and individual plots - Are these shown concurrently on the screen e.g. one plot titled 'train' and another titled 'train/loss'?

[alexx-km]

Hi @ainoam, thanks for your reply. Sorry, I completely forgot to include that information. I only have this bug for "old" tasks, all recently created tasks have both grouped and individual plots. Regarding your second question, I get a grouped plot with all metrics and an individual plot for each metric, e.g. titled train/loss or train/map.d0.loss_bbox_f.

[ainoam]

@alexx-km How are these scalars collected? The ClearML UI is designed to either show grouped plots OR individual plots (See "Group By").
If you're seeing both, it seems like they are reported twice: Once with title="Title", series="Series" , and once with title="Title/Series", series="Series"

[alexx-km]

Hey @ainoam, sorry for the late reply.

We use the TensorboardLoggerHook to log all the metrics of our training. But you are right, it seems that the scalars are reported twice. When I select group by "metric", I get the grouped plot plus a plot for each individual metric. When I select group by "none", I get the plots for each individual metric. However, there still seems to be a difference between old and new experiments, as this behavior is only true for new experiments. With the old experiments, I have the described problem, that grouping by metric results in only the grouped plots being shown and not the individual plots.

Could that be due to an issue with the database and is there any way to find out what the difference between those experiments could be?