Excessive CPU use on UReport

[matmsa27]

Hi everyone, I can share with you a strange behavior on the UReport instance. We have a machine to run the UReport instance with 2cpu cores and 14GB ram of memory available. But most of the time we have observed excessive CPU use when using totally for 2 CPU cores. This behavior Is that normal? On our instance we have 54 workspaces configured.

Beyond this, we investigate and accurate here some methods on the application that have a high cost from resources, especially from CPU. The points are:

• /engagement_data/ API
• The view that return the data
• This code calls many queries on DB and needs processing.

[norkans7]

there is a task to calculate the stats from the DB and cache them so that https://github.com/Ilhasoft/ureport/blob/main/ureport/public/views.py#L367 uses the cache results

so far we continuously recalculate the stats probably one way we can reduce the number of queries would be to reduce the number of queries by having an expiring flag like in redis that we set after we finished calculating the stats and check that the flag is expired or not present to recalculate the stats.

What do you propose as solution to the problem?

[nicpottier]

We are talking about instance CPU usage? I don't feel we see very high instance usage on U-Report. Views shouldn't be taking much of any CPU if the results are properly cached. (which they should be)

[johncordeiro]

Yes, thanks for sharing this @norkans7 and @nicpottier, we've discovered that we're not running the slow queue which avoided the task to refresh engagement data to cache this data on Redis. We've initialized it, but even with the data cached we're experiencing a lot of CPU usage to load home and engagement pages.

Memory usage is fine, but CPU is still a bottleneck. Take a look at the yellow line, it's the limit of 2 vCPUs in 2 different replicas:

How much CPU do you provide for the U-Report application?

[norkans7]

Yes, the slow queue will be running the task to update the cache.

The CPU now is from the tasks which why I mentioned that we currently continuously run the queries on a schedule even if no new stats will be updated.
We could have a smarter way to only run the queries in the tasks only when we have new data that will affect the stats

[norkans7]

I found one index that can help #723
and we can experiment more strategies later in the direction to run the queries few times that currently such as #724

[matmsa27]

Hi @norkans7 , thanks a lot for these improvements. We updated our instance here and we will observe the behavior, and any improvements that we can get we will bring here to discuss and then open a PR.