k3s server create 100% CPU load #11251
-
|
Environmental Info: Node(s) CPU architecture, OS, and Version: Cluster Configuration: Describe the bug: NewRelic CPU usage for 3 days screenshot Steps To Reproduce: As 3 days graph suggests, the CPU load increases day-by-day (while the application is having the same load), so I've tried to run: from this topic #6095 (comment), but it never helped. So I've ended up reinstalling the whole cluster on my new machine, just to find myself in the same situation 5 days after. Expected behavior: Actual behavior: Additional context / logs: |
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 20 replies
-
|
I'd probably look at what the cluster is doing? Check the logs for errors, do The fact that CPU utilization rises and falls over time suggests that the load is related to external factors, and is not just an issue with K3s itself. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed. It's also very difficult for me to understand the root of the problem. The cpu load indeed correlate to the application load, but it raises over the time non-proportionally (avg cpu usage is 10% on night 1, 20% - night 2, 50% - night 3). For me, it's an indicator of spending CPU to some kind of garbage collection problem. doesn't show anything criminal. It's a lot of I've also check journal, and didn't find anything criminal. |
Beta Was this translation helpful? Give feedback.
-
|
And there's nothing interesting in the logs? You could try enable profiling on the server node and then grab a cpu profile, when it's at a period of high resource usage. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I'm not that familiar with cpu profiling. Could you suggest any tool to do that? |
Beta Was this translation helpful? Give feedback.
-
|
you got an error from zsh not kubectl so I don't know what's going on there. Did you need to quote the url or something? I don't personally use zsh. Also note that you will want to redirect that output into a .gz file as it is a bunch of binary that you do NOT want dumped to your shell. |
Beta Was this translation helpful? Give feedback.
-
|
I've managed to resolve Should I add pod or service or smth else here? |
Beta Was this translation helpful? Give feedback.
-
|
Sounds like the flag didn't get added. Did you do a |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your patience, @brandond - I've finally managed to profile the system. Here is my profiling file: |
Beta Was this translation helpful? Give feedback.
-
|
That's not necessary, but it is another way to achieve the same result. A common mistake when modifying the systemd unit, other than not reloading before restarting, is leaving a trailing |
Beta Was this translation helpful? Give feedback.
-
|
Btw, I've removed all the agent and load from the server, but it still consumes about 50% of CPU |
Beta Was this translation helpful? Give feedback.
-
|
Pretty much the ONLY thing in your trace is sqlite: I can't say I've ever seen this before, usually sqlite is very fast. What sort of storage are you running this on? Check journald for messages in your k3s service logs about compact - is it failing to compact the datastore for some reason? |
Beta Was this translation helpful? Give feedback.
-
|
Here is my VM config. I use High-IOPS sdd, so I don't expect the problem from this point of view. As for journal, I'm using My journal looks like this: |
Beta Was this translation helpful? Give feedback.
-
|
You're enqueueing 10s of jobs per second? That seems... excessive. No wonder you're putting so much pressure on the datastore. Is there a problem here, or is this how it normally looks? Can you dump the logs with |
Beta Was this translation helpful? Give feedback.
-
|
I've replaced my real cluster name with |
Beta Was this translation helpful? Give feedback.
-
|
And yes, as I've said in the begging of the thread: |
Beta Was this translation helpful? Give feedback.
-
|
OK yeah so your compact is regularly failing with When the compact only partially completes, it skips the next cycle and then there is even more work to do the next time, so it is less likely to catch up. This also leaves more data in the database for it to sort through, which is probably the cause of your excessive load. You might consider switching to etcd? If you have a single server, you can just convert in-place by adding While we can try to handle the "database is locked" issue better, sqlite is really tuned for smaller use cases where low overhead and low disk IO is more important than raw throughput. Etcd will probably suit your use case better. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your answer, @brandond , I'm really appreciating it. I've switched my cluster to |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
@allnightlong what filesystem and disk type are you running k3s on? I am trying to reproduce the |
Beta Was this translation helpful? Give feedback.
-
|
I've already provided you the information from my hosting company. It's positioned as High-IOPS. I also can run If you need any extra information, feel free to ask - I'll run the command required. |
Beta Was this translation helpful? Give feedback.
-
I noticed something similar on two of my k3s clusters. One is single node only with rancher installed. Second one (what you see on the screenshot) CPU started increasing after I installed longhorn on it and installed my helm chart with about 18 pods. So my clusters are pretty small and nothing is enqueued constantly just some peak load when something was installed. When pods stopped scheduling and all got stalled shouldn't CPU usage be normalized back ? Just restarting k3s service helped to reduce CPU usage |
Beta Was this translation helpful? Give feedback.
-
|
These two cluster are running on not very powerful hardware. One with SSD another on HDD raid array |
Beta Was this translation helpful? Give feedback.
-
|
@brandond Just noticed I have one more time the same issue with staled single node cluster where I have only rancher running. |
Beta Was this translation helpful? Give feedback.
-
|
Disk IO is the same, about 150kb/s |
Beta Was this translation helpful? Give feedback.
-
Logs seeing right now |
Beta Was this translation helpful? Give feedback.
-
|
One think I noticed that my VM disk was on RAID 10 with HDD and Regarding second cluster: it's just running for test purposes on old mini box with proxmox and cheap SSD. Also I see it was giving small amount of IOPS (about 70). I'm going to move to another newer mini pc where I see 700 IOPS and see how it will go. |
Beta Was this translation helpful? Give feedback.
OK yeah so your compact is regularly failing with
database is lockederrors: