Is there a daemon that will kill any processes using above a specified % of CPU? I’m having issues where a system is sometimes grinding to a halt due to high CPU usage. I’m not sure what process is doing it (can’t htop as system is frozen); ideally I’d like a daemon that automatically kills processes using more than a given % of CPU, and then logs what process it was for me to look back on later. Alternatively something that just logs processes that use a given % of CPU so that I may look back on it after restarting the system.

The system is being used as a server so it’s unattended a lot of the time; it’s not a situation where I did something on the computer and then CPU usage went up.

Edit: Thanks to the comments pointing out it might be a memory leak instead of CPU usage that’s the issue. I’ve set up earlyoom which seems to have diagnosed the problem as a clamd memory leak. I’ve been running clamd on the server for ages without problems so might be the result of an update; I’ve disabled it for now, and will keep monitoring the situation to see if earlyoom catches anything else, or if the problem keeps occurring I’ll try some of the other tools people have suggested.

  • just_another_person@lemmy.world
    5·
    8 hours ago
    1. Get some sort of resource monitor running on the machine to collect timeseries data about your procs, preferably sent to another machine. Prometheus is simple enough, but SigNoz and Outrace are like DataDog alternatives if you want to go there.
    2. Identify what’s running out of control. Check CPU and Memory (most likely a memory leak)
    3. Check logs to see if something is obviously wrong
    4. Look and see if there is an update for whatever the proc is that addresses this issue
    5. If it’s a systems process, set proper limits

    In general, it’s not an out of control CPU that’s going to halt your machine, it’s memory loss. If you have an out of control process taking too much memory, it should get OOMkilled by the kernel, but if you don’t have proper swap configured, and not enough memory, it may not have time to successfully prevent the machine from running out of memory and halting.