B
BlueBuild•2mo ago
louissunk

High CPU Usage by systemd-udevd Leading to System Stuttering on Fedora Atomic-based Systems

I am experiencing severe system stuttering on Fedora Atomic-based systems, particularly noticeable when starting multiple Docker containers or performing tasks such as mvn clean install. The stuttering affects audio playback (Spotify, voice calls) and mouse movement. System Details: ryzen 7950x3d+64gb+amd 7900GRE+2tb nvme drive Previous Setup (No Issues): Fedora i3wm Spin - Used for over a year without encountering any problems. Problematic Setups: - ublue-os Aurora: Encountered the issue after switching from i3wm to this Atomic-based setup. Unable to resolve, reverted to previous setup. - Custom Image (based on wayblue sway image): Recently installed on the same machine, replicated the issue observed in Aurora. The same custom image works fine on a different machine the previous week. This issue manifests as system.slice/systemd-udevd/udev consuming an excessively high amount of CPU, up to 2000%. (only seen with systemd-cgtop, not with top,htop,btop, mission control) when monitoring udevadm monitor I do see
KERNEL[20546.524311] remove /devices/virtual/net/veth15367bb/queues/rx-0 (queues)
KERNEL[20546.524320] remove /devices/virtual/net/veth15367bb/queues/tx-0 (queues)
KERNEL[20546.524360] remove /devices/virtual/net/veth15367bb (net)
UDEV [20546.527429] add /devices/virtual/net/veth6686c49 (net)
UDEV [20546.557393] remove /devices/virtual/net/veth6686c49/queues/rx-0 (queues)
UDEV [20546.558055] remove /devices/virtual/net/veth6686c49/queues/tx-0 (queues)
UDEV [20546.596101] remove /devices/virtual/net/veth15367bb/queues/tx-0 (queues)
UDEV [20546.596834] remove /devices/virtual/net/veth6686c49 (net)
UDEV [20546.611493] remove /devices/virtual/net/veth15367bb (net)
KERNEL[20546.524311] remove /devices/virtual/net/veth15367bb/queues/rx-0 (queues)
KERNEL[20546.524320] remove /devices/virtual/net/veth15367bb/queues/tx-0 (queues)
KERNEL[20546.524360] remove /devices/virtual/net/veth15367bb (net)
UDEV [20546.527429] add /devices/virtual/net/veth6686c49 (net)
UDEV [20546.557393] remove /devices/virtual/net/veth6686c49/queues/rx-0 (queues)
UDEV [20546.558055] remove /devices/virtual/net/veth6686c49/queues/tx-0 (queues)
UDEV [20546.596101] remove /devices/virtual/net/veth15367bb/queues/tx-0 (queues)
UDEV [20546.596834] remove /devices/virtual/net/veth6686c49 (net)
UDEV [20546.611493] remove /devices/virtual/net/veth15367bb (net)
I do not know what this means, Only happens under load.... I do not know how to solve thisand how to find the cause. Are there any known issues or bugs with systemd-udevd on Fedora Atomic-based systems? Suggestions for further diagnostics ? Thanks
Solution:
So I was able to reproduce this on all my machines and on stock bluefin as well, i can remove the issue by shutting down systemd.udevd service. this can be closed, when I am back from holiday I will make a ticket with ublue os or silverblue fedora depending on if I can reproduce this on silverblue stock with docker layered as well
Jump to solution
24 Replies
louissunk
louissunk•2mo ago
custom image repo: https://github.com/tomrutsaert/sway Extra info, I start those docker containers from a distrobox , the same for running the maven commands. But is also the way I did is on my i3wm set-up this is the image I run them in: https://github.com/tomrutsaert/fedora-dev-distrobox
Luke Skywunker
Luke Skywunker•2mo ago
I've had this same issue I have no idea how to fix it though...
louissunk
louissunk•2mo ago
do you have the same issue on your current install as well? I noticed just now, That the stutter only happens with 'mvn clean install' when the testing start and the tetsing is using test containers. So I assume this issues has to to do with creating/removing docker containers... in mission control when some containers are running, I do also see now several vethxxxxx network interfaces, which corresponds with the log from udevadm monitor, I assume these are network intefaces for the docker containers..... @gmpinder what did you do to work around the problem?
fiftydinar
fiftydinar•2mo ago
Maybe some corrupted udev rule, which constantly pings systemd-udevd?
louissunk
louissunk•2mo ago
i can now reproduce on laptop as well and keep in mind this machine was installed this weekend , so brand new and spotless..... I have a create a docker-compos.yml file. If you run docker compose up -d while playing music I have stutter on all my machines (2) https://gist.github.com/tomrutsaert/0e7df26a882dcf6da0557504e29ce97e If some one would be willing to try that on their machine? a simpliefied version of what I start every day (work related) @fiftydinar I had the same problem on aurora 2 months ago
Luke Skywunker
Luke Skywunker•2mo ago
I didn't 😬
fiftydinar
fiftydinar•2mo ago
I wonder if vanilla Fedora has the same problem
Luke Skywunker
Luke Skywunker•2mo ago
Although, using docker buildx does help
louissunk
louissunk•2mo ago
vanilla none atomic does not I asked if some one in ublue-os discord could test as well https://discordapp.com/channels/1072614816579063828/1084120222400774325
Luke Skywunker
Luke Skywunker•2mo ago
I am on Bazzite
louissunk
louissunk•2mo ago
do you have docker?
Luke Skywunker
Luke Skywunker•2mo ago
Yeah From Docker's own yum repo
louissunk
louissunk•2mo ago
yeah could you test the file above? in the gist
Luke Skywunker
Luke Skywunker•2mo ago
I'm currently running in Cosmic and everything seems fine right now Let me try plasma wayland and x11
louissunk
louissunk•2mo ago
It only occurs while starting up and a bit when doing docker compose down
Want results from more Discord servers?
Add your server