Getting occasional "mcu shutdown - timer too close" shutdowns.
This time it occurred while running "beacon_ratos_calibration bed_temp=85"
When things are running normally, the short term htop load average is like .1 or .3, but when I get the timer too close shutdowns,
the load average is very high.
104 Replies
For What it's worth, I had been struggling with similar errors on my RR VCORE3.1 (https://discord.com/channels/582187371529764864/859890291591217162/1259952416619495484) and some entries in the logs pointed to USB-hub resets. It was suspected to be a USB powersupply issue (I ran a unpowered USB hub connected to Manta M8P and connected USB camera, EBB42 and Beacon of of that). I now have the Beacon directly connected to the Manta M8P and EBB42+webcam on the unpowered HUB and my errors (for now) have disappeared. (I still need to buy a power USB HUB)
Take care about powered hubs. They power the Pi too. That is something you want to prevent, because otherwise you run into issues on power-up. Ask me how I know... 😉
Doesn't the beacon have a warning about connecting it directly to the manta usb?
?.... Please enlighten me? I am not aware of that.
from their website
https://beacon3d.com/product/beacon/
If you search in this discord it was discussed, and I think the very first version was ok, but 1.1+ had the issue where the usb isn't properly grounded?
Thanx, @blacksmithforlife 🇺🇸 ! Not sure if this was already on Beacons website when I purchased the unit or I missed it... Al swap the USB connection so to connect Beacon to the Hub... Hope that that doesnt bring my errors back though...
Does anybody else get these timer too close errors?
I got some once in a while... I just toggle the power and it´s gone. 3 times if I remember correctly over the last 2 months.
I haven't had any errors/shutdowns in the last two days, but before that, I got several a day! I was doing more calibrations, PID tuning, shaper graphs, etc., and now I'm just printing. I don't understand how the Raspberry Pi load numbers can go from .1 and .2, then shoot to almost 6, but maybe the tuning is more demanding than I realize. It would suck to be several hours into a print when it happens, so I'd like to find out what's causing it.
It will not happen during printing.
I think I fixed it by reordering the USB cables on my setup. Beacon and toolboard are now using the USB3.0 connectors, Board is connected to USB2 same as my USB hub. There I just connected my cam and screen. Since then I had no such issues. Take it as a BSOD on M$. It happens, but you would not be able to localize this issue yourself, even experts don´t.
timer too close is always related to system load and doesn't actually have anything to do with USB. The connection is fine, but the system isn't fast enough to send the messages in the timeframe the other end is expecting.
Data congestion could cause it i guess, but you would have to wire your stuff in such a way that all devices would be running over the same low-speed usb connection. USB 3.0 connector on the pi to an MTT USB 3.0 hub with all the devices connected, takes care of that issue (but i'm fairly sure this isn't it - it's not something you do by accident)
Good point... I was thinking about a BIG7 7-port MTT USB hub.
On the other side: For now I don´t get this issue since I reordered my USB on the Pi.
Main issue is: To get it stored in the electronic box on the back side of the VC4... May be there is enough place in the box to get it mounted under the Pi... I don´t know tbh. Time will tell.
Read this thread at Superslicer Discord. Not sure if it is related: https://discord.com/channels/771316156203270154/858997549491027978/1277588712070316104
@miklschmidt are you aware of this issue in SS?
...read the thread and mostlikely not related as it deals with too many request for fan speed changes in the slicer (Orca in this case). Apologies for allerting you...
That's cool. I appreciate people pondering the issue. I have started to notice that most of my errors/shutdowns are during the startup phase when the bed is homing and mesh leveling, and once it gets past that, it seems to finish the print. Thanks.
for me, that is more or less the same. I also have these mcu errors during a filament runout event and specifically (within the RatOS macros only) upon resuming a print after filament reload. Once the printer is printing it normally goes smooth as a baby's bottom... (problem is not that I don't trust the filament runout / Pause Resume anymore.... kind of defeats the purpose of the filament runout sensor...)
It would be nice if you could monitor htop and find out which process is causing the cpu load spikes when you get those errors
I've taken some htop captures, but they may not show anything useful.
At least it's not doing anything here at all 😄
I thought it was weird that the short term load average was so high. I have several other captures showing pretty much 4 maxed out cores, and maxed out swap memory but I didn't save the klippy log each time which is what I figured I'd need.
I have printed more since this capture with a lower failure rate, but it's not cured.
No i need the htop screencap showing which process(es) is/are using all those resources (it's a live monitoring tool) right before during or right after a crash happens.
So far i've just seen an idling system
if you installed any thirdparty stuff, disabling those is usually a good place to start
how would you recommend doing that? SSH into py and run htop during a print? Would that then capture it, or would I need to screengrab the ssh-terminal on the moment it has the error?
the latter. You should monitor it regularly and if memory / cpu spikes, screengrab it.
You could try
sysprofIt took about 3 seconds after the shutdown before I could screen grab the htop screen. Hope it's still useful.
I had cleared the klipper and moonraker logs just before the shutdown, and they are below.
It's happening every time I start a print now. Three times in a row it has shutdown right after it does the prime blob and the gantry moves away. I'm getting MCU 'toolboard_t0' shutdown . This last time I captured the htop screen immediately after shutdown. shown below
your system is practically idling. I'd start looking at these:
Also you have modemmanager running which has been known to cause issues like this. RatOS disables that, so i don't know how you've got that enabled.
this is a different error. EDIT: Never mind, the full message is: "MCU 'toolboard_t0' shutdown: Timer too close".
If you've done anything custom, try a clean installation without it.
@millenford - VC4 300h Is this a Pi or a CB1 btw?
From the memory available i'm guessing it's Pi4 2gb
The only customizations are in printer.cfg and they are my input shaper numbers and this:
[output_pin extruder_light]
pin: toolboard_t0:PD3
pwm: true
shutdown_value: 0
value: 1
cycle_time: .01
scale: 100
I'm using a new RPi 4B 2GB with a new SD card.
I'll re-image the card from scratch.
Just for future searchers.... instead of re-imaging the card shown above, I bought another SanDisk, an Extreme Pro and installed it. The new card has a higher speed rating with a 3 inside the U character on the card, instead of 1. I don't know if the original card was too slow or defective. While it's too soon to declare it's completely fixed, it's been good so far with no shutdowns. This may be a good reason to just use a SSD instead of an SD card.
The new card that has been working without any shutdowns so far is like this, except it's 64GB.
@miklschmidt I'm still running good without shutdowns, but I was re-reading the thread and was reminded of your comment about modemmanager. It's nothing I knowingly enabled, and I see it's running in the new installation I did on the new card too. It's definitely nothing I tried to do. The only change I made to both installations was adding "dtoverlay=disable-wifi" in /boot/config.txt to kill the wifi.
It's nothing I knowingly enabled, and I see it's running in the new installation I did on the new card tooWith RC2? wtf
It's not present on my machine, and it's really weird that it would be, since the service is masked during image compilation. Makes no sense. Are you running RatOS 2.0 by chance?
wtf
Can you try and run that command?
sudo systemctl status ModemManagerpi@ratos:~ $ sudo systemctl status ModemManager
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for pi:
● ModemManager.service - Modem Manager
Loaded: loaded (/lib/systemd/system/ModemManager.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-09-20 05:42:08 BST; 2 days ago
Main PID: 449 (ModemManager)
Tasks: 3 (limit: 1389)
CPU: 186ms
CGroup: /system.slice/ModemManager.service
└─449 /usr/sbin/ModemManager
Sep 20 05:42:08 ratos systemd[1]: Starting Modem Manager...
Sep 20 05:42:08 ratos ModemManager[449]: <info> ModemManager (version 1.14.12) starting in system bus...
Sep 20 05:42:08 ratos systemd[1]: Started Modem Manager.
Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3': not supported by >
Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4': not supported by >
Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd580000.ethernet': not supported by any plugin
Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1': not supported by any plugin
It just started spilling out many lines of data.
Something ain't right
that should not be running
(and it isn't on my installs)
as you can see
It's starting to dump data to the screen
what is? the
systemctl status command?Yes. I assumed it did the one time data dump to the screen that I already pasted in, but since then it started dumping more to the screen. Maybe it's done now.
pi@ratos:~ $ pi@ratos:~ $ sudo systemctl status ModemManager
-bash: pi@ratos:~: command not found
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $ We trust you have received the usual lecture from the local System
-bash: We: command not found
pi@ratos:~ $
pi@ratos:~ $ Administrator. It usually boils down to these three things:
-bash: Administrator.: command not found
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $ #1) Respect the privacy of others.
pi@ratos:~ $
pi@ratos:~ $ #2) Think before you type.
pi@ratos:~ $
pi@ratos:~ $ #3) With great power comes great responsibility.
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $ [sudo] password for pi:
-bash: [sudo]: command not found
pi@ratos:~ $
pi@ratos:~ $ ● ModemManager.service - Modem Manager
-bash: ●: command not found
pi@ratos:~ $
pi@ratos:~ $ Loaded: loaded (/lib/systemd/system/ModemManager.service; enabled; vendor preset: enabled)
-bash: syntax error near unexpected token
('
pi@ratos:~ $
pi@ratos:~ $ Active: active (running) since Fri 2024-09-20 05:42:08 BST; 2 days ago
-bash: syntax error near unexpected token ('
pi@ratos:~ $
pi@ratos:~ $ Main PID: 449 (ModemManager)
-bash: syntax error near unexpected token ('
pi@ratos:~ $
pi@ratos:~ $ Tasks: 3 (limit: 1389)
-bash: syntax error near unexpected token ('
pi@ratos:~ $
pi@ratos:~ $ CPU: 186ms
-bash: CPU:: command not found
pi@ratos:~ $
pi@ratos:~ $ CGroup: /system.slice/ModemManager.service
-bash: CGroup:: command not found
pi@ratos:~ $
pi@ratos:~ $ └─449 /usr/sbin/ModemManager
-bash: └─449: command not found
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $
pi@ratos:~ $ Sep 20 05:42:08 ratos systemd[1]: Starting Modem Manager...
-bash: Sep: command not found
pi@ratos:~ $
pi@ratos:~ $ Sep 20 05:42:08 ratos ModemManager[449]: <info> ModemManager (version 1.14.12) starting in system bus...
-bash: syntax error near unexpected token ('
pi@ratos:~ $
pi@ratos:~ $ Sep 20 05:42:08 ratos systemd[1]: Started Modem Manager.
-bash: Sep: command not found
pi@ratos:~ $
pi@ratos:~ $ Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3': not supported by >
>
> Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4': not supported by >
-bash: syntax error near unexpected token newline'
pi@ratos:~ $
pi@ratos:~ $ Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd580000.ethernet':
not supported by any plugin
>
Sep 20 05:42:11 ratos ModemManager[449]: <info> [base-manager] couldn't check support for device '/sys/devices/platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1': not supported by any plugin-bash: info: No such file or directory pi@ratos:~ $ Sorry.
Oh you pasted
pi@ratos:~ $ in front the entire command?
Oh boy 😂
You're basically evaluating the output of that command as a bash script
Just kill the session and start a new one, should be fine 🙂
That's where it went wrong 🙂At the CLI I hand typed "sudo systemctl status ModemManager". It spit out the data I initially pasted here, then about 2 minutes later, it just started dumping all the other stuff to the screen. That's the second paste I did here.
I have no idea what's going on.
What terminal are you using?
Bitvise SSH Client 6.23
Windows? Use the built-in terminal
There's already SSH there 🙂
(been there since Win 10)
just
ssh [email protected]
Ok, I'm in.
I admit I didn't know Win10 had SSH, and I was able to login fine! But unless Bitvise is causing problems, I think I prefer it because it stores the login credentials for the many other Pi's around here, and also has a neat windowed SFTP feature when I need it.
That's why i wanted you to try the native OpenSSH on windows, since what you posted seems like a terminal problem.
I see. I'll try the same command again via the windows terminal
It ran. So far it dumped this, but it looks like it's still running as I don't have the command line prompt back... yet.
press the arrow down key
or esc to exit
it's
less on linux that paginates the output for you
Probably what made your bitvise terminal shit itself 😅
anyway... Something's wrong. You didn't install anything? this is just flashed and updated, nothing else?It's behaving in a similar way in the Windows terminal too. Since the first output, it's spit out more lines. Of course, I don't even know what sudo systemctl status ModemManager is supposed to do. I assumed it ran and spilled out the status of modem manager, and didn't keep running.
Nope. I added a few lines to printer.cfg, but the only thing I did in the OS is add the line to kill the wifi like I showed above.
Your screenshot looks normal
But since that first capture, it printed this.
Admittedly, it's WAY less addtional output as compared the first time I ran in via Bitvise SSH.
How did you get out of it?
I haven't yet, but I suspect I can hit control C
Anyway, something has enabled ModemManger on your system, probably from third party module updates. I checked the image logs and it seems like ModemManager is already disabled on the rpi images, so it only gets masked on CB1. Will fix that. You can run this:
Now it's done a third dump.
Right it's just updating your TTY in this case.
Just exit it
q or ctrl+c should do it tooOkay, control C got me out. I'll run the above lines you provided.
The first two lines completed, and it seems to be working on the third line as the prompt hasn't returned yet.
Weird. It was just sitting there with no prompt, so I hit ENTER for the third time, and it quickly did what you see here.
I guess I should reboot?
nope all is good
modemmanager is indeed gone in HTOP.
might be
ctrl+c to get out of the status report wasn't the best option. Try q next time.
and it won't start again. The service now points to nothing.Okay, will do. Will those changes stick "forever"?
yep
Okay.
So at least that's not going to be a problem.
Although sounds like it was working without it.
Pretty sure your previous troubles were caused by the SD card
Yes, it seems so. That's the second time I got double crossed by a RPi sd card. Kind of pisses me off, as both times there were "good" Sandisk cards.
I have an SSD on another unrelated Pi, but it's a bit bulky to put on the Pi in the ratrig. I mean it would fit, but I hate to clutter it up. Then I checked into NVMe boards, but they seem to be best on the RPi 5, not the 4, so I'll just let it run on the SD for now.
Jeez dude, you're up late.
The last one was pretty slow though, which i think might've been the issue
Yeah...
Alternatively there's USB sticks.
The first time an SD card failed, it just got corrupted after many, many read/write cycles.
Would a USB be more reliable than SD?
Probably similar. Only SSD's have controllers to ensure optimal disk health
SSD controllers attempt to even out writes between different sectors, has backup sectors to replace premature bad ones etc. Lots of stuff going on in those to keep them running for longer.
Not an expert though, disk stuff is my dad's area, unfortunately i was too young and stupid to understand and retain the lectures i was involuntarily subjected to in the past 😂
Because one of the warnings in the errors I had before were "overheating", I was wanting to turn the electronics enclosure fans back on after they do their inactivity shutdown. I've since decided to just add a small fan on the Pi and let a python script control it like I've done on other Pi's in the house. But is there way to command the 3 enclosure fans back on after they shutdown?
What was overheating? The pi?
Your pi really shouldn't need cooling when it's idling
Back a week or so ago, you pointed out this.
But yes, multiple ways, you could configure one of them as a generic fan, you could wire it to always on or you could do something more exotic, whatever you like.
that's just a "possible" cause
If your pi is overheating you'll get notifications about it in mainsail
The RatOS/mainsail shows the Pi temp rises quickly after the fans go down.
to how much?
As long as it's below 65 there's no need to do anything
it should sit around ~45-50 without the fans
yeah this is totally fine
Okay good.
I just did a home cycle to turn the fans back on, and they're pretty effective.
I just updated to Ratos 2.1 latest (via Update Manager) and rebooted.
"sudo systemctl status ModemManager" still gives me:
Also, it appears to start ModemManager again,... not sure if this is as expected?
Only new images will have this disabled (ie. starting with RC3). You can disable it by running this https://discord.com/channels/582187371529764864/1272979483116441630/1287569465013305354
Hi Mikkel, Thanx for the update. (I anticipated on that and ran it already)
I have many print hours now on the new SD card without shutdowns or timer too close errors, so I'm marking this issue as solved.