Webcam Goes Offline, Crowsnest Restart Crashes RatOS
I have run into an issue. The USB webcam I use (Logitech C270) will spontaneously stop displaying images after a while. I am not sure when the threshold of time is, but usually after some time it just goes out. This is the first issue.
To fix this, restarting RatOS always works. However, recently I have been experimenting with restarting Crowsnest. 2 out of 3 times i have restarted Crowsnest using the restart icon in the power menu, the whole RatOS goes down: the web UI stops working and I can not ping or ssh into the device. On the Pi, the green activity light stays unlit.
The 1 time I restarted Crowsnest and it worked was during the print… not sure if that was luck or what.
Anyways I am not sure what caused the whole OS to crash. Both times there was a crash, it was directly linked to restarting Crowsnest from the webUI.
24 Replies
Might be an issue with gpu_memory being set too high in /boot/config.txt. I changed it away from stock at some point and never exercised the crows nest restart until recently… will test and see if my current 256 setting is ok on the 8gb pi.
Issue not solved. Ran a reset for crows nest and the webcam showed “disconnected” continuously. All the rest of the numbers on the system looked normal (i.e. they keep updating). Tried to reboot host and the whole ratos system went down, similar to the issue before. It just seems like crows nest is the culprit here since i never had an issue when directly restarting the host. But a crows nest restart often crashes the system or if i try to restart the system, either crashes the system or doesn’t allow it to restart normally.
It sounds like you're talking about restart / crashing Mainsail and not RatOS, am i correct in that assumption? Restarting "RatOS" would mean rebooting the pi.
similarly "the whole RatOS goes down", you mean Mainsail?
2 out of 3 times i have restarted Crowsnest using the restart icon in the power menuPlease screenshot and mark the exact menu option you're using It's going to be hard to help you without a debug-zip, and preferably a full syslog Keep in mind Mainsail is a client application, if it crashes, it's on the client side (ie, your browser crashes, not the OS). I need a lot more clarification, preferably video
The whole OS crashes. I can’t SSH into it anymore. I will post a screenshot later.
Currently ran into a different behavior. The last “reboot host” from the RatOS power menu seems to have permanently bricked my Pi. It cannot boot at all, no activity LEDs and reflashing the bootloader doesn’t work… will need to figure out the Pi first before so can boot back in RatOS.
The same boot drive seems to boot on a friends Pi though, but i didn’t remember to grab logs.
The whole OS crashes. I can’t SSH into it anymore. I will post a screenshot later.Interesting. Almost sounds like an electrical problem.
The last “reboot host” from the RatOS power menu seems to have permanently bricked my PiYou can't brick your hardware with a reboot.
It cannot boot at all, no activity LEDs and reflashing the bootloader doesn’t work…This honestly sounds like your might have shorted something, unlikely this is a software problem Disconnect all devices and try and boot it again Check the power delivery and the 5 and 3.3v pins on the pi
Maybe. But i just ran a bunch of prints last few days. The only time I’ve witnessed crashes is restarting crowsnest, using the restart circle arrow button from the ratos power menu. other than that, ratos has been rock solid.
There are multiple ways to fry your pi, such as cutting the GND wire powering your toolboard.
to be fair i had to hard power cycle the pi like 3-4 times because ratos crashed so i have no UI or SSH access.
Easy to corrupt your filesystem that way, but you say it's booting on a friends pi?
do you have an HDMI screen available?
yes my boot drive is booting on a friends pi. have UI access on that no problem. yes i plugged in hdmi to my Pi and no image. even with a clean stock raspberry pi image
Time to inspect it for visual damage and measure the voltages
Before replacing the pi or if you somehow manage to revive it, please go over your electronics before resuming operations. Make sure you've pulled off the V_USB jumpers from your connected MCU's. Inspect wires for damage etc.
Ya no 3.3V on the Pi... It's dead. I think maybe the relay I was driving with the 3.3V got smoked. Maybe the pin was overdrawn and that killed the rail. Anyways, got a new Pi up and running and moved the relay onto a stepdown powered directly by the 24V PS so hopefully that fixes that issue... fingers crossed
Anyways, here is the ratos debug log i grabbed.
Ok, I left the Pi on overnight with the printer off. Woke up the webcam was black screen. Performed a reset using the reset button. UI became unresponsive. Refreshing the page left it stuck on "connecting". Raspberry Pi had a repeating pattern of green LED light flashing. No ability to ping or SSH into the device.
I also have a hardware pin that I can ground to safe shutdown the Pi: it doesn't work. Usually I ground the pin and it takes a few seconds and goes into its normal shutdown routine, but this time, the LED lights kept their periodic flashing. That leads me to think the OS crashed, as at minimum the hardware pin shutdown SHOULD work. To note, the previous times I've had a crashed OS, the pin method does not restart the Pi either. I've attached all the logs after reboot.
from what i can tell, there seems to be a bug related to the webcam (not a RatOS thing) you have connected next to the VAOC camera. Have you tried disconnecting that?
Seems like it's exhausting the memory, then hits swap, then basically dies.
very sus device description as well.
it has no name
Besides running the whole thing off a usb stick / disk, have you made any changes to the OS (i see at least one)?
How are you powering these? and how many leds?
I'm curious why you've put stuff on your pi, you can use your controlboard for all of this, it's much more suitable (in terms of voltage regulation, power availability and real time control).
the risk of hooking high power stuff up to your pi is quite high.
As you've found out
uhh i think i installed tmux and rpi-clone. i also changed the gpu mem to 128gb, set vm.swappiness=1 to try to utilize the ram more, and set dtoverlay to enable using pin21 to shutdown the pi safely if i don’t have gui access. also to note, i believe the usb stick i am using actually is a SSD in USB form, with a normal SSD controller built in… so hopefully more resilient to power failure.
it’s weird since it is a standard Logitech C270 (which seems to be a commonly used webcam) that i plugged into the waveshare usb expander. not sure why it’s not announcing properly. also not sure why it’s taking up all the ram… the weird thing is that the webcam dies but the OS still works until i try to restart crowsnest. wouldn’t a maxed memory/swap make the whole OS unresponsive, meaning i shouldn’t even be able to load the webUI to click the restart?
It was my bad. i read that the Pi’s 3.3V can support up to 500mA and my relays were drawing only 130mA each. I wanted to put the LED relay on the Pi since I want to control the LED through the moonraker power device menu. I didn’t see a way to put a button if I used the Octopus board. The 2nd relay (for bed heater) i powered off the Pi also because i want there to be 2 layers of protection: the relay is powered by and controlled by the Pi. this relay then switches 24V from the octopus to a bigger relay which then switches on the SSR; i heard that SSRs usually fail open and didn’t really want my house burning down. I’ve since modified the power scheme so the Pi only drives the optoisolated side of the 2 relays and the 24V PsS stepped down to 3.3V drive the relay side. seems to work ok now with minimal power draw on the Pi side…
that should really read “upper_led_relay” to be more precise, as it is a relay controlling 24V from the PS to the LED strip I installed, not directly powering the LEDs from the Pi
wouldn’t a maxed memory/swap make the whole OS unresponsiveYes indeed
meaning i shouldn’t even be able to load the webUI to click the restart?No, Mainsail is a client application, it runs on the device you're browsing with, not the pi. Your browser will cache it too, so you can launch it without even contacting the pi. It does need a socket connection to moonraker (which runs on the pi) however, to get and send information about/to your printer. Ie, your UI may still be responsive, but the data wouldn't update, and the request to restart wouldn't do anything.
set vm.swappiness=1 to try to utilize the ram morethis is likely the problem, and also why i'm seeing
zram show up in your logs. I recommend reverting to the vanilla RatOS settings, you don't need any of this, it's already using optimal settings, that's the whole point of RatOS. You're not actually meant to even touch the command line (but you can if you want to, but it's a bit beyond the scope of my support to help you with issues you cause this way).i actually installed zram after all of this crashing issue because i was ok a 8gb pi that was bricked so going back to a 1 gb i was seeing swap usage and wanted to have more overhead
so it was crashing before i installed zram
Btw, an SSD is just as vulnerable to power failure, it's not so much about the type of storage, it's that the OS hasn't finished any pending writes and thus the filesystem is corrupted. And SSD will do a lot more writes than traditional SD cards though, so it'll last longer, but you'll corrupt the fs just as fast as with an SD card.
same with the swappiness. because of the swap usage i made it utilize ram more aggressively rather than swap
It should not be swapping on a 1gb pi.
If it's swapping, something you added is causing extreme memory usage
And from what i can tell that webcam might be the culprit, i'm not sure why though, but it's the v4l2 driver that throws memory errors when shit hits the fan.
I will test it with a vanilla install, except adding the WebCam to see if it makes a difference
yeah the red flag is the swap usage in the first place, if that starts happening something is fucky, trying to facilitate more memory usage is just delaying the crash.
Something is leaking
Might be an idea running htop (or install btop) and seeing which process is eating up your memory
So I ended up checking /boot/config.txt and it looks like the gpu_mem value under RPi4 section did not match the value in raspi-config. I changed it to match and so far haven’t had a crash… so that seems to have solved the problem. Not sure why this would be as I only set gpu_mem through raspi_config, not directly i /boot/config.txt, so i’m thinking it originally came non-matching but the problem didn’t rear its head until i added a webcam…