2024-07-27-RatOS-2.1.0-RC2 issues with moonraker
As soon as I do the updates to this via Update Manager moonraker stops working and can't access GUI.
183 Replies
did you see https://discord.com/channels/582187371529764864/844364874216898562/1319098058016821349 ? There is a specific order to updating right now
Are referring to this? And is so how do I restart moonraker? After updating the configurator, you will need to restart moonraker, refresh the update manager and then update klipper, since we've changed the pinned commit as we're now using the new klipper resonance generator.
Just reboot the pi. That'll get everything started fresh. Then re run the updater
This is what I am get now. Can't get RatOS Configuration repository
Huh. You might have to ssh in and fix the repo.
Fixed that now after reboot get this:
run sudo service klipper restart and still have error
Restart moonraker the same way?
yup. Logs says klipper dirty
when do git pull in klipper doesn't work
Sounds like a pinned version mismatch. You might have to pull a specific tag in klipper and moonraker. I don't know where the pinned versions would be, but probably in the configurator somewhere
So not sure how to fix this. I have a new 500 IDEX I can get going now.
Reflash the card?
pi@RatOS:~ $ cd ~/klipper
pi@RatOS:~/klipper $ git describe --always --tags --long --dirty
v0.12.0-316-g96cceed23-dirty
pi@RatOS:~/klipper $
If you git reset --hard, will the dirty go away?
pi@RatOS:~/klipper $ git reset --hard
error: update_ref failed for ref 'ORIG_HEAD': cannot lock ref 'ORIG_HEAD': unable to resolve reference 'ORIG_HEAD': reference broken
HEAD is now at 2425a7463 virtual_sdcard: Define a default for on_gcode_error
pi@RatOS:~/klipper $
What branches are available? Do git branch (might need --list)
pi@RatOS:~ $ cd klipper
pi@RatOS:~/klipper $ git branch --list
* master
pi@RatOS:~/klipper $
Try with --remotes
pi@RatOS:~/klipper $ git branch --remotes
origin/HEAD -> origin/master
origin/dependabot/pip/scripts/jinja2-3.1.5
origin/gh-pages
origin/master
origin/work-contribute-20231030
origin/work-heaters-20240804
origin/work-nosmoothtemp-20230623
origin/work-vermsg-20240119
pi@RatOS:~/klipper $
git checkout master && git pull --all && git reset --hard origin/master,
pi@RatOS:~/klipper $ git checkout master && git pull --all && git reset --hard origin/master
Already on 'master'
Your branch is behind 'origin/master' by 245 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Fetching origin
fatal: update_ref failed for ref 'ORIG_HEAD': cannot lock ref 'ORIG_HEAD': unable to resolve reference 'ORIG_HEAD': reference broken
pi@RatOS:~/klipper $
Sounds like something broken in .git/config
Something going on. At point can't start commissioning printer.
Yeah, wiping and reflashing the card might be best. Then
Run update
Update configurator
Reboot
Update klipper
Update all
on the 2.1.0-RC2?
Right
going to reflash sd card now
when say run update which do you mean?
Click the circle to start the update manager in mainsail/machine
k. I will report back soon. Writing image now.
Fresh install and connected to WIFI, After clicking update get this:
Wow ok that's some shit.
@Helge Keck any idea why a freshly flashed rc2 card would do this?
I am trying the update to system from putty to see if helps. 114 system updates
After that update:
Try running
pi@RatOS:~ $ ratos doctor
error: unknown command 'doctor'
pi@RatOS:~ $
At the point dunno what else to do.
So to recap, you did:
* a fresh install on a SD that is new
* Followed the instructions from the announcement and updated in the correct order as the first thing you did
?
yes. I have done that all. I followed @CrashTestCharlie / VC4 400h suggestions.
TBF I was on rc3 when I updated yesterday.
I have been trying for few days now and can't get pass this. Fresh install of RC2 and after connect to wifi all happens.
Anything else we can try? Ready to return printer
sorry
i'm not sure that was the correct order, plus he had you mucking around in the command line.
If it were me I would do
* do a fresh install of ratos 2.1 rc2
* go to the mainsail interface and click the green icon to search for updates
* update ratos first
* then update ratos-configurator
* then restart moonraker - restart it manually by clicking the power icon in the top right corner, then the refresh icon next to "Moonraker" under "Service controls"
* after restarting moonraker, click the green icon to check for updates again
* then click "Update All Components"
basically following https://os.ratrig.com/docs/installation#updating-ratos
I will try all this again. If doesn't work where do I go from here?
give us the errors you are seeing then
give me about 30 mins to get all reimages etc. Should I do updates first before connecting to wifi?
you can't do updates without internet access
Starting reimage now
1. Flash image
2. Connect to hotspot
3. Setup wifi through configurator (and let it reboot)
4. Go to /config
5. Refresh update manager
6. at this point nothing should be invalid, all green. If that's not the case stop here and post your moonraker.log.
7. Update only the RatOS package (NOT configurator).
8. Update RatOS configurator (this takes a while, if the mainsail output of the update process hangs, let it sit for 10 minutes and then refresh the page, ctrl + shift + r)
9. Restart moonraker service (click power icon in the top right of mainsail, click the refresh button next to moonraker under "service controls")
10. Reload mainsail (ctrl + shift +r)
11. Refresh update manager
12. "RatOS" should be gone from the list. Everything should be green and have several updates except RatOS-configurator (ignore the klipper errors), if it's not, stop here and post your moonraker.log
13. Click update all.
14. Done.
Image flashed. Going to get on wifi.
Clicked refresh and waiting.
still spinning. how long should it take?
a minute or two
After clicking it.
yeah this is not right, moonraker.log
this looks like a network issue to me
2025-01-01 22:27:53,482 [server.py:add_log_rollover_item()] - CPU Throttled Flags: ['Under-Voltage Detected', 'Currently Throttled', 'Previously Under-Volted', 'Previously Throttled']
Your pi is not properly powered (this makes everything very slow and can cause lots of issues)
I am using it on desktop right now try to get going.
You need a proper PSU
I am switching it now
to power brick
Same issue
Also looks like it doesn't have internet
it does
Might just be things breaking apart because of all the power issues
Yes the damage was already done
Also noway you booted it and refreshed the update manager in under a minute ๐
It's likely that you might have to reflash though just to be sure, multiple things are erroring out because of the power issues (it's likely to have corrupted things).
trying again with other sd card I had so can switch fast
Aight, post a new moonraker.log if things are not all green after refreshing the update manager the first time.
Trying again now
when click green refresh doesn't do anything.
There's a "cooldown" on the refresh button built in to mainsail so you don't spam moonraker, which spams github.
It should work again in a sec
Still seems like you have network issues though
Can you try this:
2025-01-01 23:02:38,558 [server.py:add_log_rollover_item()] - Unsafe Shutdown Count: 1shut the pi down properly before removing power or you risk corrupting your file system (causing similar problems to the ones you're having)
ssh [email protected]
SSH is enabled and the default password for the 'pi' user has not been changed.
This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.
pi@RatOS:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 2c:cf:67:5e:b9:1e brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 2c:cf:67:5e:b9:1f brd ff:ff:ff:ff:ff:ff
inet 10.0.1.194/24 brd 10.0.1.255 scope global dynamic noprefixroute wlan0
valid_lft 43123sec preferred_lft 37723sec
inet6 fe80::8e69:2244:e506:40eb/64 scope link
valid_lft forever preferred_lft forever
A 10.x.x.x address, is that a work network or home network setup?
yes
home
Interesting, usually by convention home networks use 192.x.x.x
I am 10.0.1 .x since run servers at house.
500tb Plex server here
And you can confirm with ping and traceroute that the pi has Internet connection?
yup I can
moonraker keeps failing because
ip -det -json address fails.
that's why you can't refreshSo not sure what to do, My Vorons on Klipper are working
"Klipper" doesn't say anything about the OS.. RatOS is a fork of MainsailOS
You can run
ip -json -det address
And see why it fails
i would've thought ip a would fail too, but guess notpi@RatOS:~ $ ip -json -det address
[{"ifindex":1,"ifname":"lo","flags":["LOOPBACK","UP","LOWER_UP"],"mtu":65536,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"loopback","address":"00:00:00:00:00:00","broadcast":"00:00:00:00:00:00","promiscuity":0,"min_mtu":0,"max_mtu":0,"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"127.0.0.1","prefixlen":8,"scope":"host","label":"lo","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"::1","prefixlen":128,"scope":"host","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":2,"ifname":"eth0","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","group":"default","txqlen":1000,"link_type":"ether","address":"2c:cf:67:5e:b9:1e","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":1500,"num_tx_queues":5,"num_rx_queues":5,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[]},{"ifindex":3,"ifname":"wlan0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"pfifo_fast","operstate":"UP","group":"default","txqlen":1000,"link_type":"ether","address":"2c:cf:67:5e:b9:1f","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":1500,"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"10.0.1.194","prefixlen":24,"broadcast":"10.0.1.255","scope":"global","dynamic":true,"noprefixroute":true,"label":"wlan0","valid_life_time":43176,"preferred_life_time":37776},{"family":"inet6","local":"fe80::8e69:2244:e506:40eb","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]}]
pi@RatOS:~ $
yeah that seems normal
@rmilyard try this:
then go back to mainsail and click the refresh button in the update manager panel
doesn't spin
new log
2025-01-02 00:21:34,567 [update_manager.py:_handle_status_request()] - update_manager: bypassing refresh request 2025-01-02 00:21:36,443 [update_manager.py:_handle_status_request()] - update_manager: bypassing refresh requestAre you hammering that button?
nope
It seems to barely be started and then it bypasses the check
very odd..
Aight fuck it, let's do it manually.. Something is up with your particularly network that moonraker doesn't like (or so it seems), RatOS doesn't have anything to do with this unfortunately, so not much i can do.
So, here we go:
If that comes back happy you can go through the configurator and setup your printer.
so run this? It should do updates for me?
it updates the two important pieces
it is running now. Not fast but seems to be running
it takes a while, it's a big repo merge with several thousand commits
might also want to do
it is getting errors running. Not sure if ok
it's not
wait what did you do here
Did you paste every single line at once?
that is output in putty
yeah so you pasted the entire thing at once
And not line by line
which is a problem
do line by line?
no wait
Do that :this:
pi@RatOS:~ $ cd ~/ratos-configurator
pi@RatOS:~/ratos-configurator $ git fetch
pi@RatOS:~/ratos-configurator $ git reset --hard origin/v2.1.x-deployment
fatal: Unable to create '/home/pi/ratos-configurator/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
pi@RatOS:~/ratos-configurator $
๐คจ
Prolly because moonraker died at some point
Aight then...
pi@RatOS:~ $ cd ~
rm -rf ratos-configurator
git clone [email protected]:Rat-OS/RatOS-configurator ratos-configurator
cd ratos-configurator
git checkout v2.1.x-deployment
ratos doctor
Cloning into 'ratos-configurator'...
The authenticity of host 'github.com (140.82.113.3)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,140.82.113.3' (ECDSA) to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
-bash: cd: ratos-configurator: No such file or directory
fatal: not a git repository (or any of the parent directories): .git
-bash: ratos: command not found
pi@RatOS:~ $
Oh ffs, sorry. sec
There
running this now
pi@RatOS:~ $ git clone https://github.com/Rat-OS/RatOS-configurator.git ratos-configurator
cd ratos-configurator
git checkout v2.1.x-deployment
ratos doctor
Cloning into 'ratos-configurator'...
remote: Enumerating objects: 52543, done.
remote: Counting objects: 100% (3984/3984), done.
remote: Compressing objects: 100% (904/904), done.
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
error: 5541 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
-bash: cd: ratos-configurator: No such file or directory
fatal: not a git repository (or any of the parent directories): .git
-bash: ratos: command not found
pi@RatOS:~ $
GitHub
GitHub - Rat-OS/RatOS-configurator: The RatOS Configurator
The RatOS Configurator. Contribute to Rat-OS/RatOS-configurator development by creating an account on GitHub.
Yeah this is network bullshit
I very much doubt your vorons will update cleanly either.
they just did
It's essentially the same system
on the same wifi?
yup. Here is one.
Can you show me the moonraker.log from this please
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8) error: 5541 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failedThis here means your network is fucking over your connection There's nothing i can do about that
I can't see any updates actually happening in here
So if I do refresh again. What all should I do? There is so much here now. I don't want a 3k paperweight.
Your issue is your network
Your network is creating a paper weight
I tried on my backup network also and same. This is Wyerd and also have Cox.
Best i can do is tell you to wait till RC4.
eta?
within the next couple of days
k. I can keep just building stuff. I need to clean up wires etc
But you won't be able to update that either before you fix your network, but at least it'll contain up to date packages.
not sure what can do with network. Nothing else in house or other printers having issues.
Well it's dropping connections to your new pi
ยฏ\_(ใ)_/ยฏ
I might take to network closet and try is hard wired.
Oh wait
There's one thing we can try
And paste the lines one at a time
I gotta go to bed. Let me know how it goes with HTTP1
kk
So today looks like might worked. What version should stuff be on ?
Here's what I have for my Vcore4
Well I think got it all updated. But when click on CONFIGURATOR get bad gateway
Anyone have ideas?
are you using
ratos.local or the IP address to try and access?IP address.
same both ways
I am at point wish didn't start this build. Ready to get OS installed but can't.
if you SSH into the PI, can you check if nginx is running?
how?
seeing these errors now. [update_manager]: Failed to load extension ratos-configurator: Invalid path for option
install_script in section [update_manager ratos-configurator]: Path /home/pi/ratos-configurator/scripts/setup.sh does not exist
Moonraker warning
Unparsed config option 'primary_branch: v2.1.x-deployment' detected in section [update_manager ratos-configurator]. This may be an option no longer available or could be the result of a module that failed to load. In the future this will result in a startup error.
Moonraker warning
Unparsed config option 'origin: https://github.com/Rat-OS/RatOS-configurator.git' detected in section [update_manager ratos-configurator]. This may be an option no longer available or could be the result of a module that failed to load. In the future this will result in a startup error.
Moonraker warning
Unparsed config option 'is_system_service: True' detected in section [update_manager ratos-configurator]. This may be an option no longer available or could be the result of a module that failed to load. In the future this will result in a startup error.
Moonraker warning
Unparsed config option 'managed_services:
ratos-configurator' detected in section [update_manager ratos-configurator]. This may be an option no longer available or could be the result of a module that failed to load. In the future this will result in a startup error.GitHub
GitHub - Rat-OS/RatOS-configurator: The RatOS Configurator
The RatOS Configurator. Contribute to Rat-OS/RatOS-configurator development by creating an account on GitHub.
what's the output from:
sudo systemctl status ratos-configurator.servicepi@RatOS:~ $ sudo systemctl status ratos-configurator.service
[sudo] password for pi:
โ ratos-configurator.service - API Server for Klipper
Loaded: loaded (/etc/systemd/system/ratos-configurator.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2025-01-02 18:11:23 GMT; 1s ago
Process: 4519 ExecStart=pnpm start (code=exited, status=1/FAILURE)
Main PID: 4519 (code=exited, status=1/FAILURE)
CPU: 2.012s
pi@RatOS:~ $
Pulling this from thin air, but I wonder if the ratos doctor command from earlier would A: work now and B: help with this now that some progress has been made in terms of updates
which one? This post getting pretty big.
up here, I dont know how to attach text the way they did.
ssh [email protected]
ratos doctor
fyi you use the backticks: `
https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline
pi@RatOS:~ $ git clone https://github.com/Rat-OS/RatOS-configurator.git ratos-configurator
cd ratos-configurator
git checkout v2.1.x-deployment
ratos doctor
Cloning into 'ratos-configurator'...
remote: Enumerating objects: 52543, done.
remote: Counting objects: 100% (3984/3984), done.
remote: Compressing objects: 100% (907/907), done.
error: 5869 bytes of body are still expected1 MiB | 713.00 KiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
-bash: cd: ratos-configurator: No such file or directory
fatal: not a git repository (or any of the parent directories): .git
-bash: ratos: command not found
GitHub
GitHub - Rat-OS/RatOS-configurator: The RatOS Configurator
The RatOS Configurator. Contribute to Rat-OS/RatOS-configurator development by creating an account on GitHub.
no, don't try and clone the repo again. Just run
ratos doctor
but looks like you have messed up the configurator nowwhere is file?
ratos-configurator ????? I don't have folder after updates
It looks like he's pasting in the commands as one big block. There should be a
pi@RatOS before each commandthe ratos doctor not found.
no there shouldn't
I don't have ratos-configurator folder after I do the update
correct, that is how it should be. Don't try and clone it
as per the announcement they are getting merged
I am reimaging. What stepped should I do now? I am wiping this all and starting with RC2 fresh
since your network keeps causing issues, I would stay on RC2 for now and just work around the known bugs
I'm fully updated, and this is my home dir:
My network seems fine. I can access this all from other devices etc. I contacted ISP and no issues and nothing is blocked.
your network is not fine as is evidenced by all the troubles you are having contacting gitlab for updates and moonraker errors
but I can from other Klipper devices etc. I can update my vorons
now I have it hard wired
try hard wired, also may try a hotspot from your phone if possible (while phone is disconnected from wifi and on cellular)? at least eliminate your network being the issue, may go back to the pi being underpowered or a faulty pi.
Can you send us a pic real quick of your test setup? just for kicks and giggles.
Thanks!
if I could suggest running the pi and the octopus off the "final" power supply setup. I know the octopus will not flash without the 24v input, and the pi is uber sensitive to brown-outs.
I know the pi throws whack errors when undervolted, or browning out, and since it was mentioned before here where it logged undervoltage.
Its hardwired and on PI brick so not undervolt
my LS is the same
without updating, can you get to mainsail and the http://ratos.local/configure page? I mean if you can get you boards flashed, you'll be ready to move on.
I can't get to the configure. Get bad gateway
if do fresh installl when get the MCUs get version mismatch
it is actually flashed correctly, can you force it to continue on?
Is there somewhere to download a known working/updated image?
that is the RC4 Mikel talked about that will be ready in a few days
Am I correct to assume he is the dev?
Nope, -83 (and i know this is confusing) is pre-update.
Me? yes.
Cool. Well thinking put this away till next release. I am not getting anywhere if this.
Prolly best
For hell of it tried on 1gb, 4gb and 8gb pi's I have. All the same
just to try rule out the pi
It's not a pi issue, it's not a RatOS issue (well the current very annoying update path is a RatOS issue, but the errors you're getting is not RatOS's fault), it's a network issue.
You probably have a proxy, like pihole or something similar that is causing this. Or the wifi connection is unstable.
Nothing running. Also when do it hardwired the same. But yet I can get huge PI updates
116 updates
you're likely only seeing it because the size of the repository is bigger than what you normally use.
those aren't huge ๐
k. going to stop with this madness for now.
For now can use base RC2 and flash MCU's and forget about version mismatch?
And just skip updating?
Yeah, just move on, there's still a lot of building and wiring to do. Then grab the rc4 in about a week.
So I stumbled upon this thread because the exact same thing kept happening to me during installation (Bad Gateway 502 for configurator). Following the console commands above I got the configurator running again, but the wizards stops here. I cannot deselect a toolboard(and I am not using one either), the "skip" button just is missing
So guys today I got a new PNY 128gb SD card (overkill but was for something else). I did fresh RC2 and and followed the update path. This is where I am at.
But still get bad gateway when click on CONFIGURATOR.
this is good
Should get things going again. I'm assuming you ran hard recovery on the configurator which makes moonraker delete everything.
ratos doctor fixes that.running it now
that worked. So I think I am not running rc3
You are ๐
all's good now
๐
i'm guessing the reason the network requests failed on the old card is because it wasn't large enough, let me guess, 8gb?
No was 32gb. I am thinking maybe to slow to write? Dunno
Slow write speeds shouldn't cause it to fail. Bad sectors likely would though.
Well, at least you found the issue ๐