54 Replies
i've seen it nonresponsive repeatedly. the reason seems to be fly.io throttling, this is what their monitoring show when it happens
either something's changed in fly & our app setup or something external is using up our (free) cpu quota
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
@Teppo Kurki I just ran into this too trying to look up some docs
This is annoying. I don’t have any smoking gun for this
If somebody has the inclination and skills to troubleshoot I can grant access. The server is just the vanilla demo server with the admin password set
In the meanwhile: the docs are in your own server as well
@Teppo Kurki I'm happy to take a look at it
Do you have a fly account i could invite?
Fly.ii that is
In the meanwhile: the docs are in your own server as wellYes, but a lot of google results point to the demo server docs my fly account is: [email protected]
Invited
Hmm, this seems relevant, but no answers here https://community.fly.io/t/high-cpu-usage-all-of-a-sudden/17811
These fly metrics dashboards are so slow
@Teppo Kurki can you invite me to sentry too?
i don't think we have sentry enabled, let alone the app instrumented. or does it work out of the box by just enabling it?
that was incorrect,
A Sentry project named demo-signalk-org already existsI have not actually used Fly's sentry integration, so not sure.
any idea where to add you?
Do you have access to the signalk org on sentry.io?
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
i have now, shows you liko so
pressed the button
@Mantas drop me a DM
Mantas invited
Sentry complains about being over Developer plan limit if 1 developer, but there's a 2 week grace period of Business
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
My earlier investigation showed us running out of cpu quota
But we could also set up a hard limit for ws connections and not accept any when over the limit. Dropping existing connections is not an option, as they would probably just reconnect and not make situation any better
We could also add a way for admin user to get connections’ ip addresses and allow denying access by ip address/range
You can deploy a version with higher limits
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
U need to specify the version as arg to flyctl
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
Oh, good find!
You can deploy from your machine and once we have a good config PR that
so, where are we with this?
demo worked earlier today but did not a moment ago
@Teppo Kurki Sorry, I got distracted and didn't come back to it. Should we just try increasing the hard_limit for now and see if that fixes it? https://github.com/SignalK/signalk-server/compare/master...bkeepers:signalk-server:demo-hard-limit
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
Hmm, ok. So looks like that might not be the only issue? Demo is non-responsive for me right now
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
ok, responding for me now. It wasn't for me, but could also have been my connection
responds, but sluggishly on first load and now with reload stuck on some http requests pending
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
@Mantas do those graphs take a really long time for you to load? Every graph in the fly-metrics page takes like 20-30 seconds for me
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
Weird. The fly-metrics and demo server behave the same for me, where they take 20-30 seconds on first load, and then are fine
makes me wonder if there's a networking issue with fly somewhere
I was on a cell connection, but just turned on our starlink, so my network connection is fast now, but those graphs still take forever to laod
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
any news? seems to be still in bad shape
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
is it swapping? that would explain IO
or is there even swap on fly...
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
The server has been running ok for quite a long time, so something has changed
Easy enough to deploy an older server there to check if the change is in the image
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
can u do that?
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
Demo is unresponsive again
Unknown User•5w ago
Message Not Public
Sign In & Join Server To View
I already tried that, no dice, but easy enough to try agin
I think we shuld try a version much older, like months ago
Add an exit call every 10 minutes
This is rather annoying, as there is no smoking gun and this is work away from doing something useful
Unknown User•4w ago
Message Not Public
Sign In & Join Server To View
Hello! Your “demo-signalk-org” application hosted on Fly.io crashed because it ran out of memory.
Unknown User•4w ago
Message Not Public
Sign In & Join Server To View
I am not against adding resources
But at the same time i’d want to understand why this is happening
The server has no safeguards against abuse
Because that is not what it was built for
Spending time on building the safeguards is not really providing any value to the users on their systems
We could add simple brute force safeguards like disconnecting all ws connections every x seconds or not allowing anything but the simplest subscriptions
@Mantas seems like it's been pretty stable. Is v2.13.2 still deployed?
Unknown User•3w ago
Message Not Public
Sign In & Join Server To View
I don’t think this has anything to do with sk version, as it oom’d several times with the older version
And we don’t want to be stuck on an old one
So maybe deploy latest and see what happens?
Unknown User•3w ago
Message Not Public
Sign In & Join Server To View