Teppo Kurki - demo.signalk.org getting throttle...

demo.signalk.org getting throttled
47 Replies
Teppo Kurki
Teppo KurkiOP3w ago
i've seen it nonresponsive repeatedly. the reason seems to be fly.io throttling, this is what their monitoring show when it happens
No description
Teppo Kurki
Teppo KurkiOP3w ago
either something's changed in fly & our app setup or something external is using up our (free) cpu quota
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
@Teppo Kurki I just ran into this too trying to look up some docs
Teppo Kurki
Teppo KurkiOP2w ago
This is annoying. I don’t have any smoking gun for this If somebody has the inclination and skills to troubleshoot I can grant access. The server is just the vanilla demo server with the admin password set In the meanwhile: the docs are in your own server as well
Brandon Keepers
@Teppo Kurki I'm happy to take a look at it
Teppo Kurki
Teppo KurkiOP2w ago
Do you have a fly account i could invite? Fly.ii that is
Brandon Keepers
In the meanwhile: the docs are in your own server as well
Yes, but a lot of google results point to the demo server docs my fly account is: [email protected]
Teppo Kurki
Teppo KurkiOP2w ago
Invited
Brandon Keepers
Hmm, this seems relevant, but no answers here https://community.fly.io/t/high-cpu-usage-all-of-a-sudden/17811 These fly metrics dashboards are so slow @Teppo Kurki can you invite me to sentry too?
Teppo Kurki
Teppo KurkiOP2w ago
i don't think we have sentry enabled, let alone the app instrumented. or does it work out of the box by just enabling it? that was incorrect, A Sentry project named demo-signalk-org already exists
Brandon Keepers
I have not actually used Fly's sentry integration, so not sure.
Teppo Kurki
Teppo KurkiOP2w ago
any idea where to add you?
Brandon Keepers
$ flyctl ext sentry list
NAME ORG
demo-signalk-org signalk
docs-signalk-io signalk
$ flyctl ext sentry list
NAME ORG
demo-signalk-org signalk
docs-signalk-io signalk
Do you have access to the signalk org on sentry.io?
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
i have now, shows you liko so
No description
Teppo Kurki
Teppo KurkiOP2w ago
pressed the button @Mantas drop me a DM Mantas invited Sentry complains about being over Developer plan limit if 1 developer, but there's a 2 week grace period of Business
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
My earlier investigation showed us running out of cpu quota But we could also set up a hard limit for ws connections and not accept any when over the limit. Dropping existing connections is not an option, as they would probably just reconnect and not make situation any better We could also add a way for admin user to get connections’ ip addresses and allow denying access by ip address/range You can deploy a version with higher limits
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
U need to specify the version as arg to flyctl
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
Oh, good find!
Teppo Kurki
Teppo KurkiOP2w ago
You can deploy from your machine and once we have a good config PR that so, where are we with this? demo worked earlier today but did not a moment ago
Brandon Keepers
@Teppo Kurki Sorry, I got distracted and didn't come back to it. Should we just try increasing the hard_limit for now and see if that fixes it? https://github.com/SignalK/signalk-server/compare/master...bkeepers:signalk-server:demo-hard-limit
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
Hmm, ok. So looks like that might not be the only issue? Demo is non-responsive for me right now
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
ok, responding for me now. It wasn't for me, but could also have been my connection
Teppo Kurki
Teppo KurkiOP2w ago
responds, but sluggishly on first load and now with reload stuck on some http requests pending
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
@Mantas do those graphs take a really long time for you to load? Every graph in the fly-metrics page takes like 20-30 seconds for me
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
Weird. The fly-metrics and demo server behave the same for me, where they take 20-30 seconds on first load, and then are fine makes me wonder if there's a networking issue with fly somewhere I was on a cell connection, but just turned on our starlink, so my network connection is fast now, but those graphs still take forever to laod
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
any news? seems to be still in bad shape
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
is it swapping? that would explain IO or is there even swap on fly...
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
The server has been running ok for quite a long time, so something has changed Easy enough to deploy an older server there to check if the change is in the image
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP2w ago
can u do that?
Unknown User
Unknown User2w ago
Message Not Public
Sign In & Join Server To View
Brandon Keepers
Demo is unresponsive again
Unknown User
Unknown User7d ago
Message Not Public
Sign In & Join Server To View
Teppo Kurki
Teppo KurkiOP7d ago
I already tried that, no dice, but easy enough to try agin I think we shuld try a version much older, like months ago Add an exit call every 10 minutes This is rather annoying, as there is no smoking gun and this is work away from doing something useful
Unknown User
Unknown User3d ago
Message Not Public
Sign In & Join Server To View

Did you find this page helpful?