tonyfrancisv
tonyfrancisv
PostsComments
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
yeah i mean that should make it less likely to happen since there are fewer web workers. but ideally this should be dynamic so we can saturate the resources available
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
well yeah, that's a separate question
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
nothing set explicitly atm
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
im looking for where replicas are in the dashboard
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
so there is a theoretical max qps we can handle. good to know
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
oh woah, did not realize that
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
wait so you just vertically scale a fixed number of containers?
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
ahhhhh okay
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
how do you decide when to spin up another container
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
yeah, that's what numpy uses under the hood for threading. setting it to 1 disables multithreading in the c parts of numpy. bad for perfomance but something that ppl sometimes choose to do to prevent peaky behavior in their autoscaler
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
how do you decide to scale? is it % cpu usage?
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
assuming its not going to break anything on the railway side of things
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
I'm just gonna manually set that var and see if it resolves the issue
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
yeah, tbh its opaque on the usage side bc libs like OpenBLAS will just look for RLIMIT_NPROC
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
all of the services in the backend spin up on boot. some of them probably have libs that have their own threading under the hood. probably some race condition depending on the order of them actually executing/connecting
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
this issue is itermittent, most of the time it boots fine
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
yeah anything in particular you're looking for
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
yeah, vs not being set at all
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
usually set at a system level in linux systems
112 replies
RRailway
Created by justinw on 6/6/2023 in #✋｜help
Infinite worker boot/crash loop
any reason why RLIMIT_NPROC is -1 ? instead of being set to the number of threads allowed
112 replies