Increased Memory Usage in V2
It appears as though the V2 runtime is causing an increase in memory usage for my project. I am unsure if this is due to it being a .net project, or if it is something V2 will generally struggle with.
Reference Points from the image are the newest 2 deploys. The first being on V2, and the second on legacy. The spikes in both are due to roll-over issues where the previous deployment didn't exit yet and should be ignored.
Most pressing I think is the almost 100MB difference in initial memory consumption, which is very odd considering this app always has the same initial allocation.
In a previous deployment that I since removed, I also saw the single-instance memory usage grow significantly higher than 200MB, which is the hard limit I set the app to with DOTNET_GCHeapHardLimit.
76 Replies
Project ID:
fd3c34f8-2314-4d8d-8f79-2ed37386c50dfd3c34f8-2314-4d8d-8f79-2ed37386c50d
I have also tested this locally using docker. The initial heap size should be around 100MB, not teetering near 200MB
So for clarity the more left side of the graph is the v2 runtime and the right portion of the graph is the legacy runtime?
Correct
are you building with a dockerfile?
yes
are you running it locally with docker or podman?
dotnet 8 deployment with nixpacks currently does not work because the nixpkgs reference is too old
I am running locally with Rancher Desktop, which uses moby as the runtime
legacy is docker and V2 is podman
so there definitely could be some discrepancies there, but if so, it could be out of railways control
I see
I'll see if I can reproduce, and either way I'll bring this up to the team
Deploying .net applications is a bit iffy anyway, I have to be able to rely on setting the GC heap limits or the app will grow to the container limit
which on railway is 8GB
yeah same issue with Java apps too
asp.net uses a GC that pefers allocating ram over doing GC and its currently very stubborn about it
being able to set limits on container size would be a great feature
can you use a different GC or am I thinking of a different language?
I have tried setting it to workstation gc and using the new DATAS dynamic gc
but it seems to not make much of a difference
this might be a bug in the asp.net docker base image though, I saw a few reports indicating this
what's the image based on? I've heard of people having better memory usage just by switching to an alpine image, but that was node so probably not applicable but maybe worth trying?
I could try that, the regular dotnet images are all bullseye iirc
couldn't hurt, I've seen stranger things be a solution
I'll still be trying to reproduce and will report to the team
ah, its bookworm, not bullseye, but still debian
haha same difference
legacy and v2 runtime both report the same mem, 499MB on legacy and 496MB on v2
interesting, this definitely warrants more testing then
oh speaking of nixpacks, not to hijack the issue but it still installs the preview version of dotnet 8.0 when using the c# provider
yeah that provider along with nixpacks in general needs some love
while I don't mind writing dockerfiles (my day job is as a cloud consultant) nixpacks is one of my favorite features of railway
nixpacks is nice when it works the way you want it to, but it doesn't create the best dockerfiles or the smallest final images, so I've never used it for any of my projects
that is true
the dotnet file I am using is utilizing the two-stage-build pattern, so the output docker container is as small as can be
you can do something similar with nixpacks using the final image directive but it's still a pain compared to writing a Dockerfile to do it
true, but for small hacked together weekend projects it is very nice to be able to just deploy them
I agree, hopefully one of these quarters they will set aside some time to nail down some of the outdated providers and the flaws with nixpacks
It would be nice if the nixpkgs ref would get more regular updates. The current snapshot is from september :X
https://github.com/railwayapp/nixpacks/blob/9fae921e719f540965347b662571311273353a1c/src/nixpacks/nix/mod.rs#L11
it's quite scary when they update that, we get a wave of build failed help threads
the repo has a bunch of test apps, but it's not enough
I am guessing its a lot of apps that don't specify their target sdk version anywhere?
stuff like that, or I think last time I meant that nixpacks inadvertently dropped support for php 8.0
Especially node, version pinning isn't exactly common practise there
I can testify to that
dotnet has its global.json that is generated by the cli
oh my, yeah php versions disappearing is not good
for an interpreter this widely used they really love breaking changes
that one was because php 8.0 just wasn't a package in the archive version it was updated to
I have quite a few stories from my previous job with a managed hosting company that do e-learning
most e-learning software is really old, so its all written in php
anyone who wants older versions of php either use a Dockerfile or pin nixpacks to a really old version
Adventures like "no, we can't update to Ubuntu 18.04 because [software] needs [old php version]" 2 months before the EOL of 18.04
the fun world of software development and backend maintenance
its kinda awful hahaha
that's why I have only done personal projects haha
it would have been fine if my ex-employer didn't also think docker was the plague
everything had to be full vm boxes
sounds costly
it really was
just before I left they were starting to pivot to private-clouds using openstack
if only they knew about railway
I heard some rumors that they are now migrating a lot of things to an HA K8S cluster
IF ONLY!
I did a redeploy with alpine under both legacy and v2 again
still seeing about 30MB in difference
it's not as bad as the 90 I was getting before though
oddly, in docker, the alpine image made almost no difference at all
makes me think this could be a runtime oddity
I'll keep an eye on the memory for a few hours and see if it obeys the GC limit I set now
sounds good, I've sent this thread in a thread I have with the team thats for the v2 dents
fingers crossed, but I think you might be right with this being a podman issue
it might be
I've seen a user report increased network usage on the v2 runtime, but I also couldn't reproduce that
or maybe the v2 runtime now counts shm space as memory
it could be, yeah
if the ram reporting is done using the runtime's monitoring functions, the reported values could differ depending on implementation too
indeed
shm space is not counted as memory
At least the deployment is now behaving on alpine
The ram even went down! So gc is 100% running
gotta love alpine
happy to report that the container's ram usage appears to be what it should be now
although the metrics reporter seems to not quiiiite work yet
metrics reporter?
the deployment is definitely there
but the last timestamp is from 4 hours ago
on the 1d metric
and the 1h metric does not exist
this is on V2
considering the deployment exists, this seems more like a UI issue than a backend issue though
what time is where you are?
6am
Someone should take a look at how my app can be running with no 1h metric available though
The deployment is a few hours old at this point
your CEST right? can you send a link to your service?
Yea
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
!t
New reply sent from Help Station thread:
This thread has been escalated to the Railway team.You're seeing this because this thread has been automatically linked to the Help Station thread.
can you login in there for good measure?
I have logged in
perfect!
New reply sent from Help Station thread:
Sorry trying to read through this but unsure exactly where you all landed. Is the issue that in the environment labeled V2, services deployed using the V2 runtime, metrics graphs indicate increased usage? (Similar to the other one we're looking into @ Brody?)You're seeing this because this thread has been automatically linked to the Help Station thread.
if I understand correctly, memory is fine now, but the graphs have the date scale wrong