Urgent! Production DB is showing "could not resize shared memory segment" error
Project ID: 3747f720-8c9c-4d81-9b13-1e420ceabd6c
We have a Postgres service with
Max 32GB. I am getting the following error from postgreswhen querying data from our server.
I found a post from the Internet that said I need to add the following environment variable in Railway which is to give 30GB. This has worked in other services before.
As you can see I restarted the service and that's why you see the memory suddenly dropping.
I performed the query again from the server and I still get the same error. It seems like there is not enough memory at the OS level still?
Solution:Jump to solution
16777216 bytes is only 16MB.
Attempting to set the SHM size to 30GB is not something we are okay with.
500MB for the SHM size would be more than enough....
27 Replies
Project ID:
3747f720-8c9c-4d81-9b13-1e420ceabd6cThere are two Postgres DB services as you can see. The old one is just called
genie-db and the new one I am migrating to is called genie-new-db. The old one genie-db has no problem allocating memory and works fine but as soon as I use the genie-new-db , I run into the memory allocation issue.
- They both have RAILWAY_SHM_SIZE_BYTES environment variable set.
- The old genie-db is version 15.5 with timescale and PostGIS extensions by default
- The new genie-new-db is version 16.3 without timescale and PostGIS extensionsSolution
16777216 bytes is only 16MB.
Attempting to set the SHM size to 30GB is not something we are okay with.
500MB for the SHM size would be more than enough.
536870912
Only once you have changed all your shm size variables to 500MB you may switch back to the legacy runtime on your databases for the variable to take effect.
@Brody I set the env variable according to what you told me
and then I
Redeployed the service. It didn't fix the issue. Running into the exact same problem.
As mentioned before, in the same project, I have a DB called genie-db which is the old one and this one ran into exactly the same problems I am having with the genie-new-db. For that one I set it to 30GB
and it fixed the problem and with exactly the same data and SQL query, it is serving up the results without running into a memory problem.
As mentioned before, the only difference between the genie-db and genie-new-db on my side really is the Postgres template I used. Is the new Postgres 16.3 template using the RAILWAY_SHM_SIZE_BYTES properly?
As mentioned before, please do not set the shm size to 30GB, that is not something we can allow.
Ok, I can also set
RAILWAY_SHM_SIZE_BYTES="536870912" for the current DB genie-db service. But I really need the genie-new-db working. It is a blocker for us move forward in production.Please go ahead and do that
On second thought I dont' want to do that to
genie-db (the current PROD) because it's working well and serving live traffic right now. I dont' want to disturb the production.
genie-new-db is the one that needs to work and that's the one that already has
as you instructed and as I mentioned here (https://discord.com/channels/713503345364697088/1274291514268188811/1274423235470954586) It's not working stillA SHM size of 30gb is not something we can allow, it is far more than it needs to be set to
please bump that down to 500MB
I don't think you are understanding what I am saying. I said that whether you say it's not something you support or not, I made that change many months ago and that particular DB
genie-db is our current production DB and its memory issue got solved with the 30GB I set and it works. I'm not gonna bump that down to 500MB when it's serving up live users right now in production.
What I am saying is we are preparing another new DB to stop using genie-db. I need to get the new DB called genie-new-db working. And this is the one that ran into exact same memory issue and I set that one to
as you instructed and it's not working. This is the 3rd time I am mentioning this to you.Please check my first message, you need to be on the legacy runtime.
Yes and that's why I set the
RAILWAY_SHM_SIZE_BYTES to the service genie-new-db and it's not working. Are you saying both genie-new-db and genie-db services need to have the same RAILWAY_SHM_SIZE_BYTES="536870912" for this to take an effect?Only once you have changed all your shm size variables to 500MB you may switch back to the legacy runtime on your databases for the variable to take effect.
Ah ok. My apologies. I never imagined that an environment variable from one service would affect the other. What an odd user experience that is...
It doesn't.
You just said that in order for the
RAILWAY_SHM_SIZE_BYTES="536870912" to take an effect in genie-new-db service, I need to also set the same variable RAILWAY_SHM_SIZE_BYTES="536870912" to the genie-db service. That's how I understood.kindly finish reading the second half of this sentence
I am really confused now. Do you understand what I've done so far? Because that's exactly what I have already done for
genie-new-db and I tested that that didn't solve the problem.Looks like you have yet to change it to legacy
I had no idea when you said
Runtime I was supposed to go to the Settings to change that. You never mentioned that that is a settings in Railway... I just understood as some runtime you have that automatically goes into Legacy mode.
My bad. I will change that to Legacy now and redeploy.
@Brody Ok that fixed the issue. Thank you. Everything is working now.awsome
Sorry, had no idea that settings even existed... we never really use Railway because we set it up long time ago and nothing broke until now.
no worries at all
What is the new
V2 way of specifying how much shared memory can be allocated? Should I be using that instead ?at this time the v2 runtime does not support setting a shm size
Ok, so we have no choice. Thanks.
by the time the legacy runtime is deprecated the v2 runtime will have support for the shm size, so no worries there