Garbage Collection in a container?
I have a strange memory consumption problem and I’m not sure how to troubleshoot this.
I have Python web app makes a lot of external API calls, processes the data and sends it to a Postgres DB. The longer the program runs, the more container memory it consumes. When the container runs out of memory, something happens and the resources get freed up. Drops back to around 300MB.
I have confirmed the data processing is what’s consuming memory. I set up a staging environment from my dev branch on Railway, which does NOT make the API calls, and memory consumption is stable, around 300mb used.
I don’t think it’s a memory leak in my Python code. I did memory progiling on my local machine while it was making the API requests for several hours, and memory consumption looks stable. Any ideas on how I could analyze the railway container environment?
I don’t think it’s a memory leak in my Python code. I did memory progiling on my local machine while it was making the API requests for several hours, and memory consumption looks stable. Any ideas on how I could analyze the railway container environment?
7 Replies
Project ID:
34108dc5-7795-469f-b853-2863bbedfa6fCrystal uses https://www.hboehm.info/gc/ so you need to sometimes call
GC_FREE() if you really want to min/max your RAM usage.⚠️ experimental feature
34108dc5-7795-469f-b853-2863bbedfa6f
Python has notoriously horrible memory consumption. Look into the gc module
https://docs.python.org/3/library/gc.html
Python documentation
gc — Garbage Collector interface
This module provides an interface to the optional garbage collector. It provides the ability to disable the collector, tune the collection frequency, and set debugging options. It also provides acc...
Python likes to keep data for much longer than it should
@Adam thanks! I’ll play with this soon and see if it helps. Maybe I missed something when profiling the memory on my local machine.
@Adam Unfortunately, running
gc.collect() didn't fix the issue. I still see memory usage climbing, which means variables somewhere still have a reference pointing to them.
Would something like OpenTelemetry help troubleshoot this?Haven’t heard of opentelemetry, but gc has a couple of logging tools you could use.