Python Flask app file upload memory leak

I'm running a Python Flask app. When I upload a file to be stored on the server, the web service's memory usage increases by the size of the file (essentially a memory leak) and when I delete the file on the server, the memory is freed. What should happen is the memory doesn't increase significantly when the file is written to the disk. I tried this on some other platforms like Fly.io and it handled it with no problem (Fly.io's trial machine is 256MB and I uploaded multiple 200MB files in a row without it crashing). here's some code that shows how I handle each kind of upload
if file: # local upload (type: werkzeug.datastructures.FileStorage)
file.save(full_path)
elif content: # pasted text (type: str)
with open(full_path, 'wb') as f:
f.write(content)
elif stream: # url upload (type: requests.models.Response.iter_content)
with open(full_path, 'wb') as f:
for chunk in stream:
f.write(chunk)
if file: # local upload (type: werkzeug.datastructures.FileStorage)
file.save(full_path)
elif content: # pasted text (type: str)
with open(full_path, 'wb') as f:
f.write(content)
elif stream: # url upload (type: requests.models.Response.iter_content)
with open(full_path, 'wb') as f:
for chunk in stream:
f.write(chunk)
project id: 15154147-a9b4-412c-9f4a-c690ca66a087
12 Replies
Percy
Percy17mo ago
Project ID: 15154147-a9b4-412c-9f4a-c690ca66a087
Brody
Brody17mo ago
your memory leak wouldn't be a fault of railway's this would be an issue with your code
B V S S B X D I
B V S S B X D IOP17mo ago
Hi Brody, thanks for the quick response, however, I still think it is a problem with Railway. The same code doesn't cause the memory leak described above on fly.io, render.com, local docker container, local machine, etc. the code is simply getting the file from a post request and saving it to the disk, something like
@cdn.route('/upload', methods=['GET', 'POST'])
def upload_file():
if file := request.files.get('file'):
...
file.save(full_path)
@cdn.route('/upload', methods=['GET', 'POST'])
def upload_file():
if file := request.files.get('file'):
...
file.save(full_path)
notes on the .save() method:
Save the file to a destination path or file object. If the destination is a file object you have to close it yourself after the call. The buffer size is the number of bytes held in memory during the copy process. It defaults to 16KB.
Using with open() to write the file using a chunked stream also causes the memory leak. Basically anytime I write a file to the disk it causes the memory leak.
Brody
Brody17mo ago
okay so besides the fact that you are only able to see this leak on railway, what makes you think its an issue with railway, because so far this is just the "it works on my machine" meme not saying you're being silly or dumb or anything, but you should provide some kind of proof as to why this is a railway issue, not just "it happens on railway"
B V S S B X D I
B V S S B X D IOP17mo ago
What kind of proof would be sufficient
Brody
Brody17mo ago
first solve the issue, then tell me what you did to solve it
B V S S B X D I
B V S S B X D IOP17mo ago
I solved the issue by using a different hosting provider...
Brody
Brody17mo ago
one thing that pops into my mind is maybe this is an issue with the python version railway runs your code with ive ran services that saved decently sized uploaded files to disk, there was never any increase in memory, this isnt an issue with railway's platform
B V S S B X D I
B V S S B X D IOP17mo ago
Probably not because the python version is specified in the Ubuntu image in my dockerfile
Brody
Brody17mo ago
if this was an issue with railway itself, i would too see memory spikes during a file upload
B V S S B X D I
B V S S B X D IOP17mo ago
Yes that's exactly what happens
Brody
Brody17mo ago
...but i dont see any memory spikes in my applications during file uploads so that means this issue is isolated to your deployment, and not railway
Want results from more Discord servers?
Add your server