Railway•16mo ago

Python Flask app file upload memory leak

I'm running a Python Flask app. When I upload a file to be stored on the server, the web service's memory usage increases by the size of the file (essentially a memory leak) and when I delete the file on the server, the memory is freed. What should happen is the memory doesn't increase significantly when the file is written to the disk. I tried this on some other platforms like Fly.io and it handled it with no problem (Fly.io's trial machine is 256MB and I uploaded multiple 200MB files in a row without it crashing). here's some code that shows how I handle each kind of upload

if file:  # local upload (type: werkzeug.datastructures.FileStorage)
    file.save(full_path)
elif content:  # pasted text (type: str)
    with open(full_path, 'wb') as f:
        f.write(content)
elif stream:  # url upload (type: requests.models.Response.iter_content)
    with open(full_path, 'wb') as f:
        for chunk in stream:
            f.write(chunk)

if file:  # local upload (type: werkzeug.datastructures.FileStorage)
    file.save(full_path)
elif content:  # pasted text (type: str)
    with open(full_path, 'wb') as f:
        f.write(content)
elif stream:  # url upload (type: requests.models.Response.iter_content)
    with open(full_path, 'wb') as f:
        for chunk in stream:
            f.write(chunk)

project id: 15154147-a9b4-412c-9f4a-c690ca66a087

12 Replies

Percy•16mo ago

Project ID: 15154147-a9b4-412c-9f4a-c690ca66a087

Brody•16mo ago

your memory leak wouldn't be a fault of railway's this would be an issue with your code

B V S S B X D I•16mo ago

Hi Brody, thanks for the quick response, however, I still think it is a problem with Railway. The same code doesn't cause the memory leak described above on fly.io, render.com, local docker container, local machine, etc. the code is simply getting the file from a post request and saving it to the disk, something like

@cdn.route('/upload', methods=['GET', 'POST'])
def upload_file():
  if file := request.files.get('file'):
    ...
    file.save(full_path)

@cdn.route('/upload', methods=['GET', 'POST'])
def upload_file():
  if file := request.files.get('file'):
    ...
    file.save(full_path)

notes on the .save() method:

Save the file to a destination path or file object. If the destination is a file object you have to close it yourself after the call. The buffer size is the number of bytes held in memory during the copy process. It defaults to 16KB.

Using with open() to write the file using a chunked stream also causes the memory leak. Basically anytime I write a file to the disk it causes the memory leak.

Brody•16mo ago

okay so besides the fact that you are only able to see this leak on railway, what makes you think its an issue with railway, because so far this is just the "it works on my machine" meme not saying you're being silly or dumb or anything, but you should provide some kind of proof as to why this is a railway issue, not just "it happens on railway"

B V S S B X D I•16mo ago

What kind of proof would be sufficient

Brody•16mo ago

first solve the issue, then tell me what you did to solve it

B V S S B X D I•16mo ago

I solved the issue by using a different hosting provider...

Brody•16mo ago

one thing that pops into my mind is maybe this is an issue with the python version railway runs your code with ive ran services that saved decently sized uploaded files to disk, there was never any increase in memory, this isnt an issue with railway's platform

B V S S B X D I•16mo ago

Probably not because the python version is specified in the Ubuntu image in my dockerfile

Brody•16mo ago

if this was an issue with railway itself, i would too see memory spikes during a file upload

B V S S B X D I•16mo ago

Yes that's exactly what happens

Brody•16mo ago

...but i dont see any memory spikes in my applications during file uploads so that means this issue is isolated to your deployment, and not railway

Gaming

Programming

Python Flask app file upload memory leak