Files have been disappearing from mounted Volume
Project Id: dba6cd6e-5ea1-494c-b77f-81377e24fbde
I first noticed this yesterday. Some of my images and some pdfs have been lost somehow. They come up as 404. I reloaded them thru admin. And, another image is lost this morning. I'm saying lost but I don't really know what happened and I cannot browse the mounted directory. I don't have a way of knowing for sure. Are there any logs for file operations?
This incident made me question the trust issue since I cannot help to think what will happen after testing in real production. I cannot really find out what happened. Can someone look into this?
To better understand the matter, I want to ask about the mounted volume. Is it flat with keys like AWS bucket or are there real directories there?
Any help for better understanding this would be appreciated.
104 Replies
Project ID:
dba6cd6e-5ea1-494c-b77f-81377e24fbdeits an actual disk mount, though nvme based to be exact, so nothing at all like an s3 bucket.
but lets talk about the issue at hand, what is the app's tech stack?
What do you mean by tech stack? It's a FastAPI service for react native expo consumer. Thanks for help. You need other info?
what is your volume's mount point?
Well I chose to enter /assets and from what I understood it mounts at the root directory of the app. So it's either /assets or /app/assets
assets have sub directories like posters/, pdfs/
I cannot browse it so I don't exactly know
whats likely happening is that your mount point is incorrect so you end up saving files to the ephemeral container storage, then those files are lost when you deploy code again
Yes it happened after deployment but I did not deploy anything since last night and one more image is gone or I don't know what happened
what is your current mount point?
I chose /assets
where is your code saving files?
Well it is using FastAPI FileSystem Storage. I have been using it problem free for a while
right and where have you configured that to save files
In the SqlAlchemy models. I'll post a snippet
one sec.
poster_storage = FileSystemStorage(path="app/assets/posters")
`
are you using a dockerfile to deploy your app?
no. but one was created I gues automatically
do you have a dockerfile in your project?
No
is this part of your actual code?
You mean the snippet above? yes
I also have one for static files
now this is where you store your posters, but is everything you store within the assets folder?
`
assets folder have html, pdf and some templates and js and css
assets is the root of storage I have other folders there for videos, pdfs, images, html, css etc..
I have not other storage
you can only mount one volume as im sure know, so all your non static assets need their own folder, like
media/posters etc
then you want to mount the volume to /app/mediaYes. that is what I assumed. It was not very clear and you can not really browse so I dont exactly know. However ,that's what I tried to do. One volume /assets and everything inside the subfolders.
FastAPI static files mount may be confusing the directory but I cannot know for sure
try my suggested changes, once done, upload a file, then redeploy the service, if the file is still there after, you are good to go
I'm sorry but I missed what you suggested. Should I remount and if so what should be the mount point? I have it as /assets now
Mount it to /app/assets ? or /app/media?
my two suggestions where
- save your uploaded media to a path like
media/posters or media/images instead of the paths you correctly use
- mount the volume to /app/mediaOK. I'll try it
Thanks
I tried exactly what you suggested but I cannot load or find the images. So,
I have mounted the drive as /app/media
and my files system storage saves to app/media..
So what would be public address of my media folder now?
I cannot browse the drive so I cannot see where it was saved.
I am looking to find it at my app directory like <mysubdomain>/railway.app/media/posters/image.jpg
well first off, did the volume usage increase when you uploaded files?
let me check
looks flat
I mean the usage line
how big where the files that you uploaded, upload like 50mb of files
no it was an image jpg
171kb
I did upload a 56mb video yesterday but that does not show, I assumed file size might be too big but images were loading fine
I uploaded one image jpg to test
upload that 56mb video today
It may work today?
it you see a bump in the disk usage, then thats one step closer
I have this code at my app mount point for static files like html jss, they were uploaded along with and part of source code at /app/assets/html , app/assets/html/js and app/assets/html/css.
app = FastAPI()
app.mount("/static/assets", StaticFiles(directory="app/assets"), name="static_assets")
This is not confusing the volume is it?
i would like to say this since you brought it up in your original post, nothing here is due to issues with railway, this is all just because the paths are wrong in your code
please dont confuse static files like html, js, and css with the media uploads, those are completely separate
Well, that is what I thought but
I wanted to make sure
Well, I followed your document examples. You have to understand, I cannot see, browse the volume and the example in your docs was not super clear. My path was not wrong, I just don't understand it yet.
So, I managed to get them up and running, I actually saw the ,images but then they were lost since I did not understand waht was going on
my examples are only based on my best guess since i know very little about your code
That is understandable.
so did you upload that 56mb video?
For security purposes, I cannot share whole a lot but enough to let you know, that is what I'm trying 🙂 I appreciate you helping me though. Thanks a lot
yes
yes
no usage bump
let me check the mount again
whats your current volume mount point
/app/media
directly from settings
okay and can you show me the code that is responsible for saving uploaded videos to disk
Well, that is inside the SqlAlchemy admin which what I'm using. However, it works as intended. I tested it on other servers and locally. It saves a relative path.
poster_storage = FileSystemStorage(path="app/media/posters")
video_storage = FileSystemStorage(path="app/media/vid")
The rest is FastAPI FileSystemStorage
from fastapi_storages import FileSystemStorage
right so you did my second recommendation, but you haven't done my first
which was...
read the message please
save your uploaded media to a path like media/posters or media/images instead of the paths you correctly use
mount the volume to /app/media
ok I see,
You mean the storage should be media/images not app/media
media/video, media/posters, media/whateverI see..
I changed it to media but I missed taking out the app part. Trying now
once code changes are done and pushed, wait 1 minute, upload that 56mb video, wait 2 more minutes, check for bump in disk usage
Oki will do.
Thanks again 🙂 Hopefully It'll work this time
Unfortunately, no bump in the disk usage and file is 404. I will troubleshoot further and report back here. I have to go to doctor now. Thanks
file being 404 is normal for now, you haven't setup a file server, but the no disk usage bump is unfortunate
Thanks. I will try to unmount and mount again after I get back from doctor. Thanks for troubleshooting with me
it's 2:40am for me so I likely won't be available when you're back
Using Volumes | Railway Docs
https://docs.railway.app/guides/volumes
No problems. I am reading and re-reading to make sure I have not missed anything. Thanks
can you send me the docs for this FileSystemStorage stuff? I'll see if I can get a demo running
Working with Files and Images - SQLAlchemy Admin
Flexible admin dashboard for SQLAlchemy.
perfect, ill see if i can get a little example app working
That would be wonderful. However even though I do not know if I'm safe with this but I found a work around.
I point the static-files ext of FastAPI and point that to the volume mount on railway
Admin saves there
and copies the relative url
I add it to staic files base and make a auto-url.
Since they are just static files and if you store the path correctly then it works this way now
so what does your current
FileSystemStorage look like?poster_storage = FileSystemStorage(path="app/assets/posters")
and whats the volume mount point?
app.mount("/app/assets", StaticFiles(directory="app/assets"), name="static_assets")
so this static files line makes it possible for admin to write the correct url to db
After all they are static image files as long as db has correct rel path
whats the volume mount point?
/app/assets
do you see a bump in the disk usage?
the same assets folder also has an html subfolder for css and js
they work as well
yes
guess this isnt a relative path then
It takes it from the route of the server outside the app folder
I thought the static files and ( {url_for} ) was confusing it but no
do the files persist across redeploys?
I tried once with on image it was fine. I will test it again but scared a little :))
to me, what you have seems like it shoulnt work, but i could be wrong
I know it may not be the best practice but we'll see
True but if you think about it, static files ext is there for serving static files like css and js
However, if you put more sub dirs in it, it serves them as well
Some of the come from deployments but they are not in assets/posters they are in assets/html
We'll see I will redeploy for testing tomorrow.
hey you can't leave me wondering until tomorrow
client react app works fine with fastapi on railway
ok, i'll do it in a short while 🙂 I am scared that it won't work and I'll loose images but if that is the case I'll re enter after fixing
sounds good
I'll report back here
Unfortunately no :))) They did not survive this time
I had tested one image but it was. may be too soon or cached. For many images no :))
So, I guess I'll have to serve them right
the paths are still wrong, unfortunately i was right 😦
OK, so now I'll wait for you to make an example?
yep 🙂
oki, 🙂
Back in 90s I worked a lot as software developer but I'm a jazz musician and this will be for teaching 🙂 I played most NYC clubs back then 🙂
lots has changed in software dev from the 90s for sure
Yes, I found that out :))
This might be the right example may be you saw it already
first ill try to get this working https://aminalaee.dev/fastapi-storages/usage/#filesystemstorage
are you using sqlite?
I’m using MySQL
okay cool
got an crude example working, media uploads and serving the media files, uploaded media persists across multiple deploys. getting it in a repo now
Solution
heres the repo https://github.com/brody192/fastapi-FileSystemStorage
Thank you. I’ll try first thing tomorrow and report back here. Thanks a lot:))
no problem!
Hey Brody 🙂
I started working on the repo you posted yesterday. I noticed a railway.json It has a different deploy / start command than mine. It looks like it's irrelevant for what I'm doing so I don't need it right?
I also wanted to ask you about the mounting volume director. Is it one folder or can we create subdirectories. I have different file types and it looks like I have to create 5 different file uploads and It would nice to have them on separate dirs.
OK, thanks Brody 🙂 I got it working. You're probably sleeping, considering the time zone difference.
I got it working ( all five different directories )
This upload methods on the main file was not sth I wanted so I tried to change it to a router and found out that you cannot mount static files on routers. However, it does work on apps. So I created an app for uploads and mounted it on the main app with app.mount..
Then of course you need to give the app you're mounting a path as well then... it did not work. I either mixed up the paths however I tried to consider app mounting point vs media folder mounting point etc., I could not get it working yet but if I do I'll report back.
Anyways, basic way of things are working now. Thanks a lot for this 🙂
OK, I tried with right paths and could not get it working with a sub app mounting to main app with "/upload" mount point. So I gave up on that.
However, starting from your example, I managed to get it working mounted on the main app.
Thanks 🙂
I use hypercorn, but I'm sure you use uvicorn (at least I hope) so it's irrelevant, they both do the same job
you can create sub directorys, but the code would do that for you automatically as I've noticed
I have confidence that with my example you'll be able to get everything working as you want! and always make sure to redeploy the app just as a sanity check to make sure the images stick
Yes, I have double checked and the files persist 🙂
I am hopeful that I can get it working the way I want too but I have to clean up a few things first. Besides, it was just a cosmetic thing, I just did not want main to be crowded that's all 🙂
Thanks a lot for this 🙂
I really appreciate the help
awesome, happy to help!
Hey check out my recent music video if or when you have time 🙂
YAVUZ AKYAZICI
YouTube
Motor Helmet On - Official Music Video
Motor Helmet On Music Video
Funda Akyazıcı Vocal ( @fundaakyazici )
Yavuz Akyazıcı Guitar, Slide Guitar, Musical Arrangement
Kevin Fitzgerald Burke Vocal ( @kevinfitzgeraldburke )
Enver Muhamedi Bass
Cengiz Baysal Drums
Songwriters: Funda & Yavuz Akyazıcı
Complementary back vocal lyrics: Kevin Fitzgerald Burke
Websites:
https://yavuzakyazici....
cool, I will