Railway projects suddenly failing with new "Cannot find type definition file for 'mime'" Error
Seems that something changed in the Railway infra because even 'redeploying' previously working projects, I'm getting this new error during build. Any idea why redeploys suddenly error on mime types?
2158ac66-80be-42ae-b4e4-d6bbbe244d21
121 Replies
Project ID:
fb3a601a-4fe9-4358-8ffd-12d303256e6e,fb3a601a-4fe9-4358-8ffd-12d303256e6e,2158ac66-80be-42ae-b4e4-d6bbbe244d21To be clear, this suddenly started happening on multiple of my projects, that have had no changes, and deployed properly as recently as yesterday. Simply clicking 'redeploy' is enough to get the error to happen. So I've changed nothing and this new error is popping up. Very confused.
are you commiting your node_modules folder to github?
Nope!
I'm guessing something happened with Railway's nixpacks version, that has resulted in a brand new error on previously working code.
And as a result multiple projects that worked in a previous infra are now all crashing down. Shame.
have you tried looking for solutions to this error? it wouldnt be isolated to railway
I'm even trying to 'redeploy' instances that were previously working and they also crash with this new error.
Including ones from several months ago.
redeploy will redeploy, you might want a rollback instead
Yes, rollback will work, but that doesn't solve forwards.
I was just confirming that it's railway infra related.
while i agree with you when you say that a nixpacks version bump introduced something different, this would not be an error that you could only get on railway
saying that its railway infra related is not a correct sentence
I'm not deeply familiar with nixpacks, but basically multiple versions, multiple branches, multiple databases, ALL now fail on a redeploy or new deploy. Why would old (previously working) projects all suddenly fail in the same way.
Perhaps it's more correct to say the way railway deploys nixpacks is the issue? I may be wrong to even be looking at nixpacks, hence asking for help.
databases dont use nixpacks, lets keep the scope of this to your service please
I meant applications pointing at different databases.
have you tried looking for solutions to this error?
Just trying to get across that it's a sudden issue across a whole bunch of different applications with different setups, applications versions etc, which to me points to infra (railway, nixpacks, something else, not sure), but I have no idea which part of the infra to look at.
The 'error' is in a library my program made no change or has no specific reference to. And yes I've searched for solutions to it, but have found nothing useful.
what is the nixpacks version of the last working deployment?
I'm not sure how to find that, or even control it. As I'm only using nixpacks as it was recommended for railway. I'm not even sure I understand the concept of a 'nixpacks version'.
Is there a way to tell the nixpacks version on a running deployment?
look at the build table in the build logs
The most recent successful build was over 30 days ago, so I cannot see build logs. Any other way to see it? And on Railway is there even a way to force an older version of nixpacks, or is that inaccessible?
For reference, my nixpacks.toml looks like this:
Perhaps the nodejs version got bumped too far and has created this Typescript error in my projects? If so, I could perhaps lock the nodejs version (if that's even possible).
respectfully, with a nixpacks.toml like that im suprised it has ever worked
do you use yarn or npm locally?
Heh, beginning to realise this...
Yarn locally...
do you have a yarn lock file?
Yes.
do you have an npm lock file?
No, since I use yarn, I removed that and don't commit it either.
So I'm guessing I need to have a more 'locked down' nixpacks file?
do you use yarn v1?
is your yarn lock file in sync with your package.json?
Yeah, I keep yarn lock in sync with package.json with yarn install, and committing it all together.
nope, the opposite actually
why do you need
postgresql_16 ?
or.. do you need it?Because I use postgresql_16 in a script in my application that backs up my database to S3.
Since the DB on Railway are now postgres16, I needed that running on the server.
indeed
replace your nixpacks.toml with this
Note that I originally got that nixpacks recommended format from here (https://docs.medusajs.com/deployments/server/deploying-on-railway), my project is a Medusa app.
i know
Ok, I will try that now.
but i dont know why they would add that
Heh, fair enough. For interest, does that mean Railway will pick what to install? (nixpacks dummy, remember)..
nixpacks will detect this is a node app that uses yarn and install and run the applicable stuff
Ok, so do I also remove this from my nixpacks.toml?
yes
Thank you, rebuilding and the nixpacks header shows this now:
looks better
Locally my yarn version is
3.5.0 if that mattersit does
and my node version is
v20.3.0set
engines.node to 20locally? or in railway?
it would be done on your local project and you push the changes to github
I see, it's done through package.json, changing that now.
create-medusa-app use yarn or npm by default?Ooof, not sure, that was almost a year ago I ran that. I think medusa gave you the option, I seem to remember manually removing the npm stuff.
looks like its yarn lock file only https://github.com/medusajs/medusa-starter-default
Ok, that package.json worked to force it to node 20 instead of 18. Learning! (thank you btw, was scouring the internet for hours with no luck, this help is far more directed).
that would be outlined here https://nixpacks.com/docs/providers/node#setup
So now it's building, but hitting that same 'mime' error now.
use yarn v1 locally and sync your lock file again
Is there a reason to be moving backwards with yarn? Is it because railway doesn't properly support yarn v2?
railway uses yarn v1
Ok, wondering if that will create issues with medusa, which I believe uses yarn v2.
they use v1 https://github.com/medusajs/medusa-starter-default/blob/master/yarn.lock#L2C17-L2C19
so delete your lock file and then
yarn install to make a new one with v1Gotcha, now my yarn says version
1.22.19
Doing that added this to my package.json, do I want the version to be added that specifically?
"packageManager": "[email protected]"
And it also added yarnPath: .yarn/releases/yarn-1.22.19.cjs to my .yarnrc.yml file...dont do any of that please
railway uses yarn v1 already, all you need to do is use yarn v1 locally and have a v1 lock file in your project
I think when I rolled back to yarn v1 with
yarn set version classic it added those two lines...so I will remove them.Okay, so that's all re-built, still getting that mime error, any further ideas from the railway side to investigate and figure out when this error actually creeped in? That is, back to the original issue of old deploys working on the same codebase where new deploys are now failing.
now it comes down to some issue with your project / deps
what solutions have you tried to fix this?
Well that's where I'm confused, it deployed fully fine a couple months ago, it's just when I redeploy (same code, config, etc) where it starts to fail. The language around that error appears to be a typescript thing, so I'm guessing the typescript version changed in what gets deployed on railway in the past couple months (I had stepped away from the project for a bit). And so my next deploy after that gap in time just now fails. Is there any way to force Railway to deploy with nixpacks versions from a month ago? Or am I thinking about this incorrectly?
I wanted to add in the type definitions for
mime as a test, but as far as I can tell, it's typed right inside the library. So there's nothing to add. I added the latest mime-types anyways, but that didn't work.
Interestingly, this is what I get from yarn why mime - I have nothing about mime in my package.json though.
please can we take the blame away from railway, i understand how it looks but this is not an issue with the platform
Not blaming, just trying to see that it may be something to do with infra (more likely that I haven't controlled something and it's simply manifesting with an infra change)
Is there any way to force Railway to deploy with nixpacks versions from a month ago?yes
Or am I thinking about this incorrectly?yes
I simply want to investigate that as it may help me find the bug.
pinning this on railway is only going to cloud your judgment
It's more that if I can find an old infra it works on, I can step forward and may be able to discover the source of the issue or better debug. But in parallel I'm also trying to figure out the bug with this 'mime' library and typing.
Arg, somehow yarn has 're-updated' itself to v3.5.0. Not sure how that happened.
thats not ideal
Looks like the moment I removed this line from the .yarnrc.yml it reverted to the v3.5.0
so that's interesting.
remove yarn v3 from your computer
Yeah, that's what I just figured is a good next step.
Ok, ran this command
corepack prepare [email protected] --activate and that worked, will reinstall and redeploy now with yarn v1 (as apparently I never did)
So this is very weird... I ran yarn why mime :
And then checking express, it's the latest version:
So what doesn't make sense is how latest version of express seems to be using a very OLD version of mime (1.6.0) but a very NEW version of its types (4.0.1).
My package.json sets express as "express": "^4.17.2",
So, PERHAPS the bug is in express...digging in deeper but does this at all make sense to you?have you deleted your node_modules folder and then ran all the same commands railway does to install deps, build and run your project right?
I did a
yarn cache clean, but you're right, should have nuked node_modules for a fresh start.
Done, didnt' change anything.and then ran all the same commands railway does to install deps, build and run your project right?
Innnnteresting. Getting the same local error now on yarn build. Only since getting properly on yarn v1. This should help debug without having to deploy!
More interesting, I just used
medusa-create-app to make a fresh medusa install, and the response to yarn why mime is different... NO idea why this would be with my package.json, which refers to the SAME versions of express...
very odd
the joys of node dependencies
Yep, using this to dig further... thank you for your help to date. Recognise this is no longer (that is, never was) a 'railway' thing but nevertheless has been extremely useful.
So damn weird...
This is what the express and send versions are in the FRESH install:
And in my install, after removeing the node_modules folder and the yarn.lock file:
Banging head on wall at this point. Same package.json, same actual versions, but somehow this mime v4 in my project seems to be screwing everything up:
Going deeper with
yarn why @types/mime:
My project:
And the new test project:
welp time to start over with a new medusa project
Hah, no way... There must be some way to rectify this. My package.json isn't that much different from default. Is there anything to wipe aside from node_modules and the yarn cache that could be messing w this?
i honestly wouldnt know
WOW! Adding this to my package.json got past that failing build step:
Not a great fix, forcing a dependency like this, but at least helps dig deeper...
thats node for you 🙂
One last question, does this seem right in the nixpacks like this with --check-cache? Shouldn't I be wiping the cache in Railway?
its doing that because you have a yarnrc.yml file
This is what's in my yarnrc.yml file, is this needed or should i nuke that file?
yeah get rid of it
Great, so several wins:
- I undersand nixpacks better and have a healthier use of it on Railway
- You've nicely updated Medusa's documentation to avoid future mishaps like mine
- I "sorta" understand that I have a node dependencies issue and have a short-term hack around it.
Wins and beers around!!
and at this point your build has gone out and you are back up?
Nope, got a bit of a pathway to get there...
note nixpacks looks like this now:
║ install │ yarn install --frozen-lockfile ║thats what id expect it to be
what other issues are in the way?
Well now there seems to be an issue with medusa-admin not properly building before I can fully get to my prior state (says that date-fns package isn't available even though it's in the lockfile). I expect I'll get there though, and at that point, Medusa's routes suddenly stopped being registered for some of my medusa plugins and therefore the server application failed to start. Everything I've been blocked on today was simply to try to upgrade Medusa to the latest version to see if I just happened to catch a bug that only manifest in my project at that version. Have done all the necessary database migrations, etc.
All of this is very strange, as again there was no change to my codebase, environment variables, anything (hence my looking at infrastructure). All seems to be quite the tangle. Frustrating as it was all just chugging away working fine from code back before Christmas until last month and then just suddenly started falling apart. Natural return of order to chaos I suppose.
Hmmm, have just figured out one more 'Railway config' thing I don't understand. I had my nicely working from before Railway instance hooked up to github, and THAT one is failing during build, however I just made a 'fresh' one and it's working. Pointing at the same github branch though. Is there any history / memory / cache at all the Railway github instances hold onto between code deploys? Not sure why the build step would fail on one and succeed on another otherwise..
railway does do build caching on a per service bases
(it's been a service for a while so perhaps it's an old 'version' of the service too, it's been up since the fall)
Any way to 'clear' the cache?
NIXPACKS_NO_CACHE=1In the environment variables I assume... Thank you!
in the service variables
Nice. Trying that to see if I can bring my service back to life. If that doesn't work, I'll just use the 'new' one I just made (that works) and then configure it to use SSL (through DNS) etc. to match my 'old' one then retire it. Not my fave method but it'll work.
configure it with SSH (through DNS) etchuh?
Yep it worked! So that dependency thing in my admin build must have been from a cache issue. Is there a good reason to actually keep caches on in Railway? Seems unfortunate if they can mess w dependencies in this way.
Basically, I meant using the Networking to set a custom domain, and using DNS to get it to work (with SSL). Sorry spoke unclearly.
Is there a good reason to actually keep caches on in Railway?it would normally speed up builds whats ssh have to do with railway?
It's not directly, it's more that I use Railway to set redirects to https://<domain> then then I have to set up that domain DNS to the proper Railway server, AND get the SSL (https) certificate. All just tasks that come off of the change in Railway.
odd because railway doesnt offer ssh
Awwww, shoot, everytime I said SSH up there I actually meant SSL, and now I understand your confusion...Oops! Edited my messages for clarity.
haha quite confused
Just talking about this really... and how it requires DNS configuration (completely ourside and not related to Railway) in order to function properly.
btw you can move your domain over to a new railway service without touching any dns stuff
but i assume you already did all that, so this is info for a next time
Even better, I got the original one working (by nuking the cache through that service variable) so didn't need to migrate at all.
oh perfect
Hah, get this absolute node craziness. Just on a whim, I decided to remove the resolution for
mime now that it was all working, and guess what? It works both locally and on Railway. Makes no sense at all, but perhaps it 'kicked' it out of a weird dependency funk by just getting it to work once...
I mean it does sort of make sense as there's really no logical reason it should have been versioning mime in that inconsistent way...
(aka node magic)this is indeed node magic
Further magic. Have switched back to the default
NIXPACKS_NO_CACHE=false and lo-and-behold it all still works... Seems this was a 'spider-web-clearing' exercise...
Ok, full circle and in summary, I now have ALL environments [local, test (on railway), staging (on railway) and prod (on railway)] restored and properly running through a combination of:
* upgrading medusa
* cleaning up the nixpacks as per your suggestions
* clearing the nixpacks cache in Railway with the NIXPACKS_NO_CACHE service variable
* doing necessary database migrations (unrelated to this issue, but had to be done)
Of interest, to fix prod I did the least amount of manual stuff. That environment never saw the 'resolution' used to fix the 'mime' issue, I simply upgraded from November codebase to now. Then it presented the 'mime' issue, so I added the NIXPACKS_NO_CACHE, redeployed and the issue went away and everything deployed perfectly. So that's of interest for anyone presented with a similar future issue, try a pass of that (you can turn it back to default afterwards) for these weird issues presenting on Railway (or any deployment environment) when using Nixpacks. At this point, I'm actually unsure I needed to upgrade Medusa at all, perhaps the nixpacks cache fix on the old codebase would have been enough, but I can't know at this point as I fixed too fast.I'm pretty sure
NIXPACKS_NO_CACHE=false still disables cache, I don't think nixpacks checks if that value is a Boolean, I think it only checks if the value is set to anything and if it's set to anything it disables build caching
and I know that at the very least it doesn't clear cache, it just won't use cache for the builds it set onInteresting and noted, will try fully removing the variable.
side note, the maintainer of medusa seems quite reluctant to merge my changes
Can note this still deploys without the
NIXPACKS_NO_CACHE so all good over here!
Added a comment to your PR with support from my side. They should at minimum warn people of the risks associated with their guide.thank you for the backup on my pr and thank you for the trains, I really appreciate that
No worries, was exactly the right kind of help (help that helps!) I needed to unlock and unfurl a bunch of challenging (for me) issues.
sorry i just dont have the time to do the back and forth with them, what was a simple change has turned into a gigantic conversation spanning multiple topics, its in their hands now