Deployment failing since Railway incident report
I pushed a very small update to my api project, at the same time as a reported Railway incident. Since then, all my builds are failing, and I can't rollback to a previous build. I wouldn't think the two incidents would be related, but it's a bit strange in that the change was very small so it doesn't seem like a coincidence?
Project: ac828fce-6f54-43e9-825c-ac4871d226ba
Project: ac828fce-6f54-43e9-825c-ac4871d226ba
Solution:Jump to solution
yes that was the issue. langchain pinned to 0.0.173 which i had running locally. the latest is 0.0.178 which broke things.
47 Replies
Project ID:
ac828fce-6f54-43e9-825c-ac4871d226baHave you tried rolling back your code and pushing a new commit? If there's an issue on Railway's side, rolling back to a previous build may not work correctly.
im trying now. it very well could be an issue on my side. it's just weird that this happened at the same time as the incident. so i wanted to report it to see if others were having an issue.
so i rolled back successfully. i re-push the new code but that's not working. i'll try to revert my PRs
that actually won't work. so i have a stable release from before I saw the Railway Intermittent Issue notice. I pushed a small release after that and the deployment failed. I since rolled back my PR and pushed to Railway but that also failed
so to summarize:
- i had stable code in Railway
- i tried to push a release, which failed
- i then saw the Railway outage notification (after I tried to push the release)
- i tried to revert the code to the previously-stable version. this also failed.
- so now, i have a version of code running on hostway that is not the same as my current active code branch.
Interesting issue. What does the error say?
the healthcheck fails. and the deploy logs have hundreds of errors
what are those errors?
very low-level. e.g. pydantic.
Looks like a code issue
but i made a really small code change (in a test case). and then reverted the PR
Yeah sorry this isn't an issue with Railway. It's 100% an issue with your code
Your app is building, but it's failing during deployment. That's indicative of a code issue
the app is failing during the build process
full build logs please?
sure. how would you like me to send? do you just want to look at the service via the service id?
only team members can look at the service, i have access to nothing
so in a .txt file please
attached build and deploy
full build logs please
wait
why is your image 4.9 gb
torch, etc
can you send your requirements.txt
oh you should really be pinning your packages to specific versions
one of those packages must have made a breaking change
ugh
i can try to rebuild in a fresh env locally
also im sorry to tell you this but using duckduckgo_search breaks tos
as it scrapes duckduckgo
i can disable that for now. not really using
much appreciated, thank you for understanding
so you have seen this sorta issue before i assume?
yes
though i do not know what specific package was updated that results in this issue
i have a guess
let me know if you figure it out, so i can pass on this info to anyone who may have this issue in the future
long shot, but?
https://stackoverflow.com/a/76313951
ha. yeah, this was my guess 🙂 thanks for sending this. im gunna pin that version
keep me updated 🙂
ill let you kow if that works
for sure!
i assume this works locally, just pin the packages in the requirements.txt file to the versions you have installed on your computer? would that be possible?
worked!! 🙂
Solution
yes that was the issue. langchain pinned to 0.0.173 which i had running locally. the latest is 0.0.178 which broke things.
thanks a lot for the help!
latest langchain is 0.0.178?
had planned on pinning everything never got around to it. will make it a priority now 🙂
yeah, they update their package prob every day. i had 0.0.173 locally. 0.0.178 broke the build. prob 177 is ok as the build was working yesterday
oh 9.9.178 is typo lol
so your app worked as soon as you pinned langchain to 0.0.173, did you need to pin anything else?
no- just pinned that. wanted to try that first, then start pinning other stuff if it didnt fix it. i figured it was langchain as much of my code depends on their types, and the errors were pydantec related. and they update about every day. i'm in their discord- i can ping them and let them know
perfect, thank you, this info will be used to help others who may run into this issue
do you think you could fix the 9.9. typo so i can mark your message as the correct answer?
sure- whre is that typo?
^
done
thanks!
thank you!