Railway•2y ago

BREAKING BUG - Enpoints being run 2 or 4 times for 1 request

Any help will be appreciated. I have been trying to debug this for a couple of hours, ruling out every other possible source for the bug. My Python FastApi application works properly when deployed locally - so the problem comes from the Railway deployment. Minimal repro: https://github.com/gprieto/fastapi-bug-reproduction My railway.json config is:

{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "NIXPACKS"
  },
  "deploy": {
    "startCommand": "uvicorn main:app --host 0.0.0.0 --port $PORT",
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}

{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "NIXPACKS"
  },
  "deploy": {
    "startCommand": "uvicorn main:app --host 0.0.0.0 --port $PORT",
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}

GitHub

GitHub - gprieto/fastapi-bug-reproduction

Contribute to gprieto/fastapi-bug-reproduction development by creating an account on GitHub.

14 Replies

Percy•2y ago

Project ID: 4ffc6c0c-927e-4cb9-ba8c-fc0de29c5a99

Percy•2y ago

You might find these helpful: - deploy fail - Deployments, Redeployments and Restarts not working - Deployment Issues

⚠️ experimental feature

guille stOP•2y ago

4ffc6c0c-927e-4cb9-ba8c-fc0de29c5a99 It seems to happen only when an fastapi.HTTPException is raised

guille stOP•2y ago

I opened an issue here https://github.com/tiangolo/fastapi/discussions/9308

GitHub

Endpoint called/executed twice when HTTPException is raised · tiang...

First Check I added a very descriptive title here. I used the GitHub search to find a similar question and didn't find it. I searched the FastAPI documentation, with the integrated search. I al...

guille stOP•2y ago

And a minimal reproduction example here:

guille stOP•2y ago

https://github.com/gprieto/fastapi-bug-reproduction

GitHub

GitHub - gprieto/fastapi-bug-reproduction

Contribute to gprieto/fastapi-bug-reproduction development by creating an account on GitHub.

Ray•2y ago

Thank you for the repro example! Made this super easy to dig into. I can reproduce the issue. It stops happening when I changed the exception status code to 400 in https://github.com/gprieto/fastapi-bug-reproduction/blob/main/main.py#L13. I also tweaked the startCommand to run uvicorn with only one worker to make sure nothing funky's going on there (https://github.com/gprieto/fastapi-bug-reproduction/blob/main/railway.json#L7):

    "startCommand": "uvicorn main:app --workers 1 --host 0.0.0.0 --port $PORT",

    "startCommand": "uvicorn main:app --workers 1 --host 0.0.0.0 --port $PORT",

Given that it doesn't happen with a 400, is Railway somehow re-trying 404s?

guille stOP•2y ago

thanks a lot @rc I've been struggling with this! I confirm, changing all 404 into 400 in my production app did the trick! no more duplicated logs and executions

mushuw•16mo ago

@rayofbytes Me and a few people saw this bug a few months ago: <#1099377493057749134> In June 19, 2023 update notes, Railway said they fixed this 4xx/5xx retry bug: https://railway.app/changelog/2023-06-09-talk-to-an-engineer-upgrades#improvements-and-fixes

Railway

Changelog #0137

Talk to an Engineer Upgrades, Replica Stacks, Variable Highlighting

mushuw•16mo ago

(Sorry I didn't see this was just a mod resolving a bunch of tickets... )

Brody•16mo ago

what are you saying? is this not resolved?

mushuw•16mo ago

It's resolved. I'm just a fool not looking at timestamps.

Brody•16mo ago

ah okay haha

Ray•16mo ago

All g, thanks for the bump! Locking this now

Gaming

Programming

BREAKING BUG - Enpoints being run 2 or 4 times for 1 request