RunPod•4mo ago

FastAPI RunPod serverless request format

Hi everyone, Second post from me. I am (still) trying to deploy a FastAPI app in a Docker container hosted on a serverless RunPod. Here is a toy example:

import uvicorn

from fastapi import FastAPI
from pydantic import BaseModel


app = FastAPI()

class PingResponse(BaseModel):
    message: str

@app.post(path="/ping", response_model=PingResponse)
async def ping() -> dict[str, str]:

    response = {"message": "ok"}

    return response

if __name__ == "__main__":
    uvicorn.run(app=app, host="0.0.0.0", port=8080)

import uvicorn

from fastapi import FastAPI
from pydantic import BaseModel


app = FastAPI()

class PingResponse(BaseModel):
    message: str

@app.post(path="/ping", response_model=PingResponse)
async def ping() -> dict[str, str]:

    response = {"message": "ok"}

    return response

if __name__ == "__main__":
    uvicorn.run(app=app, host="0.0.0.0", port=8080)

The code is deployed as a Docker image with the port 8080 exposed. My problem is: I just can't figure out how to format my requests so that the payload hits the FastAPI path. This is the format I started with:

import requests

URL = "https://api.runpod.ai/v2/<POD_ID>/runsync"
HEADERS = {"Authorization" : "Bearer <TOKEN>"}

body = {
    "input": {
        "api": {
            "method": "POST",
            "endpoint": "/ping",
        },
        "payload": {}, # Empty for example, but holds parameters in real case
    },
}

response = requests.post(URL, json=body, headers=HEADERS)

import requests

URL = "https://api.runpod.ai/v2/<POD_ID>/runsync"
HEADERS = {"Authorization" : "Bearer <TOKEN>"}

body = {
    "input": {
        "api": {
            "method": "POST",
            "endpoint": "/ping",
        },
        "payload": {}, # Empty for example, but holds parameters in real case
    },
}

response = requests.post(URL, json=body, headers=HEADERS)

The request does get to the serverless worker, but then just hangs without being forwarded to the FastAPI path method. Am I missing something here? P.S.: last question from me I swear.

12 Replies

nerdylive•4mo ago

haha dont worry about asking questions, okay and where's your handler code? i dont see runpod.serverless.start() there

ohnoOP•4mo ago

Her @nerdylive , thanks for answering so fast. No handler code: I created the serverless through the web GUI. Here is my template, with some fields obfuscated for safety

nerdylive•4mo ago

Okay maybe you got the serverless wrong, read the docs, and read the runpod-workers github repo for example of Serverless workers But the request format seems right yes you should do it like that And also some note, that you can't run docker inside serverless or pods ( with gpu ) because of security reasons if I'm not wrong

ohnoOP•4mo ago

Yeah I know for no Docker-in-Docker, I'm just launching the base image used to make the template just like RunPod would with any pre-built image. Stuck scratching my head at the moment - I've been following the docs to the letter, and I am having the same problem with a normal Pod rather than serverless. Deploying the same image to a Pod and running the same request gives:

{'error': 'endpoint not found'}

{'error': 'endpoint not found'}

nerdylive•4mo ago

Ohh so the slot there is actually for CMD or entrypoint

ohnoOP•4mo ago

Exactly

nerdylive•4mo ago

Search on github runpod workers Or in google Runpod workers Find a repo in that org That's an example you can use

ohnoOP•4mo ago

Tried just that - found this one, which looks very similar to my use case https://github.com/dannysemi/runpod-serverless-proxy/tree/main And I can't find anything they do that I don't. Starting to think I may be a little dense.

GitHub

GitHub - dannysemi/runpod-serverless-proxy

Contribute to dannysemi/runpod-serverless-proxy development by creating an account on GitHub.

nerdylive•4mo ago

https://github.com/runpod-workers/worker-template

GitHub

GitHub - runpod-workers/worker-template: 🚀 | A simple worker that c...

🚀 | A simple worker that can be used as a starting point to build your own custom RunPod Endpoint API worker. - runpod-workers/worker-template

nerdylive•4mo ago

Start here No, its not meant to run inside serverless, but in other server it works look here it always starts from dockerfile

nerdylive•4mo ago

runpod.serverless.start({"handler": handler}) This is the important path that starts it

Gaming

Programming

FastAPI RunPod serverless request format