RunPod•5mo ago

Serverless worker doesn't run asynchronously until I request its status in local development

I'm following the docs and created a verey simple handler.py with the following:

import runpod

def handler(event):
    """
    This is a sample handler function that echoes the input
    and adds a greeting.
    """
    try:
        # Extract the prompt from the input
        prompt = event["input"]["prompt"]

        print (f"Prompt: {prompt}")

        result = f"Hello! You said: {prompt}"

        # Return the result
        return {"output": result}
    except Exception as e:
        # If there's an error, return it
        return {"error": str(e)}

# Start the serverless function
runpod.serverless.start({"handler": handler})

import runpod

def handler(event):
    """
    This is a sample handler function that echoes the input
    and adds a greeting.
    """
    try:
        # Extract the prompt from the input
        prompt = event["input"]["prompt"]

        print (f"Prompt: {prompt}")

        result = f"Hello! You said: {prompt}"

        # Return the result
        return {"output": result}
    except Exception as e:
        # If there's an error, return it
        return {"error": str(e)}

# Start the serverless function
runpod.serverless.start({"handler": handler})

` This is running inside a Docker container with a custom Dockerfile and executed with

python3 ./handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000

python3 ./handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000

The local server starts:

--- Starting Serverless Worker |  Version 1.6.2 ---
INFO   | Starting API server.
DEBUG  | Not deployed on RunPod serverless, pings will not be sent.      
INFO:     Started server process [1]                                     
INFO:     Waiting for application startup.
INFO:     Application startup complete.                                  
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

--- Starting Serverless Worker |  Version 1.6.2 ---
INFO   | Starting API server.
DEBUG  | Not deployed on RunPod serverless, pings will not be sent.      
INFO:     Started server process [1]                                     
INFO:     Waiting for application startup.
INFO:     Application startup complete.                                  
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

But when I POST localhost:8000/run with the expected prompt input, I get a id back saying the job is in progress, but nothing shows on the container logs. It doesn't run until I call /status/jobId. Only then it runs. Is this the expected behaviour? If so, why?

8 Replies

Encyrption•5mo ago

Have you tried testing your handler offline by doing the following? It will map a local version of test_input.json inside your application.

docker run -v ${PWD}/test_input.json:/app/test_input.json myimage:v3

docker run -v ${PWD}/test_input.json:/app/test_input.json myimage:v3

This will process your test_input.json through your handler locally. Without having to run Uvicorn at all. In your Dockerfile you can start your handler like this:

CMD ["python", "/app/handler.py"]

CMD ["python", "/app/handler.py"]

EMPZOP•5mo ago

I have not, but anyway it should work well by running a local uvicorn server.

Encyrption•5mo ago

Can you share your Dockerfile? Are you doing something like this?

CMD ["python", "/app/handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000"]

CMD ["python", "/app/handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000"]

You might get more info by entering your image with a bash shell and then trying to load the ./handler from there.

EMPZOP•5mo ago

Yes, as I said, the webserver works fine, I can hit the endpoints from my local machine. The jobs won't simply start if I call /run unless I request the status afterwards. If I do /runsync, they do start immediately. My Dockerfile ends with

CMD python3 -u ./handler.py

CMD python3 -u ./handler.py

But on development I'm launching it from a compose file where I override the command with

services:
  myservice:
    // Omitted
    command: python3 ./handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

services:
  myservice:
    // Omitted
    command: python3 ./handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Encyrption•5mo ago

I would still use -u in dev to prevent buffering but everything looks good.

Jan•2mo ago

I'm having the same problem, @EMPZ did you find a solution to this?

EMPZOP•2mo ago

Nope, I just do /runSync when testing locally.

yhlong00000•2mo ago

This is the expected behavior. The asynchronous /run endpoint requires communication with our system, so it can’t be tested locally. For local testing, use the /runSync endpoint instead. I’ll have the documentation team update this. Sorry for the confusion.

Gaming

Programming

Serverless worker doesn't run asynchronously until I request its status in local development