abtx Posts - Answer Overflow

abtx

Posts Comments

RRunPod

•Created by abtx on 11/19/2024 in #⚡｜serverless

Why when I try to post it already tags it Solved?

3 replies

RRunPod

•Created by abtx on 11/19/2024 in #⚡｜serverless

HF Cache

Hey I got this email from you guys

 Popular Hugging Face models have super fast cold-start times now
We know lots of our developers love working with Hugging Face models. So we decided to cache them on our GPU servers and network volumes.

 Popular Hugging Face models have super fast cold-start times now
We know lots of our developers love working with Hugging Face models. So we decided to cache them on our GPU servers and network volumes.

Can you explain what exactly what you man by Hugging Face models cache? Docker Image Templates? Or do you mean when people fetch models from HF via internet you have them cached?

4 replies

RRunPod

•Created by abtx on 8/23/2024 in #⚡｜serverless

How to force /runsync over 60 secs

Need to keep /runsync alive for over 60 seconds. No webhooks, async. Just want the /runsync to work as is just for longer exection times.

8 replies

RRunPod

•Created by abtx on 8/23/2024 in #⚡｜serverless

Sync endpoint returns prematurely

Sync endpoint sometimes randomly (about half of the time) responds prematurely with in progress json. The job finished however, I need the sync not to respond until the job is done.

15 replies

RRunPod

•Created by abtx on 3/31/2024 in #⚡｜serverless

Do endpoints support custom images?

I am able fetch custom images from my GCP artifact registry for normal Pods. However, when I create endpoint, it forever gets stuck in "initializing" without any logs. However pre-made runpod endpoints like RunDiffusion work fine.

4 replies

RRunPod

•Created by abtx on 3/29/2024 in #⚡｜serverless

Endpoint Deployment Stuck on Initializing

4 replies

RRunPod

•Created by abtx on 3/18/2024 in #⚡｜serverless

How do I write handler for /run

When I create a handler runpod.serverless.start({"handler": async_generator_handler}) Only http://localhost:8000/runsync triggers async_generator_handler However when posting against http://localhost:8000/run async_generator_handleris not triggered. Just returns

{"id":"test-20023617-4048-4f73-9511-8ae17a1ad7a5","status":"IN_PROGRESS"}

{"id":"test-20023617-4048-4f73-9511-8ae17a1ad7a5","status":"IN_PROGRESS"}

But nothing happens in the backend

12 replies

RRunPod

•Created by abtx on 3/18/2024 in #⚡｜serverless

How do indicated job status in a handler?

For example in https://docs.runpod.io/serverless/workers/handlers/handler-async

import runpod
import asyncio


async def async_generator_handler(job):
    for i in range(5):
        # Generate an asynchronous output token
        output = f"Generated async token output {i}"
        yield output

        # Simulate an asynchronous task, such as processing time for a large language model
        await asyncio.sleep(1)


# Configure and start the RunPod serverless function
runpod.serverless.start(
    {
        "handler": async_generator_handler,  # Required: Specify the async handler
        "return_aggregate_stream": True,  # Optional: Aggregate results are accessible via /run endpoint
    }
)

import runpod
import asyncio


async def async_generator_handler(job):
    for i in range(5):
        # Generate an asynchronous output token
        output = f"Generated async token output {i}"
        yield output

        # Simulate an asynchronous task, such as processing time for a large language model
        await asyncio.sleep(1)


# Configure and start the RunPod serverless function
runpod.serverless.start(
    {
        "handler": async_generator_handler,  # Required: Specify the async handler
        "return_aggregate_stream": True,  # Optional: Aggregate results are accessible via /run endpoint
    }
)

Dopes the job status automatically becomes "COMPLETED" after async_generator_handler returns? In general how do you update the status of the job in runpod python sdk? What I am trying to achieve is to use a single machine at a time for training purposes. I am not sure 1 hour long POST request is a good idea. How should this be done?

5 replies

Gaming

Programming