RunPod•10mo ago

Custom serverless deployment

I want to host https://hub.docker.com/r/konieshadow/fooocus-api on a serverless endpoint but i am unable to find the related documentation can someone guide me through this please?

58 Replies

ashleyk•10mo ago

Docker image won't work unless its designed specifically for RunPod serverless. Docker hub doesn't seem to indicate that its compatible with RunPod serverless so it won't work. Yeah its FastAPI, can work on GPU cloud, but not RunPod serverless. You either need to fork the Github or implement the cog implementation.

ashleyk•10mo ago

https://github.com/konieshadow/Fooocus-API

GitHub

GitHub - konieshadow/Fooocus-API: FastAPI powered API for Fooocus

FastAPI powered API for Fooocus. Contribute to konieshadow/Fooocus-API development by creating an account on GitHub.

ashleyk•10mo ago

https://blog.runpod.io/replicate-cog-migration/

RunPod Blog

Serverless | Migrating and Deploying Cog Images on RunPod Serverles...

Switching cloud platforms or migrating existing models can often feel like a Herculean task, especially when it necessitates additional developmental efforts. This guide aims to simplify this process for individuals who have deployed models via replicate.com or utilized the Cog framework. Through a few straightforward steps, you'll learn how

CyberTraumaOP•10mo ago

I have to clone cog-worker and cd into it and then do docker build --tag mydockeruser/fooocus-api:latest --build-arg COG_REPO=konieshadow --build-arg COG_MODEL=fooocus-api --build-arg COG_VERSION=fda927242b1db6affa1ece4f54c37f19b964666bf23b0d06ae2439067cd344a4 . right?

ashleyk•10mo ago

Don't know, never used it, @Justin Merrell should be able to advise.

Justin Merrell•10mo ago

Did you already do this?

CyberTraumaOP•10mo ago

no just wanted to confirm what i understood was right before doing it is it right?

Justin Merrell•10mo ago

Looks correct, yes

CyberTraumaOP•10mo ago

so will it work if the run the docker image that i built locally?

justin•10mo ago

? what is the question? Didn't make sense If the question is will it run - yes - the docker image once u build > and push > should work

CyberTraumaOP•10mo ago

the port number shouldn't be an issue right ? cuz fooocus api is hosted on port 8888

justin•10mo ago

So I don't think the fooocus api is relevant here so a Cog is just a standard way to wrap a model + how to use the model And then essentially you are wrapping the model into a docker container that you can deploy on runpod, and the handler.py in runpod will call that cog so there is no "backend api server" you are spinning up, you are just directly from the handler, triggering the cog

justin•10mo ago

https://github.com/runpod-workers/cog-worker/blob/main/src/handler.py

GitHub

cog-worker/src/handler.py at main · runpod-workers/cog-worker

Cog based workers to RunPod serverless workers. . Contribute to runpod-workers/cog-worker development by creating an account on GitHub.

justin•10mo ago

What we can see here is that they essentially anything in your : {input: {key key key}} gets passed along directly to Cog - the model you are prepackaging

CyberTraumaOP•10mo ago

danggg nice

CyberTraumaOP•10mo ago

i did this and pushed it to docker and used the tag to setup serverless endpoint but since like 30 min it is stuck on initialising

CyberTraumaOP•10mo ago

Justin Merrell•10mo ago

May you paste the endpoint id here?

CyberTraumaOP•10mo ago

a9qygnltdyxj54

Justin Merrell•10mo ago

Are you using network storage or locked it to a specific region?

justin•10mo ago

looks like u forgot to tag it username/fooocus-api:tagnumber

CyberTraumaOP•10mo ago

i did in the template oh yes

justin•10mo ago

siddarthreddyg/fooocus-api:0.1 should be this right? unless the screenshot was wrong and u changed after (I still feel that runpod should add a regex validation check to these inputs 😢

CyberTraumaOP•10mo ago

yess I changed it now

justin•10mo ago

Yea haha +1 my feature request 😆 https://discord.com/channels/912829806415085598/1185822883474309181https://discord.com/channels/912829806415085598/1185822883474309181 Cool hopefully that solves ur issue u wouldnt imagine how many times i did the same thing wondering why it wouldnt work lol 👁️ it's either the max worker was set to one, which for some reason sometimes causes a weird loop, so i try to set it to 2 at minimum or i forgot a tag or something about my docker container is broken itself lol

CyberTraumaOP•10mo ago

it is running now 😭

justin•10mo ago

Nice congrats U can use their web ui to test it out But make sure ur input is structured correctly such as

{
input: {
your stuff here
}
}

{
input: {
your stuff here
}
}

https://github.com/runpod-workers/cog-worker/blob/main/src/handler.py As can see here they just pass along ur full input it looks like

CyberTraumaOP•10mo ago

so input {} has the json i send as request to fooocus right? and where do i specify the endpoint?

justin•10mo ago

Yea input has the json ur sending

justin•10mo ago

Prob something like...

{
   input: {
        "prompt": "forest elf",
        "cn_type1": "ImagePrompt",
        "cn_type2": "ImagePrompt",
        "cn_type3": "ImagePrompt",
        "cn_type4": "ImagePrompt",
        "sharpness": 2,
        "image_seed": 50403806253646856,
        "uov_method": "Disabled",
        "image_number": 1,
        "guidance_scale": 4,
        "refiner_switch": 0.5,
        "negative_prompt": "",
        "style_selections": "Fooocus V2,Fooocus Enhance,Fooocus Sharp",
        "uov_upscale_value": 0,
        "outpaint_selections": "",
        "outpaint_distance_top": 0,
        "performance_selection": "Speed",
        "outpaint_distance_left": 0,
        "aspect_ratios_selection": "1152*896",
        "outpaint_distance_right": 0,
        "outpaint_distance_bottom": 0,
        "inpaint_additional_prompt": ""
    }
}
}

{
   input: {
        "prompt": "forest elf",
        "cn_type1": "ImagePrompt",
        "cn_type2": "ImagePrompt",
        "cn_type3": "ImagePrompt",
        "cn_type4": "ImagePrompt",
        "sharpness": 2,
        "image_seed": 50403806253646856,
        "uov_method": "Disabled",
        "image_number": 1,
        "guidance_scale": 4,
        "refiner_switch": 0.5,
        "negative_prompt": "",
        "style_selections": "Fooocus V2,Fooocus Enhance,Fooocus Sharp",
        "uov_upscale_value": 0,
        "outpaint_selections": "",
        "outpaint_distance_top": 0,
        "performance_selection": "Speed",
        "outpaint_distance_left": 0,
        "aspect_ratios_selection": "1152*896",
        "outpaint_distance_right": 0,
        "outpaint_distance_bottom": 0,
        "inpaint_additional_prompt": ""
    }
}
}

https://replicate.com/konieshadow/fooocus-api?input=python

konieshadow/fooocus-api – Replicate

Third party Fooocus replicate model

justin•10mo ago

That is my guess well u can run it in the web ui if u scroll down for now but if u want to run it as a programmatic python can check out their serverless doc

justin•10mo ago

https://docs.runpod.io/tutorials/ai-apis/run-your-first#run

Run your first AI API with Stable Diffusion | RunPod Documentation

Interact with your favorite models without managing the pods yourself.

CyberTraumaOP•10mo ago

and where do i pass the endpoint?

justin•10mo ago

what do u mean the endpoint? in ur request? Oh wait lol hold on i see

justin•10mo ago

in the ID section: https://docs.runpod.io/tutorials/serverless/run-your-first

Run your first serverless endpoint with Stable Diffusion | RunPod D...

Interact with your favorite models without managing the pods yourself.

justin•10mo ago

curl -X POST https://api.runpod.ai/v2/{ID}/run \
    -H 'Content-Type: application/json'                             \
    -H 'Authorization: Bearer [Your API Key]'    \
    -d '{"input": {"prompt": "A cute fluffy white dog in the style of a Pixar animation 3D drawing."}}'

curl -X POST https://api.runpod.ai/v2/{ID}/run \
    -H 'Content-Type: application/json'                             \
    -H 'Authorization: Bearer [Your API Key]'    \
    -d '{"input": {"prompt": "A cute fluffy white dog in the style of a Pixar animation 3D drawing."}}'

Damn idk what happened to the section in their old doc maybe they missed a better one Can also refer to my github repo for examples for python

justin•10mo ago

https://github.com/justinwlin/runpod_whisperx_serverless_clientside_code

GitHub

GitHub - justinwlin/runpod_whisperx_serverless_clientside_code: Hel...

Helper functions for Runpod to automatically poll my WhisperX API. Can be adapted to other use cases - justinwlin/runpod_whisperx_serverless_clientside_code

justin•10mo ago

https://docs.runpod.io/tutorials/ai-apis/run-your-first

justin•10mo ago

yea but this section

CyberTraumaOP•10mo ago

brooo lmfao

CyberTraumaOP•10mo ago

how hard do i cry? 😂

ashleyk•10mo ago

Amazing, I may have time to convert this thing into a RunPod worker for you tomorrow. Or is the bas64 actually valid?

CyberTraumaOP•10mo ago

i think it is valid i need to check its just tooo long

ashleyk•10mo ago

Sounds like a good sign then 👍

justin•10mo ago

lmao You can prob ask chatgpt to add a bit of logic to the docker from runpod:

def run_inference(inference_request):
    '''
    Run inference on a request.
    '''
    response = cog_session.post(url=f'{LOCAL_URL}/predictions',
                                json=inference_request, timeout=600)
    return response.json()


# ---------------------------------------------------------------------------- #
#                                RunPod Handler                                #
# ---------------------------------------------------------------------------- #
def handler(event):
    '''
    This is the handler function that will be called by the serverless.
    '''

    json = run_inference({"input": event["input"]})

    return json["output"]

def run_inference(inference_request):
    '''
    Run inference on a request.
    '''
    response = cog_session.post(url=f'{LOCAL_URL}/predictions',
                                json=inference_request, timeout=600)
    return response.json()


# ---------------------------------------------------------------------------- #
#                                RunPod Handler                                #
# ---------------------------------------------------------------------------- #
def handler(event):
    '''
    This is the handler function that will be called by the serverless.
    '''

    json = run_inference({"input": event["input"]})

    return json["output"]

IN this section

CyberTraumaOP•10mo ago

https://github.com/davefojtik/RunPod-Fooocus-API/tree/Standalone

GitHub

GitHub - davefojtik/RunPod-Fooocus-API at Standalone

RunPod serverless worker for Fooocus-API. Standalone or with network volume - GitHub - davefojtik/RunPod-Fooocus-API at Standalone

justin•10mo ago

rather than return the json["output"] Before that you do: base64 = base64 convert json "url" or something return base64 wow nice xD this also works hhh

justin•10mo ago

https://github.com/davefojtik/RunPod-Fooocus-API/tree/Standalone

GitHub

GitHub - davefojtik/RunPod-Fooocus-API at Standalone

RunPod serverless worker for Fooocus-API. Standalone or with network volume - GitHub - davefojtik/RunPod-Fooocus-API at Standalone

justin•10mo ago

u want the standalone one but seems like 3wad/runpod-fooocus-api:0.3.30-standalone they also just have a docker image they deployed u can prob just try to deploy that

3wad/runpod-fooocus-api:0.3.30-standalone

3wad/runpod-fooocus-api:0.3.30-standalone

but i guess the issue is ull have to figure out their api vs cog is well documented tbh i think u can just add a bit more logic to the json u got out ask chatgpt how to grab the image and do a bit more testing with it

CyberTraumaOP•10mo ago

yeah i made my own docker using cog but how i am unable to figureout the api can you help me with that ? would the input be something like this?

    input_data = {
        "api_name": "/v2/generation/image-inpaint-outpaint",
        "inpaint_additional_prompt":"veneer teeth, white",
        "input_image": "http://0x0.st/HnXJ.png",
        "cn_image": "https://i.imgur.com/RKi2BAY.png",
        "input_mask": "https://i.imgur.com/tC6DV8I.png",
        "require_base64": True,
    }

    input_data = {
        "api_name": "/v2/generation/image-inpaint-outpaint",
        "inpaint_additional_prompt":"veneer teeth, white",
        "input_image": "http://0x0.st/HnXJ.png",
        "cn_image": "https://i.imgur.com/RKi2BAY.png",
        "input_mask": "https://i.imgur.com/tC6DV8I.png",
        "require_base64": True,
    }

input_data = {
        "api_name": "/v2/generation/image-inpaint-outpaint",
        "input" : {
            "prompt":"veneer teeth, white",
            "input_image": "http://0x0.st/HnXJ.png",
            "cn_image": "https://i.imgur.com/RKi2BAY.png",
            "input_mask": "https://i.imgur.com/tC6DV8I.png",
            "require_base64": True,
        }
    }

input_data = {
        "api_name": "/v2/generation/image-inpaint-outpaint",
        "input" : {
            "prompt":"veneer teeth, white",
            "input_image": "http://0x0.st/HnXJ.png",
            "cn_image": "https://i.imgur.com/RKi2BAY.png",
            "input_mask": "https://i.imgur.com/tC6DV8I.png",
            "require_base64": True,
        }
    }

justin•10mo ago

Uh, I really dont know that highly highly depends on ur own cog. but runpod expects:

{
  input: {
   whatever u want here
  }
}

{
  input: {
   whatever u want here
  }
}

And what it will pass to ur cog is:

{
whatever you want here
}

{
whatever you want here
}

CyberTraumaOP•10mo ago

what if my cog has 3-4 apii endpoints?

justin•10mo ago

This is what runpod passes:

def run_inference(inference_request):
    '''
    Run inference on a request.
    '''
    response = cog_session.post(url=f'{LOCAL_URL}/predictions',
                                json=inference_request, timeout=600)
    return response.json()


# ---------------------------------------------------------------------------- #
#                                RunPod Handler                                #
# ---------------------------------------------------------------------------- #
def handler(event):
    '''
    This is the handler function that will be called by the serverless.
    '''

    json = run_inference({"input": event["input"]})

    return json["output"]

def run_inference(inference_request):
    '''
    Run inference on a request.
    '''
    response = cog_session.post(url=f'{LOCAL_URL}/predictions',
                                json=inference_request, timeout=600)
    return response.json()


# ---------------------------------------------------------------------------- #
#                                RunPod Handler                                #
# ---------------------------------------------------------------------------- #
def handler(event):
    '''
    This is the handler function that will be called by the serverless.
    '''

    json = run_inference({"input": event["input"]})

    return json["output"]

https://github.com/runpod-workers/cog-worker/blob/main/src/handler.py is that possible? wouldn't the cog probably be expecting something in the JSON to differentiate? Im not familiar with Cogs, but isnt the cog just a standard way to wrap an ML model for the prediction? so it probably some sort of if-statement right based off of the json?

CyberTraumaOP•10mo ago

https://github.com/konieshadow/Fooocus-API/blob/074a956d2fc6e12e9b669cfe6611fd576dd7e315/docs/api_doc_en.md#image-inpaint-outpaint

GitHub

Fooocus-API/docs/api_doc_en.md at 074a956d2fc6e12e9b669cfe6611fd576...

FastAPI powered API for Fooocus. Contribute to konieshadow/Fooocus-API development by creating an account on GitHub.

justin•10mo ago

Yea that isnt the cog that just a backend api that spins up to call the cog https://github.com/konieshadow/Fooocus-API/blob/074a956d2fc6e12e9b669cfe6611fd576dd7e315/predict.py This is the Cog https://replicate.com/konieshadow/fooocus-api?input=python U can look at their python Json to see sort of how it would look being passed into the cog Guessing what u would give runpod looks like below:

{
input: {
        "prompt": "forest elf",
        "cn_type1": "ImagePrompt",
        "cn_type2": "ImagePrompt",
        "cn_type3": "ImagePrompt",
        "cn_type4": "ImagePrompt",
        "sharpness": 2,
        "image_seed": 50403806253646856,
        "uov_method": "Disabled",
        "image_number": 1,
        "guidance_scale": 4,
        "refiner_switch": 0.5,
        "negative_prompt": "",
        "style_selections": "Fooocus V2,Fooocus Enhance,Fooocus Sharp",
        "uov_upscale_value": 0,
        "outpaint_selections": "",
        "outpaint_distance_top": 0,
        "performance_selection": "Speed",
        "outpaint_distance_left": 0,
        "aspect_ratios_selection": "1152*896",
        "outpaint_distance_right": 0,
        "outpaint_distance_bottom": 0,
        "inpaint_additional_prompt": ""
    }
}

{
input: {
        "prompt": "forest elf",
        "cn_type1": "ImagePrompt",
        "cn_type2": "ImagePrompt",
        "cn_type3": "ImagePrompt",
        "cn_type4": "ImagePrompt",
        "sharpness": 2,
        "image_seed": 50403806253646856,
        "uov_method": "Disabled",
        "image_number": 1,
        "guidance_scale": 4,
        "refiner_switch": 0.5,
        "negative_prompt": "",
        "style_selections": "Fooocus V2,Fooocus Enhance,Fooocus Sharp",
        "uov_upscale_value": 0,
        "outpaint_selections": "",
        "outpaint_distance_top": 0,
        "performance_selection": "Speed",
        "outpaint_distance_left": 0,
        "aspect_ratios_selection": "1152*896",
        "outpaint_distance_right": 0,
        "outpaint_distance_bottom": 0,
        "inpaint_additional_prompt": ""
    }
}

which then runpod would take ur input, and pass it to the Cog predictor function Cog is essentially a function So you just call the function. The backend server you showed is like, oh if you go to this URL, Ill pass these parameters to the Cog function, but u dont have a backend server ur spinning up, ur just directly calling the Cog function with the json input

CyberTraumaOP•10mo ago

ohh okay okay now it is slowly starting to make sense but in the backend i have an option to send image as base64 how do i do that here?

justin•10mo ago

Not 100% sure my guess is that you'll need to modify the handler a bit: https://github.com/konieshadow/Fooocus-API/blob/074a956d2fc6e12e9b669cfe6611fd576dd7e315/predict.py If you look here it expects a :Path type, (thank god for this author having good typing) - it probably means FilePath is my guess

uov_input_image: Path = Input(default=None, 
                                description="Input image for upscale or variation, keep None for not upscale or variation"),

uov_input_image: Path = Input(default=None, 
                                description="Input image for upscale or variation, keep None for not upscale or variation"),

What that probably means is that when you get the json:

def handler(event):
    '''
    This is the handler function that will be called by the serverless.
    '''

    json = run_inference({"input": event["input"]})

    return json["output"]

def handler(event):
    '''
    This is the handler function that will be called by the serverless.
    '''

    json = run_inference({"input": event["input"]})

    return json["output"]

Before you pass it, you prob go like: does xyz key exist, if so, write the base64 to an image, and replace the base64 value with the path i wrote too. Before it runs the inference U can try to trace down how their API endpoint works, but that just my best guess looking at it

Gaming

Programming

Custom serverless deployment