Running serverless endpoint locally
Hi,
I'm trying to run locally, but can't find a solution in documentation. I tried running handler with docker and outside docker.
running with
--rp_serve_api
ERROR: [Errno 99] error while attempting to bind on address ('::1', 8000, 0, 0): cannot assign requested address
running with --rp_serve_api --rp_api_host='0.0.0.0' exits with error:
ERROR: [Errno -2] Name or service not known
Is there some other sources I can look to solve this?
Have a nice day!
Mateusz23 Replies
another app is using the address on ('::1', 8000)
Yeah you can either stop the other app or use a different port for running the handler
Solution
that will make it listen on port 8010 instead of 8000
@digigoblin thanks! in docs and arts passing value is after
=, I didn't try without it, my brain didn't notice that it may be wrong. It's running when passing value without = sign
@digigoblin is there a way to define types for input and output to be generated in openapi file?Not sure what you mean.
openapi schema generation based on code.
by default openapi file is generated when running serverless handler. It generates schemas and for request
DefaultRequest by default. It defines input as object. When creating some custom endpoint we can make complex structures and it would be nice to put it into schemayou can check an example of schema for rp's handler in here
https://github.com/ashleykleynhans/runpod-worker-a1111/tree/main/schemas
https://github.com/ashleykleynhans/runpod-worker-a1111/blob/main/rp_handler.py#L87
GitHub
runpod-worker-a1111/schemas at main · ashleykleynhans/runpod-worker...
RunPod Serverless Worker for the Automatic1111 Stable Diffusion API - ashleykleynhans/runpod-worker-a1111
GitHub
runpod-worker-a1111/rp_handler.py at main · ashleykleynhans/runpod-...
RunPod Serverless Worker for the Automatic1111 Stable Diffusion API - ashleykleynhans/runpod-worker-a1111
it doesnt generate openapi file i guess but its for validation
RunPod allows OpenAPI compatible endpoints for the vllm worker, so you may want to look at using that.
GitHub
GitHub - runpod-workers/worker-vllm: The RunPod worker template for...
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
wdym openapi compatible endpoints?
You don't have to hack your worker to support them, its supported natively in RunPod serverless
GitHub
GitHub - runpod-workers/worker-vllm: The RunPod worker template for...
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
Oh you mean openai? I think that's not what he mean
He asked about it here.
Sorry he said openapi not openai, my bad.
You don't need openapi stuff with serverless.
yeah, I was analysing all of those workers, that's why I'm asking questions here, because I can't find answers in other places 🙂
Thanks!
yeah i think validation hasnt been documented yet..
np
why? They are still endpoints and in a team it would be nice to have openapi to share contracts automatically. you then can do some automated tests to make sure contracts are still valid. especially with proper CI/CD pipelines across the project and multiple repositories
You can't do this with serverless, only with an API on a pod. Serverless has fixed endpoints like
/run and /runsync, it doesn't use custom endpoints, and also its already an API so you don't implement an API in your handler.I was just wandering if some overwriting would be possible to extend base definition, because in most cases
input will have some structure defined. If don't then I understand. Thanks a lot for help!You have to put the entire contents of your request within the
input key of the payload, and then your response will be in the output key fo the payload. Its not possible to extend it.yes, and you can put something in this
input and it can be an object, so it may be complex structure which you can define. so you could narrow input type generated currently with your type. so in the end it could be somehow "overwritten" or "extend" (you name it, it's just semantics) to have in openapi file (which is currently generated) more precise definition what intput and output are, not some generic list of every type.