Encyrption
RRunPod
•Created by DDV-PYRMD on 7/3/2024 in #⚡｜serverless
serverless
If you haven't already I suggest you check out https://blog.runpod.io/custom-serverless-api-step-by-step-tutorial/
9 replies
RRunPod
•Created by AMooMoo on 7/6/2024 in #⚡｜serverless
Question about Network Volumes
1) Actually it has been found that using a network volume actually increases the cold startup time of serverless endpoints. It also decreases response time with Flashboot. In most every case you are better off storing models, etc., directly in container storage. Network volume is the cheapest storage on RunPod, but IMHO it is not worth using it.
2) If you attach a network volume to your image it will be attached to /runpod-volume on your serverless endpoint when it runs.
8 replies
RRunPod
•Created by DDV-PYRMD on 7/3/2024 in #⚡｜serverless
serverless
Yeah, I don't think that is a good way to go about it. You should be able to run inference from a python script without having to run a web server. Find the code that the webserver is running and convert that into a script that takes in ["input'] and returns the JSON. Put that code in your handler function. Since return is an image file you'll either need to convert it into BASE 64 or upload it to a S3 bucket.
9 replies
RRunPod
•Created by DDV-PYRMD on 7/3/2024 in #⚡｜serverless
serverless
It looks like you are trying to run a web server inside of your handler. That will not work. The RunPod module will handle gettting the POST data from the end user and passing that info to your handler function and you can get input from there. Like this:
def handler(job):
job_input = job['input']
You don't need to run a web server inside your image.
9 replies
What are the rules that RunPod follows to cache DockerHub images?
Have you heard the term "physician heal thyself"? RunPod could deploy this and others easily using their own infrastructure. Just need to run one or more persistent containers on each serverless runpod host. Then they could provide any number of services for their customers. Since such services would not consume GPU running them would not impact availability for users.
15 replies
RRunPod
•Created by ibrahim on 7/2/2024 in #⚡｜serverless
My serverless does not deploy the new releases
Have you tested your image locally?
43 replies
RRunPod
•Created by ibrahim on 7/2/2024 in #⚡｜serverless
My serverless does not deploy the new releases
43 replies
RRunPod
•Created by ibrahim on 7/2/2024 in #⚡｜serverless
My serverless does not deploy the new releases
I have found that with :latest it will not update. Rather if you specify a specific version i.e. :1.01 when you change it to 1.02 (or other) the update will work. You cannot use :latest and assume it will update. It will not. You have to edit the template and provide a tag runpod has never seen.
43 replies
RRunPod
•Created by Nafi on 6/29/2024 in #⚡｜serverless
What is meant by a runner?
There are no options for building images on RunPod. But, you can build and test it locally. In your docker run command just add a volume to test_input.json like this:
docker run --rm --gpus all -v ./test_input.json:/app/test_input.json myimage
./test_input.json should contain input for your endpoint. This example assumes that your WORKDIR is /app and you must have a GPU on your local system. This will put your image into test mode and it will parse your test_input as if it had been posted to your endpoint.
19 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
RunPod has teased about adding backend networks that serverless can connect to and along with your external servers. Until that happens OpenVPN is is pretty close to that.
105 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
Yes, combine that with openvpn connection and your in business.
105 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
With that said I am in the process of building out a front end that acts as web socker server and client... sometimes it is the best way to go.
105 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
Assuming you don't care about hyper focusing on encryption there is very little overhead. You could send frames via SCP or you could mount a disk from the server... If you want to keep it off disk then your could receive it via a webhook.. really anyway you can on network.
105 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
Also, for <- my server -><- runpod -> you are not limited to websocket. If it were me I would have servless connect to my_server via a openvpn. Really easy to setup and automate.
105 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
I'll take a look at the source, might be able to adapt.
105 replies
RRunPod
•Created by MrAssisted on 6/27/2024 in #⚡｜serverless
Exposing http ports on serverless
With GenDJ would it be possible to have it process video from a file rather than live? Take in mp4 return mp4?
105 replies
RRunPod
•Created by jvm-cb on 6/27/2024 in #⚡｜serverless
Maximum queue size
That really sheds light on the subject. With max workers set higher it really removes the concern, for me at least, about how many items can remain in QUEUE.
57 replies
RRunPod
•Created by jvm-cb on 6/27/2024 in #⚡｜serverless
Maximum queue size
Yes, I mis understood what you were saying. Right now it is just me setting up/testing so I never need more than 1 but 30 sounds good for max workers in production.
57 replies