Topics

RunPod•3mo ago

What is the meaning behind template on serverless?

I would like to understand how to create a serverless RunPod environment to run vLLM to host a LLM model. What is the purpose of template? Although it's optional, it seems a template can be pre-created and utilised. What is the meaning behind template? Thanks

2 Replies

houmie•3mo ago

I think this is just a way to save a set of settings for reusability.

haris•3mo ago

Yep pretty much! We have a vLLM template for serverless so you can deploy any LLM from huggingface pretty easily

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

12KMembers

View on Discord

Want results from more Discord servers?

Add your server

More Posts

Unable to connect to PodSince last Friday I have been unable to connect to my pod. It worked fine Thursday and now whenever I am having trouble finding the location of the model file when trying to use ComfyUI.I have edited the 'extra_model_paths.yaml' file, but I still can't seem to find it.Turn on Confidential ComputingHi! I created a pod using a H100 and tried to do some tests with Confidential Computing but it turn Poor upload speedI am uploading a large files (2GB) from a serverless endpoint in CA-MTL-1. I get consistenly low spe "SSH Public Keys" in account settings are completely ignoredHello, I am trying to access the env variables, as well as standard `PUBLIC_KEY` variable, that I sp Costs for increasing worker countHello, I wanted to know what the cost of increasing my worker count to 30 is? I know that it requi Expose S3 boto client retry config for endpoints (Dreambooth, etc)Note: posting this here since I can't in "feature requests" section. I'm currently experiencing (and Costs have skyrocketedPlease explain how my costs have skyrocketed while nothing has changed with my endpoints and my usag