Oryza sativa
Oryza sativa
PostsComments
RRunPod
Created by Oryza sativa on 3/25/2024 in #⚡｜serverless
How to Run Text Generation Inference on Serverless?
but i just curious, it is using vllm right? so is runpod now support using TGI for deploying model in serverless? https://github.com/huggingface/text-generation-inference
11 replies
RRunPod
Created by Oryza sativa on 3/25/2024 in #⚡｜serverless
How to Run Text Generation Inference on Serverless?
I am sorry, already get the response, I think becuse I hit the endpoint but my endpoint still on initializing status, isn ot ready yet. and finally get the response with this. thank you @ashleyk 
-d '{"input": {"prompt": "What is Deeplearning?", "sampling_params": {"max_tokens": 100, "n": 1, "presence_penalty": 0.2, "frequency_penalty": 0.7, "temperature": 0.3}}} '
-d '{"input": {"prompt": "What is Deeplearning?", "sampling_params": {"max_tokens": 100, "n": 1, "presence_penalty": 0.2, "frequency_penalty": 0.7, "temperature": 0.3}}} '
11 replies
RRunPod
Created by Oryza sativa on 3/25/2024 in #⚡｜serverless
How to Run Text Generation Inference on Serverless?
No description
11 replies
RRunPod
Created by Oryza sativa on 3/25/2024 in #⚡｜serverless
How to Run Text Generation Inference on Serverless?
is it also support text generation inference?
11 replies