james3000
RRunPod
•Created by james3000 on 8/20/2024 in #⚡｜serverless
Job has missing field(s): input
I try to use the response on a serverless LLM
meta-llama/Meta-Llama-3.1-8B-Instruct
with the following JSON:
{
"messages": [
{
"role": "system",
"content": "..."
},
{
"role": "user",
"content": "..."
},
{
"role": "assistant",
"content": "..."
}
]
}
and I keep getting the following in the logs:
[error]
Job has missing field(s): input.
I also tried the python code using the example code runpod recommends but I keep getting a 401? Has anyone got this working?
Also I have got authorised from Hugging face so the serverless LLM is installed ok without any errors?
3 replies
RRunPod
•Created by james3000 on 8/19/2024 in #⚡｜serverless
meta-llama/Meta-Llama-3-8B-Instruct serverless
I am bit confused, trying to get this tested using Python but it seems to point me to using openai in the tutorial @ https://docs.runpod.io/serverless/workers/vllm/get-started
Can we still use the openai python library or we need to use another one to connect to the endpoint? Can anyone help me please?
2 replies
RRunPod
•Created by james3000 on 8/19/2024 in #⚡｜serverless
With LLM on runpod is there a cost like other providers like tokens and if its serverless
Hi we want to run a LLM on runpod but I am concerned about running serverless as its pretty slow and we need the LLM to be pretty much instant? The other thing is we dont want to run a GPU all the time as it ends up costing a lot? Can someone out there give me some advice please?
7 replies
RRunPod
•Created by james3000 on 8/7/2024 in #⚡｜serverless
Hello
We are looking to build a medical app that will process Australian medical data from a AI model and we will be using Serverless to do this, does anyone know the legislation around sending Australian medical data overseas and I know that serverless doesn't store data long term but will it still be ok or so we need to have our own GPU servers in Australia? Thanks
4 replies
RRunPod
•Created by james3000 on 6/5/2024 in #⚡｜serverless
flashboot adding cost?
hello everyone!
just got a question about flashboot does it add additional cost to serverless? I cant seem to find anything definative to answer this question?
4 replies
RRunPod
•Created by james3000 on 5/30/2024 in #⚡｜serverless
is anyone experiencing a massive delay time when sending jobs to GPUs on serverless?
We are sending jobs off to our whisper serverless functions and experiencing massive delay times sometimes and sometimes it just goes through quickly? At the moment we are just testing so we are using a single 16GB GPU? Has anyone got any advice on this?
7 replies