RunPod•4mo ago

Llama 3.1 + Serveless

I´m trying to use this tutorial: https://discord.com/channels/912829806415085598/1266059553838202990/1266059553838202990 Tried to use: pooyaharatian/runpod-ollama:0.0.8 and override the default start with llama3.1 but getting this error: { "delayTime": 16752, "error": "model "llama3.1" not found, try pulling it first", "executionTime": 156, "id": "f3687a15-700f-4acf-856a-d7df024ad304-u1", "status": "FAILED" } into the logs: 2024-09-02 14:52:09.063 [info] The model you are attempting to pull requires a newer version of Ollama. Tried to update to pooyaharatian/runpod-ollama:0.0.9 but getting some JSON decoded errors.

8 Replies

Madiator2011 (Work)•4mo ago

you are using outdated version latest one is pooyaharatian/runpod-ollama:0.0.9

Gustavo MontiOP•4mo ago

Tried to update the version but getting some JsonDecoded errors using llama3.1:8b

Madiator2011 (Work)•4mo ago

what is your input? @Gustavo Monti Some user reported that the request should look like this:

{
 "input": {
   "method_name": "api/generate",
   "input": {
     "prompt": "why the sky is blue?"
   }
 }
}

{
 "input": {
   "method_name": "api/generate",
   "input": {
     "prompt": "why the sky is blue?"
   }
 }
}

Gustavo MontiOP•4mo ago

@Madiator2011 (Work) with api/generate I´m getting this error: { "delayTime": 8104, "error": "model "llama3.1:8b" not found, try pulling it first", "executionTime": 3919, "id": "fd7e2a5e-6ec7-4b98-a6c4-ec5cd23abf6d-u1", "status": "FAILED" }

Madiator2011 (Work)•4mo ago

do you use network storage? also what do you see in worker logs

Gustavo MontiOP•4mo ago

No. Only container disk!

Gustavo MontiOP•4mo ago

I´ve found this: https://github.com/pooyahrtn/RunpodOllama/issues/12 Maybe need to create another version 0.0.10 with an llama version updated

GitHub

unable to pull llama 3.1 on 0.09 · Issue #12 · pooyahrtn/RunpodOlla...

I've updated my container image to latest version 0.09 but when i try to use llama3.1 Runpod serverless Container image: pooyaharatian/runpod-ollama:0.0.9 Container start command: llama3.1:8b I...

Gaming

Programming

Llama 3.1 + Serveless