Ollama on RunPod
Hey all,
I am attempting to set up Ollama on a Nvidia GeForce RTX 4090 pod. The commands for that are pretty straightforward (link to article: https://docs.runpod.io/tutorials/pods/run-ollama#:~:text=Set%20up%20Ollama%20on%20your%20GPU%20Pod%201,4%3A%20Interact%20with%20Ollama%20via%20HTTP%20API%20). All I do is run the following two commands on the pod's web terminal after it starts up, and I'm good to go:
1) (curl -fsSL https://ollama.com/install.sh | sh && ollama serve > ollama.log 2>&1) &
2) ollama run [model_name]
However, what I would like to do is have these commands run automatically upon starting the pod. My initial thought was to enter the above two commands into the 'Container Start Command' field on the pod deployment page (as seen in image attached). I'm not sure how to write these start-up commands and would be grateful for any assistance.
Set up Ollama on your GPU Pod | RunPod Documentation
Set up Ollama server and run LLMs with RunPod GPUs
![No description](https://cdn.answeroverflow.com/1232707270316523562/image.png)
26 Replies
why not use ollama docker image instead?
I was just looking into that. Did you have any resources that maybe helpful? I was referring to this link: https://hub.docker.com/r/ollama/ollama#!, but I was wondering if there was an approach more suited to RunPod.
you want something like this
https://runpod.io/console/deploy?template=q5rqanpolz&ref=vfker49t
note this is api thingy not like chat via terminal
Thanks for the link. I went ahead and spun up a pod with the ollama/ollama container image. After the pod starts, would you know how to make inferences with a model (e.g. gemma).
you could pass gemma in container command
like this
![No description](https://cdn.answeroverflow.com/1232728447982506025/image.png)
I went ahead and tried "run gemma" (image attached), but I get an error message in the container logs that says:
Error: could not connect to ollama app, is it running?
![No description](https://cdn.answeroverflow.com/1232731584839290910/image.png)
Delete run
If I just have "gemma", the error messages is:
Error: unknown command "gemma" for "ollama"
try gemma:7b
It seems to be returning the same error.
try memby set image to ollama/ollama:latest
Tried this, and the error seems to be the same.
It looks like I just have to run two commands - "serve" and "run gemma", after which I should be able to make inferences with gemma, but I'm not sure how to implement that.
Thank you for all the support so far, but are there any other fixes I coudl implement to get this to work?
what if you put serve run gemma
It returns the following error:
Error: accepts 0 arg(s), received 2
It looks like the Container Start Command can only take one command.