justin Comments - Answer Overflow

Topics

justin

•Created by TheWitcher.... on 10/26/2024 in #⚡｜serverless

How to send an image as a prompt to vLLM?

So before sending it in json encode as base64, then decode on the other end. as long the image is not too big

5 replies

•Created by TheWitcher.... on 10/26/2024 in #⚡｜serverless

How to send an image as a prompt to vLLM?

Need to base64 encode + decode it

5 replies

•Created by lunarlander on 9/7/2024 in #⛅｜pods

Trying to install Ollama: 'could not resolve host'

maybe madiator's template would work?

5 replies

•Created by lunarlander on 9/7/2024 in #⛅｜pods

Trying to install Ollama: 'could not resolve host'

Have u tried this? just curious

5 replies

•Created by lunarlander on 9/7/2024 in #⛅｜pods

Trying to install Ollama: 'could not resolve host'

https://discord.com/channels/912829806415085598/1261031850651029524

5 replies

•Created by Gustavo Monti on 8/28/2024 in #⛅｜pods

How to override ollama/ollama image to run a model at startup

I personally couldn't ever figure out how to bake an ollama model into my template

6 replies

•Created by Gustavo Monti on 8/28/2024 in #⛅｜pods

How to override ollama/ollama image to run a model at startup

2) Uh maybe a bash script where it starts the server in the background on startup and starts the download of the model, or change the path of where it checks to ur network volume.

6 replies

•Created by Gustavo Monti on 8/28/2024 in #⛅｜pods

How to override ollama/ollama image to run a model at startup

Just b/c one, using Ollama seems to require some configurations to get it workign with GPU properly

6 replies

•Created by Gustavo Monti on 8/28/2024 in #⛅｜pods

How to override ollama/ollama image to run a model at startup

I recommend to look here

6 replies

•Created by Gustavo Monti on 8/28/2024 in #⛅｜pods

How to override ollama/ollama image to run a model at startup

https://discord.com/channels/912829806415085598/1261031850651029524

6 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

No description

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

No description

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

**make sure that the runpod by edit template, has the port exposed on port 5000

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

from flask import Flask

app = Flask(__name__)

@app.route('/hello', methods=['GET'])
def hello_world():
    return {"text": "hello"}

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

from flask import Flask

app = Flask(__name__)

@app.route('/hello', methods=['GET'])
def hello_world():
    return {"text": "hello"}

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

Okay step by step manually at least: I create a runpod template on: runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 Create a virtualEnv: python3 -m venv venv source venv/bin/activate pip install flask Run the flask app: python main.py

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

maybe someone else can help on the exposing ports programatically, i saw ur thing here

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

also not too sure about dockerfile port exposing, i tend to do it through the console. so maybe this is where we are diverging

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

basically proxy is for http requests (what u wanna do) if u wanna do more, can do the public ip method

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

https://docs.runpod.io/pods/configuration/expose-ports

29 replies

•Created by Shaplow on 8/28/2024 in #⛅｜pods

How to send request to a pod ?

u can expose over public ip if it supports it

29 replies