ENDPOINT IS
#⛅｜gpu-cloud Hi to you all, can somebody please tell where to find the "endpoint" code ? I would like to connect to my GPU Cloud based by using Python !
It would be grand if somebody can post here an example of a working Python code to connect and use the GPU !
Thanks a lot to all those that would like help 😆
53 Replies
That is for serverless if u want to send a request
if u want to ssh and write code using vscode or using the web that is another thing
U cant just connect the gpu to ur local computer, u either send it a request through an api or u ssh or connect to the cloud gpu computer and u copy / write ur files there and run it
thanks a lot Justin ! super fast...can you please provide an example Python code ? 🙂
Docs to their serverless 🙂
And then in my whisperx transcription, under clientcode i have exs of how i make requests to my endpoints for transcription through api requests
and then if u want to make ur own custom docker i recommend chatgpt + the learning how to construct a dockerfile tips as a starting point
along with runpod has blogs
u can search
If u need infras to help build the images
https://discord.com/channels/912829806415085598/1194693049897463848
U can use depot along with docker to speed things up
Justin, u'r too kind. My question is easier: can I call from my home local computer Runpod GPU passing to it data and retrieve the answer ? that is wht I think tons of people are searching for here ! 🙂
If u want to do something like that
Runpod GPU is essentially a computer in the sky with a gpu
It is probablt easier to just
work in the browser with it
and copy ur code there
and run it
and if u want the data / output
copy it to ur machine
If u use their pytorch template for ex. u can just open up their jupyter server
U can also ssh onto the server and connect it to ur vscode
but the vscode will show what is on the server
not what is on ur computer
https://discord.com/channels/912829806415085598/1194711850223415348
https://blog.runpod.io/how-to-connect-vscode-to-runpod/
RunPod Blog
How to Connect VSCode To RunPod
While working in a terminal window is sometimes good enough, it's much nicer to work within your IDE if you are doing interactive development. Fortunately, you can connect your VSCode IDE to your RunPod instance via SSH. This will allow you to edit files and code on your remote instance
But tbh i think its easier just work in the browser
and then download ur output
using ssh and scp
which my easy ssh covers
U can set that up first, then moving onwards any computer u use in the browser
u can download ur output
thro the cli
I see...so I cannot run my Python stuff locally and use the GPU JUST like a computational engine ? 😄
No
U have to imagine trying to stream compute power over a network
is inefficient
is better to either send to response to the gpu which is running like a server and processing it
but requires more setup
ud have to setup a local served on the runpod gpu and then expose ports
have it accept requests so on
Is easier to just ssh to the computer
move ur stuff there
work there
and then get ur stuff off
Or work in the web browser
then just download it from there
I see so all the data I need to reshape using RTX A6000 has to be on the server in Runpod . . . am I finally correct ? 😄
how do I work in the web browser ? 😄
Yes
If u spin up a pytorch instance under the pod
for runpod
and the pod shows up on the browser under pods
there should be a connect button
which will say
connect to jupyter labs
P.S
I HIGHLY RECOMMEND
do easy easy ssh first
just install runpod
and ur ssh keys
before making a pod
then u can go make a pod
What are u using?
this seems custom
template
this isnt a pytorch one?
nope...
Ah I see
😄 of cours you do
is there a particular reason why u wanna use thisv
just wondering
how do I connect to this ? 🙂
u looking to use it?
Yeah
My recommendation here is
yes I will be using this 😄
Sorry just wondering
are u going to be developing
or just running code
developing as in ur writing code
both of the two !
or running xode as in just pasting commands
ok
I even do not know how to SSH to it honestly....
Id follow my easy ssh guide first
Second:
U can skip the setup ssh part now in the link and connect vscode
https://blog.runpod.io/how-to-connect-vscode-to-runpod/
RunPod Blog
How to Connect VSCode To RunPod
While working in a terminal window is sometimes good enough, it's much nicer to work within your IDE if you are doing interactive development. Fortunately, you can connect your VSCode IDE to your RunPod instance via SSH. This will allow you to edit files and code on your remote instance
And then yeah
now u have an IDE conmected to ur pod
Kill the pod
completely
before doing the easy ssh
U can make a new one after
Easy SSH will be applicable to all new pods moving forward
Is ok. Most ppl dont know at first why i wrote the easy ssh guide
all u need is ur api key
Then their cli will take care making snd storing ur ssh keys
And then u can just do the vscode stuff after
then ur vscode u can run terminal command, write code, etc
let me try please...can I keep on writing here in case I cannot make ? thanks a trillion for all this time...priceless !
Yes
If u have data u must absolutely not lose btw u can add on a network storage but for now
sounds like it just testing
for now lets not comolicate it xD
yes yes...I need to send some data to the pod for summarisation and have it back. THAT'S IT 🙂 can I do it using VScode and SSH ?
Yes
lemme try . .
Interesting wanna use the llm on the gpu rather than chatgpt lol
must be alot of data
thanks a trillion for this nice and easy help super appriciateds since it's a rare thing !
lemme give it a try, im gonna use the VS Code SSH guide u pasted here.
do the easy ssh first that i linked in the discord link
THEN the vscode guide
https://discord.com/channels/912829806415085598/1194711850223415348
Just need to do steps 1-4
trying, toooo kind maaaan !!! 😄
....but even if I set up the SSH, I cannot reshape my data in VSCode locally to then send it to RunPod and have it back 😦 correct ?
sorry, I just would like to be clear for me and for all the ones that will be reading this thread
Reshaping data
depends
if it just python idk why u need to do it locally vs remote u can do either
U can either reshape it locally, then copy it over to ur pod
OR copy the data to ur pod
reshape it there
and then use it for whatever u want on the server
then u can just download the data back
https://discord.com/channels/912829806415085598/1195736144999088238/1195787659705073714
Ah someone said maybe can also just drag and drop for vscode once connected to ssh. maybe is doing scp under the hood unsure
?
ahhh I see the other thread ! but come on....what I wanna do is easy and I cannot imagine I cannot do it 😄
Let's see ... still trying what you proposed and again thanks a trillion Justin, too kinf 😄
Yeah.
1) Easy SSH
2) Connect VsCode
3) Prob see if just dragging and dropping ur files to the file section of vscode after works lol
thanks a lot ! basically...what I would like to do is connecting to my GPU Runpod instance using APIs like in this post 🙂 :
chregu on tour
How to setup and use a runpod.io pod with mixtral 8x7b
Or any other OSS LLM model from huggingface. Just quick and dirty instructions to reproduce it.
would it be possible ???
the part that I need is at the end of the post...this one:
RunPod
Expose Ports
There are a few ways to expose ports on your pod to the outside world. The first thing that you should understand is that the publicly exposed port is most likely NOT going to be the same as the port that you expose on your container. Let's look at an example to illustrate this.Let's say that I want...
Ull need to launch a local server on ur pod + then expose the port
Ive never tried this on runpod but an easy way i do it on my local computer is:
https://ngrok.com/
ngrok | Unified Application Delivery Platform for Developers
ngrok is a secure unified ingress platform that combines your global server load balancing, reverse proxy, firewall, API gateway and Kubernetes Ingress Controller to deliver applications and APIs.
that way I can start a local server on the remote server (or usually my local computer), lets say it is localhost:8081
and I just do ngrok http 8081
and I get a public URL that i can send endpoints to
one sec . . . 😄
Yeah tbh, I feel this unless ur used to backend development.. will fly above ur head xD
My recommendation is... use a pytorch template on runpod
Ollama
Get up and running with large language models, locally.
Go through the installation step for Ollama AI 🙂
Then you'll get an ollama server just running ready to take API requests locally
and from there figure out the, how do I get my port exposed issue
Ollama is way better structured imo than TheBlokeTemplateLLM
But (also if ur not planning to use this like an api.. since ur copying ur files to the remote GPU and just running it there, and copying the response out)
I see no reason to do this
Honestly, I am confused by your use case though, I think you need to clarify:
Your use case sounds like:
1) You have data
2) You need to preprocess it
3) You want to feed it to an LLM model
4) You want to copy that response out.
5) Your objective is not to have this always available, through an API call, but rather a manual workflow.
If so, I recommend, use a Runpod Pytorch template, download Ollama, work with it in the jupyter server through the browser.
https://ollama.ai/
Ollama
Get up and running with large language models, locally.