Running LLaMA remotely from a Python script
Hi, new here, relatively new to LLMs as well. VERY new to remote computing.
I've got a llama-cpp python script that I've been tinkering with on my laptop, and I'd like rent a remote GPU to speed up text generation.
I haven't found a tutorial on how to do what I really want. I'd like to be able to simply hit 'run' from vscode and have the LLM set up on a remote GPU - so that I can send my prompts over, and receive back the generated text.
I'm testing a system with an untraditional prompting system, so I can't just use an existing webUI, and I'd prefer to develop from my IDE than on Jupyter. Anyone got any tips or could point me in the right direction?
8 Replies
im no expert in vs code, but i dont think you can execute remote commands like that.
what you can do, is "deploy" your python script into the runpod servers and execute manually, but that would require you to know how to work the cli im afraid.
Maybe a jupiter notebook can help
I'm not opposed to learning, it's the age of AI after all
I found out there's actually a VScode template, so I deployed a GPU on that and that was nice, but now I'm trying to see how I can get it to work with the nice tkinter GUI I made
I was banging my head against the wall trying to figure out ssh displays with ChatGPT to no avail
TKinter doesn't work with RunPod. The Docker containers are headless and don't have a physical display attached.
Yeah, ChatGPT said something about X11 forwarding
I tried to set that up but it doesn't work and I'm not sure why
Like the ssh config says X11 forwarding is enabled but it doesn't seem to connect to VCXSRV on my computer
use putty for X forwarding in windows
ChatGPT suggested that as well, I tried that, turned my private key into a ppk, and then opened it up and got a "login as" prompt, and if I interpreted correctly the container didn't really work on that basis
I mean should Putty be working just fine with runpod?
Am I missing something here?
putty is a terminal emulator
with tons of features, like full ssh support, including X11 forwarding
what happens is, the app runs in the remote server, and tryes to connect to the screen manager (aso known as X11). the remote server is not runing X11, so it fails
with X11 forwarding, the remote app tryes to connect a X11 runing on another machine
putty creates the ssh tunnel between the remote mahcine and your local machine
so, in teory, your app should be able to contact X11 on your local machine, efectively running the app remotely while showing the GUI locally
one of the funnest couple of nights i spend figuring out tech, in my younger linux days
thing is, nowaways, linux doesnt use X anymore. and dont know how it would work with runpod proxying and port forwarding, so i cant even begin to tell if if it will work
Hahaha okay, thanks for letting me know though, I'm giving putty another shot
I'm gonna make another post just for putty stuff