zfmoodydub
RRunPod
•Created by zfmoodydub on 11/4/2024 in #⚡|serverless
not getting any serverless logs using runpod==1.6.2
using FROM runpod/worker-vllm:stable-cuda12.1.0
base image
2 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
it seems as though downgrading the package worked. thank you very much for your help
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
i see what you mean now. downgrade the version in the container i build. sorry for my misunderstanding
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
the whole process would probably be abt 200 lines
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
~15 seconds in. and i cant see any of my debug logs because of that
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
right, but this whole thread ive been talking about how those logs disappear halfway through the execution.
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
thats what im trying to do as there seems to be a bug in the UI
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
i still think i am unable to retrieve container logs via the runpod library though, is that correct?
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
got it
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
also on github there is no release for 1.6.2, only 1.6.1 and 1.7.0.
brew install returns this:
brew install [email protected]
No available formula with the name "[email protected]". Did you mean runpodctl?
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
great thanks. just to be clear, im not able to retrieve any logs via the ctl tool. testing via the UI, the logs stop, perhaps they are too verbose, i get a message in the container logs saying:
No Container logs yet, this usually means that the pod is still initializing
and my process halts about halfway through, and the worker goes idle. so even if im downgrading the ctl tool to try to extract the logs, im not sure they will even be available. i will check though.
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
@yhlong00000 not sure if you can help...
the requests are now getting lost as well, they get placed in a queue but then are never returned as failed.
here is another example:
pod id: h6uc0sa88m2n5t
request id: e2980df5-7861-46b7-8c9a-65a4171c30ad-u1
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
no luck getting log stream with runpodclt
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
runpodctl v1.14.4
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
so after reading about runpodclt, this library is meant for streaming logs to your machine, and you were not asking what version of the runpod library i was using in my code.
i have only been using the UI for testing. I will now try to use runpodclt to stream logs to see if i can get a better idea of what is happenning later in my process
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
import runpod
print(f"Runpod version: {runpod.version}")
Runpod version: 1.7.3
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Runpod GPU use when using a docker image built on mac
i have not, not the most seasoned engineer, and havent had much luck successfully deploying my apps with anything else but with this base image:
runpod/base:0.4.0-cuda11.8.0
so ive really only been using that base image for my apps for about a year
25 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
while im waiting to get you the exact version, im using python 3.8. so whatever version gets installed automatically in that python version.
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Worker frozen during long running process
going to take me 10 mins to build the image again to run a version command and check but will shoot it in here asap
38 replies
RRunPod
•Created by zfmoodydub on 10/24/2024 in #⚡|serverless
Runpod GPU use when using a docker image built on mac
when i run a process in a particular pod that im seeing the issue, it does say that "cuda is not available using cpu" but another serverless pod (the one im talking to you about in a different thread) using the same base image does not have this problem. so i believe this is an internal code thing in my repository.
after noticing that, i do not think this is a runpod problem. I can dig deeper there. thanks for your responses!
25 replies