RunPod•2mo ago

Automatically shut down pod after being idle for X mins

Hi guys, a general question: I'd like to automatically shut down my pod after being idle for 30 mins (so I can complete several trainings overnight). I've found this command bash -c "nohup sleep 2h; runpodctl stop pod $RUNPOD_POD_ID" & but the thing is that I have to pre-define after which period the pod gets shut down. Is there any functionality that automatically monitors the pod's status and shuts it down after being idle for X mins?

4 Replies

Encyrption•2mo ago

Could you monitor the GPU usage and then shut down X minutes after GPU VRAM goes to 0, or similar?

cyberzen•2mo ago

Yes that would be a nice solution. How could I achieve this? 🙂

Encyrption•2mo ago

I'm not sure if this is official or not but below is how I poll the GPU in JavaScript, using graphQL.

const myJSON = JSON.stringify({query: `
        query Pod {
                pod(input: {podId: "${POD_ID}"}) {
                        id
                        name
                        runtime {
                                uptimeInSeconds
                                ports {
                                        ip
                                        isIpPublic
                                        privatePort
                                        publicPort
                                        type
                                }
                                gpus {
                                        id
                                        gpuUtilPercent
                                        memoryUtilPercent
                                }
                                container {
                                        cpuPercent
                                        memoryPercent
                                }
                        }
                }
        }`
});

const myJSON = JSON.stringify({query: `
        query Pod {
                pod(input: {podId: "${POD_ID}"}) {
                        id
                        name
                        runtime {
                                uptimeInSeconds
                                ports {
                                        ip
                                        isIpPublic
                                        privatePort
                                        publicPort
                                        type
                                }
                                gpus {
                                        id
                                        gpuUtilPercent
                                        memoryUtilPercent
                                }
                                container {
                                        cpuPercent
                                        memoryPercent
                                }
                        }
                }
        }`
});

cyberzen•2mo ago

Thanks a lot!

Gaming

Programming

Automatically shut down pod after being idle for X mins