Notification When Pod Is Deployed
Hello,
I'm an admin of a research group which uses runpod. I would like to receive a notification anytime a pod is deployed, along with it's specs (most importantly the GPU type, num GPUs, storage info, community or secure cloud, the image which is being used, and the account which launched it).
My hope is to then automate some admin things and run commands (such as automatically terminating the pod) after init.
What's the best way to do this?
Solution:Jump to solution
@muddyfootprints. we don't currently offer notifications for when a new pod is created, however I have taken note of this thread for feedback for the team which will be reviewed by leadership tomorrow.
27 Replies
You can set the startup command to a script if you want to
Or from the template itself
The thing which I'd be checking is if people are using the correct template
Not sure if there is a notification for everytime pod is deployed
So that wouldn't work
just deploy from graphql api if you want
Or a tech guy should deploy it
??
The pods
I think youre misunderstanding my use case
hmm?
You are using teams on runpod, allowing other users to create pods using their own configuration and you need the platform to report to you automatically?
Also you need to run commands inside the pod after initializing
From runpod itself, the notification available is for outage, billings, and some other but there is no notification for every each pod created i think
so you would have to query them via the graphql api and check the pods deployed, template, gpu type, etc ( by polling )
Oh wait, there is another way. by using a startup script on the pod, it has to hit a webhook endpoint with the data of the pod
are those good enough?
The thing is that I don't have guarantees that any script I write will be executed by the pod because there's no way to enforce that through runpod.
Using templates
I can't enforce a user to use a templste
Yeah you'll have to communicate them through your team
Or make a simple web ui for them to create pods
Using predefined templates
Check the graphql reference on the docs
Yes, that's the whole issue of it. I can communicate and I can set up interfaces but people can (and prob will) still set up pods without it, and what I'm looking for is an enforcement mechanism.
The hacky way around it is setting up a cronjob on a server which periodically checks the pods deployed, which seems like is what I'll need to do. It's just a pain to do so and I wish runpod had better admin infrastructure in place
Well don't invite them to runpod as administrator
Just as basic to connect only
But then someone has to be I charge of deploying pods
With the interfaces you can enforce which template to use and not let them modify the value right?
Save the API keys on the back end, not in the front end
So you can just put a fixed value under the Templateid on the graphql api
Yeah, I guess that could work instead of periodically checking. That still requires me to put up a server which I'd have to separately manage, but I can just byte the bullet if need be....
I'll have to do some thinking on which exactly is the right call. Thanks
Yeah or a light cronjob would do but it cant enforce as much as the interface
I thought about it and the cronjob is the way to go. Having a separate server is too large of a security liability. The poor user experience of getting booted is only there for people who don't follow the recs, so I'm okay with that.
Is there any chance my cronjob would get perceived as spam?
Not sure what throttling guidelines runpod has on its backend
Use a good smtp
What throttling?
You can use something like this for sending mails:
https://www.mailjet.com/
Mailjet
Email Delivery Service for Marketing & Developer Teams | Mailjet
Streamline your emailing and boost your sales with Mailjet. We serve all your email and SMS needs with one simple and powerful service.
They have a free tier and then after that you have to start paying.
You get 6000 emails per month/200 emails per day on free tier, which is probably more than enough for this.
Rate limiting. Like Serverless has an IP rate limit so you can't send too many /status requests or your IP gets rate limited. I am not sure whether GraphQL API also has rate limiting in place.
i thought you would run cronjob inside the pod and check for env variables huh
there will be no throttling there
No, the service wouldn't be coming from a pod. I can draw a systems diagram later on if that helps
Ooh its fine no need to do extra for me 🙂
Solution
@muddyfootprints. we don't currently offer notifications for when a new pod is created, however I have taken note of this thread for feedback for the team which will be reviewed by leadership tomorrow.