drained of my funds somehow. HELP??
hey guys i dont know who would be able to help me out here but i had set up a serverless endpoint with a custom template. all it does is generate a custom image when the user clicks to generate one. it runs me less than $0.20 a day, usually less. But on one particular day, I was charged me entire account funds ($24) and i truly dont know why that's happened. how could the worker be running all day? how didn't it time out? and also, im pretty sure it wasn't on my end because i have an idle timeout set to 5 minutes maximum so i truly don't know what's going on. can someone help me?
attached is the screenshot of average usage + the time i was charged everything:
it's funny because the day before, i reloaded funds (Nov 22, $25), and then the next day i was essentially drained of all my funds (Nov 23, a little more than $24).
does any one know what could possibly be going wrong?
for context: i have only 1 serverless endpoint running a custom a1111 docker image with 5 serverless workers configured on RTX A4000 and RTX A4500 machiens
37 Replies
im assuming there was a job that just kept running and was never killed? is there a work around for this?
OOh it shows big usage in your serverless
i guess, maybe send your endpoint id here, maybe a staff can check the log
or check the worker logs if it its still ther
@nerdylive
Endpoint: grrjm4di7nvl75
It just happened again over night
I changed my code and everything so that it was near impossible to keep a working running for long but it still happened
@chee
Escalated To Zendesk
The thread has been escalated to Zendesk!
You can try changing it to go to the hour granularity to check which hour it happens, maybe you can get more information there
Ok I’ll do that in a sec not on my laptop rn
Send the ticket too
Okay nice
@nerdylive
so it was only 5 requests it seems like and none of them were constantly running if im reading this right
somehow still went from $16 to 0
so there were only 5 requests on dec 11 and it said they were all completed but my account was still drained
Woah, what's in the logs
Check the logs of each worker if the number of workers is small
wait a second:
there was one running until 4am
how is this even possible
Maybe there is something wrong in your code, or a bug
Any logs you can get?
What are you running there btw
so basically it's jsut a simple image generation. it's running a custom a1111 stable diffusion image generation
so each worker that's running should just generate an image then just turn off
trying to figure that out now
You took from an open source one?
Maybe check also your changes, that might lead to this problem
uhhh, i believe so? what i did to get here was that i was running the a1111 ui client on my own pc but i wanted that specific, customized version to be hosted on a server for my app and cost the least amount possible for genreations so i found out how to make it into a docker image/template and made it serverless
does that answer the question
also when should i expect a response from zendesk
also i super appreciate your help @nerdylive
just dont want the account to get drained like this when my app is live, you know?
unless im doing it wrong, i cant find any logs for yesterday
Yeah I understand, but don't worry tho because something like this most likely is because your code, yeah maybe best zendesk can help you, response around a day or two after creating I guess
hmm ok ill take your word for it.
would zendesk or anyone be able to take a look at my code to help me out with it?
it shouldnt be too complicated
Maybe they'll try to check the endpoint first if they need any more information, then yep they will try to ask you for your code
of course im using cursor to help but i dojnt know if its understanding the xontext well
ok
Have you tried recreating your endpoint too?
no, why would that help
Nah, I was guessing there can be something wrong with he worker but yeah it should act the same
Meanwhile check your code too, and logs if you want to debug it yourself
To see which errors out ( don't leave the endpoint running, keep an eye for it)
ok, yeah ill look at the logs while its running htis time but is there a way to check the logs from yesterday that im not seeing?
On the logs tab supposedly
If you log anything, the history will be there
this is all i see
Hmm yeah seems like that's an error, I'm not sure of what it means any other?
also for more context, i have each worker set to 1 second idle timeout but the execution of each worker to run before being terminated set to 600 seconds (10 minutes)
dont know if that has to do with anything but just thought id put it in here
Maybe try rebuilding your image with the latest runpod python sdk ( the container image)
hmm, i would do that normally but it took me so long to set up that whole thing
it probably IS a code problem like you're saying
no i just see that one
I'm guessing it's the return code
@nerdylive is it possible to get on a call with you and show you my code
K sure quick one