RunPod•3mo ago

Jobs randomly dropping - {'error': 'request does not exist'}

RunPod worker errors:

2024-10-12T18:25:21.522075786Z {"requestId": "51124010-27f8-4cfa-b737-a50e6d436623-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:25:22.723756821Z {"requestId": "51124010-27f8-4cfa-b737-a50e6d436623-u1", "message": "Finished.", "level": "INFO"}
2024-10-12T18:27:09.433322101Z {"requestId": null, "message": "Failed to get job, status code: 404", "level": "ERROR"}
2024-10-12T18:27:09.602268203Z {"requestId": "b88fe3f1-1212-4eee-acda-e5c58626b69a-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:27:11.082924318Z {"requestId": "b88fe3f1-1212-4eee-acda-e5c58626b69a-u1", "message": "Finished.", "level": "INFO"}
2024-10-12T18:29:43.434273977Z {"requestId": null, "message": "Failed to get job, status code: 404", "level": "ERROR"}
2024-10-12T18:29:43.613420319Z {"requestId": "d964329c-2abc-4931-bd8e-53f7d5089d59-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:29:44.956554990Z {"requestId": "d964329c-2abc-4931-bd8e-53f7d5089d59-u1", "message": "Finished.", "level": "INFO"}
2024-10-12T18:29:49.734447718Z {"requestId": "4cc76d9e-7e65-4b3f-afaf-5382d0bd8dd6-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:29:50.975923513Z {"requestId": "4cc76d9e-7e65-4b3f-afaf-5382d0bd8dd6-u1", "message": "Finished.", "level": "INFO"}

2024-10-12T18:25:21.522075786Z {"requestId": "51124010-27f8-4cfa-b737-a50e6d436623-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:25:22.723756821Z {"requestId": "51124010-27f8-4cfa-b737-a50e6d436623-u1", "message": "Finished.", "level": "INFO"}
2024-10-12T18:27:09.433322101Z {"requestId": null, "message": "Failed to get job, status code: 404", "level": "ERROR"}
2024-10-12T18:27:09.602268203Z {"requestId": "b88fe3f1-1212-4eee-acda-e5c58626b69a-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:27:11.082924318Z {"requestId": "b88fe3f1-1212-4eee-acda-e5c58626b69a-u1", "message": "Finished.", "level": "INFO"}
2024-10-12T18:29:43.434273977Z {"requestId": null, "message": "Failed to get job, status code: 404", "level": "ERROR"}
2024-10-12T18:29:43.613420319Z {"requestId": "d964329c-2abc-4931-bd8e-53f7d5089d59-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:29:44.956554990Z {"requestId": "d964329c-2abc-4931-bd8e-53f7d5089d59-u1", "message": "Finished.", "level": "INFO"}
2024-10-12T18:29:49.734447718Z {"requestId": "4cc76d9e-7e65-4b3f-afaf-5382d0bd8dd6-u1", "message": "Started.", "level": "INFO"}
2024-10-12T18:29:50.975923513Z {"requestId": "4cc76d9e-7e65-4b3f-afaf-5382d0bd8dd6-u1", "message": "Finished.", "level": "INFO"}

from lbeoz75vjlfck0 the request ID does not show up on the Requests tab. The error also does not get logged to the daily statistics as it seems to be a RunPod job routing issue, not a worker image runtime error. What we receive from the endpoint:

https://api.runpod.ai/v2/***/status/5c7ae484-b1df-4efd-a06d-a283b6d42e3a-u1 {'error': 'request does not exist'}

https://api.runpod.ai/v2/***/status/5c7ae484-b1df-4efd-a06d-a283b6d42e3a-u1 {'error': 'request does not exist'}

worker runpod SDK version: 1.6.2 might update once https://discord.com/channels/912829806415085598/1293773578738864158 is fixed

3 Replies

yhlong00000•3mo ago

It seems like after you sent the request, the worker woke up but was immediately stopped due to the code update you triggered. Has this happened multiple times, or was it just this one time?

marshallOP•3mo ago

It happens multiple times... What "code update" though? it seems to happen even back then

yhlong00000•3mo ago

I mean when you modify your endpoint settings, change to new docker image and etc.

Gaming

Programming

Jobs randomly dropping - {'error': 'request does not exist'}