Jamb
Runpod occasionally fails to pull from ECR
As per this message , it looks like the
containerRegistryAuthId input field for the REST API isn't working properly. We will have to wait for a fix on runpod's side before we can test if that's the actual issue or not.11 replies
Runpod occasionally fails to pull from ECR
Update: I was able to delete my entries via API.
We worked with dstack and believe the issue is due to graphQL not having an input for registry auth on pods creation/start, so you have to edit the pod after and assign the registry auth...and we think it's a weird timing issue.
Looks like the REST API has this input , so they will change their automation to use REST to see if it resolves this issue
11 replies
Runpod occasionally fails to pull from ECR
They do, however with
dstack, I automatically take care of this process and make sure I have a valid ECR password every time.
Looking into it, I looks like dstack creates Container Registry Auth entries with the username and password and then links the pull command to the correct entry. This could an issue on their end, as it's not cleaning up old entries. I now have 6 of them (with the docs stating I can only have a max of 4) and I can't manually delete any of them. 😦11 replies
Unable to modify owner of network volume
A bit more testing here and I find network volume behaviour strange. Maybe someone can clarify what's going on here.
- I'll mount the network volume to a more normal place, say
/data/mount . Whatever user you ssh into the container as, is who the owner is? This is weird. ie, if I ssh as root , it's owned by root. If I ssh as <my-container-user> it's now owned by that user
- I can't properly change permissions on files on the network mount. It applies the same permissions across owner, group and everybody and I can't set different permissions for these.20 replies
Unable to modify owner of network volume
We use
/home inside our dev environment containers though. We build up an environment our researchers and devs are familiar with, with the stipulation that their user is <our-company's-generic-user> home dir is /home/<our-company's-generic-user> and not /home/<individual-user> . This works great with AWS and EBS volumes and creates a persistant home dir for users.
BTW we utilize dstack20 replies