RunPod•3mo ago

AWS ECR Registry Authentication

Is it possible to use a private AWS ECR as a container registry as opposed to Docker Hub? Although RunPod allows username and password credentials to be configured (which would work with Docker Hub) there does not appear to be a feature to use AWS IAM credentials. The AWS CLI allows an IAM principal to obtain a password using aws ecr get-login-password however this is only valid for 12 hours so would need to be cycled regularly but there does not appear to be a programatic way of doing this in RunPod. This question was put to the ai-helper but it was unable to provide a resolution on this (https://discord.com/channels/912829806415085598/1118945694863065230/1288092289977290762). I know this question isn't Serverless specific but I couldn't find a better place to put this question and we're using RunPod Serverless. I also think it might apply to Serverless more so than Pods due to the potential nature of having to pull images more frequently (potentially every cold start?).

8 Replies

nerdylive•3mo ago

Try searching on #⛅｜pods too for similar thread discussing this

JackOP•3mo ago

Ah yes, there is a thread discussing this issue however I was unable to find it initally due to the incorrectly named post title. For ref it is this post. Unfortunately this post does not resolve the second point in my question with regards to the 12 hour expiry of the AWS provided password. This makes using AWS ECR very limited as, realistically, only public container images could be used because the credentials to get a private image would otherwise expire after 12 hours. The AWS documentation on this is here. Are there any work arounds for this and does RunPod plan to support AWS credentials in the near future?

yhlong00000•3mo ago

short term, you could consider create a cron lambda function, call our graphql to update the saved reg credential every couple hours. We plan to support token based registry in Q4, will let you know.

JackOP•3mo ago

Thanks for getting back @yhlong00000. To confirm, this would look like calling the saveRegistryAuth GraphQL mutation every couple of hours which looks like it would create a new credential each time. And then, with this new credential we'd need to update the template using the saveTemplate mutation (including the template ID to distinguish between creating and updating the template). Does this sound right to you?

yhlong00000•3mo ago

actually, you don't need to change the template, only need to call

// update key:
mutation UpdateRegistryAuth($input: UpdateRegistryAuthInput) {
  updateRegistryAuth(input: $input) {
    id
    name
  }
}

// input
{
  "input": {
    "id": "your_auth_id",
    "username": "username",
    "password": "password"
  }
}

// update key:
mutation UpdateRegistryAuth($input: UpdateRegistryAuthInput) {
  updateRegistryAuth(input: $input) {
    id
    name
  }
}

// input
{
  "input": {
    "id": "your_auth_id",
    "username": "username",
    "password": "password"
  }
}

you can also check this PR: https://github.com/runpod/runpod-python/issues/317

GitHub

Update & delete registry auth credentials · Issue #317 · runpod/run...

Is your feature request related to a problem? Please describe. We're using RunPod pods to train AI models based on our Docker images, which we store in ECR. The ECR credentials expire every 12 ...

JackOP•3mo ago

Ah that looks perfect! Exactly what I was looking for. Thanks for sending this over. I wasn't able to find this from the docs or the GraphQL reference though. I'll have a look at the GitHub issue 👍 Is there some up to date documentation for the GraqhQL API? I'm struggling to find these and other queries & mutations. The GraphQL Spec here seems to be out of date and doesn't have the endpoints you've mentioned. This doesn't prevent me from using the mutations you've mentioned but it would be nice to know what else I can do with this API

yhlong00000•3mo ago

yeah, sorry we current don't, we are working on a rest api, and build better doc for that, will share it when it available.

JackOP•3mo ago

ok, thanks for confirming 👍

Gaming

Programming

AWS ECR Registry Authentication