R
RunPod•2mo ago
zkreutzjanz

Clone endpoint failing in UI

{
    "errors": [
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        },
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        }
    ]
}
{
    "errors": [
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        },
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        }
    ]
}
User input,sensitive information removed: 
{"operationName":"saveEndpoint","variables":{"input":{"gpuIds":"AMPERE_48,ADA_48_PRO,-NVIDIA A40,-NVIDIA L40","gpuCount":1,"allowedCudaVersions":"","idleTimeout":5,"locations":null,"name":"myendpoint-dev (cloned)","networkVolumeId":null,"scalerType":"QUEUE_DELAY","scalerValue":4,"workersMax":1,"workersMin":0,"executionTimeoutMs":600000,"template":{"containerDiskInGb":25,"containerRegistryAuthId":"","dockerArgs":"","env":[{},{],"imageName":"my-imaage","name":"endpoint-dev (cloned)__template"}}},"query":"mutation saveEndpoint($input: EndpointInput!) {\n  saveEndpoint(input: $input) {\n    gpuIds\n    id\n    idleTimeout\n    locations\n    name\n    networkVolumeId\n    scalerType\n    scalerValue\n    templateId\n    userId\n    workersMax\n    workersMin\n    gpuCount\n    __typename\n  }\n}"}
{"operationName":"saveEndpoint","variables":{"input":{"gpuIds":"AMPERE_48,ADA_48_PRO,-NVIDIA A40,-NVIDIA L40","gpuCount":1,"allowedCudaVersions":"","idleTimeout":5,"locations":null,"name":"myendpoint-dev (cloned)","networkVolumeId":null,"scalerType":"QUEUE_DELAY","scalerValue":4,"workersMax":1,"workersMin":0,"executionTimeoutMs":600000,"template":{"containerDiskInGb":25,"containerRegistryAuthId":"","dockerArgs":"","env":[{},{],"imageName":"my-imaage","name":"endpoint-dev (cloned)__template"}}},"query":"mutation saveEndpoint($input: EndpointInput!) {\n  saveEndpoint(input: $input) {\n    gpuIds\n    id\n    idleTimeout\n    locations\n    name\n    networkVolumeId\n    scalerType\n    scalerValue\n    templateId\n    userId\n    workersMax\n    workersMin\n    gpuCount\n    __typename\n  }\n}"}
17 Replies
zkreutzjanz
zkreutzjanz•2mo ago
seems to be related to "allowedCudaVersions" changing in some way
digigoblin
digigoblin•2mo ago
I think its better to remove allowedCudaVersions from your payload than to leave it empty
zkreutzjanz
zkreutzjanz•2mo ago
Can’t control that because it is how the UI is built
digigoblin
digigoblin•2mo ago
oh damn, yeah sorry oversight from my part, I think @nerdylive or @justin [Not Staff] can help by pinging the UI dev(s).
justin
justin•2mo ago
@zkreutzjanz @flash-singh Bug? I think if you want to provide more info @zkreutzjanz can maybe put under #🧐｜feedback as a [Bug]
flash-singh
flash-singh•2mo ago
@zkreutzjanz can you share endpoint id? we can try to replicate using that
zkreutzjanz
zkreutzjanz•2mo ago
6fiz1j5a45xg0u Looks like only reason why is because it now adds 
"allowedCudaVersions"
:""
"allowedCudaVersions"
:""
which is invalid graphql maybe?
flash-singh
flash-singh•2mo ago
oh empty one is not allowed, @Zeke
zkreutzjanz
zkreutzjanz•2mo ago
But also fails when you add fyi see: "allowedCudaVersions":"12.1,12.2"
kaj
kaj•2mo ago
hm, I can't seem to replicate this on my own, can you detail what steps in the UI you take to reproduce this issue?
kaj
kaj•2mo ago
empty string works just fine for me
No description
No description
kaj
kaj•2mo ago
doesn't seem to be allowedCudaVersions causing the issue might be that the GPU IDs you're sending aren't quite right?
flash-singh
flash-singh•2mo ago
is it possible you reached max workers?
digigoblin
digigoblin•2mo ago
You never really know what the actual problem is with GraphQL, it never gives a useful error. It is a nightmare trying to debug GraphQL payloads.
zkreutzjanz
zkreutzjanz•2mo ago
No, have plenty of workers left Just hit three dots, clone, deploy And here is the exact request: 
{"operationName":"saveEndpoint","variables":{"input":{"gpuIds":"AMPERE_48,ADA_48_PRO,-NVIDIA A40,-NVIDIA L40","gpuCount":1,"allowedCudaVersions":"","idleTimeout":5,"locations":null,"name":"facerecognition-dev (cloned)","networkVolumeId":null,"scalerType":"QUEUE_DELAY","scalerValue":4,"workersMax":1,"workersMin":0,"executionTimeoutMs":600000,"template":{"containerDiskInGb":5,"containerRegistryAuthId":"clhpo24ed0008le085dfb6npb","dockerArgs":"","env":[{"__typename":"EnvironmentVariable","key":"RUNPOD_BUCKET","value":"deep-test-bucket"},{"__typename":"EnvironmentVariable","key":"CREDS_PATH","value":"/app/creds.json"}],"imageName":"zachdeepshot/deep-facerecognition-service-dev:c44a24d9806136cb2df0a7a36e4d8bcfed5a8961-serverless","name":"facerecognition-dev (cloned)__template"}}},"query":"mutation saveEndpoint($input: EndpointInput!) {\n  saveEndpoint(input: $input) {\n    gpuIds\n    id\n    idleTimeout\n    locations\n    name\n    networkVolumeId\n    scalerType\n    scalerValue\n    templateId\n    userId\n    workersMax\n    workersMin\n    gpuCount\n    __typename\n  }\n}"}
{"operationName":"saveEndpoint","variables":{"input":{"gpuIds":"AMPERE_48,ADA_48_PRO,-NVIDIA A40,-NVIDIA L40","gpuCount":1,"allowedCudaVersions":"","idleTimeout":5,"locations":null,"name":"facerecognition-dev (cloned)","networkVolumeId":null,"scalerType":"QUEUE_DELAY","scalerValue":4,"workersMax":1,"workersMin":0,"executionTimeoutMs":600000,"template":{"containerDiskInGb":5,"containerRegistryAuthId":"clhpo24ed0008le085dfb6npb","dockerArgs":"","env":[{"__typename":"EnvironmentVariable","key":"RUNPOD_BUCKET","value":"deep-test-bucket"},{"__typename":"EnvironmentVariable","key":"CREDS_PATH","value":"/app/creds.json"}],"imageName":"zachdeepshot/deep-facerecognition-service-dev:c44a24d9806136cb2df0a7a36e4d8bcfed5a8961-serverless","name":"facerecognition-dev (cloned)__template"}}},"query":"mutation saveEndpoint($input: EndpointInput!) {\n  saveEndpoint(input: $input) {\n    gpuIds\n    id\n    idleTimeout\n    locations\n    name\n    networkVolumeId\n    scalerType\n    scalerValue\n    templateId\n    userId\n    workersMax\n    workersMin\n    gpuCount\n    __typename\n  }\n}"}
and response 
{
    "errors": [
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        },
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        }
    ]
}
{
    "errors": [
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        },
        {
            "message": "Something went wrong. Please try again later or contact support.",
            "locations": [
                {
                    "line": 1,
                    "column": 23
                }
            ],
            "extensions": {
                "code": "BAD_USER_INPUT"
            }
        }
    ]
}
same response here so not a gpu is issue: 
{"operationName":"saveEndpoint","variables":{"input":{"gpuIds":"ADA_24","gpuCount":1,"allowedCudaVersions":"","idleTimeout":5,"locations":null,"name":"facerecognition-dev (cloned)","networkVolumeId":null,"scalerType":"QUEUE_DELAY","scalerValue":4,"workersMax":1,"workersMin":0,"executionTimeoutMs":600000,"template":{"containerDiskInGb":5,"containerRegistryAuthId":"clhpo24ed0008le085dfb6npb","dockerArgs":"","env":[{"__typename":"EnvironmentVariable","key":"RUNPOD_BUCKET","value":"deep-test-bucket"},{"__typename":"EnvironmentVariable","key":"CREDS_PATH","value":"/app/creds.json"}],"imageName":"zachdeepshot/deep-facerecognition-service-dev:c44a24d9806136cb2df0a7a36e4d8bcfed5a8961-serverless","name":"facerecognition-dev (cloned)__template"}}},"query":"mutation saveEndpoint($input: EndpointInput!) {\n  saveEndpoint(input: $input) {\n    gpuIds\n    id\n    idleTimeout\n    locations\n    name\n    networkVolumeId\n    scalerType\n    scalerValue\n    templateId\n    userId\n    workersMax\n    workersMin\n    gpuCount\n    __typename\n  }\n}"}
{"operationName":"saveEndpoint","variables":{"input":{"gpuIds":"ADA_24","gpuCount":1,"allowedCudaVersions":"","idleTimeout":5,"locations":null,"name":"facerecognition-dev (cloned)","networkVolumeId":null,"scalerType":"QUEUE_DELAY","scalerValue":4,"workersMax":1,"workersMin":0,"executionTimeoutMs":600000,"template":{"containerDiskInGb":5,"containerRegistryAuthId":"clhpo24ed0008le085dfb6npb","dockerArgs":"","env":[{"__typename":"EnvironmentVariable","key":"RUNPOD_BUCKET","value":"deep-test-bucket"},{"__typename":"EnvironmentVariable","key":"CREDS_PATH","value":"/app/creds.json"}],"imageName":"zachdeepshot/deep-facerecognition-service-dev:c44a24d9806136cb2df0a7a36e4d8bcfed5a8961-serverless","name":"facerecognition-dev (cloned)__template"}}},"query":"mutation saveEndpoint($input: EndpointInput!) {\n  saveEndpoint(input: $input) {\n    gpuIds\n    id\n    idleTimeout\n    locations\n    name\n    networkVolumeId\n    scalerType\n    scalerValue\n    templateId\n    userId\n    workersMax\n    workersMin\n    gpuCount\n    __typename\n  }\n}"}
flash-singh
flash-singh•2mo ago
@kaj did we get to the bottom of this?
kaj
kaj•2mo ago
not yet, looking into a possible solution
Want results from more Discord servers?
Add your server
More Posts
Automatically Terminate Idle PodsI want to write a daemon which will automatically terminate my pod if the GPU has sat idle for x amoIs there any limit on how many environment variables can be added per container?It seems i can't add any more environment variables on my serverless. the button is greyd out disablSlow downloadI'm currently getting 2mb/s download on 2xa100 pod; normally get way higher than this -- anyone else`ERROR | InternalServerError: None is not in list`I was using several machines but faced the same error(above). Someone said it was due to the ddos athow to host 20gb models + fastapi code on serverlessI have 20gb model files and a fastapi pipeline code to perform preprocessing and inference+ trainingNeed help putting 23 GB .pt file in serverless enviornmentI have a 23 GB .pt file containing tensors for 36 attention processors for each step, cannot reduce Increase Spot Warning TimeI see in the docs that there is a 5s window before a spot instance is interrupted. 5s isn't really eControlNet does not seem to work on Serverless APIHey guys, before everything, I'm not a native English speaker so sorry in advance if I got somethinhow to route docker secrets to pod automaticallyI have some credentials saved as runpod secrets. After creating a new pod using runpodctl, I manuallimage deprecated?When i put all the variables in (trying with phi-4k-mini) it says image is deprecated