RunPod•2mo ago

Errors while downloading image from s3 using presigned urls

Hey! I am having occasional errors when I try to download image from s3 bucket using presigned urls inside my serverless worker. The worker is processing around 100K requests per hour and only around (rather less) 10K of them fail due to s3 download error. The error message is (I replaced keys and buckets names here):

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='bucketname.s3-accelerate.amazonaws.com', port=443): Max retries exceeded with url: /bucket/f0bb45e2.png?AWSAccessKeyId=AAWSAccessKeyId&Signature=Signature%3D&Expires=1715540566 (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))

I am not sure this is the right place to ask this question, but maybe some of you faced anything similar. I also have additional api layer between runpod worker and client application which also uses s3 buckes for upload and dowload and it never had any issues regarding downloading from presigned urls, that's why I start to think some issue may be withitn my runpod workers

7 Replies

Madiator2011•2mo ago

for me it looks like presigned url is not complited?

agentpietrucha•2mo ago

what do you mean exactly? I am using the following to get presigned url:

boto_client.generate_presigned_url(
            "get_object",
            Params={
                "Bucket": bucket,
                "Key": key,
            },
            ExpiresIn=36000,  # 10 hours
        )

And most of the time it works. But sometimes it doesn't

Madiator2011•2mo ago

why not use rp_download?

agentpietrucha•2mo ago

what is rp_download? I couldn't find anything related to python

Madiator2011•2mo ago

https://github.com/runpod/runpod-python/blob/main/runpod/serverless/utils/rp_download.py

GitHub

runpod-python/runpod/serverless/utils/rp_download.py at main · runp...

🐍 | Python library for RunPod API and serverless worker SDK. - runpod/runpod-python

Madiator2011•2mo ago

here is good place to see good worker examples https://github.com/runpod-workers