'Background' options for Pod Initiated file transfer
I'm trying to scope out if there's a solution to have a runpod send me back a small .db/txt file on completion of task, or of progress before closing due to being outbid and closed (Community pods)
I've been looking at rsync, runpodctl, SSH, and they all seem to require transfer to be 'initiated' from the recipient machine
I'm looking at the google drive API, which I think is going to be my best bet for an 'always ready to receive' solution.
Anyone else already have experience working around something similar?
Solution:Jump to solution
You might need something like this, detect the signal and do something:
import signal
import boto3
import os...
7 Replies
Solution
You might need something like this, detect the signal and do something:
import signal
import boto3
import os
from time import sleep
S3_BUCKET = 'your-s3-bucket-name'
FILE_TO_UPLOAD = '/path/to/your/file.txt'
S3_KEY = 'uploaded-file.txt'
def upload_to_s3():
s3_client = boto3.client('s3')
try:
s3_client.upload_file(FILE_TO_UPLOAD, S3_BUCKET, S3_KEY)
print(f"File {FILE_TO_UPLOAD} uploaded to S3 bucket {S3_BUCKET} as {S3_KEY}")
except Exception as e:
print(f"Failed to upload file: {e}")
def handle_signal(signal_number, frame):
print(f"Received signal {signal_number}. Saving file to S3.")
upload_to_s3()
exit(0)
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
print("Running... Press Ctrl+C to exit.")
try:
while True:
sleep(1)
except KeyboardInterrupt:
handle_signal(signal.SIGINT, None)
Thanks for the signal handling stub, I'll definitely use it. I don't have S3, but should work with gdrive api as well
I'm not a huge fan of the gdrive API, I've used it for another project, and it requires you to renew your credential auth, with a webpage warning you have to click through, which really throws a wrench in things since it'll probably trigger a request if being accessed by a foreign IP on whatever pod tries to use it
You can get the sig signals on outbid
and do that in the 5 second range i guess
But for googledrive, i don't think its an api except you're using colab, so s3 is the best alternative yeah
I'll have to look at S3, if it's easier to set up than google drive api, I'll definitely use it. Some of my other projects are going to be AWS so it's already on the todo list
Yes, gdrive outside of colab (I don't use colab) still req you to either register your api access as an app or juggle reauth tokens with a webpage permission window. Unless, I'm doing it some profoundly wrong way, which could be the case since it looks like theres a handful of different options for getting API access (registering a service, etc) that are way too deep into the google-sphere for me
Haha yeah okay Goodluck on that
do you know of another way to get files into gdrive without wrestling with API permissions? Just for clarity sake, I'd like to know if I'm missing something obvious
I'm plugging s3 into my image right now, the python boto3 library for AWS stuff looks pretty straightforward, at least for something simple like file up/downloading
Yeah s3 is meant for that, and for gdrive not really.... Never used that