ImportError: version conflict: '/opt/micromamba/envs/comfyui/lib/python3.10/site-packages/psutil/_ps
I'm spinning up a new pod and copying from backblaze B2, it works just fine before the download but after I get a 502 error when I try to load the Jupyter notebook. I updated SSL to 3.2.0. Using the community ghcr.io/ai-dock/comfyui:latest-jupyter template.
Any idea what I can do here?
Any idea what I can do here?
16 Replies
Try
micromamba install -n jupyter -c conda-forge openssl=3.2 && supervisorctl restart jupyter@reb82 Thank you for responding so quickly! Unfortunately I'm still getting the same error:* ImportError: version conflict: '/opt/micromamba/envs/comfyui/lib/python3.10/site-packages/psutil/_psutil_linux.abi3.so' C extension module was built for another version of psutil (5.9.5 instead of 5.9.0); you may try to 'pip uninstall psutil', manually remove /opt/micromamba/envs/comfyui/lib/python3.10/site-packages/psutil/_psutil_linux.abi3.so or clean the virtual env somehow, then reinstall*
I tried pip uninstall psutil but that created a problem with ssh
You mentioned that you upgraded openssl initially. How did you do that? The default environment ships with 3.2 through mamba. Was it an apt update?
Unless it's critical to fix the existing install it may just be simpler and more cost effective to destroy the pod. If it's a synced environment I'd probably just delete the /workspace/environments directory and start over.
Yes @reb82 it was an apt update, because the initial error I got said that openssl_3.2.0 was not found. This is a backup that I started on another pod, same template, but I can barely use it because I am getting '0 GPUs' most of the time. I've tried to spin up a new pod and download my backup to it 3 times and the same thing keeps happening - works fine until I download my backup to it. And I don't believe there is anything wrong with that pod since I just used it last night. Is it possible that the backup has a different openssl that's causing this?
@reb82 I tried manually removing /opt/micromamba/envs/comfyui/lib/python3.10/site-packages/psutil/_psutil_linux.abi3.so now I'm getting the following error: ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/opt/micromamba/envs/comfyui/lib/python3.10/site-packages/psutil/init.py)
Yes that's entirely possible. Once the sync is done then it's never touched again by the image. That way you can be sure that even if the docker image is updated then no new environment stuff will be brought in. It's assumed that if you sync it then you want to maintain it so my scripts should never interfere.
Moving a backup in is (probably) possible, but you'd need to delete any existing environment at /workspace/environments, move your backup into its place and then restart the pod -- and hope for the best. It might not work and it's not designed to... But it might.
All that being said, I'd still recommend starting over. You shouldn't lose anything you'd just need to reinstall the nodes
@reb82 Got it. I've been having an issue retaining GPUs -= pretty much whenever I spin up a pod, I'll get 2-3 days out of it before it has 0 GPUs so frequently that it's almost impossible to use. Is there something that I'm missing here about ways to work consistently? It took me a while to install the nodes and troubleshoot the inevitable conflicts that arise when I drop in a workflow.
The absolute best way is to build your own image containing everything you need for your workflow.
You can do it fairly easily by forking my repo and editing build/COPY_ROOT_EXTRA/opt/ai-dock/bin/build/layer1/init.sh and pushing. GitHub will build the image for you.
Alternatively you can use a network volume (rather than a pod volume). You're restricted to secure cloud and it can be slow (really slow) to do the initial sync because of moving small files onto MooseFS but once it's on it should be fine
@reb82 Ok the network volume route is probably the best for me. Does the pod that I'm trying to move the backup from also have to be on the network volume?
Yes - if you got backup data on a current pod without a network volume - to my knowledge your best bet is to download the files using runpodctl, or just right clicking and clickign download
(if they are small enough the right click method is prob fine)
and then spin up a new pod and do whatever you need to
if you got multiple files, and just in general, zip first, then download or push to whatever data storage u need
@justin It's like 200GB of data, with all the nodes and models. Is it possible to use my backblaze backup and sync to cloud to do this or must I use runpodctl? (I'm not great with command line yet, if I do have to use runpodctl, is there a script somewhere that I can copy?).
I think it would be faster to just download them again into the new pod, assuming they are public models.
Backup images you've generated and any workflow stuff, but I wouldn't download models to then have to reupload
@reb82 ok cool, I've deployed your jupyter template to my network volume, cloud syncing my backup now, I'll get back in a little bit when it's done. Really appreciate the help!
No problem. It's late here so I won't be around for a bit but ping me if you have issues and I'll pick back up when I can. Hopefully all goes smoothly anyway 🤞
@reb82 Welp that didn't work, not sure what happened but the logs kept printing the same stuff over and over again. At this point I'm just going to redownload everything from my original pod onto one that's on the network volume. If I'm correct in understanding, that should alleviate the problem if I want to migrate my /workspace to a new pod when the current one has 0 GPUs, yes?
Hm. it isn't downloading?
It should be that you should be able to do something like, mount a pod to a volume, run download to the pod
then mv the files to the runpod-volume folder
i wouldn't download straight to the runmpod-volume due to speed issues
@justin Syncing with Backblaze actually seems to work as far as downloading straight to the pod', but it's just that once the sync completes the pod won't launch Jupyter