Training for days
I want to train my model for days using a single GPU. How do I keep my Jupyter Notebook session to persist even after I close my laptop so that training continues?
6 Replies
If its on runpod, then it should just keep running, but ur best bet in case jupyter notebook crash is use something like tmux to execute your script in the background, then u can turn off ur terminal and stuff, and not worry about the script just stopping.
https://youtu.be/nTqu6w2wc68?si=eT995bcH8hXZ7xEi
Make sure you backup some sort of checkpoint to a network drive or third party storage tho, every so often
NetworkChuck
YouTube
you need to learn tmux RIGHT NOW!!
Spin up your next project with Linode: https://ntck.co/linode –You get a $100 Credit good for 60 days as a new user!
I just started using Tmux……it’s amazing! If you use a terminal or CLI in any capacity Tmux will 10x your productivity in 10 seconds. From creating multiple panes and windows with ease to leaving your terminal sessions active as...
in case something goes wrong
Will the notebook keep running if I get disconnected from WiFi or shutdown the laptop?
If it in runpod it shouldn't but you can give it a try, b/c you have to imagine
its essentially another computer
out there
jupyter notebook is just a local server running on runpod's infrastructure
so even if u disconnect or connect to it, doesn't affect whether it keeps running or not
for long term training I recomend running with ssh and tmux
Yep, tmux is slightly more difficult to use than screen though, so screen may be easier.