Beta test Llama3 serving and GUI on MAX Nightly
You can beta test the upcoming MAX serving and Llama3 chatbot GUI using the
nightly
release, I'd love feedback if you run into any problems.
Start the GUI by running these commands on arm64 macOS
or x86/arm64 Linux
(Note the smallest llama3 model is 4.5GB):
You can also ssh into a machine with vscode and run the above commands in the terminal, which will forward ports so you can run the GUI in your local browser.
Work is being done to vastly improve the experience of getting up and running and MAX, stay tuned for that.27 Replies
@Jack Clayton It appears you have permission to the max/repo that I do not.
Is it the
git
in the [email protected]
that I haven't given permissions or ?
Ubuntu 22.04 clean install
Simply changed it to git clone https://github.com/modularml/max.git
and am able to move onOk cool thanks I'll fix the command, yes need to have a ssh key set for that to work thanks for raising
š¦ Llama3
Select a quantization encoding to download model from a predefined Model URL. If the model exists at Model Path it won't be downloaded again. You can set a custom Model URL or Model Path that matches the quantization encoding.
It does say 30minute download time btw
2.34MB/s. Is Telstra serving this up from mid 2005? š
Of course 5Gb download would have been 1/2 monthly at the time. I lived in Wollongong for a year.
Yeah it's a 8B model so 4.5GB, but with the Q4_K it's very fast on CPU and high quality outputs
Oh nice my Dad is from the Gong, very nice area
yes, we learned to put beets on our chicken burgers, š¤Æ . I went to uni for a year while my wife worked and volunteered and travelled.
Anything you want to me try once it downloads?
You can ask it any question and make sure it's working with context for the conversation, the system prompt down the bottom says it's a coding assistant, you can change that without rebuilding the model
It's llama doing the llama thing. š¦ I asked it for some mojo code and it suggested
import parallel
and used let
but it's responding and working just fine. I'm asking about going to San Francisco now with a different context.
I tried "When is the best day and time to travel into San Francisco?" and it started giving a very general answer. I asked another question before it was done, and it appears to be hung. Going out to dinner, will try some more when I get back.Cool thanks for the feedback, will make it so it locks sending another message until the current stream is finished or cancelled like ChatGPT. Much appreciated.
I stopped via the button on the upper right, I chose
q6_k
, it started the download process, I went to dinner and came back to...Killing the app and restarting worked fine (defaulted to q4), I switched it q6_k and it built find and then (eventually) started. Old AMD Ryzen 3 1700 , 64GB ram.
Thanks for that cheers
rm -rf the whole thing, got back to q4, flipped it to q6 and it loaded fine and then ran fine
Installation went without issues on my M2 Mac Book Pro, thanks for sharing this app. Works well except the issue @Darin Simmons already mentioned. Cancel button would be great
Awesome thanks for the report
I tried a frankenstein of
magic
and git
and this (great) but it didn't work. First it complained that there is no name
field in the modular/max pyproject.toml so I added name = "max"
and it appeared to work but then crashed for not finding max. I wasn't surprised that it didn't work...
But then I went to verify that the clean original is working and it is not working either. Streamlit fires, the UI works, but choosing a model returns an error about not finding max
.
Should this module be working ?
I drank coffee, removed the idiocy from my brain, installed max, confirmed max and it returns same error.Do
pixi run setup
first and it should find your max installationreturns "āØ Pixi task (setup): python find-max.py" , do I need to run python find-max.py?
Works without running. Thanks for the help. I'll try the same in the magic project and see if it fires off as well.
Cheers, when we have
magic
nightly releases going, you won't have to do the setup stepJust tried this on the latest nightly with nighty mojo & max installed. I've confirmed
mojo -v
returns mojo 2024.8.1005 (34d14290)
and max -v
returns max 2024.8.914 (96cec001) Modular version 2024.8.914
. However I still get this errorI also get
ModuleNotFoundError: No module named 'max'
for StableDiffusion, even though nightly max and nightly mojo are both on the path in my .zshrc
and my .zprofile
For stable diffusion, run
pixi run setup
first to fix it, for the first one it should be fixed on the next nightly release thanks for raisinghm no luck even after running
pixi run setup
š¦ I'm not sure if I'm doing something offcan you paste your
pyproject.toml
after running pixi run setup
?it's not finding the MAX wheels, have you installed with
modular install nightly/max
?
Do you have MODULAR_HOME
set to the correct spot?Confirming that I've just run
modular clean
and modular install nightly/max
Both my zshrc and zprofile have the following lines
modular-cli is also up to date at modular 0.9.1 (3460d12e)