how to make an ai
ive never made an ai chat bot model work because my pc doesnt have the power to run python models wondering if i can run one on an i7-870 and nvidia geforce gt 710
6 Replies
ping me please if anyone can help
@LunaTheEquinoctialCat as in make / train your own custom model or use a pretrained one ?
pretrained
i havent done it myself, so im not too much of use but go on youtube and figure out how to load very small pretrained models first, then use QLoRA to load in larger models
QLoRA quantizes the larger models making them significantly easier to run / use, but not for training, though im not sure how this translates to finetuning
ok i have no clue what im looking for
If you just want a GUI for running models locally you can use https://github.com/oobabooga/text-generation-webui
Or if you want to integrate it into a python program you can use https://huggingface.co/docs/transformers/index
Your GPU is very low end though so it'll only be able to run smaller models if it works at all. Something like https://huggingface.co/EleutherAI/gpt-neo-2.7B is probably the largest you can possibly run and that's if you use 4-bit quantization. Using the 7B param models that are popular nowadays would require at least 6GB of VRAM. You could also try running 7B param models on your CPU if you've got enough system memory but it'll be slow. Though honestly probably not much slower than that GPU.