Geri Comments - Answer Overflow

Topics

Geri

•Created by Volko on 4/17/2024 in #⛅｜pods

is AWQ faster than GGUF ?

hi do you use tensorrt llm?

10 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

hi does someone know to configure a config.pbtxt for onnx or pytorch?

66 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

and does torch-tensorrt work with an older gpu like a g4dn.xlarge?

66 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

where can i find which torch-tensorrt version is compatibel with cuda, torch etc? is it expected that pip install torch-tensorrt==2.2.0 installs both: nvidia-cuda-runtime-cu11 and nvidia-cuda-runtime-cu12 .. same for nvidia-cudnn-cu11 and nvidia-cudnn-cu12 ... and some other nvidia packages?

66 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

has someone tried torch.compile?

66 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

i want to use pytorch with sentence transformers from huggingface (https://github.com/huggingface/setfit) and do a torch.compile and run predictions

66 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

can i just do a docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:23.10-py3?

66 replies

•Created by sbhavani on 4/1/2024 in #⚡｜serverless

NGC containers

im looking for a pytorch docker container without runpod

66 replies

•Created by Dhruv Mullick on 4/5/2024 in #⛅｜pods

TensorRT-LLM setup

hi guys - is someone using torch tensorrt?

53 replies

•Created by aikitoria on 2/24/2024 in #⛅｜pods

ngc tritonserver container image not usable?

hi - is anybody here to run llama2 with tensorrt-llm and trition inference server backend?

14 replies