Runpod•13mo ago

Enable performance counter on runpod

Hi, I'm trying to profile some CUDA kernels on a pod with A100 in order to improve its performance. Is there a way to enable the performance counters as per https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters on pods? I've tried to enable it by creating necessary config files on

/etc/modprobe.d

/etc/modprobe.d

but no avail

It seems that the permission needs to be enabled on the host

When profiling within a container, access must be enabled on the host, or the container must be started with the appropriate permissions by passing --cap-add=SYS_ADMIN as an admin user.

When profiling within a container, access must be enabled on the host, or the container must be started with the appropriate permissions by passing --cap-add=SYS_ADMIN as an admin user.

Happy to provide more details and even a temporary grant of permission is sufficient. Thanks!

NVIDIA Developer

NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission i...

Jason•12/16/24, 12:14 AM

I think it's possible only if you do want to rent a full machine with like 8x gpu for a longer time

Jason•12/16/24, 12:14 AM

But try asking a support instead

Jason•12/16/24, 12:15 AM

A staff: you cant do it on RunPod as containers are not provilaged and exposing --cap-add=SYS_ADMIN would cause security risk