Check lsmod and see if the kernel modules are loaded. Nvidia smi is probably loading them for you on first run.