saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-26 19:52:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/docs/genai/howto/setup-cuda-env.md
Baiju Meswani 5e549a5c24
How to setup the cuda env for windows (#20444)
2024-04-24 16:35:47 -07:00

944 B
Raw Blame History

title description has_children parent grand_parent nav_order
Setup CUDA env Instructions to setup the CUDA environtment to run onnxruntime-genai-cuda false How to Generative AI (Preview) 4

Setup the CUDA Environment

{: .no_toc }

  • TOC placeholder {:toc}

Install the CUDA Toolkit

On a CUDA capable machine, install the CUDA toolkit. onnxruntime-genai-cuda is built and packaged with CUDA-11.8.

The CUDA toolkit can be downloaded from the CUDA Toolkit Archive.

Special Instructions on Windows

After CUDA toolkit installation completed on windows, ensure that the CUDA_PATH system environment variable has been set to the path where the toolkit was installed. This variable will be used when importing the onnxruntime_genai python module on windows. Unset or incorrectly set CUDA_PATH variable may lead to a DLL load failed while importing onnxruntime_genai.