saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-18 18:52:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/docs/genai/howto/setup-cuda-env.md
Nat Kershaw (MSFT) 35cd15e6a9
Generate() API updates (#20739) 
Staged here: https://natke.github.io/onnxruntime/docs/genai/
2024-05-20 21:12:19 -07:00

943 B
Raw Blame History

title description has_children parent grand_parent nav_order
Setup CUDA env Instructions to setup the CUDA environtment to run onnxruntime-genai-cuda false How to Generate API (Preview) 4

Setup the CUDA Environment

{: .no_toc }

  • TOC placeholder {:toc}

Install the CUDA Toolkit

On a CUDA capable machine, install the CUDA toolkit. onnxruntime-genai-cuda is built and packaged with CUDA-11.8.

The CUDA toolkit can be downloaded from the CUDA Toolkit Archive.

Special Instructions on Windows

After CUDA toolkit installation completed on windows, ensure that the CUDA_PATH system environment variable has been set to the path where the toolkit was installed. This variable will be used when importing the onnxruntime_genai python module on windows. Unset or incorrectly set CUDA_PATH variable may lead to a DLL load failed while importing onnxruntime_genai.