From 35cd15e6a96dfdc251d91fd58a34d64c05ffd0ac Mon Sep 17 00:00:00 2001 From: "Nat Kershaw (MSFT)" Date: Mon, 20 May 2024 21:12:19 -0700 Subject: [PATCH] Generate() API updates (#20739) Staged here: https://natke.github.io/onnxruntime/docs/genai/ --- docs/genai/api/c.md | 88 +++++++++----- docs/genai/api/cpp.md | 0 docs/genai/api/csharp.md | 24 +++- docs/genai/api/index.md | 4 +- docs/genai/api/python.md | 63 +++++++--- docs/genai/howto/build-from-source.md | 2 +- docs/genai/howto/build-model.md | 44 +++++-- docs/genai/howto/index.md | 4 +- docs/genai/howto/install.md | 42 ++++--- docs/genai/howto/setup-cuda-env.md | 2 +- docs/genai/index.md | 10 +- docs/genai/reference/config.md | 4 +- docs/genai/reference/index.md | 2 +- docs/genai/tutorials/index.md | 4 +- docs/genai/tutorials/phi2-python.md | 12 +- docs/genai/tutorials/phi3-python.md | 167 ++++++++++++++++++++++++++ 16 files changed, 373 insertions(+), 99 deletions(-) create mode 100644 docs/genai/api/cpp.md create mode 100644 docs/genai/tutorials/phi3-python.md diff --git a/docs/genai/api/c.md b/docs/genai/api/c.md index 63f1dfe801..06da3647ea 100644 --- a/docs/genai/api/c.md +++ b/docs/genai/api/c.md @@ -1,13 +1,13 @@ --- title: C API -description: C API reference for ONNX Runtime GenAI +description: C API reference for ONNX Runtime generate() API has_children: false parent: API docs -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 3 --- -# ONNX Runtime GenAI C API +# ONNX Runtime generate() C API _Note: this API is in preview and is subject to change._ @@ -23,7 +23,7 @@ _Note: this API is in preview and is subject to change._ ### Create model -Creates a model from the given configuration directory and device type. +Creates a model from the given directory. The directory should contain a file called `genai_config.json`, which corresponds to the [configuration specification](../reference/config.md). #### Parameters * Input: config_path The path to the model configuration directory. The path is expected to be encoded in UTF-8. @@ -224,6 +224,23 @@ Set a search option where the option is a bool. OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetSearchBool(OgaGeneratorParams* generator_params, const char* name, bool value); ``` +### Try graph capture with max batch size + +Graph capture fixes the dynamic elements of the computation graph to constant values. It can provide more efficient execution in some environments. To execute in graph capture mode, the maximum batch size needs to be known ahead of time. This function can fail if there is not enough memory to allocate the specified maximum batch size. + +#### Parameters + +* generator_params: The generator params object to set the parameter on +* max_batch_size: The maximum batch size to allocate + +#### Returns + +`OgaResult` containing the error message if graph capture mode could not be configured with the specified batch size + +```c +OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsTryGraphCaptureWithMaxBatchSize(OgaGeneratorParams* generator_params, int32_t max_batch_size); +``` + ### Set inputs Sets the input ids for the generator params. The input ids are used to seed the generation. @@ -255,12 +272,30 @@ Sets the input id sequences for the generator params. The input id sequences are #### Returns - OgaResult containing the error message if the setting of the input id sequences failed. +OgaResult containing the error message if the setting of the input id sequences failed. ```c OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetInputSequences(OgaGeneratorParams* generator_params, const OgaSequences* sequences); ``` +### Set model input + +Set an additional model input, aside from the input_ids. For example additional inputs for LoRA adapters. + +### Parameters + +* generator_params: The generator params to set the input on +* name: the name of the parameter to set +* tensor: the value of the parameter + +### Returns + +OgaResult containing the error message if the setting of the input failed. + +```c +OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetWhisperInputFeatures(OgaGeneratorParams*, OgaTensor* tensor); +``` + ## Generator API @@ -330,7 +365,7 @@ OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_ComputeLogits(OgaGenerator* gene ### Generate next token -Generates the next token based on the computed logits using the greedy search. +Generates the next token based on the computed logits using the configured generation parameters. #### Parameters @@ -341,32 +376,13 @@ Generates the next token based on the computed logits using the greedy search. OgaResult containing the error message if the generation of the next token failed. ```c -OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_Top(OgaGenerator* generator); +OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken(OgaGenerator* generator); ``` -### Generate next token with Top K sampling - -#### Parameters - -#### Returns - -```c -OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_TopK(OgaGenerator* generator, int k, float t); -``` - -### Generate next token with Top P sampling - -#### Parameters - -#### Returns - -```c -OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_TopP(OgaGenerator* generator, float p, float t); -``` ### Get number of tokens - Returns the number of tokens in the sequence at the given index. +Returns the number of tokens in the sequence at the given index. #### Parameters @@ -378,12 +394,12 @@ OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_TopP(OgaGenera The number tokens in the sequence at the given index. ```c -OGA_EXPORT size_t OGA_API_CALL OgaGenerator_GetSequenceLength(const OgaGenerator* generator, size_t index); +OGA_EXPORT size_t OGA_API_CALL OgaGenerator_GetSequenceCount(const OgaGenerator* generator, size_t index); ``` ### Get sequence -Returns a pointer to the sequence data at the given index. The number of tokens in the sequence is given by OgaGenerator_GetSequenceLength. +Returns a pointer to the sequence data at the given index. The number of tokens in the sequence is given by `OgaGenerator_GetSequenceCount`. #### Parameters @@ -395,7 +411,7 @@ Returns a pointer to the sequence data at the given index. The number of tokens A pointer to the token sequence ```c -OGA_EXPORT const int32_t* OGA_API_CALL OgaGenerator_GetSequence(const OgaGenerator* generator, size_t index); +OGA_EXPORT const int32_t* OGA_API_CALL OgaGenerator_GetSequenceData(const OgaGenerator* generator, size_t index); ``` ## Enums and structs @@ -419,6 +435,18 @@ typedef struct OgaBuffer OgaBuffer; ## Utility functions +### Set the GPU device ID + +```c +OGA_EXPORT OgaResult* OGA_API_CALL OgaSetCurrentGpuDeviceId(int device_id); +``` + +### Get the GPU device ID + +```c +OGA_EXPORT OgaResult* OGA_API_CALL OgaGetCurrentGpuDeviceId(int* device_id); +``` + ### Get error message #### Parameters diff --git a/docs/genai/api/cpp.md b/docs/genai/api/cpp.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/genai/api/csharp.md b/docs/genai/api/csharp.md index 86b566f451..915e6e2bf5 100644 --- a/docs/genai/api/csharp.md +++ b/docs/genai/api/csharp.md @@ -1,13 +1,13 @@ --- title: C# API -description: C# API reference for ONNX Runtime GenAI +description: C# API reference for ONNX Runtime generate() API has_children: false parent: API docs -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 2 --- -# ONNX Runtime GenAI C# API +# ONNX Runtime generate() C# API _Note: this API is in preview and is subject to change._ @@ -98,6 +98,12 @@ public void SetSearchOption(string searchOption, double value) public void SetSearchOption(string searchOption, bool value) ``` +### Try graph capture with max batch size + +```csharp + public void TryGraphCaptureWithMaxBatchSize(int maxBatchSize) +``` + ### Set input ids method ```csharp @@ -110,8 +116,11 @@ public void SetInputIDs(ReadOnlySpan inputIDs, ulong sequenceLength, ulong public void SetInputSequences(Sequences sequences) ``` +### Set model inputs - +```csharp +public void SetModelInput(string name, Tensor value) +``` ## Generator class @@ -137,9 +146,14 @@ public void ComputeLogits() ### Generate next token method ```csharp -public void GenerateNextTokenTop() +public void GenerateNextToken() ``` +### Get sequence + +```csharp +public ReadOnlySpan GetSequence(ulong index) +``` ## Sequences class diff --git a/docs/genai/api/index.md b/docs/genai/api/index.md index 1684099508..8b05e69c25 100644 --- a/docs/genai/api/index.md +++ b/docs/genai/api/index.md @@ -1,7 +1,7 @@ --- title: API docs -description: API documentation for ONNX Runtime GenAI -parent: Generative AI (Preview) +description: API documentation for ONNX Runtime generate() API +parent: Generate API (Preview) has_children: true nav_order: 2 --- diff --git a/docs/genai/api/python.md b/docs/genai/api/python.md index 7a80ebc5bb..0ff27a0002 100644 --- a/docs/genai/api/python.md +++ b/docs/genai/api/python.md @@ -1,9 +1,9 @@ --- title: Python API -description: Python API reference for ONNX Runtime GenAI +description: Python API reference for ONNX Runtime generate() API has_children: false parent: API docs -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 1 --- @@ -30,7 +30,7 @@ import onnxruntime_genai ## Model class -### Load the model +### Load a model Loads the ONNX model(s) and configuration from a folder on disk. @@ -59,22 +59,14 @@ onnxruntime_genai.Model.generate(params: GeneratorParams) -> numpy.ndarray[int, `numpy.ndarray[int, int]`: a two dimensional numpy array with dimensions equal to the size of the batch passed in and the maximum length of the sequence of tokens. +### Device type -## GeneratorParams class - -### Create GeneratorParams object +Return the device type that the model has been configured to run on. ```python -onnxruntime_genai.GeneratorParams(model: onnxruntime_genai.Model) -> onnxruntime_genai.GeneratorParams +onnxruntime_genai.Model.device_type ``` -#### Parameters - -- `model`: (required) The model that was loaded by onnxruntime_genai.Model() - -#### Returns - -`onnxruntime_genai.GeneratorParams`: The GeneratorParams object ## Tokenizer class @@ -193,18 +185,49 @@ onnxruntime_genai.TokenizerStream.decode(token: int32) -> str onnxruntime_genai.GeneratorParams(model: Model) -> GeneratorParams ``` -### Input_ids member +### Pad token id member ```python -onnxruntime_genai.GeneratorParams.input_ids = numpy.ndarray[numpy.int32, numpy.int32] +onnxruntime_genai.GeneratorParams.pad_token_id ``` +### EOS token id member + +```python +onnxruntime_genai.GeneratorParams.eos_token_id +``` + +### vocab size member + +```python +onnxruntime_genai.GeneratorParams.vocab_size +``` + +### input_ids member + +```python +onnxruntime_genai.GeneratorParams.input_ids: numpy.ndarray[numpy.int32, numpy.int32] +``` + +### Set model input + +```python +onnxruntime_genai.GeneratorParams.set_model_input(name: str, value: []) +``` + + ### Set search options method ```python onnxruntime_genai.GeneratorParams.set_search_options(options: dict[str, Any]) ``` +### Try graph capture with max batch size + +```python +onnxruntime_genai.GeneratorParams.try_graph_capture_with_max_batch_size(max_batch_size: int) +``` + ## Generator class ### Create a Generator @@ -242,6 +265,14 @@ Runs the model through one iteration. onnxruntime_genai.Generator.compute_logits() ``` +### Get output + +Returns the output logits of the model. + +```python +onnxruntime_genai.Generator.get_output() +``` + ### Generate next token Using the current set of logits and the specified generator parameters, calculates the next batch of tokens, using Top P sampling. diff --git a/docs/genai/howto/build-from-source.md b/docs/genai/howto/build-from-source.md index 3dad67c336..6332d03f11 100644 --- a/docs/genai/howto/build-from-source.md +++ b/docs/genai/howto/build-from-source.md @@ -3,7 +3,7 @@ title: Build from source description: How to build the ONNX Runtime generate() API from source has_children: false parent: How to -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 2 --- diff --git a/docs/genai/howto/build-model.md b/docs/genai/howto/build-model.md index 408710d34e..fe8b91e03d 100644 --- a/docs/genai/howto/build-model.md +++ b/docs/genai/howto/build-model.md @@ -1,9 +1,9 @@ --- title: Build models -description: How to build models with ONNX Runtime GenAI +description: How to build models with ONNX Runtime generate() API has_children: false parent: How to -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 2 --- @@ -13,7 +13,7 @@ nav_order: 2 * TOC placeholder {:toc} -The model builder greatly accelerates creating optimized and quantized ONNX models that run with ONNX Runtime GenAI. +The model builder greatly accelerates creating optimized and quantized ONNX models that run with the ONNX Runtime generate() API. ## Current Support The tool currently supports the following model architectures. @@ -23,11 +23,33 @@ The tool currently supports the following model architectures. - Mistral - Phi -## Usage +## Installation -### Full Usage -For all available options, please use the `-h/--help` flag. +Model builder is available as an [Olive](https://github.com/microsoft/olive) pass. It is also shipped as part of the onnxruntime-genai Python package. You can also download and run it standalone. + +In any case, you need to have the following packages installed. + +```bash +pip install torch transformers onnx onnxruntime ``` + +### Install from package + +```bash +pip install --pre onnxruntime-genai +``` + +#### Direct download + +```bash +curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/src/python/py/models/builder.py -o builder.py +``` + +### Usage + +For all available options, please use the `-h/--help` flag. + +```bash # From wheel: python3 -m onnxruntime_genai.models.builder --help @@ -35,9 +57,12 @@ python3 -m onnxruntime_genai.models.builder --help python3 builder.py --help ``` -### Original PyTorch Model from Hugging Face +### Original PyTorch Model from HuggingFace + This scenario is where your PyTorch model is not downloaded locally (either in the default Hugging Face cache directory or in a local folder on disk). -``` + +```bash + # From wheel: python3 -m onnxruntime_genai.models.builder -m model_name -o path_to_output_folder -p precision -e execution_provider -c cache_dir_to_save_hf_files @@ -46,6 +71,7 @@ python3 builder.py -m model_name -o path_to_output_folder -p precision -e execut ``` ### Original PyTorch Model from Disk + This scenario is where your PyTorch model is already downloaded locally (either in the default Hugging Face cache directory or in a local folder on disk). ``` # From wheel: @@ -87,7 +113,7 @@ python3 builder.py -m model_name -o path_to_output_folder -p precision -e execut To see all available options through `--extra_options`, please use the `help` commands in the `Full Usage` section above. ### Config Only -This scenario is for when you already have your optimized and/or quantized ONNX model and you need to create the config files to run with ONNX Runtime GenAI. +This scenario is for when you already have your optimized and/or quantized ONNX model and you need to create the config files to run with ONNX Runtime generate() API. ``` # From wheel: python3 -m onnxruntime_genai.models.builder -m model_name -o path_to_output_folder -p precision -e execution_provider -c cache_dir_for_hf_files --extra_options config_only=true diff --git a/docs/genai/howto/index.md b/docs/genai/howto/index.md index 06847318ef..73d48b9950 100644 --- a/docs/genai/howto/index.md +++ b/docs/genai/howto/index.md @@ -1,7 +1,7 @@ --- title: How to -description: How to perform specific tasks with ONNX Runtime GenAI -parent: Generative AI (Preview) +description: How to perform specific tasks with ONNX Runtime generate() API +parent: Generate API (Preview) has_children: true nav_order: 3 --- diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index c9acf5c1da..46e7e9ff64 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -1,52 +1,58 @@ --- title: Install -description: Instructions to install ONNX Runtime GenAI on your target platform in your environment +description: Instructions to install ONNX Runtime generate() API on your target platform in your environment has_children: false parent: How to -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 1 --- -# Install ONNX Runtime GenAI +# Install ONNX Runtime generate() API {: .no_toc } * TOC placeholder {:toc} -## Python package release candidates +## Python packages ```bash pip install numpy -pip install onnxruntime-genai --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ +pip install onnxruntime-genai --pre +``` +Append `-directml` for the library that is optimized for DirectML on Windows + +```bash +pip install numpy +pip install onnxruntime-genai-directml --pre ``` Append `-cuda` for the library that is optimized for CUDA environments ```bash +pip install numpy pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` -## Nuget package release candidates +## Nuget packages -To install the NuGet release candidates, add a new package source in Visual Studio, go to `Project` -> `Manage NuGet Packages`. +```bash +dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease +``` -1. Click on the `Settings` cog icon +For the package that has been optimized for CUDA: -2. Click the `+` button to add a new package source +```bash +dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease +``` - - Change the Name to `onnxruntime-genai` - - Change the Source to `https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/nuget/v3/index.json` +For the package that has been optimized for DirectML: -3. Check the `Include prerelease` button +```bash +dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease +``` -4. Add the `Microsoft.ML.OnnxRuntimeGenAI` package -5. Add the `Microsoft.ML.OnnxRuntime` package -To run with CUDA, use the following packages instead: - -- `Microsoft.ML.OnnxRuntimeGenAI.Cuda` -- `Microsoft.ML.OnnxRuntime.Gpu` diff --git a/docs/genai/howto/setup-cuda-env.md b/docs/genai/howto/setup-cuda-env.md index f9153aefbe..b87c83d8cb 100644 --- a/docs/genai/howto/setup-cuda-env.md +++ b/docs/genai/howto/setup-cuda-env.md @@ -3,7 +3,7 @@ title: Setup CUDA env description: Instructions to setup the CUDA environtment to run onnxruntime-genai-cuda has_children: false parent: How to -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 4 --- diff --git a/docs/genai/index.md b/docs/genai/index.md index 917dccc51c..2da5906a57 100644 --- a/docs/genai/index.md +++ b/docs/genai/index.md @@ -1,15 +1,17 @@ --- -title: Generative AI (Preview) -description: Run generative models with ONNX Runtime GenAI +title: Generate API (Preview) +description: Run generative models with the ONNX Runtime generate() API has_children: true nav_order: 6 --- -# Generative AI with ONNX Runtime +# ONNX Runtime generate() API _Note: this API is in preview and is subject to change._ -Run generative AI models with ONNX Runtime. Source code: (https://github.com/microsoft/onnxruntime-genai) +Run generative AI models with ONNX Runtime. + +See the source code here: [https://github.com/microsoft/onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai) This library provides the generative AI loop for ONNX models, including inference with ONNX Runtime, logits processing, search and sampling, and KV cache management. diff --git a/docs/genai/reference/config.md b/docs/genai/reference/config.md index e1fba73e5e..6b8218bb96 100644 --- a/docs/genai/reference/config.md +++ b/docs/genai/reference/config.md @@ -1,9 +1,9 @@ --- title: Config reference -description: Reference for the ONNX Runtime Generative AI configuration file +description: Reference for the ONNX Runtime generate() API configuration file has_children: false parent: Reference -grand_parent: Generative AI (Preview) +grand_parent: Generate API (Preview) nav_order: 1 --- diff --git a/docs/genai/reference/index.md b/docs/genai/reference/index.md index f34d266ded..13da613853 100644 --- a/docs/genai/reference/index.md +++ b/docs/genai/reference/index.md @@ -1,7 +1,7 @@ --- title: Reference description: Reference information for ONNX Runtime Generative AI -parent: Generative AI (Preview) +parent: Generate API (Preview) has_children: true nav_order: 4 --- diff --git a/docs/genai/tutorials/index.md b/docs/genai/tutorials/index.md index c05d1a1797..dc4b4adc14 100644 --- a/docs/genai/tutorials/index.md +++ b/docs/genai/tutorials/index.md @@ -1,7 +1,7 @@ --- title: Tutorials -description: Build your application with ONNX Runtime GenAI -parent: Generative AI (Preview) +description: Build your application with ONNX Runtime generate() API +parent: Generate API (Preview) has_children: true nav_order: 1 --- diff --git a/docs/genai/tutorials/phi2-python.md b/docs/genai/tutorials/phi2-python.md index 77f975b465..a2205667ba 100644 --- a/docs/genai/tutorials/phi2-python.md +++ b/docs/genai/tutorials/phi2-python.md @@ -1,17 +1,17 @@ --- title: Python phi-2 tutorial -description: Learn how to write a language generation application with ONNX Runtime GenAI in Python using the phi-2 model +description: Learn how to write a language generation application with ONNX Runtime generate() API in Python using the phi-2 model has_children: false parent: Tutorials -grand_parent: Generative AI (Preview) -nav_order: 1 +grand_parent: Generate API (Preview) +nav_order: 2 --- # Language generation in Python with phi-2 ## Setup and installation -Install the ONNX Runtime GenAI Python package using the [installation instructions](../howto/install.md). +Install the ONNX Runtime generate() API Python package using the [installation instructions](../howto/install.md). ## Build phi-2 ONNX model @@ -31,11 +31,11 @@ python -m onnxruntime_genai.models.builder -m microsoft/phi-2 -e cpu -p int4 -o ``` You can replace the name of the output folder specified with the `-o` option with a folder of your choice. -After you run the script, you will see a series of files generated in this folder. They include the HuggingFace configs for your reference, as well as the following generated files used by ONNX Runtime GenAI. +After you run the script, you will see a series of files generated in this folder. They include the HuggingFace configs for your reference, as well as the following generated files used by ONNX Runtime generate() API. - `model.onnx`: the phi-2 ONNX model - `model.onnx.data`: the phi-2 ONNX model weights -- `genai_config.json`: the configuration used by ONNX Runtime GenAI +- `genai_config.json`: the configuration used by ONNX Runtime generate() API You can view and change the values in the `genai_config.json` file. The model section should not be updated unless you have brought your own model and it has different parameters. diff --git a/docs/genai/tutorials/phi3-python.md b/docs/genai/tutorials/phi3-python.md new file mode 100644 index 0000000000..d93eed15cd --- /dev/null +++ b/docs/genai/tutorials/phi3-python.md @@ -0,0 +1,167 @@ +--- +title: Python phi-3 tutorial +description: Small but mighty. Run Phi-3 with ONNX Runtime. +has_children: false +parent: Tutorials +grand_parent: Generate API (Preview) +nav_order: 1 +--- + +# Run the Phi-3 Mini models with the ONNX Runtime generate() API + +## Steps +1. [Setup](#setup) +2. [Choose your platform](#choose-your-platform) +3. [Run with DirectML](#run-with-directml) +4. [Run with NVDIA CUDA](#run-with-nvidia-cuda) +5. [Run on CPU](#run-on-cpu) + +## Introduction + +There are two Phi-3 mini models to choose from: the short (4k) context version or the long (128k) context version. The long context version can accept much longer prompts and produce longer output text, but it does consume more memory. + +The Phi-3 ONNX models are hosted on HuggingFace: [short](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx) and [long](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx). + +This tutorial downloads and runs the short context model. If you would like to use the long context model, change the `4k` to `128k` in the instructions below. + +## Setup + +1. Install the git large file system extension + + HuggingFace uses `git` for version control. To download the ONNX models you need `git lfs` to be installed, if you do not already have it. + + * Windows: `winget install -e --id GitHub.GitLFS` (If you don't have winget, download and run the `exe` from the [official source](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage?platform=windows)) + * Linux: `apt-get install git-lfs` + * MacOS: `brew install git-lfs` + + Then run `git lfs install` + +2. Install the HuggingFace CLI + + ```bash + pip install huggingface-hub[cli] + ``` + +## Choose your platform + +Are you on a Windows machine with GPU? +* I don't know → Review [this guide](https://www.microsoft.com/en-us/windows/learning-center/how-to-check-gpu) to see whether you have a GPU in your Windows machine. +* Yes → Follow the instructions for [DirectML](#run-with-directml). +* No → Do you have an NVIDIA GPU? + * I don't know → Review [this guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#verify-you-have-a-cuda-capable-gpu) to see whether you have a CUDA-capable GPU. + * Yes → Follow the instructions for [NVIDIA CUDA GPU](#run-with-nvidia-cuda). + * No → Follow the instructions for [CPU](#run-on-cpu). + +**Note: Only one package and model is required based on your hardware. That is, only execute the steps for one of the following sections** + +## Run with DirectML + +1. Download the model + + ```bash + huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include directml/* --local-dir . + ``` + + This command downloads the model into a folder called `directml`. + + +2. Install the generate() API + + ``` + pip install numpy + pip install --pre onnxruntime-genai-directml + ``` + + You should now see `onnxruntime-genai-directml` in your `pip list`. + +3. Run the model + + Run the model with [phi3-qa.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py). + + ```cmd + curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py + python phi3-qa.py -m directml\directml-int4-awq-block-128 + ``` + + Once the script has loaded the model, it will ask you for input in a loop, streaming the output as it is produced the model. For example: + + ```bash + Input: Tell me a joke about GPUs + + Certainly! Here\'s a light-hearted joke about GPUs: + + + Why did the GPU go to school? Because it wanted to improve its "processing power"! + + + This joke plays on the double meaning of "processing power," referring both to the computational abilities of a GPU and the idea of a student wanting to improve their academic skills. + ``` + +## Run with NVIDIA CUDA + +1. Download the model + + ```bash + huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cuda/cuda-int4-rtn-block-32/* --local-dir . + ``` + + This command downloads the model into a folder called `cuda`. + +2. Install the generate() API + + ``` + pip install numpy + pip install --pre onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ + ``` + +3. Run the model + + Run the model with [phi3-qa.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py). + + ```bash + curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py + python phi3-qa.py -m cuda/cuda-int4-rtn-block-32 + ``` + + Once the script has loaded the model, it will ask you for input in a loop, streaming the output as it is produced the model. For example: + + ```bash + Input: Tell me a joke about creative writing + + Output: Why don\'t writers ever get lost? Because they always follow the plot! + ``` + +## Run on CPU + +1. Download the model + + ```bash + huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir . + ``` + + This command downloads the model into a folder called `cpu_and_mobile` + +2. Install the generate() API for CPU + + ``` + pip install numpy + pip install --pre onnxruntime-genai + ``` + +3. Run the model + + Run the model with [phi3-qa.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py). + + ```bash + curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py + python phi3-qa.py -m cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 + ``` + + Once the script has loaded the model, it will ask you for input in a loop, streaming the output as it is produced the model. For example: + + ```bash + Input: Tell me a joke about generative AI + + Output: Why did the generative AI go to school? + + To improve its "creativity" algorithm!