mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-15 18:23:41 +00:00
Generate() API updates (#20739)
Staged here: https://natke.github.io/onnxruntime/docs/genai/
This commit is contained in:
Nat Kershaw (MSFT)
GitHub
parent
2fe196ea39
commit
35cd15e6a9
16 changed files with 373 additions and 99 deletions
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
title: C API
|
||||
description: C API reference for ONNX Runtime GenAI
|
||||
description: C API reference for ONNX Runtime generate() API
|
||||
has_children: false
|
||||
parent: API docs
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# ONNX Runtime GenAI C API
|
||||
# ONNX Runtime generate() C API
|
||||
|
||||
_Note: this API is in preview and is subject to change._
|
||||
|
||||
|
|
@ -23,7 +23,7 @@ _Note: this API is in preview and is subject to change._
|
|||
|
||||
### Create model
|
||||
|
||||
Creates a model from the given configuration directory and device type.
|
||||
Creates a model from the given directory. The directory should contain a file called `genai_config.json`, which corresponds to the [configuration specification](../reference/config.md).
|
||||
|
||||
#### Parameters
|
||||
* Input: config_path The path to the model configuration directory. The path is expected to be encoded in UTF-8.
|
||||
|
|
@ -224,6 +224,23 @@ Set a search option where the option is a bool.
|
|||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetSearchBool(OgaGeneratorParams* generator_params, const char* name, bool value);
|
||||
```
|
||||
|
||||
### Try graph capture with max batch size
|
||||
|
||||
Graph capture fixes the dynamic elements of the computation graph to constant values. It can provide more efficient execution in some environments. To execute in graph capture mode, the maximum batch size needs to be known ahead of time. This function can fail if there is not enough memory to allocate the specified maximum batch size.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* generator_params: The generator params object to set the parameter on
|
||||
* max_batch_size: The maximum batch size to allocate
|
||||
|
||||
#### Returns
|
||||
|
||||
`OgaResult` containing the error message if graph capture mode could not be configured with the specified batch size
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsTryGraphCaptureWithMaxBatchSize(OgaGeneratorParams* generator_params, int32_t max_batch_size);
|
||||
```
|
||||
|
||||
### Set inputs
|
||||
|
||||
Sets the input ids for the generator params. The input ids are used to seed the generation.
|
||||
|
|
@ -255,12 +272,30 @@ Sets the input id sequences for the generator params. The input id sequences are
|
|||
|
||||
#### Returns
|
||||
|
||||
OgaResult containing the error message if the setting of the input id sequences failed.
|
||||
OgaResult containing the error message if the setting of the input id sequences failed.
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetInputSequences(OgaGeneratorParams* generator_params, const OgaSequences* sequences);
|
||||
```
|
||||
|
||||
### Set model input
|
||||
|
||||
Set an additional model input, aside from the input_ids. For example additional inputs for LoRA adapters.
|
||||
|
||||
### Parameters
|
||||
|
||||
* generator_params: The generator params to set the input on
|
||||
* name: the name of the parameter to set
|
||||
* tensor: the value of the parameter
|
||||
|
||||
### Returns
|
||||
|
||||
OgaResult containing the error message if the setting of the input failed.
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetWhisperInputFeatures(OgaGeneratorParams*, OgaTensor* tensor);
|
||||
```
|
||||
|
||||
|
||||
## Generator API
|
||||
|
||||
|
|
@ -330,7 +365,7 @@ OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_ComputeLogits(OgaGenerator* gene
|
|||
|
||||
### Generate next token
|
||||
|
||||
Generates the next token based on the computed logits using the greedy search.
|
||||
Generates the next token based on the computed logits using the configured generation parameters.
|
||||
|
||||
#### Parameters
|
||||
|
||||
|
|
@ -341,32 +376,13 @@ Generates the next token based on the computed logits using the greedy search.
|
|||
OgaResult containing the error message if the generation of the next token failed.
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_Top(OgaGenerator* generator);
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken(OgaGenerator* generator);
|
||||
```
|
||||
|
||||
### Generate next token with Top K sampling
|
||||
|
||||
#### Parameters
|
||||
|
||||
#### Returns
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_TopK(OgaGenerator* generator, int k, float t);
|
||||
```
|
||||
|
||||
### Generate next token with Top P sampling
|
||||
|
||||
#### Parameters
|
||||
|
||||
#### Returns
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_TopP(OgaGenerator* generator, float p, float t);
|
||||
```
|
||||
|
||||
### Get number of tokens
|
||||
|
||||
Returns the number of tokens in the sequence at the given index.
|
||||
Returns the number of tokens in the sequence at the given index.
|
||||
|
||||
#### Parameters
|
||||
|
||||
|
|
@ -378,12 +394,12 @@ OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken_TopP(OgaGenera
|
|||
The number tokens in the sequence at the given index.
|
||||
|
||||
```c
|
||||
OGA_EXPORT size_t OGA_API_CALL OgaGenerator_GetSequenceLength(const OgaGenerator* generator, size_t index);
|
||||
OGA_EXPORT size_t OGA_API_CALL OgaGenerator_GetSequenceCount(const OgaGenerator* generator, size_t index);
|
||||
```
|
||||
|
||||
### Get sequence
|
||||
|
||||
Returns a pointer to the sequence data at the given index. The number of tokens in the sequence is given by OgaGenerator_GetSequenceLength.
|
||||
Returns a pointer to the sequence data at the given index. The number of tokens in the sequence is given by `OgaGenerator_GetSequenceCount`.
|
||||
|
||||
#### Parameters
|
||||
|
||||
|
|
@ -395,7 +411,7 @@ Returns a pointer to the sequence data at the given index. The number of tokens
|
|||
A pointer to the token sequence
|
||||
|
||||
```c
|
||||
OGA_EXPORT const int32_t* OGA_API_CALL OgaGenerator_GetSequence(const OgaGenerator* generator, size_t index);
|
||||
OGA_EXPORT const int32_t* OGA_API_CALL OgaGenerator_GetSequenceData(const OgaGenerator* generator, size_t index);
|
||||
```
|
||||
|
||||
## Enums and structs
|
||||
|
|
@ -419,6 +435,18 @@ typedef struct OgaBuffer OgaBuffer;
|
|||
|
||||
## Utility functions
|
||||
|
||||
### Set the GPU device ID
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaSetCurrentGpuDeviceId(int device_id);
|
||||
```
|
||||
|
||||
### Get the GPU device ID
|
||||
|
||||
```c
|
||||
OGA_EXPORT OgaResult* OGA_API_CALL OgaGetCurrentGpuDeviceId(int* device_id);
|
||||
```
|
||||
|
||||
### Get error message
|
||||
|
||||
#### Parameters
|
||||
|
|
|
|||
0
docs/genai/api/cpp.md
Normal file
0
docs/genai/api/cpp.md
Normal file
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
title: C# API
|
||||
description: C# API reference for ONNX Runtime GenAI
|
||||
description: C# API reference for ONNX Runtime generate() API
|
||||
has_children: false
|
||||
parent: API docs
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# ONNX Runtime GenAI C# API
|
||||
# ONNX Runtime generate() C# API
|
||||
|
||||
_Note: this API is in preview and is subject to change._
|
||||
|
||||
|
|
@ -98,6 +98,12 @@ public void SetSearchOption(string searchOption, double value)
|
|||
public void SetSearchOption(string searchOption, bool value)
|
||||
```
|
||||
|
||||
### Try graph capture with max batch size
|
||||
|
||||
```csharp
|
||||
public void TryGraphCaptureWithMaxBatchSize(int maxBatchSize)
|
||||
```
|
||||
|
||||
### Set input ids method
|
||||
|
||||
```csharp
|
||||
|
|
@ -110,8 +116,11 @@ public void SetInputIDs(ReadOnlySpan<int> inputIDs, ulong sequenceLength, ulong
|
|||
public void SetInputSequences(Sequences sequences)
|
||||
```
|
||||
|
||||
### Set model inputs
|
||||
|
||||
|
||||
```csharp
|
||||
public void SetModelInput(string name, Tensor value)
|
||||
```
|
||||
|
||||
|
||||
## Generator class
|
||||
|
|
@ -137,9 +146,14 @@ public void ComputeLogits()
|
|||
### Generate next token method
|
||||
|
||||
```csharp
|
||||
public void GenerateNextTokenTop()
|
||||
public void GenerateNextToken()
|
||||
```
|
||||
|
||||
### Get sequence
|
||||
|
||||
```csharp
|
||||
public ReadOnlySpan<int> GetSequence(ulong index)
|
||||
```
|
||||
|
||||
## Sequences class
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: API docs
|
||||
description: API documentation for ONNX Runtime GenAI
|
||||
parent: Generative AI (Preview)
|
||||
description: API documentation for ONNX Runtime generate() API
|
||||
parent: Generate API (Preview)
|
||||
has_children: true
|
||||
nav_order: 2
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
---
|
||||
title: Python API
|
||||
description: Python API reference for ONNX Runtime GenAI
|
||||
description: Python API reference for ONNX Runtime generate() API
|
||||
has_children: false
|
||||
parent: API docs
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
|
|
@ -30,7 +30,7 @@ import onnxruntime_genai
|
|||
|
||||
## Model class
|
||||
|
||||
### Load the model
|
||||
### Load a model
|
||||
|
||||
Loads the ONNX model(s) and configuration from a folder on disk.
|
||||
|
||||
|
|
@ -59,22 +59,14 @@ onnxruntime_genai.Model.generate(params: GeneratorParams) -> numpy.ndarray[int,
|
|||
|
||||
`numpy.ndarray[int, int]`: a two dimensional numpy array with dimensions equal to the size of the batch passed in and the maximum length of the sequence of tokens.
|
||||
|
||||
### Device type
|
||||
|
||||
## GeneratorParams class
|
||||
|
||||
### Create GeneratorParams object
|
||||
Return the device type that the model has been configured to run on.
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams(model: onnxruntime_genai.Model) -> onnxruntime_genai.GeneratorParams
|
||||
onnxruntime_genai.Model.device_type
|
||||
```
|
||||
|
||||
#### Parameters
|
||||
|
||||
- `model`: (required) The model that was loaded by onnxruntime_genai.Model()
|
||||
|
||||
#### Returns
|
||||
|
||||
`onnxruntime_genai.GeneratorParams`: The GeneratorParams object
|
||||
|
||||
## Tokenizer class
|
||||
|
||||
|
|
@ -193,18 +185,49 @@ onnxruntime_genai.TokenizerStream.decode(token: int32) -> str
|
|||
onnxruntime_genai.GeneratorParams(model: Model) -> GeneratorParams
|
||||
```
|
||||
|
||||
### Input_ids member
|
||||
### Pad token id member
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.input_ids = numpy.ndarray[numpy.int32, numpy.int32]
|
||||
onnxruntime_genai.GeneratorParams.pad_token_id
|
||||
```
|
||||
|
||||
### EOS token id member
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.eos_token_id
|
||||
```
|
||||
|
||||
### vocab size member
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.vocab_size
|
||||
```
|
||||
|
||||
### input_ids member
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.input_ids: numpy.ndarray[numpy.int32, numpy.int32]
|
||||
```
|
||||
|
||||
### Set model input
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.set_model_input(name: str, value: [])
|
||||
```
|
||||
|
||||
|
||||
### Set search options method
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.set_search_options(options: dict[str, Any])
|
||||
```
|
||||
|
||||
### Try graph capture with max batch size
|
||||
|
||||
```python
|
||||
onnxruntime_genai.GeneratorParams.try_graph_capture_with_max_batch_size(max_batch_size: int)
|
||||
```
|
||||
|
||||
## Generator class
|
||||
|
||||
### Create a Generator
|
||||
|
|
@ -242,6 +265,14 @@ Runs the model through one iteration.
|
|||
onnxruntime_genai.Generator.compute_logits()
|
||||
```
|
||||
|
||||
### Get output
|
||||
|
||||
Returns the output logits of the model.
|
||||
|
||||
```python
|
||||
onnxruntime_genai.Generator.get_output()
|
||||
```
|
||||
|
||||
### Generate next token
|
||||
|
||||
Using the current set of logits and the specified generator parameters, calculates the next batch of tokens, using Top P sampling.
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Build from source
|
|||
description: How to build the ONNX Runtime generate() API from source
|
||||
has_children: false
|
||||
parent: How to
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
---
|
||||
title: Build models
|
||||
description: How to build models with ONNX Runtime GenAI
|
||||
description: How to build models with ONNX Runtime generate() API
|
||||
has_children: false
|
||||
parent: How to
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
|
|
@ -13,7 +13,7 @@ nav_order: 2
|
|||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
The model builder greatly accelerates creating optimized and quantized ONNX models that run with ONNX Runtime GenAI.
|
||||
The model builder greatly accelerates creating optimized and quantized ONNX models that run with the ONNX Runtime generate() API.
|
||||
|
||||
## Current Support
|
||||
The tool currently supports the following model architectures.
|
||||
|
|
@ -23,11 +23,33 @@ The tool currently supports the following model architectures.
|
|||
- Mistral
|
||||
- Phi
|
||||
|
||||
## Usage
|
||||
## Installation
|
||||
|
||||
### Full Usage
|
||||
For all available options, please use the `-h/--help` flag.
|
||||
Model builder is available as an [Olive](https://github.com/microsoft/olive) pass. It is also shipped as part of the onnxruntime-genai Python package. You can also download and run it standalone.
|
||||
|
||||
In any case, you need to have the following packages installed.
|
||||
|
||||
```bash
|
||||
pip install torch transformers onnx onnxruntime
|
||||
```
|
||||
|
||||
### Install from package
|
||||
|
||||
```bash
|
||||
pip install --pre onnxruntime-genai
|
||||
```
|
||||
|
||||
#### Direct download
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/src/python/py/models/builder.py -o builder.py
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
For all available options, please use the `-h/--help` flag.
|
||||
|
||||
```bash
|
||||
# From wheel:
|
||||
python3 -m onnxruntime_genai.models.builder --help
|
||||
|
||||
|
|
@ -35,9 +57,12 @@ python3 -m onnxruntime_genai.models.builder --help
|
|||
python3 builder.py --help
|
||||
```
|
||||
|
||||
### Original PyTorch Model from Hugging Face
|
||||
### Original PyTorch Model from HuggingFace
|
||||
|
||||
This scenario is where your PyTorch model is not downloaded locally (either in the default Hugging Face cache directory or in a local folder on disk).
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
# From wheel:
|
||||
python3 -m onnxruntime_genai.models.builder -m model_name -o path_to_output_folder -p precision -e execution_provider -c cache_dir_to_save_hf_files
|
||||
|
||||
|
|
@ -46,6 +71,7 @@ python3 builder.py -m model_name -o path_to_output_folder -p precision -e execut
|
|||
```
|
||||
|
||||
### Original PyTorch Model from Disk
|
||||
|
||||
This scenario is where your PyTorch model is already downloaded locally (either in the default Hugging Face cache directory or in a local folder on disk).
|
||||
```
|
||||
# From wheel:
|
||||
|
|
@ -87,7 +113,7 @@ python3 builder.py -m model_name -o path_to_output_folder -p precision -e execut
|
|||
To see all available options through `--extra_options`, please use the `help` commands in the `Full Usage` section above.
|
||||
|
||||
### Config Only
|
||||
This scenario is for when you already have your optimized and/or quantized ONNX model and you need to create the config files to run with ONNX Runtime GenAI.
|
||||
This scenario is for when you already have your optimized and/or quantized ONNX model and you need to create the config files to run with ONNX Runtime generate() API.
|
||||
```
|
||||
# From wheel:
|
||||
python3 -m onnxruntime_genai.models.builder -m model_name -o path_to_output_folder -p precision -e execution_provider -c cache_dir_for_hf_files --extra_options config_only=true
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: How to
|
||||
description: How to perform specific tasks with ONNX Runtime GenAI
|
||||
parent: Generative AI (Preview)
|
||||
description: How to perform specific tasks with ONNX Runtime generate() API
|
||||
parent: Generate API (Preview)
|
||||
has_children: true
|
||||
nav_order: 3
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,52 +1,58 @@
|
|||
---
|
||||
title: Install
|
||||
description: Instructions to install ONNX Runtime GenAI on your target platform in your environment
|
||||
description: Instructions to install ONNX Runtime generate() API on your target platform in your environment
|
||||
has_children: false
|
||||
parent: How to
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# Install ONNX Runtime GenAI
|
||||
# Install ONNX Runtime generate() API
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Python package release candidates
|
||||
## Python packages
|
||||
|
||||
```bash
|
||||
pip install numpy
|
||||
pip install onnxruntime-genai --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/
|
||||
pip install onnxruntime-genai --pre
|
||||
```
|
||||
Append `-directml` for the library that is optimized for DirectML on Windows
|
||||
|
||||
```bash
|
||||
pip install numpy
|
||||
pip install onnxruntime-genai-directml --pre
|
||||
```
|
||||
|
||||
Append `-cuda` for the library that is optimized for CUDA environments
|
||||
|
||||
```bash
|
||||
pip install numpy
|
||||
pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/
|
||||
```
|
||||
|
||||
## Nuget package release candidates
|
||||
## Nuget packages
|
||||
|
||||
To install the NuGet release candidates, add a new package source in Visual Studio, go to `Project` -> `Manage NuGet Packages`.
|
||||
```bash
|
||||
dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease
|
||||
```
|
||||
|
||||
1. Click on the `Settings` cog icon
|
||||
For the package that has been optimized for CUDA:
|
||||
|
||||
2. Click the `+` button to add a new package source
|
||||
```bash
|
||||
dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease
|
||||
```
|
||||
|
||||
- Change the Name to `onnxruntime-genai`
|
||||
- Change the Source to `https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/nuget/v3/index.json`
|
||||
For the package that has been optimized for DirectML:
|
||||
|
||||
3. Check the `Include prerelease` button
|
||||
```bash
|
||||
dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease
|
||||
```
|
||||
|
||||
4. Add the `Microsoft.ML.OnnxRuntimeGenAI` package
|
||||
|
||||
5. Add the `Microsoft.ML.OnnxRuntime` package
|
||||
|
||||
To run with CUDA, use the following packages instead:
|
||||
|
||||
- `Microsoft.ML.OnnxRuntimeGenAI.Cuda`
|
||||
- `Microsoft.ML.OnnxRuntime.Gpu`
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Setup CUDA env
|
|||
description: Instructions to setup the CUDA environtment to run onnxruntime-genai-cuda
|
||||
has_children: false
|
||||
parent: How to
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,15 +1,17 @@
|
|||
---
|
||||
title: Generative AI (Preview)
|
||||
description: Run generative models with ONNX Runtime GenAI
|
||||
title: Generate API (Preview)
|
||||
description: Run generative models with the ONNX Runtime generate() API
|
||||
has_children: true
|
||||
nav_order: 6
|
||||
---
|
||||
|
||||
# Generative AI with ONNX Runtime
|
||||
# ONNX Runtime generate() API
|
||||
|
||||
_Note: this API is in preview and is subject to change._
|
||||
|
||||
Run generative AI models with ONNX Runtime. Source code: (https://github.com/microsoft/onnxruntime-genai)
|
||||
Run generative AI models with ONNX Runtime.
|
||||
|
||||
See the source code here: [https://github.com/microsoft/onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai)
|
||||
|
||||
This library provides the generative AI loop for ONNX models, including inference with ONNX Runtime, logits processing, search and sampling, and KV cache management.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
---
|
||||
title: Config reference
|
||||
description: Reference for the ONNX Runtime Generative AI configuration file
|
||||
description: Reference for the ONNX Runtime generate() API configuration file
|
||||
has_children: false
|
||||
parent: Reference
|
||||
grand_parent: Generative AI (Preview)
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: Reference
|
||||
description: Reference information for ONNX Runtime Generative AI
|
||||
parent: Generative AI (Preview)
|
||||
parent: Generate API (Preview)
|
||||
has_children: true
|
||||
nav_order: 4
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: Tutorials
|
||||
description: Build your application with ONNX Runtime GenAI
|
||||
parent: Generative AI (Preview)
|
||||
description: Build your application with ONNX Runtime generate() API
|
||||
parent: Generate API (Preview)
|
||||
has_children: true
|
||||
nav_order: 1
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,17 +1,17 @@
|
|||
---
|
||||
title: Python phi-2 tutorial
|
||||
description: Learn how to write a language generation application with ONNX Runtime GenAI in Python using the phi-2 model
|
||||
description: Learn how to write a language generation application with ONNX Runtime generate() API in Python using the phi-2 model
|
||||
has_children: false
|
||||
parent: Tutorials
|
||||
grand_parent: Generative AI (Preview)
|
||||
nav_order: 1
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# Language generation in Python with phi-2
|
||||
|
||||
## Setup and installation
|
||||
|
||||
Install the ONNX Runtime GenAI Python package using the [installation instructions](../howto/install.md).
|
||||
Install the ONNX Runtime generate() API Python package using the [installation instructions](../howto/install.md).
|
||||
|
||||
## Build phi-2 ONNX model
|
||||
|
||||
|
|
@ -31,11 +31,11 @@ python -m onnxruntime_genai.models.builder -m microsoft/phi-2 -e cpu -p int4 -o
|
|||
```
|
||||
You can replace the name of the output folder specified with the `-o` option with a folder of your choice.
|
||||
|
||||
After you run the script, you will see a series of files generated in this folder. They include the HuggingFace configs for your reference, as well as the following generated files used by ONNX Runtime GenAI.
|
||||
After you run the script, you will see a series of files generated in this folder. They include the HuggingFace configs for your reference, as well as the following generated files used by ONNX Runtime generate() API.
|
||||
|
||||
- `model.onnx`: the phi-2 ONNX model
|
||||
- `model.onnx.data`: the phi-2 ONNX model weights
|
||||
- `genai_config.json`: the configuration used by ONNX Runtime GenAI
|
||||
- `genai_config.json`: the configuration used by ONNX Runtime generate() API
|
||||
|
||||
You can view and change the values in the `genai_config.json` file. The model section should not be updated unless you have brought your own model and it has different parameters.
|
||||
|
||||
|
|
|
|||
167
docs/genai/tutorials/phi3-python.md
Normal file
167
docs/genai/tutorials/phi3-python.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
---
|
||||
title: Python phi-3 tutorial
|
||||
description: Small but mighty. Run Phi-3 with ONNX Runtime.
|
||||
has_children: false
|
||||
parent: Tutorials
|
||||
grand_parent: Generate API (Preview)
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# Run the Phi-3 Mini models with the ONNX Runtime generate() API
|
||||
|
||||
## Steps
|
||||
1. [Setup](#setup)
|
||||
2. [Choose your platform](#choose-your-platform)
|
||||
3. [Run with DirectML](#run-with-directml)
|
||||
4. [Run with NVDIA CUDA](#run-with-nvidia-cuda)
|
||||
5. [Run on CPU](#run-on-cpu)
|
||||
|
||||
## Introduction
|
||||
|
||||
There are two Phi-3 mini models to choose from: the short (4k) context version or the long (128k) context version. The long context version can accept much longer prompts and produce longer output text, but it does consume more memory.
|
||||
|
||||
The Phi-3 ONNX models are hosted on HuggingFace: [short](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx) and [long](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx).
|
||||
|
||||
This tutorial downloads and runs the short context model. If you would like to use the long context model, change the `4k` to `128k` in the instructions below.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install the git large file system extension
|
||||
|
||||
HuggingFace uses `git` for version control. To download the ONNX models you need `git lfs` to be installed, if you do not already have it.
|
||||
|
||||
* Windows: `winget install -e --id GitHub.GitLFS` (If you don't have winget, download and run the `exe` from the [official source](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage?platform=windows))
|
||||
* Linux: `apt-get install git-lfs`
|
||||
* MacOS: `brew install git-lfs`
|
||||
|
||||
Then run `git lfs install`
|
||||
|
||||
2. Install the HuggingFace CLI
|
||||
|
||||
```bash
|
||||
pip install huggingface-hub[cli]
|
||||
```
|
||||
|
||||
## Choose your platform
|
||||
|
||||
Are you on a Windows machine with GPU?
|
||||
* I don't know → Review [this guide](https://www.microsoft.com/en-us/windows/learning-center/how-to-check-gpu) to see whether you have a GPU in your Windows machine.
|
||||
* Yes → Follow the instructions for [DirectML](#run-with-directml).
|
||||
* No → Do you have an NVIDIA GPU?
|
||||
* I don't know → Review [this guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#verify-you-have-a-cuda-capable-gpu) to see whether you have a CUDA-capable GPU.
|
||||
* Yes → Follow the instructions for [NVIDIA CUDA GPU](#run-with-nvidia-cuda).
|
||||
* No → Follow the instructions for [CPU](#run-on-cpu).
|
||||
|
||||
**Note: Only one package and model is required based on your hardware. That is, only execute the steps for one of the following sections**
|
||||
|
||||
## Run with DirectML
|
||||
|
||||
1. Download the model
|
||||
|
||||
```bash
|
||||
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include directml/* --local-dir .
|
||||
```
|
||||
|
||||
This command downloads the model into a folder called `directml`.
|
||||
|
||||
|
||||
2. Install the generate() API
|
||||
|
||||
```
|
||||
pip install numpy
|
||||
pip install --pre onnxruntime-genai-directml
|
||||
```
|
||||
|
||||
You should now see `onnxruntime-genai-directml` in your `pip list`.
|
||||
|
||||
3. Run the model
|
||||
|
||||
Run the model with [phi3-qa.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py).
|
||||
|
||||
```cmd
|
||||
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py
|
||||
python phi3-qa.py -m directml\directml-int4-awq-block-128
|
||||
```
|
||||
|
||||
Once the script has loaded the model, it will ask you for input in a loop, streaming the output as it is produced the model. For example:
|
||||
|
||||
```bash
|
||||
Input: Tell me a joke about GPUs
|
||||
|
||||
Certainly! Here\'s a light-hearted joke about GPUs:
|
||||
|
||||
|
||||
Why did the GPU go to school? Because it wanted to improve its "processing power"!
|
||||
|
||||
|
||||
This joke plays on the double meaning of "processing power," referring both to the computational abilities of a GPU and the idea of a student wanting to improve their academic skills.
|
||||
```
|
||||
|
||||
## Run with NVIDIA CUDA
|
||||
|
||||
1. Download the model
|
||||
|
||||
```bash
|
||||
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cuda/cuda-int4-rtn-block-32/* --local-dir .
|
||||
```
|
||||
|
||||
This command downloads the model into a folder called `cuda`.
|
||||
|
||||
2. Install the generate() API
|
||||
|
||||
```
|
||||
pip install numpy
|
||||
pip install --pre onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/
|
||||
```
|
||||
|
||||
3. Run the model
|
||||
|
||||
Run the model with [phi3-qa.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py).
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py
|
||||
python phi3-qa.py -m cuda/cuda-int4-rtn-block-32
|
||||
```
|
||||
|
||||
Once the script has loaded the model, it will ask you for input in a loop, streaming the output as it is produced the model. For example:
|
||||
|
||||
```bash
|
||||
Input: Tell me a joke about creative writing
|
||||
|
||||
Output: Why don\'t writers ever get lost? Because they always follow the plot!
|
||||
```
|
||||
|
||||
## Run on CPU
|
||||
|
||||
1. Download the model
|
||||
|
||||
```bash
|
||||
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .
|
||||
```
|
||||
|
||||
This command downloads the model into a folder called `cpu_and_mobile`
|
||||
|
||||
2. Install the generate() API for CPU
|
||||
|
||||
```
|
||||
pip install numpy
|
||||
pip install --pre onnxruntime-genai
|
||||
```
|
||||
|
||||
3. Run the model
|
||||
|
||||
Run the model with [phi3-qa.py](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py).
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py -o phi3-qa.py
|
||||
python phi3-qa.py -m cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4
|
||||
```
|
||||
|
||||
Once the script has loaded the model, it will ask you for input in a loop, streaming the output as it is produced the model. For example:
|
||||
|
||||
```bash
|
||||
Input: Tell me a joke about generative AI
|
||||
|
||||
Output: Why did the generative AI go to school?
|
||||
|
||||
To improve its "creativity" algorithm!
|
||||
Loading…
Reference in a new issue