onnxruntime/docs/genai/api/python.md

---
title: Python API
description: Python API reference for ONNX Runtime generate() API
has_children: false
parent: API docs
grand_parent: Generate API (Preview)
nav_order: 1
---

# Python API

_Note: this API is in preview and is subject to change._

{: .no_toc }

* TOC placeholder
{:toc}

## Install and import

The Python API is delivered by the onnxruntime-genai Python package.

```bash
pip install onnxruntime-genai
```

```python
import onnxruntime_genai
```

## Model class

### Load a model

Loads the ONNX model(s) and configuration from a folder on disk.

```python
onnxruntime_genai.Model(model_folder: str) -> onnxruntime_genai.Model
```

#### Parameters

- `model_folder`: Location of model and configuration on disk

#### Returns

`onnxruntime_genai.Model`

### Generate method

```python
onnxruntime_genai.Model.generate(params: GeneratorParams) -> numpy.ndarray[int, int]
```

#### Parameters
- `params`: (Required) Created by the `GeneratorParams` method.

#### Returns

`numpy.ndarray[int, int]`: a two dimensional numpy array with dimensions equal to the size of the batch passed in and the maximum length of the sequence of tokens.

### Device type

Return the device type that the model has been configured to run on.

```python
onnxruntime_genai.Model.device_type
```

#### Returns

`str`: a string describing the device that the loaded model will run on


## Tokenizer class

### Create tokenizer object

```python
onnxruntime_genai.Model.Tokenizer(model: onnxruntime_genai.Model) -> onnxruntime_genai.Tokenizer
```

#### Parameters

- `model`: (Required) The model that was loaded by the `Model()`

#### Returns

- `Tokenizer`: The tokenizer object

### Encode

```python
onnxruntime_genai.Tokenizer.encode(text: str) -> numpy.ndarray[numpy.int32]
```

#### Parameters

- `text`: (Required)

#### Returns

`numpy.ndarray[numpy.int32]`: an array of tokens representing the prompt

### Decode

```python
onnxruntime_genai.Tokenizer.decode(tokens: numpy.ndarry[int]) -> str 
```

#### Parameters

- `numpy.ndarray[numpy.int32]`: (Required) a sequence of generated tokens


#### Returns

`str`: the decoded generated tokens


### Encode batch

```python
onnxruntime_genai.Tokenizer.encode_batch(texts: list[str]) -> numpy.ndarray[int, int]
```

#### Parameters

- `texts`: A list of inputs

#### Returns

`numpy.ndarray[int, int]`: The batch of tokenized strings

### Decode batch

```python
onnxruntime_genai.Tokenize.decode_batch(tokens: [[numpy.int32]]) -> list[str]
```

#### Parameters

- tokens

#### Returns

`texts`: a batch of decoded text


### Create tokenizer decoding stream


```python
onnxruntime_genai.Tokenizer.create_stream() -> TokenizerStream
```

#### Parameters

None

#### Returns

`onnxruntime_genai.TokenizerStream` The tokenizer stream object

## TokenizerStream class

This class accumulates the next displayable string (according to the tokenizer's vocabulary).

### Decode method

 
```python
onnxruntime_genai.TokenizerStream.decode(token: int32) -> str
```
  
#### Parameters

- `token`: (Required) A token to decode

#### Returns

`str`: If a displayable string has accumulated, this method returns it. If not, this method returns the empty string.

## GeneratorParams class

### Create a Generator Params object

```python
onnxruntime_genai.GeneratorParams(model: Model) -> GeneratorParams
```

### Pad token id member

```python
onnxruntime_genai.GeneratorParams.pad_token_id
```

### EOS token id member

```python
onnxruntime_genai.GeneratorParams.eos_token_id
```

### vocab size member

```python
onnxruntime_genai.GeneratorParams.vocab_size
```

### input_ids member

```python
onnxruntime_genai.GeneratorParams.input_ids: numpy.ndarray[numpy.int32, numpy.int32]
```

### Set model input

```python
onnxruntime_genai.GeneratorParams.set_model_input(name: str, value: [])
```


### Set search options method

```python
onnxruntime_genai.GeneratorParams.set_search_options(options: dict[str, Any])
```

### Try graph capture with max batch size

```python
onnxruntime_genai.GeneratorParams.try_graph_capture_with_max_batch_size(max_batch_size: int)
```

## Generator class

### Create a Generator

```python
onnxruntime_genai.Generator(model: Model, params: GeneratorParams) -> Generator
```

#### Parameters

- `model`: (Required) The model to use for generation
- `params`: (Required) The set of parameters that control the generation

#### Returns

`onnxruntime_genai.Generator` The Generator object


### Is generation done

```python
onnxruntime_genai.Generator.is_done() -> bool
```

#### Returns

Returns true when all sequences are at max length, or have reached the end of sequence.


### Compute logits

Runs the model through one iteration.

```python
onnxruntime_genai.Generator.compute_logits()
```

### Get output

Returns an output of the model.

```python
onnxruntime_genai.Generator.get_output(str: name) -> numpy.ndarray
```

#### Parameters
- `name`: the name of the model output

#### Returns
- `numpy.ndarray`: a multi dimensional array of the model outputs. The shape of the array is shape of the output.

#### Example

The following code returns the output logits of a model.

```python
logits = generator.get_output("logits")
```


### Generate next token

Using the current set of logits and the specified generator parameters, calculates the next batch of tokens, using Top P sampling.

```python
onnxruntime_genai.Generator.generate_next_token()
```

### Get next tokens

```python
onnxruntime_genai.Generator.get_next_tokens() -> numpy.ndarray[numpy.int32]
```

Returns

`numpy.ndarray[numpy.int32]`: The most recently generated tokens

### Get sequence

```python
onnxruntime_genai.Generator.get_sequence(index: int) -> numpy.ndarray[numpy.int32] 
```

- `index`: (Required) The index of the sequence in the batch to return