mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-14 18:12:05 +00:00
612 lines
12 KiB
Markdown
612 lines
12 KiB
Markdown
---
|
|
title: Java API
|
|
description: Java API reference for ONNX Runtime generate() API
|
|
has_children: false
|
|
parent: API docs
|
|
grand_parent: Generate API (Preview)
|
|
nav_order: 4
|
|
---
|
|
|
|
# ONNX Runtime generate() Java API
|
|
|
|
_Note: this API is in preview and is subject to change._
|
|
|
|
{: .no_toc }
|
|
|
|
* TOC placeholder
|
|
{:toc}
|
|
|
|
## Install and import
|
|
The Java API is delivered by the ai.onnxruntime.genai Java package. Package publication is pending. To build the package from source, see the [build from source guide](../howto/build-from-source.md).
|
|
|
|
```java
|
|
import ai.onnxruntime.genai.*;
|
|
```
|
|
|
|
## SimpleGenAI Class
|
|
|
|
The `SimpleGenAI` class provides a simple usage example of the GenAI API. It works with a model that generates text based on a prompt, processing a single prompt at a time.
|
|
Usage:
|
|
|
|
Create an instance of the class with the path to the model. The path should also contain the GenAI configuration files.
|
|
|
|
```java
|
|
SimpleGenAI genAI = new SimpleGenAI(folderPath);
|
|
```
|
|
|
|
Call createGeneratorParams with the prompt text.
|
|
Set any other search options via the GeneratorParams object as needed using `setSearchOption`.
|
|
|
|
```java
|
|
GeneratorParams generatorParams = genAI.createGeneratorParams(promptText);
|
|
// .. set additional generator params before calling generate()
|
|
```
|
|
|
|
Call generate with the GeneratorParams object and an optional listener.
|
|
|
|
```java
|
|
String fullResponse = genAI.generate(generatorParams, listener);
|
|
```
|
|
|
|
The listener is used as a callback mechanism so that tokens can be used as they are generated. Create a class that implements the `Consumer<String>` interface and provide an instance of that class as the `listener` argument.
|
|
|
|
### Constructor
|
|
|
|
```java
|
|
public SimpleGenAI(String modelPath) throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- on failure.
|
|
|
|
### Generate Method
|
|
|
|
Generate text based on the prompt and settings in GeneratorParams.
|
|
|
|
NOTE: This only handles a single sequence of input (i.e. a single prompt which equates to batch size of 1).
|
|
|
|
```java
|
|
public String generate(GeneratorParams generatorParams, Consumer<String> listener) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `generatorParams`: the prompt and settings to run the model with.
|
|
- `listener`: optional callback for tokens to be provided as they are generated.
|
|
|
|
NOTE: Token generation will be blocked until the listener's `accept` method returns.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- on failure.
|
|
|
|
#### Returns
|
|
|
|
The generated text.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
SimpleGenAI generator = new SimpleGenAI(modelPath);
|
|
GeneratorParams params = generator.createGeneratorParams("What's 6 times 7?");
|
|
Consumer<String> listener = token -> logger.info("onTokenGenerate: " + token);
|
|
String result = generator.generate(params, listener);
|
|
|
|
logger.info("Result: " + result);
|
|
```
|
|
|
|
### createGenerateParams Method
|
|
|
|
Create the generator parameters and add the prompt text. The user can set other search options via the GeneratorParams object prior to running `generate`.
|
|
|
|
```java
|
|
public GeneratorParams createGeneratorParams(String prompt) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `prompt`: the prompt text to encode.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- on failure.
|
|
|
|
#### Returns
|
|
|
|
The generator parameters.
|
|
|
|
## Exception Class
|
|
|
|
An exception which contains the error message and code produced by the native layer.
|
|
|
|
### Constructor
|
|
|
|
```java
|
|
public GenAIException(String message)
|
|
```
|
|
|
|
#### Example
|
|
|
|
```java
|
|
catch (GenAIException e) {
|
|
throw new GenAIException("Token generation loop failed.", e);
|
|
}
|
|
```
|
|
|
|
## Model class
|
|
|
|
### Constructor
|
|
|
|
```java
|
|
Model(String modelPath)
|
|
```
|
|
|
|
### Create Tokenizer Method
|
|
|
|
Creates a Tokenizer instance for this model. The model contains the configuration information that determines the tokenizer to use.
|
|
|
|
```java
|
|
public Tokenizer createTokenizer() throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails
|
|
|
|
#### Returns
|
|
|
|
The tokenizer instance.
|
|
|
|
### Generate Method
|
|
|
|
```java
|
|
public Sequences generate(GeneratorParams generatorParams) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `generatorParams`: the generator parameters.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
The generated sequences.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
Sequences output = model.generate(generatorParams);
|
|
```
|
|
|
|
### createGeneratorParams Method
|
|
|
|
Creates a GeneratorParams instance for executing the model.
|
|
|
|
NOTE: GeneratorParams internally uses the Model, so the Model instance must remain valid.
|
|
|
|
```java
|
|
public GeneratorParams createGeneratorParams() throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
The GeneratorParams instance.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
GeneratorParams params = generator.createGeneratorParams("What's 6 times 7?");
|
|
```
|
|
|
|
## Tokenizer class
|
|
|
|
### Encode Method
|
|
|
|
Encodes a string into a sequence of token ids.
|
|
|
|
```java
|
|
public Sequences encode(String string) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `string`: text to encode as token ids.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
A Sequences object with a single sequence in it.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
Sequences encodedPrompt = tokenizer.encode(prompt);
|
|
```
|
|
|
|
### Decode Method
|
|
|
|
Decodes a sequence of token ids into text.
|
|
|
|
```java
|
|
public String decode(int[] sequence) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `sequence`: collection of token ids to decode to text
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
The text representation of the sequence.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
String result = tokenizer.decode(output_ids);
|
|
```
|
|
|
|
### encodeBatch Method
|
|
|
|
Encodes an array of strings into a sequence of token ids for each input.
|
|
|
|
```java
|
|
public Sequences encodeBatch(String[] strings) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `strings`: collection of strings to encode as token ids.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
A Sequences object with one sequence per input string.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
Sequences encoded = tokenizer.encodeBatch(inputs);
|
|
```
|
|
|
|
### decodeBatch Method
|
|
|
|
Decodes a batch of sequences of token ids into text.
|
|
|
|
```java
|
|
public String[] decodeBatch(Sequences sequences) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `sequences`: a Sequences object with one or more sequences of token ids.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
An array of strings with the text representation of each sequence.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
String[] decoded = tokenizer.decodeBatch(encoded);
|
|
```
|
|
|
|
### createStream Method
|
|
|
|
Creates a TokenizerStream object for streaming tokenization. This is used with Generator class to provide each token as it is generated.
|
|
|
|
```java
|
|
public TokenizerStream createStream() throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
The new TokenizerStream instance.
|
|
|
|
## TokenizerStream class
|
|
|
|
This class is used to convert individual tokens when using Generator.generateNextToken.
|
|
|
|
### Decode method
|
|
|
|
```java
|
|
public String decode(int token) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `token`: int value for token
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`
|
|
|
|
## Tensor Class
|
|
|
|
Constructs a Tensor with the given data, shape, and element type.
|
|
|
|
```java
|
|
public Tensor(ByteBuffer data, long[] shape, ElementType elementType) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `data`: the data for the Tensor. Must be a direct ByteBuffer.
|
|
- `shape`: the shape of the Tensor.
|
|
- `elementType`: the Type of elements in the Tensor.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`
|
|
|
|
#### Example
|
|
|
|
Create a 2x2 Tensor with 32-bit float data.
|
|
|
|
```java
|
|
long[] shape = {2, 2};
|
|
ByteBuffer data = ByteBuffer.allocateDirect(4 * Float.BYTES);
|
|
FloatBuffer floatBuffer = data.asFloatBuffer();
|
|
floatBuffer.put(new float[] {1.0f, 2.0f, 3.0f, 4.0f});
|
|
|
|
Tensor tensor = new Tensor(data, shape, Tensor.ElementType.float32);
|
|
```
|
|
|
|
## GeneratorParams class
|
|
|
|
The `GeneratorParams` class represents the parameters used for generating sequences with a model. Set the prompt using setInput, and any other search options using setSearchOption.
|
|
|
|
### Create a Generator Params object
|
|
|
|
```java
|
|
GeneratorParams params = new GeneratorParams(model);
|
|
```
|
|
|
|
### setSearchOption Method
|
|
|
|
```java
|
|
public void setSearchOption(String optionName, double value) throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`
|
|
|
|
#### Example
|
|
|
|
Set search option to limit the model generation length.
|
|
|
|
```java
|
|
generatorParams.setSearchOption("max_length", 10);
|
|
```
|
|
|
|
### setSearchOption Method
|
|
|
|
```java
|
|
public void setSearchOption(String optionName, boolean value) throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`
|
|
|
|
#### Example
|
|
|
|
```java
|
|
generatorParams.setSearchOption("early_stopping", true);
|
|
```
|
|
|
|
### setInput Method
|
|
|
|
Sets the prompt/s for model execution. The `sequences` are created by using Tokenizer.Encode or EncodeBatch.
|
|
|
|
```java
|
|
public void setInput(Sequences sequences) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
- `sequences`: sequences containing the encoded prompt.
|
|
|
|
#### Throws
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Example
|
|
```java
|
|
generatorParams.setInput(encodedPrompt);
|
|
```
|
|
|
|
### setInput Method
|
|
|
|
Sets the prompt/s token ids for model execution. The `tokenIds` are the encoded parameters.
|
|
|
|
```java
|
|
public void setInput(int[] tokenIds, int sequenceLength, int batchSize)
|
|
throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `tokenIds`: the token ids of the encoded prompt/s
|
|
- `sequenceLength`: the length of each sequence.
|
|
- `batchSize`: size of the batch.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
NOTE: all sequences in the batch must be the same length.
|
|
|
|
#### Example
|
|
|
|
```java
|
|
generatorParams.setInput(tokenIds, sequenceLength, batchSize);
|
|
```
|
|
|
|
## Generator class
|
|
|
|
The Generator class generates output using a model and generator parameters.
|
|
The expected usage is to loop until isDone returns false. Within the loop, call computeLogits followed by generateNextToken.
|
|
|
|
The newly generated token can be retrieved with getLastTokenInSequence and decoded with TokenizerStream.Decode.
|
|
|
|
After the generation process is done, GetSequence can be used to retrieve the complete generated sequence if needed.
|
|
|
|
### Create a Generator
|
|
|
|
Constructs a Generator object with the given model and generator parameters.
|
|
|
|
```java
|
|
Generator(Model model, GeneratorParams generatorParams)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `model`: the model.
|
|
- `params`: the generator parameters.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
### isDone Method
|
|
|
|
Checks if the generation process is done.
|
|
|
|
```java
|
|
public boolean isDone()
|
|
```
|
|
|
|
#### Returns
|
|
|
|
Returns true if the generation process is done, false otherwise.
|
|
|
|
### computeLogits Method
|
|
|
|
Computes the logits for the next token in the sequence.
|
|
|
|
```java
|
|
public void computeLogits() throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
### getSequence Method
|
|
|
|
Retrieves a sequence of token ids for the specified sequence index.
|
|
|
|
```java
|
|
public int[] getSequence(long sequenceIndex) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
- `sequenceIndex`: the index of the sequence.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
An array of integers with the sequence of token ids.
|
|
|
|
#### Exmaple
|
|
```java
|
|
int[] outputIds = output.getSequence(i);
|
|
```
|
|
|
|
### generateNextToken Method
|
|
|
|
Generates the next token in the sequence.
|
|
|
|
```java
|
|
public void generateNextToken() throws GenAIException
|
|
```
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
### getLastTokenInSequence Method
|
|
|
|
Retrieves the last token in the sequence for the specified sequence index.
|
|
|
|
```java
|
|
public int getLastTokenInSequence(long sequenceIndex) throws GenAIException
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `sequenceIndex`: the index of the sequence.
|
|
|
|
#### Throws
|
|
|
|
`GenAIException`- if the call to the GenAI native API fails.
|
|
|
|
#### Returns
|
|
|
|
The last token in the sequence.
|
|
|
|
## Sequences Class
|
|
|
|
Represents a collection of encoded prompts/responses.
|
|
|
|
### numSequences Method
|
|
|
|
Gets the number of sequences in the collection. This is equivalent to the batch size.
|
|
|
|
```java
|
|
public long numSequences()
|
|
```
|
|
|
|
### Returns
|
|
|
|
The number of sequences.
|
|
|
|
#### Example
|
|
```java
|
|
int numSequences = (int) sequences.numSequences();
|
|
```
|
|
|
|
### getSequence Method
|
|
|
|
Gets the sequence at the specified index.
|
|
|
|
```java
|
|
public int[] getSequence(long sequenceIndex)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
- `sequenceIndex`: The index of the sequence.
|
|
|
|
#### Returns
|
|
|
|
The sequence as an array of integers.
|
|
|