diff --git a/docs/genai/api/java.md b/docs/genai/api/java.md new file mode 100644 index 0000000000..3f69ce76b1 --- /dev/null +++ b/docs/genai/api/java.md @@ -0,0 +1,612 @@ +--- +title: Java API +description: Java API reference for ONNX Runtime generate() API +has_children: false +parent: API docs +grand_parent: Generate API (Preview) +nav_order: 1 +--- + +# ONNX Runtime generate() Java API + +_Note: this API is in preview and is subject to change._ + +{: .no_toc } + +* TOC placeholder +{:toc} + +## Install and import +The Java API is delivered by the ai.onnxruntime.genai Java package. Package publication is pending. To build the package from source, see the [installation guide](../howto/build-from-source.md). + +```java +import ai.onnxruntime.genai.*; +``` + +## SimpleGenAI Class + +The `SimpleGenAI` class provides a simple usage example of the GenAI API. It works with a model that generates text based on a prompt, processing a single prompt at a time. +Usage: + +Create an instance of the class with the path to the model. The path should also contain the GenAI configuration files. + +```java +SimpleGenAI genAI = new SimpleGenAI(folderPath); +``` + +Call createGeneratorParams with the prompt text. +Set any other search options via the GeneratorParams object as needed using `setSearchOption`. + +```java +GeneratorParams generatorParams = genAI.createGeneratorParams(promptText); +// .. set additional generator params before calling generate() +``` + +Call generate with the GeneratorParams object and an optional listener. + +```java +String fullResponse = genAI.generate(generatorParams, listener); +``` + +The listener is used as a callback mechanism so that tokens can be used as they are generated. Create a class that implements the `Consumer` interface and provide an instance of that class as the `listener` argument. + +### Constructor + +```java +public SimpleGenAI(String modelPath) throws GenAIException +``` + +#### Throws + +`GenAIException`- on failure. + +### Generate Method + +Generate text based on the prompt and settings in GeneratorParams. + +NOTE: This only handles a single sequence of input (i.e. a single prompt which equates to batch size of 1). + +```java +public String generate(GeneratorParams generatorParams, Consumer listener) throws GenAIException +``` + +#### Parameters + +- `generatorParams`: the prompt and settings to run the model with. +- `listener`: optional callback for tokens to be provided as they are generated. + +NOTE: Token generation will be blocked until the listener's `accept` method returns. + +#### Throws + +`GenAIException`- on failure. + +#### Returns + +The generated text. + +#### Example + +```java +SimpleGenAI generator = new SimpleGenAI(modelPath); +GeneratorParams params = generator.createGeneratorParams("What's 6 times 7?"); +Consumer listener = token -> logger.info("onTokenGenerate: " + token); +String result = generator.generate(params, listener); + +logger.info("Result: " + result); +``` + +### createGenerateParams Method + +Create the generator parameters and add the prompt text. The user can set other search options via the GeneratorParams object prior to running `generate`. + +```java +public GeneratorParams createGeneratorParams(String prompt) throws GenAIException +``` + +#### Parameters + +- `prompt`: the prompt text to encode. + +#### Throws + +`GenAIException`- on failure. + +#### Returns + +The generator parameters. + +## Exception Class + +An exception which contains the error message and code produced by the native layer. + +### Constructor + +```java +public GenAIException(String message) +``` + +#### Example + +```java +catch (GenAIException e) { + throw new GenAIException("Token generation loop failed.", e); +} +``` + +## Model class + +### Constructor + +```java +Model(String modelPath) +``` + +### Create Tokenizer Method + +Creates a Tokenizer instance for this model. The model contains the configuration information that determines the tokenizer to use. + +```java +public Tokenizer createTokenizer() throws GenAIException +``` + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails + +#### Returns + +The tokenizer instance. + +### Generate Method + +```java +public Sequences generate(GeneratorParams generatorParams) throws GenAIException +``` + +#### Parameters + +- `generatorParams`: the generator parameters. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +The generated sequences. + +#### Example + +```java +Sequences output = model.generate(generatorParams); +``` + +### createGeneratorParams Method + +Creates a GeneratorParams instance for executing the model. + +NOTE: GeneratorParams internally uses the Model, so the Model instance must remain valid. + +```java +public GeneratorParams createGeneratorParams() throws GenAIException +``` + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +The GeneratorParams instance. + +#### Example + +```java +GeneratorParams params = generator.createGeneratorParams("What's 6 times 7?"); +``` + +## Tokenizer class + +### Encode Method + +Encodes a string into a sequence of token ids. + +```java +public Sequences encode(String string) throws GenAIException +``` + +#### Parameters + +- `string`: text to encode as token ids. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +A Sequences object with a single sequence in it. + +#### Example + +```java +Sequences encodedPrompt = tokenizer.encode(prompt); +``` + +### Decode Method + +Decodes a sequence of token ids into text. + +```java +public String decode(int[] sequence) throws GenAIException +``` + +#### Parameters + +- `sequence`: collection of token ids to decode to text + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +The text representation of the sequence. + +#### Example + +```java +String result = tokenizer.decode(output_ids); +``` + +### encodeBatch Method + +Encodes an array of strings into a sequence of token ids for each input. + +```java +public Sequences encodeBatch(String[] strings) throws GenAIException +``` + +#### Parameters + +- `strings`: collection of strings to encode as token ids. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +A Sequences object with one sequence per input string. + +#### Example + +```java +Sequences encoded = tokenizer.encodeBatch(inputs); +``` + +### decodeBatch Method + +Decodes a batch of sequences of token ids into text. + +```java +public String[] decodeBatch(Sequences sequences) throws GenAIException +``` + +#### Parameters + +- `sequences`: a Sequences object with one or more sequences of token ids. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +An array of strings with the text representation of each sequence. + +#### Example + +```java +String[] decoded = tokenizer.decodeBatch(encoded); +``` + +### createStream Method + +Creates a TokenizerStream object for streaming tokenization. This is used with Generator class to provide each token as it is generated. + +```java +public TokenizerStream createStream() throws GenAIException +``` + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +The new TokenizerStream instance. + +## TokenizerStream class + +This class is used to convert individual tokens when using Generator.generateNextToken. + +### Decode method + +```java +public String decode(int token) throws GenAIException +``` + +#### Parameters + +- `token`: int value for token + +#### Throws + +`GenAIException` + +## Tensor Class + +Constructs a Tensor with the given data, shape, and element type. + +```java +public Tensor(ByteBuffer data, long[] shape, ElementType elementType) throws GenAIException +``` + +#### Parameters + +- `data`: the data for the Tensor. Must be a direct ByteBuffer. +- `shape`: the shape of the Tensor. +- `elementType`: the Type of elements in the Tensor. + +#### Throws + +`GenAIException` + +#### Example + +Create a 2x2 Tensor with 32-bit float data. + +```java +long[] shape = {2, 2}; +ByteBuffer data = ByteBuffer.allocateDirect(4 * Float.BYTES); +FloatBuffer floatBuffer = data.asFloatBuffer(); +floatBuffer.put(new float[] {1.0f, 2.0f, 3.0f, 4.0f}); + +Tensor tensor = new Tensor(data, shape, Tensor.ElementType.float32); +``` + +## GeneratorParams class + +The `GeneratorParams` class represents the parameters used for generating sequences with a model. Set the prompt using setInput, and any other search options using setSearchOption. + +### Create a Generator Params object + +```java +GeneratorParams params = new GeneratorParams(model); +``` + +### setSearchOption Method + +```java +public void setSearchOption(String optionName, double value) throws GenAIException +``` + +#### Throws + +`GenAIException` + +#### Example + +Set search option to limit the model generation length. + +```java +generatorParams.setSearchOption("max_length", 10); +``` + +### setSearchOption Method + +```java +public void setSearchOption(String optionName, boolean value) throws GenAIException +``` + +#### Throws + +`GenAIException` + +#### Example + +```java +generatorParams.setSearchOption("early_stopping", true); +``` + +### setInput Method + +Sets the prompt/s for model execution. The `sequences` are created by using Tokenizer.Encode or EncodeBatch. + +```java +public void setInput(Sequences sequences) throws GenAIException +``` + +#### Parameters +- `sequences`: sequences containing the encoded prompt. + +#### Throws +`GenAIException`- if the call to the GenAI native API fails. + +#### Example +```java +generatorParams.setInput(encodedPrompt); +``` + +### setInput Method + +Sets the prompt/s token ids for model execution. The `tokenIds` are the encoded parameters. + +```java +public void setInput(int[] tokenIds, int sequenceLength, int batchSize) + throws GenAIException +``` + +#### Parameters + +- `tokenIds`: the token ids of the encoded prompt/s +- `sequenceLength`: the length of each sequence. +- `batchSize`: size of the batch. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +NOTE: all sequences in the batch must be the same length. + +#### Example + +```java +generatorParams.setInput(tokenIds, sequenceLength, batchSize); +``` + +## Generator class + +The Generator class generates output using a model and generator parameters. +The expected usage is to loop until isDone returns false. Within the loop, call computeLogits followed by generateNextToken. + +The newly generated token can be retrieved with getLastTokenInSequence and decoded with TokenizerStream.Decode. + +After the generation process is done, GetSequence can be used to retrieve the complete generated sequence if needed. + +### Create a Generator + +Constructs a Generator object with the given model and generator parameters. + +```java +Generator(Model model, GeneratorParams generatorParams) +``` + +#### Parameters + +- `model`: the model. +- `params`: the generator parameters. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +### isDone Method + +Checks if the generation process is done. + +```java +public boolean isDone() +``` + +#### Returns + +Returns true if the generation process is done, false otherwise. + +### computeLogits Method + +Computes the logits for the next token in the sequence. + +```java +public void computeLogits() throws GenAIException +``` + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +### getSequence Method + +Retrieves a sequence of token ids for the specified sequence index. + +```java +public int[] getSequence(long sequenceIndex) throws GenAIException +``` + +#### Parameters +- `sequenceIndex`: the index of the sequence. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +An array of integers with the sequence of token ids. + +#### Exmaple +```java +int[] outputIds = output.getSequence(i); +``` + +### generateNextToken Method + +Generates the next token in the sequence. + +```java +public void generateNextToken() throws GenAIException +``` + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +### getLastTokenInSequence Method + +Retrieves the last token in the sequence for the specified sequence index. + +```java +public int getLastTokenInSequence(long sequenceIndex) throws GenAIException +``` + +#### Parameters + +- `sequenceIndex`: the index of the sequence. + +#### Throws + +`GenAIException`- if the call to the GenAI native API fails. + +#### Returns + +The last token in the sequence. + +## Sequences Class + +Represents a collection of encoded prompts/responses. + +### numSequences Method + +Gets the number of sequences in the collection. This is equivalent to the batch size. + +```java +public long numSequences() +``` + +### Returns + +The number of sequences. + +#### Example +```java +int numSequences = (int) sequences.numSequences(); +``` + +### getSequence Method + +Gets the sequence at the specified index. + +```java +public int[] getSequence(long sequenceIndex) +``` + +#### Parameters + +- `sequenceIndex`: The index of the sequence. + +#### Returns + +The sequence as an array of integers. + diff --git a/docs/genai/howto/build-from-source.md b/docs/genai/howto/build-from-source.md index 71971239cf..683dd26739 100644 --- a/docs/genai/howto/build-from-source.md +++ b/docs/genai/howto/build-from-source.md @@ -15,7 +15,8 @@ nav_order: 2 ## Pre-requisites -`cmake` +- `cmake` +- `.Net v6` (if building C# library or API) ## Clone the onnxruntime-genai repo @@ -154,36 +155,45 @@ This step assumes that you are in the root of the onnxruntime-genai repo, and yo cd ../onnxruntime-genai ``` -### Build for Windows CPU +### Build Python API + +#### Build for Windows CPU ```bash python build.py ``` -### Build for Windows DirectML +#### Build for Windows DirectML ```bash python build.py --use_dml ``` -### Build on Linux +#### Build on Linux ```bash python build.py ``` -### Build on Linux with CUDA +#### Build on Linux with CUDA ```bash python build.py --use_cuda ``` -### Build on Mac +#### Build on Mac ```bash python build.py ``` - + +### Build Java API + +```bash +python build.py --build_java --config Release +``` +Change config to Debug for debug builds. + ## Install the library into your application ### Install Python wheel @@ -193,10 +203,12 @@ cd build/wheel pip install *.whl ``` +### Install .jar + +Copy `build/Windows/Release/src/java/build/libs/*.jar` into your application. + ### Install Nuget package - - ### Install C/C++ header file and library _Coming soon_