# On-Device Training: Building an Android Application
In this tutorial, we will explore how to build an Android application that incorporates ONNX Runtime's On-Device Training solution. On-device training refers to the process of training a machine learning model directly on an edge device without relying on cloud services or external servers.
Here is what the application will look like at the end of this tutorial:
We will guide you through the steps to create an Android app that can train a simple image classification model using on-device training techniques. This tutorial showcases the `transfer learning` technique where knowledge gained from training a model on one task is leveraged to improve the performance of a model on a different but related task. Instead of starting the learning process from scratch, transfer learning allows us to transfer the knowledge or features learned by a pre-trained model to a new task.
For this tutorial, we will leverage the `MobileNetV2` model which has been trained on large-scale image datasets such as ImageNet (which has 1,000 classes). We will use this model for classifying custom data into one of four classes. The initial layers of MobileNetV2 serve as a feature extractor, capturing generic visual features applicable to various tasks, and only the final classifier layer will be trained for the task at hand.
In this tutorial, we will use data to learn to:
- Classify animals into one of four categories using a pre-packed animals dataset.
- Classify celebrities into one of four categories using a custom celebrities dataset.
## Contents
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Offline Phase - Building the training artifacts](#offline-phase---building-the-training-artifacts)
- [Export the model to ONNX](#op1)
- [Define the trainable and non trainable parameters](#op2)
- [Setting up the project in Android Studio](#tp1)
- [Adding the ONNX Runtime dependency](#tp2)
- [Packaging the Prebuilt Training Artifacts and Dataset](#tp3)
- [Interfacing with ONNX Runtime - C++ Code](#tp4)
- [Image Preprocessing](#tp5)
- [Application Frontend](#tp6)
- [Training Phase - Running the application on a device](#training-phase---running-the-application-on-a-device)
- [Running the application on a device](#tp7)
- [Training with a pre-loaded dataset - Animals](#tp8)
- [Training with a custom dataset - Celebrities](#tp9)
- [Conclusion](#conclusion)
## Prerequisites
To follow this tutorial, you should have a basic understanding of Android app development using Java or Kotlin. Familiarity with C++ as well as familiarity with machine learning concepts such as neural networks and image classification will help as well.
- Python development environment to prepare the training artifacts
- Android Studio 4.1+
- Android SDK 29+
- Android NDK r21+
- An Android device with a camera in [developer mode](https://developer.android.com/studio/debug/dev-options) with USB debugging enabled
> **Note** The entire android application is also made available on the [`onnxruntime-training-examples`](https://github.com/microsoft/onnxruntime-training-examples/tree/master/on_device_training/mobile/android/c-cpp) GitHub repository.
## Offline Phase - Building the training artifacts
1.<aname="op1"></a>Export the model to ONNX.
We start with a pre-trained PyTorch model and export it to ONNX. The `MobileNetV2` model has been pretrained on the imagenet dataset that has data in 1000 categories. For our task of image classification, we want to only classify images in 4 classes. So, we change the last layer of the model to output 4 logits instead of 1,000.
More details around how to export PyTorch models to ONNX can be found [here](https://pytorch.org/docs/stable/onnx.html).
We will use the `CrossEntropyLoss` loss and the `AdamW` optimizer for this tutorial. More details around artifact generation can be found [here](../../../docs/api/python/on_device_training/training_artifacts.html).
That's all! The training artifacts have been generated in the `training_artifacts` folder. This marks the end of the offline phase. These artifacts are ready to be deployed to the Android device for training.
## Training Phase - Android application development
1.<aname="tp1"></a>Setting up the project in Android Studio
b. Head over to [Maven Central](https://central.sonatype.com/artifact/com.microsoft.onnxruntime/onnxruntime-training-android/). Go to `Versions`->`Browse`-> and download the `onnxruntime-training-android` archive package (aar file).
c. Rename the `aar` extension to `zip`. So `onnxruntime-training-android-1.15.0.aar` becomes `onnxruntime-training-android-1.15.0.zip`.
d. Extract the contents of the zip file.
e. Copy the `libonnxruntime.so` shared library from the `jni\arm64-v8a` folder to your Android project under the newly created `lib` folder.
f. Copy the contents of the `headers` folder to the newly created `include\onnxruntime` folder.
g. In the `native-lib.cpp` file, include the training cxx header file.
```cpp
#include "onnxruntime_training_cxx_api.h"
```
h. Add `abiFilters` to the `build.gradle (Module)` file so as to select `arm64-v8a`. This setting must be added under `defaultConfig` in `build.gradle`:
```gradle
ndk {
abiFilters 'arm64-v8a'
}
```
Note that the `defaultConfig` section of the `build.gradle` file should look like:
i. Add the `onnxruntime` shared library to the `CMakeLists.txt` so that `cmake` can find and build against the shared library. To do this, add these lines after the `ortpersonalize` library is added in the `CMakeLists.txt`:
find_library( # Sets the name of the path variable.
log-lib
# Specifies the name of the NDK library that
# you want CMake to locate.
log)
target_link_libraries( # Specifies the target library.
ortpersonalize
# Links the target library to the log library
# included in the NDK.
${log-lib}
+ onnxruntime)
```
j. Build the application and wait for success to confirm that the app has included the ONNX Runtime headers and can link against the shared onnxruntime library successfully.
3.<aname="tp3"></a>Packaging the Prebuilt Training Artifacts and Dataset
a. Create a new `assets` folder inside the `app` from the left pane of the Android Studio project by right click app -> New -> Folder -> Assets Folder and place it under main.
b. Copy the training artifacts generated in step 2 to this folder.
c. Now, head over to the [`onnxruntime-training-examples`](https://github.com/microsoft/onnxruntime-training-examples/tree/master/on_device_training/mobile/android/c-cpp/data) repo and download the dataset (`images.zip`) to your machine and extract it. This dataset was modified from the orignal [`animals-10`](https://www.kaggle.com/datasets/alessiocorrado99/animals10) dataset available on Kaggle created by [Corrado Alessio](https://www.kaggle.com/alessiocorrado99).
d. Copy the downloaded `images` folder to `assets/images` directory in Android Studio.
The left pane of the project should look like this:
4.<aname="tp4"></a>Interfacing with ONNX Runtime - C++ Code
a. We will implement the following four functions in C++ that will be called from the application:
-`createSession`: Will be invoked on the application startup. It will create a new `CheckpointState` and `TrainingSession` objects.
-`releaseSession`: Will be invoked when the application is about to close. This function will free up resources that were allocated at the start of the application.
-`performTraining`: Will be invoked when the user clicks the `Train` button on the UI.
-`performInference`: Will be invoked when the user clicks the `Infer` button on the UI.
b. Create Session
This function gets called when the application is launched. This will use the training artifacts assets to create the `C++` CheckpointState and TrainingSession objects. These objects will be used for training the model on the device.
The arguments to `createSession` are:
-`checkpoint_path`: Cached path to the checkpoint artifact.
-`train_model_path`: Cached path to the training model artifact.
-`eval_model_path`: Cached path to the eval model artifact.
-`optimizer_model_path`: Cached path to the optimizer model artifact.
-`cache_dir_path`: Path to the cache dir on the android device. The cache dir is used as a way to access the training artifacts from the C++ code.
The function returns a `long` that represents the pointer to the `session_cache` object. This `long` can be cast to `SessionCache` whenever we need access to the training session.
As can be seen from the function body above, this function creates a unique pointer to an object of the class `SessionCache`. The definition of `SessionCache` is provided below.
This function gets called when the application is about to shutdown. It releases the resources that were created when the application was launched, mainly the CheckpointState and the TrainingSession.
The arguments to `releaseSession` are:
-`session`: `long` representing the `SessionCache` object.
auto *session_cache = reinterpret_cast<SessionCache *>(session);
delete session_cache->inference_session;
delete session_cache;
}
```
d. Perform Training
This function gets called for every batch that needs to be trained. The training loop is written on the application side in Kotlin, and within the training loop, the `performTraining` function gets invoked for every batch.
The arguments to `performTraining` are:
-`session`: `long` representing the `SessionCache` object.
-`batch`: Input images as a float array to be passed in for training.
-`labels`: Labels as an int array associated with the input images provided for training.
-`batch_size`: Number of images to process with each `TrainStep`.
-`channels`: Number of channels in the image. For our example, this will always be invoked with the value `3`.
-`frame_rows`: Number of rows in the image. For our example, this will always be invoked with the value `224`.
-`frame_cols`: Number of columns in the image. For our example, this will always be invoked with the value `224`.
The function returns a `float` that represents the training loss for this batch.
a. The `MobileNetV2` model expects that the input image provided be
- of size `3 x 224 x 224`.
- a normalized image with the mean `(0.485, 0.456, 0.406)` subtracted and divided by the standard deviation `(0.229, 0.224, 0.225)`
This preprocessing is done in Java/Kotlin using the android provided libraries.
Let's create a new file called `ImageProcessingUtil.kt` under the `app/src/main/java/com/example/ortpersonalize` directory. We will add the utility methods for cropping and resizing, and normalizing the images in this file.
b. Cropping and resizing the image.
```java
fun processBitmap(bitmap: Bitmap) : Bitmap {
// This function processes the given bitmap by
// - cropping along the longer dimension to get a square bitmap
// If the width is larger than the height
// ___+_________________+___
// | + + |
// | + + |
// | + + + |
// | + + |
// |__+_________________+__|
// <--------width-------->
// <-----height---->
// <--> cropped <-->
//
// If the height is larger than the width
// _________________________ ʌ ʌ
// | | | cropped
// |+++++++++++++++++++++++| | ʌ v
// | | | |
// | | | |
// | + | height width
// | | | |
// | | | |
// |+++++++++++++++++++++++| | v ʌ
// | | | cropped
// |_______________________| v v
//
//
//
// - resizing the cropped square image to be of size (3 x 224 x 224) as needed by the
// mobilenetv2 model.
lateinit var bitmapCropped: Bitmap
if (bitmap.getWidth() >= bitmap.getHeight()) {
// Since height is smaller than the width, we crop a square whose length is the height
// So cropping happens along the width dimesion
val width: Int = bitmap.getHeight()
val height: Int = bitmap.getHeight()
// left side of the cropped image must begin at (bitmap.getWidth() / 2 - bitmap.getHeight() / 2)
// so that the cropped width contains equal portion of the width on either side of center
// top side of the cropped image must begin at 0 since we are not cropping along the height
// dimension
val x: Int = bitmap.getWidth() / 2 - bitmap.getHeight() / 2
val y: Int = 0
bitmapCropped = Bitmap.createBitmap(bitmap, x, y, width, height)
} else {
// Since width is smaller than the height, we crop a square whose length is the width
// So cropping happens along the height dimesion
val width: Int = bitmap.getWidth()
val height: Int = bitmap.getWidth()
// left side of the cropped image must begin at 0 since we are not cropping along the width
// dimension
// top side of the cropped image must begin at (bitmap.getHeight() / 2 - bitmap.getWidth() / 2)
// so that the cropped height contains equal portion of the height on either side of center
val x: Int = 0
val y: Int = bitmap.getHeight() / 2 - bitmap.getWidth() / 2
bitmapCropped = Bitmap.createBitmap(bitmap, x, y, width, height)
}
// Resize the image to be channels x width x height as needed by the mobilenetv2 model
val width: Int = 224
val height: Int = 224
val bitmapResized: Bitmap = Bitmap.createScaledBitmap(bitmapCropped, width, height, false)
return bitmapResized
}
```
c. Normalizing the image.
```java
fun processImage(bitmap: Bitmap, buffer: FloatBuffer, offset: Int) {
// This function iterates over the image and performs the following
// on the image pixels
// - normalizes the pixel values to be between 0 and 1
// - substracts the mean (0.485, 0.456, 0.406) (derived from the mobilenetv2 model configuration)
// from the pixel values
// - divides by pixel values by the standard deviation (0.229, 0.224, 0.225) (derived from the
// mobilenetv2 model configuration)
// Values are written to the given buffer starting at the provided offset.
a. For this tutorial, we will be using the following user interface elements:
- Train and Infer buttons
- Class buttons
- Status message text
- Image display
- Progress dialogue
b. This tutorial does not intend to show how the graphical user interface is created. For this reason, we will simply re-use the files available on [GitHub](https://github.com/microsoft/onnxruntime-training-examples/).
c. Copy all the string definitions from [`strings.xml`](https://github.com/microsoft/onnxruntime-training-examples/blob/206521eb22e496b2ea50bef956e63273b6b1d5bf/on_device_training/mobile/android/c-cpp/app/ORTPersonalize/app/src/main/res/values/strings.xml) to `strings.xml` local to your Android Studio.
d. Copy the contents from [`activity_main.xml`](https://github.com/microsoft/onnxruntime-training-examples/blob/206521eb22e496b2ea50bef956e63273b6b1d5bf/on_device_training/mobile/android/c-cpp/app/ORTPersonalize/app/src/main/res/layout/activity_main.xml) to `activity_main.xml` local to your Android Studio.
e. Create a new file under the `layout` folder called `dialog.xml`. Copy the contents from [`dialog.xml`](https://github.com/microsoft/onnxruntime-training-examples/blob/206521eb22e496b2ea50bef956e63273b6b1d5bf/on_device_training/mobile/android/c-cpp/app/ORTPersonalize/app/src/main/res/layout/dialog.xml) to the newly created `dialog.xml` local to your Android Studio.
f. The remainder of the changes in this section need to be made in the [MainActivity.kt](https://github.com/microsoft/onnxruntime-training-examples/blob/master/on_device_training/mobile/android/c-cpp/app/ORTPersonalize/app/src/main/java/com/example/ortpersonalize/MainActivity.kt) file.
g. Launching the application
When the application launches, the `onCreate` function gets invoked. This function is responsible for setting up the session cache and the user interface handlers.
Please refer to the `onCreate` function in the `MainActivity.kt` file for the code.
h. Custom class button handlers - We would like to use the class buttons for users to select their custom images for training. We need to add the listeners for these buttons to do this. These listeners will do exactly that.
Please refer to these button handlers in `MainActivity.kt`:
- onClassAClickedListener
- onClassBClickedListener
- onClassXClickedListener
- onClassYClickedListener
i. Personalize the custom class labels
By default, the custom class labels are `[A, B, X, Y]`. But, let's allow users to rename these labels for clarity. This is achieved by long click listeners namely (defined in `MainActivity.kt`):
- onClassALongClickedListener
- onClassBLongClickedListener
- onClassXLongClickedListener
- onClassYLongClickedListener
j. Toggling the custom classes.
When the custom class toggle is turned off, the prepackaged animals dataset is run. And when it is turned on, the user is expected to bring their own dataset for training. To handle this transition, the `onCustomClassSettingChangedListener` switch handler is implemented in `MainActivity.kt`.
k. Training handler
When each class has at least 1 image, the `Train` button can be enabled. When the `Train` button is clicked, training kicks in for the selected images. The training handler is responsible for:
- collecting the training images into one container.
- shuffling the order of the images.
- cropping and resizing the images.
- normalizing the images.
- batching the images.
- executing the training loop (invoking the C++ `performTraining` function in a loop).
The `onTrainButtonClickedListener` function defined in `MainActivity.kt` does the above.
l. Inference handler
Once training is complete, the user can click the `Infer` button to infer any image. The inference handler is responsible for
- collecting the inference image.
- cropping and resizing the image.
- normalizing the image.
- invoking the C++ `performInference` function.
- Reporting the inferred output to the user interface.
This is achieved by the `onInferenceButtonClickedListener` function in `MainActivity.kt`.
m. Handler for all the activities mentioned above
Once the image(s) have been selected for inference or for the custom classes, they need to be processed. The `onActivityResult` function defined in `MainActivity.kt` does that.
n. One last thing. Add the following in the `AndroidManifest.xml` file to use the camera:
Congratulations! You have successfully built an Android application that learns to classify images using ONNX Runtime on the device. The application is also made available on GitHub at [`onnxruntime-training-examples`](https://github.com/microsoft/onnxruntime-training-examples/tree/master/on_device_training/mobile/android/c-cpp).
