diff --git a/docs/tutorials/mobile/deploy-ios.md b/docs/tutorials/mobile/deploy-ios.md index b3d9ebdb46..6667c138d4 100644 --- a/docs/tutorials/mobile/deploy-ios.md +++ b/docs/tutorials/mobile/deploy-ios.md @@ -3,7 +3,7 @@ title: Mobile objection detection on iOS description: Build an iOS object detection application with ONNX Runtime parent: Deploy on mobile grand_parent: Tutorials -nav_order: 1 +nav_order: 4 --- # Write a mobile object detection iOS application diff --git a/docs/tutorials/mobile/helpers/index.md b/docs/tutorials/mobile/helpers/index.md index 7cd444e569..0fa071f00f 100644 --- a/docs/tutorials/mobile/helpers/index.md +++ b/docs/tutorials/mobile/helpers/index.md @@ -3,7 +3,7 @@ title: ORT Mobile Model Export Helpers descriptions: Helpers to assist with export and usage of models with ORT Mobile parent: Deploy on mobile grand_parent: Tutorials -nav_order: 4 +nav_order: 5 --- # ORT Mobile Model Export Helpers diff --git a/docs/tutorials/mobile/pose-detection.md b/docs/tutorials/mobile/pose-detection.md new file mode 100644 index 0000000000..d07da3d90b --- /dev/null +++ b/docs/tutorials/mobile/pose-detection.md @@ -0,0 +1,153 @@ +--- +title: Object detection and pose estimation with YOLOv8 +description: Run the YOLOv8 model on mobile (iOS and Android) with built-in pre and post processing to perform object detection and pose estimation +image: /images/pose-thumbnail.png +parent: Deploy on mobile +grand_parent: Tutorials +nav_order: 1 +--- + +# Object detection and pose estimation on mobile with YOLOv8 + +Learn how to build and run ONNX models on mobile with built-in pre and post processing for object detection and pose estimation. + +## Contents +{: .no_toc } + +* TOC placeholder +{:toc} + +## Object detection with YOLOv8 + +You can find the full source code for the [Android](https://github.com/microsoft/ app in the ONNX Runtime inference examples repository. + +### Build the ONNX model with built-in pre and post processing + +This step is optional as the model is available in the examples repository in the applications folders above. If you are interested, the following steps show you how to build the model yourself. + +Create a Python environment and install the following packages. + +```bash +pip install --upgrade onnx onnxruntime onnxruntime-extensions pillow +``` + +Download the following script to build the model. + +```bash +curl https://raw.githubusercontent.com/microsoft/onnxruntime-extensions/main/tutorials/yolo_e2e.py > yolo_e2e.py +``` + +Run the script. + +```bash +python yolo_e2e.py [--test_image ] +``` + +After the script has run, you will see one PyTorch model and two ONNX models: +* `yolov8n.pt`: The original YOLOv8 PyTorch model +* `yolov8n.onnx`: The exported YOLOv8 ONNX model +* `yolov8n.with_pre_post_processing.onnx`: The ONNX model with pre and post processing included in the model +* `.out.jpg`: Your test image with bounding boxes supplied. + +For example, the wolves test image in the extensions repo: + +![Image of three white wolves with red bounding boxes](../../../images/wolves-with-bounding-boxes.png) + +### Build an Android application + +Load the Android application into Android Developer Studio. + +You see the main inference code in [ObjectDetector.kt](https://github.com/microsoft/onnxruntime-inference-examples/blob/main/mobile/examples/object_detection/android/app/src/main/java/ai/onnxruntime/example/objectdetection/ObjectDetector.kt). It's as simple as loading the image image into byte array, and running it through the model with ONNX Runtime to get the original image with boxes. + +```java + fun detect(inputStream: InputStream, ortEnv: OrtEnvironment, ortSession: OrtSession): Result { + // Step 1: convert image into byte array (raw image bytes) + val rawImageBytes = inputStream.readBytes() + + // Step 2: get the shape of the byte array and make ort tensor + val shape = longArrayOf(rawImageBytes.size.toLong()) + + val inputTensor = OnnxTensor.createTensor( + ortEnv, + ByteBuffer.wrap(rawImageBytes), + shape, + OnnxJavaType.UINT8 + ) + inputTensor.use { + // Step 3: call ort inferenceSession run + val output = ortSession.run(Collections.singletonMap("image", inputTensor), + setOf("image_out","scaled_box_out_next") + ) + + // Step 4: output analysis + output.use { + val rawOutput = (output?.get(0)?.value) as ByteArray + val boxOutput = (output?.get(1)?.value) as Array + val outputImageBitmap = byteArrayToBitmap(rawOutput) + + // Step 5: set output result + var result = Result(outputImageBitmap,boxOutput) + return result + } + } + } +``` + +![Image of person with bicycle](../../../images/person-with-bicycle-and-bounding-boxes.png) + +## Pose estimation with YOLOv8 + +### Build the pose estimation model + +Note: this part of the tutorial uses Python. Android and iOS samples are coming soon! + +Create a Python environment and install the following packages. + +```bash +pip install --upgrade onnx onnxruntime onnxruntime-extensions pillow +``` + +Download the following script to build the model. + +```bash +curl https://raw.githubusercontent.com/microsoft/onnxruntime-extensions/main/tutorials/yolov8_pose_e2e.py > yolov8_pose_e2e.py +``` + +Run the script. + +```bash +python yolov8_pose_e2e.py +``` + +After the script has run, you will see one PyTorch model and two ONNX models: +* `yolov8n-pose.pt`: The original YOLOv8 PyTorch model +* `yolov8n-pose.onnx`: The exported YOLOv8 ONNX model +* `yolov8n-pose.with_pre_post_processing.onnx`: The ONNX model with pre and post processing included in the model + + +### Run examples of pose estimation + +You can use the same script to run the model, supplying your own image to detect poses. + +```bash +python yolov8_pose_e2e.py --test_image person.jpg --run_model +``` + +And the output is drawn on the original image! + +![Person with pose drawn](../../../images/person-with-pose.png) + + +### Develop your mobile application + +You can use the Python inference code as a basis for developing your mobile application. + +## Additional resources + +[ONNX Runtime examples repository](https://github.com/microsoft/onnxruntime-inference-examples) +[ONNX Runtime extentions repository](https://github.com/microsoft/onnxruntime-extensions) + + + + + diff --git a/images/person-with-bicycle-and-bounding-boxes.png b/images/person-with-bicycle-and-bounding-boxes.png new file mode 100644 index 0000000000..e9613b062e Binary files /dev/null and b/images/person-with-bicycle-and-bounding-boxes.png differ diff --git a/images/person-with-pose.png b/images/person-with-pose.png new file mode 100644 index 0000000000..466779b362 Binary files /dev/null and b/images/person-with-pose.png differ diff --git a/images/pose-thumbnail.png b/images/pose-thumbnail.png new file mode 100644 index 0000000000..657a882670 Binary files /dev/null and b/images/pose-thumbnail.png differ diff --git a/images/wolves-with-bounding-boxes.png b/images/wolves-with-bounding-boxes.png new file mode 100644 index 0000000000..ab02f58b18 Binary files /dev/null and b/images/wolves-with-bounding-boxes.png differ