Added accelerating phi 2 blog. (#19657)

Changes staged here: 
https://maanavd.github.io/onnxruntime/blogs

---------

Co-authored-by: MaanavD <maanavdalal@microsoft.com>
Co-authored-by: Sophie Schoenmeyer <107952697+sophies927@users.noreply.github.com>

package.json

@@ -47,6 +47,7 @@
 	"type": "module",
 	"dependencies": {
 		"animejs": "^3.2.1",
+		"mdsvex-relative-images": "^1.0.3",
 		"saos": "^1.3.1",
 		"svelte-icons": "^2.1.0",
 		"sveltejs-adapter-ipfs": "^0.4.12",

src/routes/blogs/+page.svelte

@@ -9,6 +9,7 @@
 	import HFImage from '../../images/blogs/hugging-face-blog-img.png';
 	import LlamaImage from '../../images/blogs/accelerating-llama-2/Figure1-LLaMA-2-7B-E2E-Throughput.png';
 	import SDXLTurboImage from '../../images/blogs/sdxl_blog_thumbnail.png';
+	import Phi2Image from '../../routes/blogs/accelerating-phi-2/Phi2_Int4_TokenGenerationTP.png';
 	import { createEventDispatcher } from 'svelte';
 	import WebTrainingImage from '../../images/blogs/webtraining_blog_thumbnail.png';
 	onMount(() => {
@@ -38,6 +39,15 @@
 		dispatch('switchTab', tab);
 	}
 	let featuredblog = [
+		{
+			title: 'Accelerating Phi-2, CodeLlama, Gemma and other Gen AI models with ONNX Runtime',
+			date: 'February 26th, 2024',
+			blurb:
+				'Improvements with ONNX Runtime for inferencing popular Gen AI models.',
+			link: 'blogs/accelerating-phi-2',
+			image: Phi2Image,
+			imgalt: 'Phi2 float16 token generation throughput comparison'
+		},
 		{
 			title: 'On-Device Training: Training a model in browser',
 			date: 'February 6th, 2024',
@@ -56,6 +66,8 @@
 			image: SDXLTurboImage,
 			imgalt: 'SD Turbo and SDXL Turbo models with ONNX Runtime and Olive'
 		},
+	];
+	let blogs = [
 		{
 			title: 'Accelerating LLaMA-2 Inference with ONNX Runtime',
 			date: 'November 14th, 2023',
@@ -63,9 +75,7 @@
 			link: 'blogs/accelerating-llama-2',
 			image: LlamaImage,
 			imgalt: 'LLaMA-2 e2e throughput'
-		}
-	];
-	let blogs = [
+		},
 		{
 			title: 'Run PyTorch models on the edge',
 			date: 'October 12th, 2023',

src/routes/blogs/accelerating-phi-2/+page.svx

@@ -0,0 +1,231 @@
+---
+title: Accelerating Phi-2, CodeLlama, Gemma and other Gen AI models with ONNX Runtime
+date: '26th February, 2024'
+description: 'Improvements with ONNX Runtime for inferencing popular Gen AI models'
+keywords: 'GenAI , LLM, SLM, ONNXRuntime, ORT, Phi, Mistral, Mixtral, LLama, Gemma, Stable Diffusion, Orca'
+authors:
+  [
+    'Parinita Rahi',
+    'Sunghoon Choi',
+    'Yufeng Li',
+    'Kshama Pawar',
+    'Ashwini Khade',
+    'Ye Wang'
+   ]
+authorsLink:
+  [
+    'https://www.linkedin.com/in/sunghoon/',
+    'https://www.linkedin.com/in/parinitaparinita/',
+    'https://www.linkedin.com/in/yufeng-li-940412159/',
+    'https://www.linkedin.com/in/kshama-pawar/',
+    'https://www.linkedin.com/in/ashwini-khade-0266541a/',
+    'http://linkedin.com/in/ye-wang-0ab35812a'
+  ]
+image: 'accelerating-phi-2/Phi2_Int4_TokenGenerationTP.png'
+url: 'https://onnxruntime.ai/blogs/accelerating-phi-2'
+---
+
+
+In a fast-moving landscape where speed and efficiency are paramount, [ONNX Runtime](https://onnxruntime.ai/) (ORT) allows users to easily integrate the power of generative AI models into their apps and services with improved optimizations that yield faster inferencing speeds and effectively lowers costs. These include state-of-the-art fusion and kernel optimizations to help improve model performance. The recent [ONNX Runtime 1.17 release](https://github.com/microsoft/onnxruntime/releases/tag/v1.17.0) improves inference performance of several Gen AI models including Phi-2, Mistral, CodeLlama, Orca-2 and more. ONNX Runtime is a complete solution for small language models (SLMs) from training to inference, showing significant speedups compared to other frameworks. With support for float32, float16, and int4, ONNX Runtime's inference enhancements provide maximum flexibility and performance.
+
+In this blog, we will cover significant optimization speed up for both training and inference for the latest GenAI models like Phi-2, Mistral, CodeLlama, SD-Turbo, SDXL-Turbo, Llama2, and Orca-2. For these model architectures, ONNX Runtime significantly improves performance across a spectrum of batch sizes and prompt lengths when compared against other frameworks like PyTorch, and Llama.cpp. These optimizations using ONNX Runtime are now also available using [Olive](https://github.com/microsoft/Olive/tree/main/examples/).
+# Quick Links
+- [Phi-2](#phi-2)
+- [Mistral](#mistral)
+- [CodeLlama](#codellama)
+- [SD-Turbo and SDXL-Turbo](#sd-turbo-and-sdxl-turbo)
+- [Llama-2](#llama-2)
+- [Orca-2](#orca-2)
+- [Gemma](#gemma)
+
+<div class="anchor" id="phi-2"/>
+
+# Phi-2
+
+[Phi-2](https://huggingface.co/microsoft/phi-2) is a 2.7 billion parameter transformer model developed by Microsoft. It is an SLM that exhibits excellent reasoning and language comprehension skills. With its small size, Phi-2 is a great platform for researchers, who can explore various aspects such as mechanistic interpretability, safety improvements, and fine-tuning experiments on different tasks.
+
+ONNX Runtime 1.17 introduces kernels changes that support the Phi-2 model, including optimizations for Attention, Multi-Head Attention, Grouped-Query Attention, and RotaryEmbedding for Phi-2. Specifically, support has been added for the following:
+
+- causal mask in the Multi-Head Attention CPU kernel
+- rotary_embedding_dim in the Attention and Rotary Embedding kernels
+- bfloat16 in the Grouped-Query Attention kernel
+
+TorchDynamo-based ONNX export for Phi-2 is supported, and the optimization script is built on top.
+
+For Phi-2 inference, ORT with float16 and int4 quantization performs better than ORT with float32, PyTorch, and Llama.cpp for all prompt lengths.
+
+## Inferencing
+
+### ORT gains with float16
+
+Optimized CUDA performance for prompt throughput (i.e., the rate at which the model processes and generates responses based on input prompts) is **up to 7.39x** faster than PyTorch Compile. We also observe ONNX Runtime is significantly faster for larger batch size and prompt lengths compared to Llama.cpp. For example, it is **up to 13.08x faster** for batch size =16, prompt length =2048.
+
+
+
+Token generation throughput is the average throughput of the first 256 tokens generated. ONNX Runtime with float16 is **on average 6.6x faster** than torch.compile and **as high as 18.55x** faster. It also performs **up to 1.64x** faster than Llama.cpp.
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto w50" src="./Phi2_Float16_PromptThroughput.png" alt="Phi2 float16 prompt throughput comparison">
+
+<img class="m-auto w50" src="./Phi2_Float16_TokenGenerationThroughput.png" alt="Phi2 float16 token generation throughput comparison">
+</div>
+
+### ORT gains with int4
+
+ORT provides support for int4 quantization. ORT with int4 quantization can provide **up to 20.48x** improved performance compared to PyTorch. It is 3.9x better than Llama.cpp on average and **up to 13.42x** faster for large sequence lengths. ONNX Runtime with int4 quantization typically performs best with batch size 1 due to a special kernel for GemV.
+
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto" src="./Phi2_Int4_PromptTP.png" alt="Phi2 int4 prompt throughput comparison">
+
+<img class="m-auto" src="./Phi2_Int4_TokenGenerationTP.png" alt="Phi2 int4 token generation throughput comparison">
+</div>
+
+<i>Note: torch.compile does not work well with 4-bit quantization. Also, Llama.cpp does not use FlashAttention, its attention implementation is slower for large sequence lengths.</i> 
+<br/>
+<br/>
+
+- _Phi-2 benchmarks is done on 1 A100 GPU (SKU: Standard_ND96amsr_A100_v4).Packages: torch: 2.3.0. dev20231221+cu121; pytorch-triton: 2.2.0+e28a256d71;ort-nightly-gpu: 1.17.0.dev20240118001;deepspeed: 0.12_
+- _Batch is a set of input sentences of varying lengths; prompt length refers to the size or length of the input text._
+
+Here is an example of [Phi-2 optimizations with Olive](https://github.com/microsoft/Olive/tree/main/examples/phi2), which utilizes the ONNX Runtime optimizations highlighted in this blog using easy-to-use hardware-aware model optimization tool, [Olive](https://microsoft.github.io/Olive/).
+
+## Training
+
+In addition to inference, ONNX Runtime also provides training speedup for Phi-2 and other LLMs. ORT training is part of the PyTorch Ecosystem and is available via the torch-ort python package as part of the [Azure Container for PyTorch (ACPT)](https://learn.microsoft.com/en-us/azure/machine-learning/resource-azure-container-for-pytorch?view=azureml-api-2). It provides flexible and extensible hardware support, where the same model and APIs works with both NVIDIA and AMD GPUs. ORT accelerates training through optimized kernels and memory optimizations which show significant gains in reducing end-to-end training time for large model training. This involves changing a few lines of code in the model to wrap it with the ORTModule API. It is also composable with popular acceleration libraries like DeepSpeed and Megatron for faster and more efficient training.
+
+[Open AI's Triton](https://openai.com/research/triton) is a domain specific language and compiler to write highly efficient custom deep learning primitives. ORT supports Open AI Triton integration (ORT+Triton), where all element wise operators are converted to Triton ops and ORT creates custom fused kernels in Triton.
+
+ORT also performs sparsity optimization to assess input data sparsity and perform graph optimizations leveraging this sparsity. This reduces the compute FLOP requirements and increases performance.
+
+Low-Rank Adapters (LoRA) based fine-tuning makes training more efficient by training only a small number of additional parameters (the adapters) while freezing the original model's weights. These adapters adapt the model to specific tasks. Quantization and LoRA (QLoRA) combines quantization with LoRA where the weights are represented using fewer bits, while preserving the performance and quality of the model. ONNX Runtime training composes with both LoRA and QLoRA to provide gains in memory efficiency and training time acceleration for LLMs. LoRA and QLoRA techniques enable very large models like LLMs to fit in the GPU memory to efficiently complete training.
+
+The Phi-2 model trained using ORT shows performance gains against PyTorch Eager mode and torch.compile. Phi-2 was trained using a mixture of synthetic and web datasets. We measured gains against ORT and the ORT+Triton mode, and gains increased with larger batch sizes. The model was trained using DeepSpeed Stage-2 for 5 epochs, with increasing batch sizes on the wikitext dataset. The gains are summarized in the charts below for V100 and A100.
+
+The training benchmarks were run on 8 V100 and measured throughput in iterations/second (higher is better):
+
+<img class="m-auto w50" src="./Phi2_trainingTP.png" alt="Phi2 training throughput comparison">
+
+The training benchmarks below were run on 2 A100 and measured throughput in iterations/second (higher is better):
+<img class="m-auto w50" src="./Phi2_training_2a100.png" alt="Phi2 training benchmarks on 2 A100">
+
+<i>Note: PyTorch Stable 2.2.0 and ONNXRuntime Training: Stable 1.17.0 versions were used.</i>
+
+<div class="anchor" id="mistral"/>
+
+# Mistral
+
+## Inferencing
+
+[Mistral7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) is a pretrained generative text LLM with 7 billion parameters. ONNX Runtime improves inference performance significantly for Mistral with both float16 and int4 models. With float16, ONNX Runtime is **as high as 9.46x** compared to Llama.cpp. Token generation throughput significantly improves with int4 quantization for batch size 1 and is **up to 18.25x** faster than PyTorch Eager.
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto" src="./Mistral_float16_PromptTP.png" alt="Mistral float16 prompt throughput comparison">
+
+<img class="m-auto" src="./Mistral_float16_TokenGenerationTP.png" alt="Mistral float16 token generation throughput comparison">
+
+<img class="m-auto" src="./Mistral_int4_promptTP.png" alt="Mistral int4 prompt throughput comparison">
+
+<img class="m-auto" src="./Mistral_int4_TokenGenTP.png" alt="Mistral int4 token generation throughput comparison">
+</div>
+
+
+You can now access the optimized Mistral model on Huggingface [here.](https://huggingface.co/microsoft/Mistral-7B-v0.1-onnx)
+
+## Training
+
+Similar to Phi-2, Mistral also benefits from training acceleration using ORT. We trained Mistral-7B using the following configuration to see gains with ORT, including when composed with LoRA and QLoRA. The model was trained using DeepSpeed Stage-2 for 5 epochs, with batch size 1 on the wikitext dataset.
+<img class="m-auto w50" src="./Mistral_Training.png" alt="Mistral training benchmarks">
+
+<div class="anchor" id="codellama"/>
+
+# CodeLlama
+
+[Codellama-70B](https://huggingface.co/codellama/CodeLlama-70b-hf) is a programming-focused model developed on the Llama-2 platform. This model can produce code and generate discussions around code in natural language. Since CodeLlama-70B is a finetuned Llama model, existing optimizations can be applied directly. We compared a 4bit quantized ONNX model with PyTorch Eager and Llama.cpp. For prompt throughput, ONNX Runtime is **at least 1.4x faster** than PyTorch Eager for all batch sizes. ONNX Runtime produces tokens at an average speed that is **3.4x** higher than PyTorch Eager for any batch size and **1.5x** higher than Llama.cpp for batch size 1.
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto" src="./CodeLLama_int4_PromptTP.png" alt="CodeLLama int4 prompt throughput comparison">
+
+<img class="m-auto" src="./CodeLlama_int4_TokenGenTP.png" alt="CodeLLama int4 token generation throughput comparison">
+</div>
+
+
+<div class="anchor" id="sd-turbo-and-sdxl-turbo"/>
+
+# SD-Turbo and SDXL-Turbo
+
+ONNX Runtime provides inference performance benefits when used with [SD Turbo](https://huggingface.co/stabilityai/sd-turbo) and [SDXL Turbo](https://huggingface.co/stabilityai/sdxl-turbo), and it also makes the models accessible in languages other than Python, like C# and Java. ONNX Runtime achieved a higher throughput than PyTorch for all (batch size, number of steps) combinations evaluated, with throughput improvements **up to 229%** for the SDXL Turbo model and **120%** for the SD Turbo model. ONNX Runtime CUDA is especially good at handling dynamic shape, but it also shows a significant advantage over PyTorch for static shape.
+
+<img class="m-auto" src="./SDXL.jpg" alt="Stable Diffusion XL Turbo Speedup">
+
+To read more about accelerating SD-Turbo and SDXL-Turbo inference with ONNX Runtime, check out our recent [blog](https://huggingface.co/blog/sdxl_ort_inference) with Hugging Face.
+
+<div class="anchor" id="llama-2"/>
+
+# Llama-2
+
+We published a separate blog for Llama-2 improvements with ORT for Inference [here](https://onnxruntime.ai/blogs/accelerating-llama-2). Additionally, Llama-2-7B and Llama-2-13B show good gains with ORT for training, especially when combined with LoRA and QLoRA. [These](https://github.com/huggingface/optimum/blob/main/examples/onnxruntime/training/text-classification/README.md#onnx-runtime-training) scripts can be used as an example to finetune Llama-2 with ORT using Optimum. The numbers below are for Llama-2 models training with ORT using DeepSpeed Stage-2 for 5 epochs, with batch size 1 on the wikitext dataset.
+
+<img class="m-auto w50" src="./Llama2_Training.png" alt="Llama2 training benchmarks">
+
+<div class="anchor" id="orca-2"/>
+
+# Orca-2
+
+## Inference
+
+[Orca-2](https://www.microsoft.com/en-us/research/blog/orca-2-teaching-small-language-models-how-to-reason/) is a research-only system that gives a one-time answer in tasks such as reasoning with user-provided data, understanding texts, solving math problems, and summarizing texts. Orca-2 has two versions (7 billion and 13 billion parameters); they are both made by fine-tuning the respective Llama-2 base models on customized, high-quality artificial data. ONNX Runtime helps optimize Orca-2 inferencing for using graph fusions and kernel optimizations like those for Llama-2.
+
+### ORT gains with int4
+
+Orca-2-7B int4 quantization performance comparison indicated **up to 26X** increase in performance in prompt throughput, and **up to 16.5X** improvement in token generation throughput over PyTorch. It also shows over **4.75X** improvement in prompt throughput, and **3.64X** improvement in token generation throughput compared to Llama.cpp.
+
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto" src="./Orca2_7b_int4_promptTP.png" alt="Orca2 7b int4 prompt throughput comparison">
+
+<img class="m-auto" src="./Orca2_7b_int4_TokenGenTP.png" alt="Orca2 7b int4 token generation throughput comparison">
+
+<img class="m-auto" src="./Orca2_13b_int4_promptTP.png" alt="Orca2 13b int4 prompt throughput comparison">
+
+<img class="m-auto" src="./Orca2_13b_int4_tokengenTP.png" alt="Orca2 13b int4 token generation throughput comparison">
+</div>
+
+Orca-2 7b with ONNX runtime float16 performance comparison also shows significant gains in prompt and token generation throughput.
+
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto" src="./Orca2_7b_float16_PromptTP.png" alt="Orca2 7b float16 prompt throughput comparison">
+
+<img class="m-auto" src="./Orca2_7b_float16_TokengenTP.png" alt="Orca2 7b float16 token generation throughput comparison">
+
+<img class="m-auto" src="./Orca2_13b_float16_PromptTP.png" alt="Orca2 13b float16 prompt throughput comparison">
+
+<img class="m-auto" src="./Orca2_13b_TokengenTP.png" alt="Orca2 13b float16 token generation throughput comparison">
+</div>
+
+
+_Orca-2 benchmarking done on1 A100 GPU, SKU: Standard_ND96amsr_A100_v4 , Packages torch 2.2.0, triton 2.2.0, onnxruntime-gpu 1.17.0, deepspeed 0.13.2, llama.cpp - commit 594fca3fefe27b8e95cfb1656eb0e160ad15a793, transformers 4.37.2_
+
+## Training
+
+Orca-2-7B also benefits from training acceleration using ORT. We trained the Orca-2-7B model for a sequence length of 512 with LoRA and with the sparsity optimization enabled and saw good gains in performance. The numbers below are for Orca-2-7B models trained with ORT using DeepSpeed Stage-2 for 5 epochs, with batch size 1 on the wikitext dataset.
+
+<img class="m-auto w50" src="./Orca2_Training.png" alt="Orca2 training benchmarks">
+<i>Uses ACPT image: nightly-ubuntu2004-cu118-py38-torch230dev:20240131</i>
+
+<div class="anchor" id="gemma"/>
+
+# Gemma
+
+[Gemma](https://ai.google.dev/gemma/docs) is a family of lightweight, open models built from the research and technology that Google used to create Gemini models. It is available in two sizes: 2B and 7B. Each size is released with pre-trained and instruction-tuned variants. ONNX Runtime can be used to optimize and efficiently run any open-source model. We benchmarked against the [Gemma-2B](https://huggingface.co/google/gemma-2b) model, and ONNX Runtime with float16 is **up to 7.47x** faster than PyTorch Compile and **up to 3.47x** faster than Llama.cpp. ORT with int4 quantization is **up to 19.81x** faster than PyTorch Eager and **2.62x** faster than Llama.cpp.
+<div class="grid grid-cols-1 lg:grid-cols-2">
+<img class="m-auto" src="./Gemma2_int4_tokengenTP.png" alt="Gemma2b int4 token generation throughput comparison">
+
+<img class="m-auto" src="./Gemma2b_TokenGenTP.png" alt="Gemma2b token generation throughput comparison">
+</div>
+
+<div class="anchor" id="conclusion"/>
+
+# Conclusion
+
+In conclusion, ONNX Runtime (ORT) provides significant performance improvements for several models, including Phi-2, Mistral, CodeLlama, SDXL-Turbo, Llama-2, Orca-2, and Gemma. ORT offers state-of-the-art fusion and kernel optimizations, including support for float16 and int4 quantization, resulting in faster inferencing speeds and lower costs. ORT outperforms other frameworks like PyTorch and Llama.cpp in terms of prompt and token generation throughput. ORT also shows significant benefits for training LLMs, with increasing gains for larger batch sizes, and composes well with state-of-the-art techniques to enable efficient large model training.
+<style>
+	.anchor {
+		scroll-margin-top: 40px;
+	}
+</style>

src/routes/blogs/github-markdown-light.css

@@ -1,3 +1,10 @@
+ul {
+  list-style: circle !important;
+}
+
+.w50{
+  width: 35em;
+}
 /*light*/
 
 .markdown-body {
@@ -972,7 +979,6 @@
   width: 1em;
   height: 1em;
 }
-
 .markdown-body .task-list-item {
   list-style-type: none;
 }
@@ -1080,3 +1086,4 @@
 .markdown-body .markdown-alert.markdown-alert-caution .markdown-alert-title {
   color: #d1242f;
 }
+

src/routes/blogs/ort-1-17-release/+page.svx

@@ -0,0 +1,72 @@
+---
+title: Unlock new functionality with ONNX Runtime 1.17
+date: '2024-02-26'
+description: 'From Phi-2 model optimizations to CUDA 12 support, read this post to learn more about some of the exciting new functionality introduced in the ONNX Runtime 1.17 release.'
+keywords: 'ORT, ONNX Runtime, ONNX, machine learning, deep learning, model optimization, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8, pose detection, CUDA 12, GPU, Windows, browser, WebGPU, DirectML, NPU, Phi-2, Mistral, CodeLlama, SDXL-Turbo, on-device training, DirectML, NPU, WebGPU, Yolov8'
+authors:
+  [
+    'Sophie Schoenmeyer',
+    'Parinita Rahi',
+    'Kshama Pawar',
+    'Emma Ning',
+    'Natalie Kershaw',
+    'Jian Chen'
+  ]
+authorsLink:
+  [
+    'https://www.linkedin.com/in/sophieschoenmeyer/',
+    https://www.linkedin.com/in/parinitaparinita/,
+    'https://www.linkedin.com/in/kshama-pawar',
+    '',
+    'https://www.linkedin.com/in/natkershaw/',
+    ''
+  ]
+image: ''
+url: 'https://onnxruntime.ai/blogs/ort-1_17-release-blog'
+---
+
+# ONNX Runtime 1.17 Release Blog
+
+Recently, we released ONNX Runtime 1.17, which includes a host of new features to further streamline the process of inferencing and training machine learning models across various platforms faster than ever. The release includes improvements to some of our existing features, along with exciting new features like Phi-2 optimizations, training a model in-browser with on-device training, ONNX Runtime Web with WebGPU, and more.
+
+For a complete list of new features, along with various assets, check out the release on GitHub: [ONNX Runtime v1.17.0](https://github.com/microsoft/onnxruntime/releases/tag/v1.17.0).
+
+## Models Optimization
+
+The ONNX Runtime (ORT) 1.17 release provides improved inference performance for several models, such as [Phi-2](https://huggingface.co/microsoft/phi-2), [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1), [CodeLlama](https://huggingface.co/codellama), and [SDXL-Turbo](https://huggingface.co/stabilityai/sdxl-turbo), by using state-of-the-art fusion and kernel optimizations and including support for Float16 and Int4 quantization. The specific ORT optimizations added in this release are Attention, Multi-Head Attention, Grouped-Query Attention, and Rotary Embedding ORT kernel changes. ORT outperforms other frameworks like PyTorch, DeepSpeed, and Llama.cpp in terms of prompt and token generation throughput, with speedups as high as **18.55x** for **Phi-2** with Float16, **20.48x** for Phi-2 with Int4, and **4.1x** for Mistral with Float16 (see linked blogs below for additional details).
+
+ONNX Runtime also shows significant benefits for training LLMs, and these gains typically increase with batch size. For example, ORT is 1.2x faster than PyTorch Eager mode and 1.5x faster than torch.compile for Phi-2 with LoRA on 2 A100 GPUs. ORT also shows benefits for other LLMs, like Llama, Mistral, and Orca-2, with combinations of LoRA or QLoRA.
+
+To read more about accelerating Phi-2, Mistral, CodeLlama, SDXL-Turbo, and more with ONNX Runtime 1.17, check out this recent post on the ONNX Runtime blog: **_Phi-2 newsletter link_**.
+
+## On-Device Training
+
+On-device training allows you to improve the user experience for developer applications using device data. It supports scenarios like federated learning, which trains a global model using data on the device. With the 1.17 release, ORT will now enable training machine learning models in the browser using on-device training.
+
+To learn more about training a model in browser with on-device training, check out this recent post on the Microsoft Open Source Blog: [On-Device Training: Training a model in browser](https://cloudblogs.microsoft.com/opensource/2024/02/06/on-device-training-training-a-model-in-browser/).
+
+## DirectML NPU Support
+
+With the release of [DirectML 1.13.1](https://github.com/microsoft/DirectML/blob/master/Releases.md) and ONNX Runtime 1.17, developer preview support for neural processing unit (NPU) acceleration is now available in DirectML, the machine learning platform API for Windows. This developer preview enables support for a subset of models on new Windows 11 devices with Intel® Core™ Ultra processors with Intel® AI boost.
+
+To learn more about NPU support in DirectML, check out this recent post on the Windows Developer Blog: [Introducing Neural Processor Unit (NPU) support in DirectML (developer preview)](https://blogs.windows.com/windowsdeveloper/2024/02/01/introducing-neural-processor-unit-npu-support-in-directml-developer-preview/).
+
+## ONNX Runtime Web with WebGPU
+
+WebGPU enables web developers to harness GPU hardware for high-performance computations. The ONNX Runtime 1.17 release introduces the official launch of the WebGPU execution provider in ONNX Runtime Web, allowing sophisticated models to run entirely and efficiently within the browser (see the [list of WebGPU browser compatibility](https://github.com/gpuweb/gpuweb/wiki/Implementation-Status)). This advancement, demonstrated by the effective execution of models such as SD-Turbo, unlocks new possibilities in scenarios where CPU-based in-browser machine learning faces challenges in meeting performance standards.
+
+To learn more about how ONNX Runtime Web further accelerates in-browser machine learning with WebGPU, stay tuned for our upcoming blog post.
+
+## Yolov8 Pose Detection Scenario
+
+This release adds support for running the Yolov8 model for pose detection. Pose detection involves processing the objects detected in an image and identifying the position and orientation of people in the image. The core Yolov8 model returns a set of key points, representing specific parts of the detected person's body, such as joints and other distinctive features. Including the pre- and post-processing in the ONNX model allows developers to supply an input image directly, either in common image formats or raw RGB values, and output the image with bounding boxes and key points.
+
+**_TODO: Add output image_**
+
+**_TODO: Add link_**
+
+## CUDA 12 packages - Jian
+
+As part of the 1.17 release, ONNX Runtime now ensures compatibility across multiple versions of Nvidia's CUDA execution provider by introducing CUDA 12 packages for Python and NuGet. With this more flexible methodology, users will now have access to both CUDA 11 and CUDA 12, allowing for more seamless integration of cutting-edge hardware acceleration technologies.
+
+To install CUDA 12 for ONNX Runtime GPU, refer to the instructions in the ONNX Runtime docs: [Install ONNX Runtime GPU (CUDA 12.X)](https://onnxruntime.ai/docs/install/#install-onnx-runtime-gpu-cuda-12x).

src/routes/blogs/post.svelte

@@ -1,47 +1,51 @@
 <script>
 	import Header from '../components/header.svelte';
 	import Footer from '../components/footer.svelte';
-	import './github-markdown-light.css'
+	import './github-markdown-light.css';
+	import { onMount } from 'svelte';
 	/**
 	 * @type {any}
 	 */
-	 export let title;
+	export let title;
 	/**
 	 * @type {any}
 	 */
-	 export let description;
+	export let description;
 	/**
 	 * @type {any}
 	 */
-	 export let keywords;
+	export let keywords;
 	/**
 	 * @type {any[]}
 	 */
-	 export let authors;
+	export let authors;
 	/**
 	 * @type {string[]}
 	 */
-	 export let authorsLink;
+	export let authorsLink;
 	/**
 	 * @type {string}
 	 */
-	 export let date;
+	export let date;
 	/**
 	 * @type {undefined}
 	 */
-	 export let updated;
+	export let updated;
 	/**
 	 * @type {any}
 	 */
-	 export let image;
+	export let image;
 	/**
 	 * @type {any}
 	 */
-	 export let url;
+	export let url;
+	/**
+	 * @type {any}
+	 */
+	export let robots;
 	/**
 	 * @type {any}
 	 */
-	 export let robots;
 </script>
 
 <svelte:head>
@@ -50,16 +54,16 @@
 	<meta name="keywords" content={keywords} />
 	<meta name="author" content={authors.join(', ')} />
 	<meta name="date" content={date} />
-	<meta name="image" content={image} />
 	<meta name="og:title" content={title} />
 	<meta name="og:description" content={description} />
 	<meta name="og:type" content="article" />
 	<meta name="og:url" content={url} />
-	<meta name="og:image" content={image} />
 	<meta name="twitter:title" content={title} />
 	<meta name="twitter:description" content={description} />
-	<meta name="twitter:card" content={image} />
-	<meta name="twitter:image" content={image} />
+	<!-- <meta name="image" content={image} />
+	<meta name="og:image" content={image}  />
+	<meta name="twitter:card" content={image}  />
+	<meta name="twitter:image" content={image}  /> -->
 
 	<meta name="robots" content={robots} />
 </svelte:head>
@@ -68,12 +72,12 @@
 	<article class="">
 		<h1 class="text-5xl pb-2">{title}</h1>
 		<p class="text-neutral">
-            By: 
+			By:
 			{#each authors as author, i}
-            <a href={authorsLink[i]} class="text-blue-500"
-            >{author}</a
-        >{i + 1 === authors.length ? '' : ', '}
-            {/each}
+				<a href={authorsLink[i]} class="text-blue-500">{author}</a>{i + 1 === authors.length
+					? ''
+					: ', '}
+			{/each}
 		</p>
 		<p class="text-neutral">
 			{date.toLocaleUpperCase()}
@@ -81,9 +85,10 @@
 				<span class="italic text-stone-500">(Updated {updated})</span>
 			{/if}
 		</p>
-        <div class="py-4 markdown-body">
-            <slot />
-        </div>
+		<div class="py-4 markdown-body">
+			<slot />
+		</div>
 	</article>
 </div>
 <Footer pathvar="" />
+

svelte.config.js

@@ -1,6 +1,8 @@
 import adapter from '@sveltejs/adapter-static';
 import { vitePreprocess } from '@sveltejs/kit/vite';
 import { mdsvex } from 'mdsvex';
+import relativeImages from "mdsvex-relative-images";
+
 
 /** @type {import('@sveltejs/kit').Config} */
 const config = {
@@ -13,7 +15,8 @@ const config = {
 			extensions: ['.md', '.svx'],
 			layout: {
 				blogs: 'src/routes/blogs/post.svelte'
-			}
+			},
+			remarkPlugins: [relativeImages],
 		})
 	],