saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-30 20:18:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add cuda visible devices for Mistral benchmark (#18764)

Browse source 
### Description
<!-- Describe your changes. -->
Add cuda visible devices for Mistral benchmark as it is not working for
Torch compile and throwing an error.


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Error: 
File
"/opt/conda/envs/ptca/lib/python3.8/site-packages/torch/_inductor/triton_heuristics.py",
line 556, in run
    return launcher(
  File "<string>", line 8, in launcher
RuntimeError: Triton Error [CUDA]: invalid device context
This commit is contained in:
Abhishek Jindal 2023-12-08 23:12:48 -08:00 committed by GitHub
parent c7799d7058
commit 2f93d97fd0
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
onnxruntime/python/tools/transformers/models/llama/README.md
View file

@ -412,7 +412,7 @@ python -m models.llama.convert_to_onnx -i /path/to/model/directory -o /path/to/o
The benchmarking scripts in the LLaMA directory support Mistral benchmarking. To benchmark the ORT version, you can run:
```
python -m models.llama.benchmark \
CUDA_VISIBLE_DEVICES=0 python -m models.llama.benchmark \
-bt ort-convert-to-onnx \
-p fp16 \
-m mistralai/Mistral-7B-v0.1 \
@ -422,7 +422,7 @@ python -m models.llama.benchmark \
To benchmark the Hugging Face implementation without `torch.compile`:
```
python -m models.llama.benchmark \
CUDA_VISIBLE_DEVICES=0 python -m models.llama.benchmark \
-bt hf-pt-eager \
-p fp16 \
-m mistralai/Mistral-7B-v0.1
@ -431,7 +431,7 @@ python -m models.llama.benchmark \
And to benchmark the Hugging Face implementation with `torch.compile`:
```
python -m models.llama.benchmark \
CUDA_VISIBLE_DEVICES=0 python -m models.llama.benchmark \
-bt hf-pt-compile \
-p fp16 \
-m mistralai/Mistral-7B-v0.1