mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-30 20:18:08 +00:00
cbb29d80ff
### Description These changes add rotary embedding and packed qkv input to gqa. As of now, the changes are only supported with Flash-Attention (SM >= 80) but should soon be supported with Memory Efficient Attention as well. ### Motivation and Context With the fusion of rotary embedding into this Attention op, we hope to observe some perf gain. The packed QKV should also provide some perf gain in the context of certain models, like Llama2, that would benefit from running ops on the fused QKV matrix, rather than the separate Q, K, and V. --------- Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>
10 lines
255 B
Text
10 lines
255 B
Text
# packages used by transformers python unittest (only enabled in Linux CPU CI Pipeline)
|
|
packaging
|
|
protobuf==3.20.2
|
|
numpy==1.24.0 ; python_version < '3.12'
|
|
numpy==1.26.0 ; python_version >= '3.12'
|
|
torch
|
|
coloredlogs==15.0
|
|
transformers==4.36.0
|
|
psutil
|
|
einops
|