saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-30 20:18:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/cgmanifests
History
Ye Wang 2ee822d483
Extend memory efficient attention coverage in Attention/MHA cuda op (#15064) 
### Description
<!-- Describe your changes. -->

1. upgrade cutlass to 3.0 that containing attn_bias support.
2. extend Attention/MHA to use memory efficient attention when
rel_pos_bias with [1, num_head, s, s*] and 1d mask with [2 * batch_size
+ 1] are present.

new mask format introduction:
MASK_1D_KEY_SEQ_LEN_START,  
[3 * batch_size + 2] with [key_len[0], ..., key_len[batch_size - 1],
query_start[0], ..., query_start[batch_size - 1], query_end[batch_size -
1], key_start[0], ..., key_start[batch_size - 1], key_end[batch_size -
1]]

e.g
2D mask with [[1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 0]] converts to this
1D mask is [3, 5, 0, 6, 12, 0, 6, 12]


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

It potentially benefits tnlrv6 and t5(encoder)

---------

Co-authored-by: Ubuntu <wy@v100-2.0cdb2e52twzevn1i4fi45bylyg.jx.internal.cloudapp.net>
Co-authored-by: Kunal Vaishnavi <kvaishnavi@microsoft.com>
Co-authored-by: Kunal Vaishnavi <kvaishnavi@microsoft.com@orttrainingdev7.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
2023-03-23 11:05:17 -07:00
..
generated Extend memory efficient attention coverage in Attention/MHA cuda op (#15064) 2023-03-23 11:05:17 -07:00
cgmanifest.json [CPU] Resize of Opset 18 (#13890) 2023-01-14 08:57:23 +10:00
generate_cgmanifest.py Use onnxruntime_fetchcontent_makeavailable cmake function for TRT (#13918) 2022-12-12 11:27:46 -08:00
print_submodule_info.py Format all python files under onnxruntime with black and isort (#11324) 2022-04-26 09:35:16 -07:00
README.md Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00

README.md

CGManifest Files

This directory contains CGManifest (cgmanifest.json) files. See here for details.

cgmanifests/generated/cgmanifest.json

This file contains generated CGManifest entries.

It covers these dependencies:

  • git submodules
  • dependencies from the Dockerfile tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_cuda11
  • the entries in ../cmake/deps.txt

If any of these dependencies change, this file should be updated. When updating, please regenerate instead of editing manually.

How to Generate

  1. Change to the repository root directory.
  2. Ensure the git submodules are checked out and up to date. For example, with: 
    $ git submodule update --init --recursive
  3. Run the generator script: 
    $ python cgmanifests/generate_cgmanifest.py --username <xxx> --token <your_access_token>

Please supply your github username and access token to the script. If you don't have a token, you can generate one at https://github.com/settings/tokens. This is for authenticating with Github REST API so that you would not hit the rate limit.

cgmanifests/cgmanifest.json

This file contains non-generated CGManifest entries. Please edit directly as needed.