saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-14 20:58:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix(FA): QKV not being casted to target_dtype for FA with dpo lora (#35834)

Browse source 
fix(FA): QKV not being casted to target_dtype due to dtype check
This commit is contained in:
NanoCode012 2025-01-28 23:06:56 +07:00 committed by GitHub
parent ece8c42488
commit 478c4f2d0d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
src/transformers/modeling_flash_attention_utils.py
View file

@ -209,7 +209,7 @@ def fa_peft_integration_check(
if target_dtype is None:
return query, key, value
input_dtype = value.dtype
input_dtype = query.dtype
if input_dtype == torch.float32:
logger.warning_once(
f"The input hidden states seems to be silently casted in float32, this might be related to"