transformers/tests
Joao Gante b0f0c61899
Add SynthID (watermerking by Google DeepMind) (#34350)
* Add SynthIDTextWatermarkLogitsProcessor

* esolving comments.

* Resolving comments.

* esolving commits,

* Improving SynthIDWatermark tests.

* switch to PT version

* detector as pretrained model + style

* update training + style

* rebase

* Update logits_process.py

* Improving SynthIDWatermark tests.

* Shift detector training to wikitext negatives and stabilize with lower learning rate.

* Clean up.

* in for 7B

* cleanup

* upport python 3.8.

* README and final cleanup.

* HF Hub upload and initiaze.

* Update requirements for synthid_text.

* Adding SynthIDTextWatermarkDetector.

* Detector testing.

* Documentation changes.

* Copyrights fix.

* Fix detector api.

* ironing out errors

* ironing out errors

* training checks

* make fixup and make fix-copies

* docstrings and add to docs

* copyright

* BC

* test docstrings

* move import

* protect type hints

* top level imports

* watermarking example

* direct imports

* tpr fpr meaning

* process_kwargs

* SynthIDTextWatermarkingConfig docstring

* assert -> exception

* example updates

* no immutable dict (cant be serialized)

* pack fn

* einsum equivalent

* import order

* fix test on gpu

* add detector example

---------

Co-authored-by: Sumedh Ghaisas <sumedhg@google.com>
Co-authored-by: Marc Sun <marc@huggingface.co>
Co-authored-by: sumedhghaisas2 <138781311+sumedhghaisas2@users.noreply.github.com>
Co-authored-by: raushan <raushan@huggingface.co>
2024-10-23 21:18:52 +01:00
..
agents Decorator for easier tool building (#33439) 2024-09-18 11:07:51 +02:00
benchmark
bettertransformer
deepspeed Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
extended [tests] skip tests for xpu (#33553) 2024-09-19 19:28:04 +01:00
fixtures
fsdp
generation Add SynthID (watermerking by Google DeepMind) (#34350) 2024-10-23 21:18:52 +01:00
models skip test_pipeline_depth_estimation temporarily (#34316) 2024-10-23 17:27:51 +02:00
optimization
peft_integration [PEFT] Support low_cpu_mem_usage option for PEFT loading adapters (#33725) 2024-10-03 16:15:36 +02:00
pipelines Add option for running ffmpeg_microphone_live as a background process (#32838) 2024-10-22 15:56:41 +02:00
quantization Add GGUF for starcoder2 (#34094) 2024-10-14 10:22:49 +02:00
repo_utils Refactor CI: more explicit (#30674) 2024-08-30 18:17:25 +02:00
sagemaker Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
tokenization Fix for slow the bug tokenizer adding spaces to single id decodes (#32564) 2024-09-18 12:32:02 +02:00
trainer Enable users to use their own loss functions + deal with prefetching for grad accum (#34198) 2024-10-17 17:01:56 -04:00
utils Attn implementation for composite models (#32238) 2024-10-22 06:54:44 +02:00
__init__.py
test_backbone_common.py
test_configuration_common.py
test_feature_extraction_common.py
test_image_processing_common.py Add DetrImageProcessorFast (#34063) 2024-10-21 09:05:05 -04:00
test_image_transforms.py
test_modeling_common.py T5 compile compatibilty (#34089) 2024-10-22 08:23:53 +02:00
test_modeling_flax_common.py
test_modeling_tf_common.py [TF] Fix Tensorflow XLA Generation on limited seq_len models (#33903) 2024-10-05 16:20:50 +02:00
test_pipeline_mixin.py Sync video classification pipeline with huggingface_hub spec (#34288) 2024-10-22 13:33:49 +01:00
test_processing_common.py Uniformize model processors (#31368) 2024-10-02 10:41:08 +02:00
test_sequence_feature_extraction_common.py
test_tokenization_common.py Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00