saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-14 20:58:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Fix hashing for deduplication (#17048)

Browse source
This commit is contained in:
Thomas Wang 2022-05-04 08:40:24 +02:00 committed by GitHub
parent 39f8eafc1b
commit db034660fb
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
examples/research_projects/codeparrot/scripts/preprocessing.py
View file

@ -1,4 +1,5 @@
import gzip
import hashlib
import multiprocessing
import os
import shutil
@ -13,7 +14,7 @@ from transformers import HfArgumentParser
def get_hash(example):
"""Get hash of content field."""
return {"hash": hash(example["content"])}
return {"hash": hashlib.md5(example["content"].strip().encode("utf-8")).hexdigest()}
def line_stats(example):