#

semantic-deduplication

Here is 1 public repository matching this topic...

NVIDIA / NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs

python data data-processing data-preparation deduplication data-quality data-curation data-prep fine-tuning fast-data-processing data-processing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes semantic-deduplication llm-data-quality

Updated Nov 15, 2024
Jupyter Notebook

Improve this page

Add a description, image, and links to the semantic-deduplication topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the semantic-deduplication topic, visit your repo's landing page and select "manage topics."