Datasets

Training datasets with quantum-safe provenance

jobs-git/Zyda-2 HF Unverified

Zyda-2 Zyda-2 is a 5 trillion token language modeling dataset created by collecting open and high quality datasets and combining them and cross-deduplication and model-based quality filtering. Zyda-2 comprises diverse sources of web data, highly educational content, math, code, and scientific papers. To construct Zyda-2, we took the best open-source datasets available: Zyda, FineWeb, DCLM, and Dolma. Models trained on Zyda-2 significantly outperform identical models trained on the… See the full description on the dataset page: https://huggingface.co/datasets/jobs-git/Zyda-2.

Task_categories:text-GenerationLanguage:enSize_categories:n>1T
leggedrobotics/grand_tour_dataset HF Unverified

The GrandTour Dataset A project brought to you by RSL - ETH Zurich. References • Contributing • Citation References Official dataset webpage: grand-tour.leggedrobotics.com Getting started & examples: github.com/leggedrobotics/grand_tour_dataset Boxi used to collect the data: github.com/leggedrobotics/grand_tour_box Contributing We warmly welcome contributions to improve and expand this project. Whether it's new examples, enhancements, or… See the full description on the dataset page: https://huggingface.co/datasets/leggedrobotics/grand_tour_dataset.

Task_categories:roboticsSize_categories:1M<n<10MRoboticsLidarCameraVision
zai-org/RPC-Bench HF Unverified

RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension 🌐 Project Page • 💻 GitHub • 📖 Paper RPC-Bench is a fine-grained benchmark for research paper comprehension. It is built from review-rebuttal exchanges of high-quality academic papers and supports both text-only and visual evaluation through complementary paper representations. Data Structure RPC-Bench is organized into train, dev, and test subsets. Split assignments… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/RPC-Bench.

Task_categories:question-AnsweringLanguage:enSize_categories:100K<n<1MModality:imageResearch-PaperDocument-Understanding
codeparrot/github-code HF PQC Verified

The GitHub Code dataest consists of 115M code files from GitHub in 32 programming languages with 60 extensions totalling in 1TB of text data. The dataset was created from the GitHub dataset on BiqQuery.

Task_categories:text-GenerationTask_ids:language-ModelingLanguage_creators:crowdsourcedLanguage_creators:expert-GeneratedMultilinguality:multilingualLanguage:code
mlfoundations/MINT-1T-PDF-CC-2023-14 HF PQC Verified

🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens 🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2023-14.

Task_categories:image-To-TextTask_categories:text-GenerationLanguage:enSize_categories:1M<n<10MFormat:webdatasetModality:image
cadene/droid HF Unverified

This dataset was created using LeRobot. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset One of the biggest open-source dataset for robotics with 27.044,326 frames, 92,223 episodes, 31,308 unique task description in natural language. Ported from Tensorflow Dataset format (2TB) to LeRobotDataset format (400GB) with the help from IPEC-COMMUNITY. Visualization: LeRobot Homepage: Droid Paper: Arxiv License: apache-2.0 Dataset Structure meta/info.json: {… See the full description on the dataset page: https://huggingface.co/datasets/cadene/droid.

Task_categories:roboticsLanguage:enSize_categories:10M<n<100MModality:videoLeRobotOpenx
ATH-MaaS/Marco_Longspeech HF Unverified

Marco-LongSpeech Dataset Marco-LongSpeech is a multi-task long speech understanding dataset containing 8 different speech understanding tasks designed to benchmark Large Language Models on lengthy audio inputs. 📊 Dataset Statistics Task Statistics Task Train Val Test Total Unique Audios ASR 71,275 15,273 15,274 101,822 101,822 Temporal_Relative_QA 5,886 1,261 1,262 8,409 8,409 summary 4,366 935 937 6,238 6,238… See the full description on the dataset page: https://huggingface.co/datasets/ATH-MaaS/Marco_Longspeech.

Task_categories:automatic-Speech-RecognitionTask_categories:audio-ClassificationTask_categories:text-GenerationLanguage:enLanguage:zhSize_categories:10K<n<100K
SWE-bench/SWE-smith HF Unverified

SWE-smith Dataset Code • Paper • Site [12/14/2025] NOTE: We will no longer actively update this dataset. While this dataset is still functional and usable, we recommend you use the `SWE-bench/SWE-smith-[lang]` datasets. For better maintainability and ease-of-use, we are maintaining language-specific datasets in lieu of this mono-repo. The SWE-smith Dataset is a training dataset of 50137 task instances from 128 GitHub repositories, collected using the SWE-smith toolkit.… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-smith.

Task_categories:text-GenerationLanguage:enSize_categories:10K<n<100KFormat:parquetModality:textLibrary:datasets
builddotai/Egocentric-100K HF Unverified

Egocentric-100K is the largest dataset of manual labor. You can visualize the dataset here. Egocentric-100K is state-of-the-art in hand visibility and active manipulation density compared to previous in-the-wild egocentric datasets. The complete 30,000 frame evaluation set is available at Egocentric-100K-Evaluation. Dataset Statistics Attribute Value Total Hours 100,405 Total Frames 10.8 billion Video Clips 2,010,759 Median Clip Length 180.0 seconds Mean… See the full description on the dataset page: https://huggingface.co/datasets/builddotai/Egocentric-100K.

Size_categories:1M<n<10MFormat:webdatasetModality:textModality:videoLibrary:datasetsLibrary:webdataset
google/physical-engineerings-last-exam HF Unverified

Physical-Engineering’s Last Exam (Dev Set) Physical-Engineering’s Last Exam (PLE) is a benchmark of long-horizon engineering tasks built by licensed practicing engineers across 4 disciplines. In each task, an agent works inside a realistic project workspace containing drawings, specifications, spreadsheets, solver scripts, and CAD/BIM or simulation tools to produce engineering deliverables: code, calculations, models, schedules, and technical reports. This repository contains… See the full description on the dataset page: https://huggingface.co/datasets/google/physical-engineerings-last-exam.

Benchmark:officialBenchmark:eval-YamlTask_categories:text-GenerationTask_categories:question-AnsweringLanguage:enSize_categories:n<1K
airtrain-ai/fineweb-edu-fortified HF Unverified

Fineweb-Edu-Fortified The composition of fineweb-edu-fortified, produced by automatically clustering a 500k row sample in Airtrain What is it? Fineweb-Edu-Fortified is a dataset derived from Fineweb-Edu by applying exact-match deduplication across the whole dataset and producing an embedding for each row. The number of times the text from each row appears is also included as a count column. The embeddings were produced using TaylorAI/bge-micro Fineweb and… See the full description on the dataset page: https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified.

Task_categories:text-GenerationLanguage:enSize_categories:100M<n<1BFormat:parquetModality:tabularModality:text
allenai/qasc HF Unverified

Dataset Card for "qasc" Dataset Summary QASC is a question-answering dataset with a focus on sentence composition. It consists of 9,980 8-way multiple-choice questions about grade school science (8,134 train, 926 dev, 920 test), and comes with a corpus of 17M sentences. Supported Tasks and Leaderboards More Information Needed Languages More Information Needed Dataset Structure Data Instances default Size of… See the full description on the dataset page: https://huggingface.co/datasets/allenai/qasc.

Task_categories:question-AnsweringTask_categories:multiple-ChoiceTask_ids:extractive-QaTask_ids:multiple-Choice-QaAnnotations_creators:crowdsourcedLanguage_creators:found
allenai/dolma3_mix-6T-1025-7B HF Unverified

⚠️ WARNING: This dataset is intended ONLY for reproducing Olmo 3 7B ⚠️ For all other training use cases, including training from scratch, please utilize our primary dolma 3 data mix: https://huggingface.co/datasets/allenai/dolma3_mix-6T. Note: Some olmOCR science PDFs in the current dataset have been redacted following the training of Olmo 3 7B. These texts are indicated with [REMOVED] in the text field. This will affect reproducibility of Olmo 3 7B. For this reason, please use our… See the full description on the dataset page: https://huggingface.co/datasets/allenai/dolma3_mix-6T-1025-7B.

Task_categories:text-GenerationLanguage:en
phanerozoic/hi-21cm-survey HF Unverified

21cm Hydrogen Line Sky Survey Continuous 1-second-cadence power spectra of the 21cm neutral hydrogen (HI) line at 1420.405 MHz, recorded 24/7 from a fixed omnidirectional observer on the US East Coast. What this is A radio telescope pointed at the whole sky, recording one spectrum per second, indefinitely. The Earth's rotation scans the beam across the galactic plane daily, producing a natural drift scan. Every row is a self-timestamped power spectrum spanning 2… See the full description on the dataset page: https://huggingface.co/datasets/phanerozoic/hi-21cm-survey.

Task_categories:time-Series-ForecastingSize_categories:1M<n<10MFormat:parquetModality:tabularModality:textLibrary:datasets
open-index/hacker-news HF Unverified

Hacker News - Complete Archive Every Hacker News item since 2006, live-updated every 5 minutes What is it? This dataset contains the complete Hacker News archive: every story, comment, Ask HN, Show HN, job posting, and poll ever submitted to the site. Hacker News is one of the longest-running and most influential technology communities on the internet, operated by Y Combinator since 2007. It has become the de facto gathering place for founders, engineers… See the full description on the dataset page: https://huggingface.co/datasets/open-index/hacker-news.

Task_categories:text-GenerationTask_categories:feature-ExtractionTask_categories:text-ClassificationTask_categories:question-AnsweringLanguage:enSize_categories:10M<n<100M
vincewin/CREST_data HF Unverified

CREST forcing (parquet) EF5/CREST hourly forcing for CONUS, 2016-present, packed as one tar per variable/year. dir variable source cadence mrms/ precipitation MRMS QPE (corrected) hourly temp/ 2 m temperature NLDAS-2 FORA hourly pet/ potential ET FEWS NET daily PET daily Each *.tar expands to individual .pqf (Apache Arrow parquet) grids readable by the EF5 v4.5 native parquet reader. Used by the Space vincewin/CREST_AI. Download + extract one year, e.g.: from… See the full description on the dataset page: https://huggingface.co/datasets/vincewin/CREST_data.

Size_categories:10M<n<100MFormat:parquetModality:imageModality:tabularLibrary:datasetsLibrary:dask
chenxran/uspto_full HF Unverified

Dataset Card for "uspto_full" More Information needed

Size_categories:1M<n<10MFormat:parquetModality:textLibrary:datasetsLibrary:pandasLibrary:mlcroissant
StringFellow/fusion-dw HF Unverified

Size_categories:1M<n<10MFormat:parquetModality:textLibrary:datasetsLibrary:daskLibrary:polars
allenai/sciq HF Unverified

Dataset Card for "sciq" Dataset Summary The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. For the majority of the questions, an additional paragraph with supporting evidence for the correct answer is provided. Supported Tasks and Leaderboards More Information Needed Languages More Information Needed… See the full description on the dataset page: https://huggingface.co/datasets/allenai/sciq.

Task_categories:question-AnsweringTask_ids:closed-Domain-QaAnnotations_creators:no-AnnotationLanguage_creators:crowdsourcedMultilinguality:monolingualSource_datasets:original
openbmb/Ultra-FineWeb HF Unverified

Ultra-FineWeb 📜 Technical Report | 📦 UltraData Collection | 🌐 UltraData | 🤗 MiniCPM4 Series | 🤗 MiniCPM5 Series English | 中文 📚 Introduction Ultra-FineWeb is a large-scale, high-quality, and efficiently-filtered dataset. We use the proposed efficient verification-based high-quality filtering pipeline to the FineWeb and Chinese FineWeb datasets (source data from Chinese FineWeb-edu-v2, which includes IndustryCorpus2, MiChao, WuDao, SkyPile… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/Ultra-FineWeb.

Task_categories:text-GenerationLanguage:enLanguage:zhSize_categories:1B<n<10BModality:textLlm
Showing 20 of 341 datasets (page 8 of 18)