FineWeb Tokenized > 4 trillion tokens of the pre-tokenized data the š web has to offer What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre-tokenized corpus, we eliminateā¦ See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.
Use this model
Pull with QuantumShield
quantumshield pull anisoleai/fineweb-tokenized Verify integrity
quantumshield verify anisoleai/fineweb-tokenized pip install
pip install quantumshield && quantumshield pull anisoleai/fineweb-tokenized PQC-Verified with ML-DSA-87
This model has a real FIPS 204 ML-DSA-87 (Dilithium5) signature from the platform signing authority. Signature chain includes 2 verification(s). Last verified 2026-07-23.
README.md
fineweb-tokenized
FineWeb Tokenized > 4 trillion tokens of the pre-tokenized data the š web has to offer What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre-tokenized corpus, we eliminateā¦ See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.
Intended Uses
This model is registered on the QuantaMrkt quantum-safe registry. All files have been cryptographically verified using post-quantum signatures.
Quick Start
# Install the CLI pip install quantumshield # Pull the model quantumshield pull anisoleai/fineweb-tokenized # Verify file integrity quantumshield verify anisoleai/fineweb-tokenized
About
FineWeb Tokenized > 4 trillion tokens of the pre-tokenized data the š web has to offer What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre-tokenized corpus, we eliminateā¦ See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.
Get this model
Pull with QuantumShield
quantumshield pull anisoleai/fineweb-tokenized Verify signatures
quantumshield verify anisoleai/fineweb-tokenized