{"id":592,"slug":"anisoleai--fineweb-tokenized","name":"fineweb-tokenized","author":"anisoleai","description":"\n\t\n\t\t\n\t\n\t\n\t\tFineWeb Tokenized\n\t\n\n\n    \n\n> 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer\n\n\n\t\n\t\t\n\t\n\t\n\t\tWhat is it?\n\t\n\nThis is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards.\nBy distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.","tags":"[\"Task_categories:text-Generation\",\"Language:en\",\"Size_categories:n>1T\",\"Modality:tabular\",\"Modality:text\",\"Tabular\"]","license":null,"framework":null,"parameters":null,"downloads":8553360,"likes":18,"verified":1,"created_at":"2026-06-23 18:23:36","updated_at":"2026-07-23 21:25:40","source_url":"https://huggingface.co/datasets/anisoleai/fineweb-tokenized","source_platform":"huggingface","hf_repo_id":"anisoleai/fineweb-tokenized","ollama_name":"","category":"dataset","latest_version":"v1.0.0","version_count":1,"signature_count":2,"risk_level":null,"risk_score":null,"versions":[{"id":591,"model_id":592,"version":"v1.0.0","manifest_hash":"2c6c7f1c28f4c1f99dc4cd04366665b0dbb1e76c1ba9ba609ebcf810997b4c01","file_count":0,"total_size":0,"r2_manifest_key":"manifests/datasets/anisoleai--fineweb-tokenized/v1.0.0.json","created_at":"2026-06-23 18:23:36"}],"files":[],"signatures":[{"id":1125,"version_id":591,"signer_did":"did:quantamrkt:registry:shield-v1","algorithm":"ML-DSA-65","signature_hex":"b8b3fb4609ccd80008f4b8f715591a6d40d6ea4e8fa7c8aaab0c59c8047abf5b","attestation_type":"registry","signed_at":"2026-06-23 18:23:36"},{"id":1241,"version_id":591,"signer_did":"did:web:quantamrkt.com:chain:authority","algorithm":"ML-DSA-87","signature_hex":"c2e19bd40db07771fbad1b8b9739c82b46451b44f8623d8d83210c9e64c1dcd5a96ac3810726ca1cc1ad68cfff856d995663522637706fe28388c4a5fa82473377c5bd4bf47f33dee9d50610fe5e348fe341aedab52215d2a73bb15e8f07d93b19887c9169f01e2a080901b8731bb941624952006eb4a6f753ca46cf16839acb44a7e6546201767b73a03d9a705563e71385ae9499388b7e2fa03c34949ea50b1993e11ba225a28b5abe5f42d0320daa01e98c335c4608d58a998d63a0d114a360ed1f1aeb5a86d44629269a331b09e9486cd776ec0c6cc041359e375f7b395e459a44d5ac49e43b77c1580536fe928cad7c4a0cf1ff4dab778a86ba432bf71139e4e3d35a34ab2eadfc0f8c9f57572e11d9e9d3f85150ddf52e27faca0356c1cb0ed3da6717cf968aef6089bd0991b90b06bb1a47ece6e27c96f2ecfd5bb60dcfff378ad79fc12ebdec8aed7121d5d15209ffeeaba5aebca37cc47bdd05dd2b33562fd90e10c486cfd4a2b523029770b0881cf0e45ead65a168dde3881be81514b96cb8f6e3eeca3f9d68849f8f3cf7a986e1ea8b8955153cae99896910b837fc48b8a53605f6a46801c72a0eed26424f5a697f559a36bd5e90c0b2bb7bb0f43d0bcc45ca2f165bdf1ed1c613ada1ed12794840bb0504561adc3a86fd9f66150439029331eea6dcdaf27aeacfcd83454fa877336cb2fc53dec6d97e6bd3441247116f89a845ca07c1971be31bf847fca546a2aebc7b45d05db21355a5252e6981c7b95266ee665ae279e4233538b2d2545ea227f73e898b5b5bac6b8fbc8bc66d957f2aaffced5d6ec26091a0cfd3e900cb2f525de2de10e4991b49f21b114bf80925dba3f78acaf2f5a173ddb6fcdcf37014f3dc358bb71d5be060215a6a02be0567842595464675ba996da3f81ae807729fa3be289228a683d25177c0253316212fbcb81e80ac215885925c2688290ff1adc7a11d902b3e313e8cd95fd879d732c6ccb13a47c0c865c50b0426e62662a3d57c03190844824636d60a666e4e589bc46157fdf62d6ee919e21c2105aa52e3dcf5c4b9105a7bdbb373cd4c3bc9ef4bed3bed1a2955f9486567322fa9cc427a8dd8adf4d0abfe8d472d4cde3c047642fb99ebf27926ffbe9384964c2834a05e8c5cf2473a5331fa489f27c6e8d17fb0736f5c8d222cdc0a35ce7a8204e92e2c9be5a7cf42e859b4a8151e8f4fa27dbc4d1bfe7db8e81a2a34e2c3e21f7d5a48fd4ddef172af2e62145b7ae07a8d135eda9c3a2e26bad74d1544c1eb31d8317c7f377a91ab864efbed6ab5d681ab8996725f6bb736aa66b3ce2fa45ba08f734ebb46c372554c9c301db2c7ac5aaab76393878d1fd86a9c12bfacd075412daa792f1eac47a528c7191416161f2cec94f472202765cd531fd25aa30d11d56b588c71e7f25aac4d8adbedf490b33b48d79528d25cb442c161f5904ef3f7f4447c45283125a583eaec2223a81dffb7b4515b0091196a9d1c783af1975e343e8e2a78e33b4bb51aefa3df265c3e08c8ab31db2799430f0ce7ae8dd1fa143f8943b62f50867f8ae150c5453ce8eae410627c20510719914bc2bd77f480e69c4f8e94392ba09aaf37d0e7899b28a587e909024a06c6d3a40dfd3ec7e38f906a307f96e49017b7243ed6057aac39c70fbabef6ed2d3e434b5b97a54c5e5fa4d143e451e5540c5038e6673026b130cb9f17a25d25747eac745b9a370adab145f3a30d97d0c9e93c475a1868d9b4086f26c0e9c97c6dfc75f5b28db57f046fa7843b71f64c8aa1f2b3d917bb3be2e3d62a3ce2c09f4732d661eef8c1ffaebb456e22d6f9c46910c1689898b0646b5f76a7741526657a085907050f97f2a5dda63d93f72dbedcf3ff19a85465f1d3b67734b66d3e8e5552f12e4710588b5f378d3b95c5eaf6757a76a603d06630f73b96d7f88a576176850a017480f766b4ea0981bbe59a9d27a5c5a828602000fa8fccfd0aa17a5ebf9076d6310e1f4ca7338386eec51fc35eae1b99d29ecc6cc248ab0c3a2f94881b6e9548ff6851d4b297b75742ac89f63f0f4b259f669063a87c0a903f791ffe8a643f491a05b80828f9cb8a850e820ca3e40281c3f41dfa1e9928e191987a6d4d171edf39c3e7bb3ca98d67b17f78d12a20e7269566ca5dfed4c6483f25c574e71fcadb19babc7f8f46f2aa3050d30392979473da153813c8d22744c49031c24d0976b7aaef6275d131a45b4c494ce8145f75366dc29815d9c38300dcf42e0b4cd05520e03f6e39ad7df8211de4b16a4b26507fcf4f6df212cf75f2871851a9af120159bcd46347476d072541ea2ee92103a1f6f45fdcf53e796dbc35da85a111624e72d7af8ddde9facbf74b72830e4a1bc099d84d26c58c86bac7bb5c4d64a93b1b05d31eaf0e23546341c3be4793faafccd31781b411b93704e95c04137c46653a2fba2fb72f544d62e627fb8f7f8f2c9096472cbb3477f5d6eb7fbc305409fc2d213ec84ce1158d4ee03426028cce37d6017a3339820bef699ad5c50739d2c7475372f1673daea532b870e6c2a6f326c9c4be1852c0271f455c32e14fb900b2115783faf4effa49657f8c7747bdb2db0fa62542d9a72a2a4d55d514f592e2045145a8d4417a299f15ba8e96d865cb9e7505f7b5dabb96a4e6fcb0893ea96dbcd6d305b32b72983cc348c11ae059b641dba8285394fdd4f2f4dc42f5f8741ed34ef0d5f4744681715030ab938e5f330c9f7c6a1b3ab71babf1c7bf4b7de0324434451d17424457736e0a31d1aff3001c1cf6c5afa1850e68bd37e69639c0453ac6da57f0036fb9f33a1c9f43306ce0421490852ea73dafca22367b6fa55be8154b08f342ee159d12d466a54a2b3750f5fcfd5372eb5613a95fbd63483becf6d5a5e1f6ab561728a27e45fe072ed06935182ca15b41f6a53847a59655c0db8908ee64d28f2d1f17de641d0962b54230e4a18d00adf1dbc7dc31ba7f3b332e864691a9704b3414ede8abdd38b5d6c82a785eecdb1ed36d38cc635df54f403998970372f11d97e8d44093f3b3ee456b869194a4f762ee06ec319f01e30a2771cfdf69cd08705f8a56cc83e90a00f3278b6399448a4a788d4d5f05d98f2ce9d734272a8b149cd8b8509d35c37be97c7038cfb42a6f14b5d8367afb311be7571584c27756aaa4f2cf4473d3945bade559d6583f3bb035d2280250cd66050337f391fd53def31bac50348327fc972d3eab08b3b223e4ae21baf1f345ef0b937b8b45378ae4702fc723cfff5aa4cc41fc86db2fbc227c71ff461853927385078753e1188e168fb2106d5fd7af2b94ce37f735dab2e60edaddb7b9babf6617921e62f928dad2534eed02295f3c9939e7195fe9e66f9c41e0827cf7f85fe845113b74a29c5baba9ee6017425743d1bffcd45aad4751158e4c310ebd2487c58a59ab69f61ce7f3da0918f3726732c7a779583e8ab5eaaa156bfc3a7d716c3fdc67e130eb5560642d2bc512aaead5639f25191bbe9bf535eb3df6efefbcdfe08566d750783e8b0d311f5492b09b340c6fe6aca193e2d6c098f025b6a98f9fbcd222941cecb209254f11b5bd414c1754059b671a8a9b654bc3fd4cc1fea3ee6898d155c1356a850722f58b666982f41ef010d86c38c427ec030f36cf5622b82d2b2731a70b287128c8df29c2b23bc2ff1539234bcef657e104b79f5a1cea9845c22c6fa5453b376d28f24bf632fc20e6b76ec2bef9a27d3a5555d58e2713c76c17a8589a88e63aa8299c91900d479419e8080bf27b7dacb5c12ad1fd0c8189d9a60e77f4fa66fdcdc414c21c5b2d90bf21e1aaef41b81c688ab848513f3ba1402f039ddadfc6dde350e2b54e73a2e27bfb80b2ec73478397661bd80ca4eaf0faca2fa240c57846c8d0de7d0656af86687d3f4b5cd2e42d2d47e590d1289836ae979872f4741a3033f1df6bfa40681b6213509faa8048b37abfda91ac95c97312b494ad1913152be5f96f13dbaffd3acefc57e30ad08fd5470529f0b0e6f1b203364e929c9e0a3823f298dfb6a2f2d6fe76269ad7aee894f93975c3e1c8180aa437ab8cfc4eecdfb7b5f22c64bab505883c5b4d16e2e63d3606e46d75a5b8d723aa6743cea3bcbfdb536912fc4f86b0131e4ef72a8e094a3c9d2df4ef182fffdc4a4d6351aa6dc4bd2e4241c4b844face28314774ef03216405a593679ec8dcd5a73f353c25ffff11b6895e826782d3fb398e0ed2c063430adfdd63c451d187c88b77ebfac0e4521eb84c7af81a428de7acc1e0d994be1f206ba256e544f1f9a8dfb9772f7795ba1b734fd307eba90784a31abba83bc055a55b0223a13322c27933fbc725f7eadf821f73ecd30e33d379b22376ed36fa4fa0d816b43dfe026012e746e416cc7419e9a3cddeb670cf92bbde106a9196f23a80eb69209e52d1391912e3a51b47661ef068169ed8b09860e65266bc3f9e41ad8f454d2c16304f8a2417dec0a75b6218bcc0b2329d60dae68323774a5861f06a9f5cad640b72bca9f9526294c2373a3a8c4d98e5f0332339586ca02c0e4d0bd8ca28a349a2ade4829110b013e4d5b7904154c2b76773283e6ef810846ef11a98fd925577a433c87aaeb959359825016904d7131cdda50d934b24aa470d941c8efd526a9708848f49a702d00b9502813c45942d5670f9bf189b92c15b574b5095f56ce876e7129b866779980b8a8e90e83e6b16f66320e539aead9cd48f9cf169e850867322cf24f0bc6a50b9d9b80d91194a599baaa817b8040891b8e6517a2f8472307a72dc25d72e8032f243bee39d0a97da6a20872c08aceb8a30351a8f186955c6ee301c833bd05483e9745da84b1cf129b07d05e189bc8db734c8c79a8bd784eab782f312801a736d9e8c921efac268af1642b665b0cd1c88a4f5e1ed6edf4067c995d2a39b641d5654420b04ebfbf784056c8a0c81df197517aeeda492d49987cb234ad4f58c651712f701aa907a16dacf8dc9d84c54bab3ef8323238ad9849751a50c5aa62cc135e27940e7df0464da33f0b2fa4f65356c20a70e4ad08c2ba0abdebb92309812580ef57b908c257c27d00a667741598c69e96578a838d171fde68b959a20b35e9c676200347fdf26dd247c6fbabc97b6eb41f196e71ac24d809dae3cf206f4cf6b7b1e561a7c230d720d6c5b0ae4c16f81fbae77f489a4e47a2908a30f9e08efe7cdb8a6044d7e394931ea282a8142d98ce2ee1c9e13eade3e30fb96fb75a872973a158b5c6e494394d16f616c0aba8354cd7d927712be4af3b91de127ebd2a20f4256cddafa006d34dade413198973d75e5ce8e649946332d34ecc8f6f005728827ef3760eb4511b113e8232b4aa89ead8b579a362b6f5e3e90040ed1ae2d8affc4ee7d76a89334ada8ab784fd1cf845aa55f531f529cc66e49593acf5ca61d4fe0069d5099b60a68b47167f7591ef57a3f3b3e8dbd4e2bf2a6a01f5d7614965cfb2fc5d980959676f8e2c01fbab75484174054a42393734fffa19a3a01afba54d479185b3b3d765fe4bcea2a441beceae2217f6040650f68cccb7022aad64a74f710efe356c0ab39cad9c69a03f17601c2d59b8e7352886974da9cd49209825393c6c2dc2527fd3b9301376a33d54ff0fb1c4f0f61d0f7929ff0ad8bffab10ef998700afe911a48c97bf862e75803cf4faebf784c94178dec8039acbe102fac2bea9da4a64236918a4e1c2cb7537015d07004bc334598a93a3414da7317b2e83dfd64f882bbca8f1d194c5ac55a3b46577c25942acef9ec07ecf0b0ee05b52eb659108909b6f5c9f81cf8f657abadb79e0a29f8edf36c73cbf40f284191ff8ea622c1d5a6091e70f4ed3a94a48237a547117e967203be9768d077ee4a5116145895782894026984165fb411e2bdbf8c69b86bf9696c3e653cfe9219e24c029584bd5b08c81c6dcd91618a05e115d58c316102f96f6a5ad78c998192ef7cc9d3b4c5d8e1c4c4f98f049248cec02381fb883af1c2d1870c336968d6bfae832a573079657a07c9077fbbe497507f517fc11d05d635da461a705868a71a5192b7cd5783087ef3ab767a8b72fa3ecc406ecdb10f2b24a8907f7b1dc44befd536478ae85042fd093e22cc556b60d34dae3d88e926bba648c0eb89617a57a67eb94c22a2f26c6dababf6dac9962e2bc9aef0af4f471ece5d5bbf36895ed225b5d78849cf2fffd089265661b36447606b3ff986c3f4093774d9d03a795ae4ec1b7408a51fad5e5ec1aedfd5cba8788c1be31ca098f763c2f76170d1065b851951af2118377d90b4f7ba78209561212e45beb4b3022c58ed71d3663dd19961317b1fa1a4fcb590c1005f6c4a6b15867cdb0f62c4a343931f25bffdbbe76deaa4c0fa924634d2b2a6c868fedd3cf7b640571b9d0e6399fa2bf4defc46a3fcf4de9dc60d18109121327475398c7d1062b41626d86c3e335484d0e1020657f93adbec4cae1f1111925536e7b7f828ea23c587c8bbc04496091b2be03040d4072f100000000000000000000000000000000091114202a2f353b","attestation_type":"pqc_registry","signed_at":"2026-07-01 12:34:35"}],"hndl":null}