# Compression Test Data

This repository contains a compilation of compression test datasets, found across the internet. The data was put in this repository for use as a submodule in [rzip-rs](https://github.com/amkillam/rzip-rs), where it is used in benchmarking and tests.

## Original Sources

- [The Artificial Corpus](https://corpus.canterbury.ac.nz/descriptions/)
- [The Calgary Corpus](https://corpus.canterbury.ac.nz/descriptions/)
- [The Canterbury Corpus](https://corpus.canterbury.ac.nz/descriptions/)
- [The Large Corpus](https://corpus.canterbury.ac.nz/descriptions/)
- [The Lukas Corpus](https://www.data-compression.info/Corpora/LukasCorpus/index.html)
- [The Miscellaneous Corpus](https://corpus.canterbury.ac.nz/descriptions/)
- [The Protein Corpus](https://www.data-compression.info/Corpora/ProteinCorpus/index.html#ProteinCorpus)
- [The Silesia Corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia)
- [enwik](https://www.mattmahoney.net/dc/textdata.html)

## FAQ

- **Q**: Why copy this data into a new repository? Why not just download it from the original location where it was hosted?
  - **A**: The original source of most of this data was either of a hobbyist or academic nature. As such, the idea of repeatedly downloading this data from the original hosts for testing and benchmarking seemed to me to be morally wrong; I would hate to cost these organizations money by repeatedly downloading these extremely helpful datasets which they have freely and generously provided. Thus, reuploading the data on Github for future download seemed preferable.