# Krummholz LLM Training: Historical Reasoning & Forestry Pilot

**PRELIMINARY - 2026-10-23, v1.** A small, curated text sample for inspecting an LLM ingestion pipeline, tokenization, provenance handling, document retrieval and deduplication. This is **not a representative training corpus**, a benchmark, or a validated source of current scientific advice. No model was trained or evaluated on it, and no improvement in model performance is claimed.

## What is included

37 non-overlapping text excerpts, containing 8,801 words by the documented ASCII word-token counter. They were selected from three existing Common Corpus source fragments, representing two books and one journal volume:

| Area | Excerpts | Original material |
| --- | ---: | --- |
| Reasoning and language | 30 | Carveth Read, *Logic: Deductive and Inductive*, fourth edition (1914; reprint history through May 1920) |
| Forestry and botany | 5 | J. C. Loudon, *Arboretum et fruticetum britannicum*, volume 1 (1854) |
| Forestry and botany | 2 | *The Journal of Forestry and Estates Management*, volume 1 (1877–1878) |

These are short excerpts of existing source fragments, not complete books or volumes. This release includes no fisheries or engineering excerpts. The title of the website category is “LLM training”; the actual downloadable artifact is a curation pilot.

## Source and rights

The text comes from [Pleias Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus), revision `307910e4c5d040d6f318e6edf2a2b97849155771`, file `common_corpus_1/subset_100_1.parquet`. Only this one 429,962,586-byte shard was downloaded; its SHA-256 is `4ee719a130b7f86978b08c20cc9f490309e2bae2a5c0df4f04fd427365530f07`.

All retained records have an upstream “Public Domain” label. Original library/catalog records were also reviewed for matching editions, explicit public-domain/no-copyright statements and dates. `sources.json` preserves bibliographic authors, original URLs, contributors, exact edition/volume labels and the evidence used. Publication dates were not inferred from journal series-start dates.

This selection uses a **United States public-domain assessment**; it does not provide a global rights clearance. Assess the applicable jurisdiction and intended use before deploying or redistributing a derivative. Provider names are used solely for source attribution. There is no affiliation with or endorsement by Harvard, Pleias, Project Gutenberg or the contributing libraries. This is **not** Harvard’s Institutional Data Initiative Institutional Books release.

The original Project Gutenberg catalog and policy pages are linked in `LICENSES.txt`; its trademark/license boilerplate is absent from the training text. Bibliographic credit and links are retained as provenance. No license restricting the underlying public-domain text has been added by Krummholz.

## Cleaning and selection

1. Inspect the shard's 69,907 records; select relevant English public-domain source fragments and verify their provenance. Screen historical education/expository prose as well as industry topics.
2. Exclude material with unresolved rights, noncommercial/restricted terms, Google-sponsored scans for this conservative first release, biographies, medical/legal advice, news, catalogues, financial filings and off-topic material.
3. Normalize Unicode and whitespace. Remove detected boilerplate and page/header lines. Join a split word only where its joined spelling already appears at least twice as a complete lowercase word in the approved upstream text. No semantic rewriting is performed.
4. Make non-overlapping, sentence-based candidates, generally 120–360 words. Reject table-heavy, fragmented, repetitive, obviously damaged, weakly topical and unwanted historical content.
5. Omit whole chunks matching contact, address, SSN-like or selected personal-narrative patterns. Omit source `creator` fields from model-text rows; retain historical bibliographic attribution in the source register.
6. Screen normalized exact duplicates and heuristic near-duplicates. No duplicates were detected among the final selected passages.
7. Read every final excerpt and conduct an independent agent-assisted content and risk review. Exclude damaged passages rather than guess repairs. This is not human domain-expert validation.

The detailed funnel and rejection counts are in `audit.json`. Of 314 candidates from 11 rights-reviewed fragments, 37 were retained. The manual passage selection and exclusions are shipped so that the same output can be reproduced. The word-wrap repair count in the audit includes rejected candidates as well as retained text.

## Important limits

- Historical science, taxonomy, terminology, social assumptions and forestry practices may be inaccurate, biased, incomplete or unsafe today. Treat the passages as historical source text, not operational guidance.
- Residual OCR mistakes remain, including a small number of misspellings and awkward punctuation. The corpus provider may already have applied model-based OCR correction. **Krummholz generated no new text or completions; upstream OCR/model corrections may remain.**
- Personal contact details were screened, but this is not a guarantee of exhaustive anonymization. Historical author names, citations, institutions and geographic references remain where relevant.
- Only three upstream source fragments contribute retained text. The sample is small, deliberately selected and unbalanced; it is unsuitable for conclusions about training quality, coverage or downstream performance.
- No train/test split, benchmark contamination assessment or model evaluation is included. Any downstream evaluation should split by original work rather than randomly splitting adjacent excerpts.
- Topic labels are curation aids. Word counts are approximate ASCII word-token counts, not model-token counts.

## Files and reproduction

- `krummholz-llm-training-preliminary.jsonl`: one UTF-8 JSON object per line, with text, topic, original source identifier, rights provenance and content hash.
- `schema.json`: field definitions and validation schema.
- `sources.json`: original bibliography and reviewed rights evidence.
- `audit.json`: filters, counts and limitations.
- `LICENSES.txt`: attribution and reuse notes.
- `reproduce/`: pinned download details, source allowlist, editorial choices and Python script.
- `CHECKSUMS.sha256`: hashes of every other package file.

To reproduce, use Python 3.11+ with `pyarrow==26.0.0`. In the `reproduce` directory run `python download_source.py` (downloads about 430 MB), then `python curate.py --out ../reproduced`. Compare the resulting JSONL SHA-256 with the package checksum. The original raw shard is deliberately not bundled. All curation is local; the script sends no text to an external model or NLP service.

For a client training project, use this package to review the data contract and screening approach before commissioning a larger, domain-specific corpus and an evaluation plan.