Terminal & CLI command reference
Every command works in Windows CMD, PowerShell, and Git Bash / WSL unless noted. Replace python with python3 where needed. The complete written guide is docs/COMMANDS.md.
SetupClone, install, verify
git clone <repo-url> && cd anime-text-dataset python -m venv .venv .venv\Scripts\activate # Windows CMD/PowerShell # source .venv/bin/activate # Git Bash / WSL / macOS pip install -r requirements.txt python -m src.cli validate # repository checks python -m src.cli checksums # SHA-256 manifest
AccessWays to access, clone, or train
| # | Method | What you get |
|---|---|---|
| 1 | Clone the repository | code, schemas, docs, pipeline |
| 2 | GitHub Releases (gh release download) | the full exports/ tree |
| 3 | Stream JSONL in Python | train-ready rows, no full load |
| 4 | Hugging Face datasets | sharded, streamable HF Dataset |
| 5 | SQLite database | joins and referential queries |
| 6 | Parquet + DuckDB / pandas | fast analytics |
| 7 | CSV | spreadsheet-friendly flat view |
| 8 | Training recipes | fine-tuned models (training page) |
ExploreRead the data from any shell
# count rows — PowerShell (Get-Content exports\anime.jsonl | Measure-Object -Line).Lines # count rows — Bash / WSL / macOS wc -l exports/anime.jsonl # first record — PowerShell Get-Content exports\anime.jsonl -TotalCount 1
# SQLite — Python stdlib, no install python -c "import sqlite3; con=sqlite3.connect('exports/anime_dataset.sqlite'); [print(r) for r in con.execute('select year,count(*) from anime group by year order by 2 desc limit 5')]" # DuckDB over Parquet (pip install duckdb) duckdb -c "select year, count(*) from read_parquet('exports/anime.parquet') group by year order by 1 desc limit 5;"
# one-line Python stream (no deps)
python -c "import json; A=[json.loads(l) for l in open('exports/anime.jsonl',encoding='utf-8')]; print(len(A), sum(1 for a in A if a['year']==2020))"
PipelineThe src.cli driver
One module regenerates every release artifact. Each phase is resumable.
| Command | Produces |
|---|---|
python -m src.cli discover | IDs per source |
python -m src.cli dedup | 35,766 canonical works |
python -m src.cli enrich | depth-tier detail (episodes, roles, relations) |
python -m src.cli entities | characters · staff · studios · episodes |
python -m src.cli normalize | duplicate works merged, enums canonicalized, values repaired, entities deduped |
python -m src.cli relationships | 39,226 relationship edges |
python -m src.cli descriptions | 63,396 descriptive texts |
python -m src.cli qa | 317,907 QA pairs |
python -m src.cli instructions | 235,395 instruction examples |
python -m src.cli franchise | franchise clusters |
python -m src.cli rag | 206,906 RAG chunks |
python -m src.cli split | 90/5/5 splits per work |
scripts/license_tiers.py | per-anime license tier, manifest and subset id lists |
python -m src.cli export | JSONL + SQLite + Parquet + CSV, with split and license_tier joined |
python -m src.cli validate | 21 quality checks over the exported files |
python -m src.cli checksums | SHA-256 for every export |
python -m src.cli report | reports/final_report.md + summary.json |
TrainTraining entry points
Full install lines and code for all nine recipes live in the training page.
pip install -r examples/training-requirements.txt python examples/finetune_lora.py \ --model mistralai/Mistral-7B-v0.1 \ --data exports/instructions.jsonl \ --split train --epochs 2 --output runs/anime-lora
PreviewServe the site locally
python -m http.server 8000 --directory docs
# open http://localhost:8000
TroubleshootCommon issues
| Symptom | Fix |
|---|---|
database is locked | run pipeline phases in sequence; no parallel writers |
Unknown command: X | run python -m src.cli to list valid commands |
UnicodeEncodeError in Windows | $env:PYTHONIOENCODING='utf-8' (PowerShell) or chcp 65001 (cmd) |
| Checksum mismatch | re-download the release asset or python -m src.cli export |
| Memory spike on large JSONL | stream line-by-line (see explore examples above) |