Engineering Documentation
The plan & science: linguistic context, the descriptive-corpus approach, data analysis & outputs, resources, community quality, standards and the roadmap — the why behind the project. Hands-on engineering (architecture, code, APIs, DevOps, onboarding) lives on the workflow page. Tap a section to expand.
Read this first
- Org: github.com/KorpusaKurdi
- Repo: github.com/KorpusaKurdi/korpusa-kurdi
- Kanban: GitHub Project · Board
- PROD env: korpusa-kurdi.pages.dev
Welcome. KorpusaKurdî is a Kurdish language variation tracking and documentation platform — not a standardisation project. The technical north star is to collect real usage from speakers, attach rigorous geographic metadata to every contribution, and let statistical and LLM-based analysis surface the most popular forms while preserving minority variants.
This document is a working manual. Treat it as living: open a PR if you spot anything stale.
How to read this doc
- Sections 2–4 are required reading: linguistic context, the scientific approach, and what the data analysis produces.
- Section 5 lists resources & prior art to reuse before building.
- Sections 6–7 cover community quality (gamification) and standards (licensing, privacy, ethics).
- Section 8 is the roadmap; Section 9 the reference appendix (sources, glossary).
- Hands-on engineering — architecture, code, APIs, DevOps, onboarding — lives on the workflow page.
Linguistic context engineers must know
You don't need to be a linguist, but you need to know enough not to design naïve schemas. Three facts will shape almost every technical decision.
Dialect taxonomy
| Dialect | Primary regions | Speakers | Structure | Script |
|---|---|---|---|---|
| Kurmanji (North) | Turkey, Syria, N. Iraq (Badinan), Armenia | 15–20M | Synthetic / inflected; gender, case, ergativity | Latin (Hawar) |
| Sorani (Central) | Iraqi Kurdistan, W. Iran (Rojhelat) | 6–12M | Analytic; no gender/case; word order | Perso-Arabic |
| Southern | Kermanshah, Ilam, Khanaqin | 3–5M | Transitions Sorani ↔ Laki/Pehlewani | Perso-Arabic |
| Zaza-Gorani | Tunceli, Bingöl, Hawraman | 2–3M | Distinct branch; high complexity | Varied |
Source: cross-checked against multi-LLM research (Gemini deep-research output) and Wikipedia / Translators without Borders factsheet.
Scripts & the bi-directional problem
| Feature | Hawar (Latin) | Sorani (Arabic) | Engineering impact |
|---|---|---|---|
| Direction | LTR | RTL | UI must support bi-directional layout, mirror icons |
| Vowels | 8 distinct (A, E, Ê, I, Î, O, U, Û) | Markers (و, ێ, ە) | Diacritic-aware tokenisation; never strip marks |
| Consonants | Velar / alveolar stops | Pharyngeals, uvulars (/ʕ/, /ħ/) | Phonological mapping needed for TTS / ASR |
Three-tier geographic hierarchy
Every contribution must be tagged at three levels — this is non-negotiable and shapes the database schema.
- Country level — TR / IQ / IR / SY (loanword influence: Turkish, Arabic, Persian).
- City level — major hubs (Erbil, Duhok, Diyarbakir, Mahabad…) act as dialect anchors.
- Region / village level — preserves hyper-local accents and lexical variants that broader classifications drop.
region string field. Always model country / city / locality as separate normalised columns + PostGIS point.
Scientific approach: algorithmic democracy
"Algorithmic democracy" is the working name for the four-stage pipeline that turns raw contributions into usable, regionally-tagged variant statistics — without imposing a top-down standard.
1. Collect & label
Every datum tagged with country/city/locality + dialect self-ID + user profile metadata.
2. Pattern extraction
LLMs + classical NLP (clustering, frequency mining) find variants for any concept.
3. Democratic selection
For each context, compute most-popular form per region; never delete minorities.
4. Living reference
Surface results via dashboards + open API. Re-runs as new data arrives.
Corpus vs. agent: a hybrid
We deliberately build both:
- Static corpus — the ground-truth dataset, exported under
CC-BY 4.0, suitable for training third-party models. - Dynamic agent — an interactive layer that does active learning: it identifies under-represented regions and prompts users to fill those gaps, plus real-time validation hints.
Sample tasks the agent surfaces
- "Translate word X in your dialect."
- "Correct this sentence from a Rudaw article."
- "Record this 5-second daily snippet."
- "Confirm whether you'd say ez diçim bazarê or ez bazarê diçim."
Data analysis & outputs
Data analysis is the bridge between contributions and insight. Every analysis we run is descriptive: we surface what speakers actually do, never decide what they should do. The result is a corpus that reveals variation rather than flattening it.
Pipeline (raw → output)
Six stages, each one auditable. Any contribution can be traced from any output back to the original speaker (anonymised, with consent), so any claim the platform makes is reproducible.
Speakers submit text, audio or labels with regional, dialect, age and register metadata.
Script unification (Hawar / Sorani / Cyrillic), token cleaning, metadata validation.
Bucket contributions by region, dialect, cohort, register and time window.
Statistical: which forms exist, at what frequency per region. Clustering: which regions group together.
Semantic clustering, novel-form detection, lemma proposals — always reviewed by humans before publication.
Outputs are versioned, signed and shipped: datasets, dashboards, APIs, papers.
What the analysis measures
| Metric | What it shows |
|---|---|
form_distribution | Frequency of a form across regions and dialects. |
dialect_cohesion | How much speakers within a region agree on a single form. |
variant_rarity | How rare a minority form is — drives the preservation index. |
geographic_spread | How wide the geographic footprint of a form is. |
temporal_drift | How a form's frequency changes over time (years, generations). |
cross_script_map | The same word across Hawar, Sorani and Cyrillic spellings. |
What the corpus enables
Each output below is something the analysis stage can produce or feed. Some are live dashboards, some are versioned datasets, some are training corpora consumed by other projects.
Public datasets
Versioned Parquet / JSON-L releases with full metadata, CC-BY licensed, citable.
Variation map
Interactive geographic map: pick a word, see how it is said across regions.
Open API
Query: "most common form for X in region Y" — JSON in, JSON out.
Living dictionary
A reference where every entry carries regional usage frequencies — not a single "standard" form.
Academic papers
Replicable methodology + dataset versioning lets researchers cite the exact slice they used.
Training corpora
Cleaned, region-tagged data for TTS, STT and Kurdish-aware LLMs.
Educational materials
Variation-aware schoolbooks: students see the range and where each form is used, not just one.
Preservation index
A live measure of how endangered a rare form is — a signal for archivists and educators.
Journalism dashboards
Reporters can verify "is this word really common?" and find where to interview real speakers.
Resources & prior art
Before building anything, audit what already exists. The Kurdish NLP space has scattered but real resources — datasets, toolkits, voice corpora, comparable platforms. Reuse where possible; complement rather than duplicate.
Direct links below are deliberately conservative — only canonical URLs we are confident about. For everything else, names are leads — search HuggingFace Datasets and GitHub by name to find the current home.
Datasets (text)
kurdish-ai / kurdish-corpus
Mixed-source Kurdish text corpus on HuggingFace Datasets — fast starting point for tokenisation, embedding and language-model work.
huggingface.co →OSCAR
Massively multilingual web corpus with Kurdish slices (Kurmanji + Sorani). Useful for LM pre-training. Search HuggingFace for oscar-corpus/oscar.
CC-100
Filtered Common Crawl per-language corpora with Kurdish splits. Search HuggingFace Datasets for cc100.
Wikipedia (Kurdish)
Periodic dumps of Kurmanji + Sorani Wikipedia. Clean encyclopedic register — small, but high quality. Search the Wikimedia dumps site for kuwiki / ckbwiki.
Tatoeba (Kurdish)
Crowdsourced sentence pairs with translations — useful for parallel data and sanity checks across scripts.
AgaCKNER
Annotated NER dataset for Sorani. Smaller scale, but rare for being supervised. Search HuggingFace / GitHub by name.
Voice & speech
Mozilla Common Voice
Open multilingual voice corpus with growing Kurdish coverage. Aligned MP3 + transcript pairs, contributor pipeline we can study.
commonvoice.mozilla.org →kurdishtts.com
External TTS API used for synthesis. Possible STT integration. Already wired into the planned audio pipeline.
RHVoice
Open-source TTS engine with experimental Kurdish voices. A solid base for offline / low-resource synthesis.
NLP libraries & tools
KLPT
Kurdish Language Processing Toolkit (Sina Ahmadi). Tokenisation, lemmatisation, transliteration, normalisation across scripts. The de-facto baseline.
github.com/sinaahmadi/KLPT →Tesseract 5
OCR engine with Kurdish-trained models for both Hawar (Latin) and Sorani (Arabic) scripts.
spaCy + custom
Tokenisation / POS pipelines extended for Kurdish. No first-party model — community-trained, project-specific.
awesome-kurdish
Community-maintained index of Kurdish NLP work — datasets, papers, tools. Search GitHub for awesome-kurdish.
Comparable platforms (prior art)
Common Voice (Mozilla)
The reference for crowdsourced multilingual voice collection. Mature contribution UX, validation flow, donor experience to study.
Dia-Lingle
ETH Zürich's gamified Swiss German dialect collection. Inspiration for a playful, low-friction contribution flow.
CorCenCC
National Corpus of Contemporary Welsh — mobile recording, register tagging, public-facing dashboards. Closest analogue in scope and intent.
standwithkurds.org
Action-focused single-page UX — pattern for clarity over feature density on the public-facing side.
Gamification & quality control
Contribution levels are both an engagement loop and a quality gate. Levels unlock progressively more sensitive tasks — e.g. only Validators and above may vote on other people's contributions — so trust is earned, not assumed.
| Level | Tasks unlocked | Reward |
|---|---|---|
| Newcomer | Profile + region setup | XP |
| Contributor | Free-type, correct-a-word | Badges, micro-tokens |
| Validator | Vote on others' data | Higher token multiplier |
| Expert (Linguist) | Manage OCR datasets, localise UI | Staking rights, governance votes |
The technical acceptance thresholds (CER/WER, vote counts, ensemble agreement) live in Quality & metrics on the workflow page.
Standards: licensing, privacy & ethics
Data licensing
- Corpus releases: CC-BY 4.0.
- Code: Apache 2.0 unless a repo says otherwise.
- Models we fine-tune: same licence as the base model unless we own all the training data.
Privacy & ethics
- GDPR-aligned consent flow on first launch and before any voice capture.
- Right to deletion: hard-delete a user → soft-anonymise their contributions (keep the data for the corpus, drop the link).
- No political bias or content moderation by the team — descriptive only. Hate-speech / spam handled by community + automated filters, never by editorial choice on linguistic content.
- Underrepresented regions are actively targeted by outreach, never algorithmically penalised.
Code style and engineering standards live in Code standards on the workflow page.
Roadmap & milestones
| Phase | Engineering deliverable | Exit criteria |
|---|---|---|
| P0 · Foundations | Repos, CI, infra terraformed, schema v1, auth | Hello-world contribution lands in DB end-to-end |
| P1 · MVP (text only) | Web app: contribute / correct, basic stats (mobile later) | ≥ 100 active testers, 10k contributions across 1 country |
| P2 · Seed data | Import existing corpora; manual cleanup | Bootstrapped corpus ≥ 1M tokens, geo-tagged |
| P3 · Voice | Audio capture, opt-in, ASR ingestion | ≥ 100 hours regionally-tagged audio |
| P4 · LLM analysis | Pattern extraction, dashboards, agent prompts | Live "popular per region" view, < 1h latency |
| P5 · Open API | Public read API, dataset releases | First external integration (TTS / translation) |
Appendix & references
Research sources behind this doc
This document was distilled from a multi-LLM research pass — Claude, Grok, Perplexity and Gemini — cross-checked to reduce single-model bias. Where they disagreed we kept the most cautious option and flagged the disagreement here for future review.
External datasets, tools and comparable platforms are in §5 (Resources & prior art) so they are easy to find before diving into the codebase.
Glossary
- CER / WER — character / word error rate.
- KLPT — Kurdish Language Processing Toolkit.
- NER — named entity recognition.
- POS — part-of-speech tagging.
- MRWL — Max Rightmost White Line segmentation (Kurdish OCR).
- Algorithmic democracy — our four-stage pipeline (collect → extract → select → surface) for finding popular forms without imposing a standard.