Workflow & Current State

Who owns what, and when. Three horizons — work moves left → right as it ships. Weekly team sync. Last updated 10 Jun 2026.

Team & roles

Who	Role
Armanc	Project lead — vision, ideas, priorities
Shero	Backend / API
Sidar	Frontend / web

Standing rules & decisions

• R1 · Web first, mobile later. Build and ship web; mobile comes after. All korpusa-kurdi pages must reflect this.
• R2 · Hosting. Frontend → Cloudflare · Backend → Render · Database → Render Postgres (Postgres 16 + PostGIS).
• R3 · Always deployable. Every change auto-deploys to cloud on PR (Cloudflare preview + Render auto-deploy).

Scope — 3 web surfaces

Surface	What	Owner
Explainer site	Project pages (index / docs) — keep current	Armanc Sidar
User web app	Contribution app — the main idea (mockup → real)	Sidar Shero
Admin / researcher panel	Moderate, choose what users see, analyze collected data	Sidar Shero

Now · Week 1 (10–16 Jun) — foundation + finish in-flight

#	Item	Owner	State
13	Deploy kk-api → Render (finish PR #23)	Shero	PR #23
15	Cloudflare deploy kk-web (finish)	Sidar	In progress
14	Shared team accounts & secrets	Armanc	In review
🆕	Provision Render Postgres + connect kk-api	Shero	to file
🆕	DB schema v1 — users + contributions + migration	Shero	to file

Next · Week 2 (17–23 Jun) — connect + single page

#	Item	Owner	State
🆕	API client / contract in kk-web (FE calls BE)	Sidar Shero	to file
🆕	Auto-deploy on PR — CF preview + Render auto-deploy	Sidar Shero	to file
🆕	kk-web single explainer page (light, no deep content)	Sidar	to file
16	kk-web i18n — 8 languages (finish)	Sidar	PR #24

Then · Week 3 (24–30 Jun) — user setup + first flow

#	Item	Owner	State
🆕	User setup / onboarding (anonymous-first) — FE	Sidar	to file
🆕	Users + first contribution endpoint (writes to DB)	Shero	to file
🆕	Minimal user-app screen wired to contribution API	Sidar	to file

Later — backlog

#	Item	Owner	State
—	Mobile app (after web)	?	Rule: later
2	Linguistic research & decisions	Armanc	Backlog
5	Change profile pic	?	Backlog

KorpusaKurdî runs plan-first: nothing gets built before a short, written plan is approved by the team. New people — and every LLM tool — start from the same entry path, then pick up live work.

Boot sequence (people & LLMs)

1. CLAUDE.md                  → project overview (what + why)
2. .ai/README.md              → how to work
3. .ai/sessions/_tracker.md   → where we left off (history)
4. main-plans/_tracker.md     → what we're about to do (intent)
5. active session files       → pick up ongoing work
6. docs.html / workflow.html  → plan & science / how we build

Where things live

Folder	Holds	When
main-plans/	What we're about to do — intent, awaiting approval	future
main-plans/_approved/	Plans the team signed off (proposed → approved → done)	active
.ai/sessions/	What was done — one file per shipped session	past

Principles every change serves

• Descriptive, not prescriptive — observe and surface; never "fix" a dialect.
• Web-first, mobile later — ship web now; mobile follows (contributors are mostly on phones, so it comes next).
• Ethically sourced — consent, right-to-delete, licensed sources only.
• One identifier across ticket title · plan file · plan branch · impl branch.

Visible GitHub writes pause first. Before creating/closing PRs or issues, pushing or deleting branches, or changing settings — say what changes, where it's visible, and whether it's reversible, then wait for a human.

Humans only react on the Plan PR. Once a plan is approved, the Impl PR is a mechanical follow-through reviewed by an LLM tool.

TICKET    lead authors (with LLM help) · standard template
   │      "How to start" section is mandatory
   ▼
PLAN PR   branch: plan/<date>-<slug>  ·  adds ONE file: main-plans/<date>-<slug>.md
   │      reviewed by HUMANS — Armanc · Sidar · Shero
   │      3 approvals → merge → file moves to main-plans/_approved/
   ▼
IMPL PR   branch: impl/<date>-<slug>  ·  the actual work (code/design/devops/content)
   │      reviewed by an LLM tool (e.g. CodeRabbit) · dev self-merges on green
   ▼
CLOSE     session file in .ai/sessions/  ·  plan status: done

Plans live in main-plans/ here, even when the impl targets another repo; approved plans move to main-plans/_approved/.

Plan shape (one screen, every time)

• Understanding — problem, how it serves principles, smallest shippable version, main risk.
• Flow — always a diagram (ASCII or Mermaid), even 5 lines.
• Approach — numbered steps, short, no code.
• Out of scope · Open questions (mark unknowns ?) · Approval (3 humans).

Status values

Status	Meaning	Lives in
proposed	Drafted, awaiting review	main-plans/
approved	All three reviewers signed off	main-plans/_approved/
in_progress	Impl PR is open	main-plans/_approved/
done	Shipped — matching session file	main-plans/_approved/

LLMs are part of the team, with clear boundaries: they help author and review, but humans own every plan approval.

In the workflow

• LLM-agnostic entry — Claude, Cursor, Copilot all boot from .ai/README.md; each tool's own config just points there.
• Authoring — LLMs help draft tickets and plans, but must read context first and mark unknowns with ? instead of guessing.
• Plan PR — reviewed by humans only.
• Impl PR — reviewed by an LLM tool (e.g. CodeRabbit); the developer self-merges on green.

Rules of thumb for the LLM

• Read context before proposing (follow the boot sequence).
• One main-plan per ticket — split the ticket if it needs more.
• No code in plan files; the Approach section is prose + bullets.
• Plans always live in main-plans/, even when the impl targets another repo.
• Never bypass the ticket template — the How to start section is mandatory.

How LLMs process the collected corpus (a different job) → see “AI / LLM data pipeline” below.

• Backend + web scaffolds: done, on main ✅
• Open plan branches: i18n · render-kk-api · kk-webapp-bootstrap
• render-kk-api impl ready in kk-api — not merged

Five logical stages, each independently scalable.

Stage	Component	Tech & responsibilities
1 · Clients	Mobile	React Native / Flutter — contribute UI, voice capture, offline queue
1 · Clients	Web	Next.js dashboard — dashboards, moderation console
2 · Edge	API gateway	Cloudflare / Nginx — rate limit, WAF, TLS
2 · Edge	Auth	Auth0 / Firebase — JWT issuance, OAuth
3 · Services	Core API	FastAPI (Python) or Node — contribution CRUD, voting, moderation queue
3 · Services	Workers	Celery / BullMQ — OCR jobs, LLM jobs, stats refresh
4 · Data	Primary	PostgreSQL + PostGIS
4 · Data	Vector	Pinecone / Weaviate
4 · Data	Object	S3-compatible (audio, scans)
5 · Intelligence	LLM layer	Hosted (OpenAI, Anthropic, xAI) + OSS fallback — pattern extraction, variant clustering
5 · Intelligence	NLP toolkit	KLPT, spaCy, custom — tokenise / lemmatise, POS tagging

Stages are deployed independently. The intelligence layer is async — clients never block on LLM calls.

The target setup is polyrepo under the KorpusaKurdi GitHub org. A monorepo was rejected because mobile and ML pipelines have very different CI shapes.

Today vs. target. Only korpusa-kurdi (plus scaffolded kk-api and kk-web) exists right now — korpusa-kurdi serves as project homepage, development sandbox, and the docs/workflow pages (deployed to korpusa-kurdi.pages.dev). The rest of the table below is the target shape and will be spun up as each workstream starts.

Repo	Status	Purpose	Stack	CI target
korpusa-kurdi	live	Homepage + docs/workflow + ADRs (today also dev sandbox)	HTML / static	Cloudflare Pages
kk-api	scaffolded	Core REST + GraphQL API	Python · FastAPI	Docker / k8s
kk-web	scaffolded	Dashboards & moderation console	Next.js + TS	Vercel
kk-mobile	planned	Cross-platform contribution app	React Native + TS	EAS / Fastlane
kk-workers	planned	OCR, LLM, ETL jobs	Python · Celery	Docker / k8s
kk-nlp	planned	Tokenisation, lemmatisation, POS	Python · KLPT	PyPI publish
kk-corpus	planned	Public dataset releases (CC-BY)	Parquet, JSON-L	HF datasets
kk-infra	planned	Terraform, k8s manifests, Helm	Terraform · Helm	Atlantis

The originally-planned kk-docs role is currently filled by korpusa-kurdi; a dedicated docs repo may split off later if scope grows.

What the live repos run today

kk-api Python — FastAPI + Python 3.12, uv; PostgreSQL 16 + PostGIS, SQLAlchemy 2 + Alembic; ruff + mypy + pytest (80% cov), Docker, GitHub Actions CI. Live: GET /v1/health. Render deploy ready on a branch — not merged.

kk-web TypeScript — pnpm + Turbo monorepo: apps/admin = Next.js 15, apps/app = Expo 52 / React Native (iOS · Android · web), packages/core = shared types + API stub; NativeWind + Tailwind. i18n = not built yet (plan only).

korpusa-kurdi HTML — static site (this page), Cloudflare Pages; holds all plans, tickets and team rules.

Reference code layout (kk-api)

kk-api/
├── app/
│   ├── api/           # FastAPI routers (v1, v2)
│   ├── domain/        # business logic, no I/O
│   ├── infra/         # db, queues, object storage
│   ├── workers/       # celery tasks
│   └── schemas/       # pydantic models
├── migrations/        # alembic
├── tests/             # pytest, >80% coverage gate
├── pyproject.toml
└── Dockerfile

Branching: trunk-based on main; short-lived feature branches; squash-merge with semantic commit messages (feat:, fix:, chore:, docs:).

Web ships first; mobile comes later (rule R1). The notes below are the mobile target for when that phase starts — contributors are overwhelmingly on phones, often in the diaspora or in regions with patchy connectivity, so mobile is the next surface after web.

Stack decision: React Native (TypeScript)

• Cross-platform, large hiring pool, web team can review.
• i18next for UI strings; ICU pluralisation.
• react-native-mmkv for offline contribution queue.
• expo-av for voice capture; opus codec for compact uploads.

Required UX rules

• RTL toggle when the user picks a Sorani-script keyboard. Test both directions.
• Never auto-correct contributions — that violates the descriptive principle.
• Always show the user's region tag at the top of the contribute screen so they can correct it.
• Voice capture is opt-in per session; explicit consent banner before mic access.

Screen inventory (MVP)

• Onboarding — country/city/region picker, dialect self-ID (or "guess for me").
• Contribute — quick type, correct-a-source, record snippet.
• Profile / Stats — contributions, badges, level, regional impact.
• Map — visual contributions per region (uses PostGIS aggregates).

FastAPI for the synchronous edges, Celery (Redis broker) for any task that touches an LLM, OCR, or batch stats.

Sample endpoints

# Submit a free-form contribution
POST /v1/contributions
{
  "type": "text",
  "content": "Ez diçim bazarê",
  "context_concept": "go-to-market.1sg.present",
  "region": { "country": "TR", "city": "Diyarbakir", "locality": "Sur" },
  "dialect_self_id": "kurmanji",
  "source": { "type": "freetype" }
}

# Vote on a candidate variant
POST /v1/contributions/{id}/votes
{ "value": 1 }   # +1 / -1

# Read the live "most popular" form for a concept + region
GET /v1/concepts/{concept_id}/popular?country=TR&city=Diyarbakir

Moderation queue

Every contribution lands in a pending state. A row promotes to accepted when:

• ≥ 3 community up-votes and agent-side spam check passes, or
• An expert moderator (role: linguist) confirms it.

PostgreSQL 16 + PostGIS + pgvector. The schema centres on three things: what was contributed, where it was contributed from, and how confident we are.

Core tables (DDL excerpt)

CREATE TABLE users (
  id          uuid PRIMARY KEY,
  display     text,
  origin_country  char(2),
  origin_city     text,
  origin_locality text,
  geo         geography(Point, 4326),
  dialect_self_id  text,         -- kurmanji|sorani|southern|zaza-gorani|other
  role        text DEFAULT 'contributor',
  created_at  timestamptz DEFAULT now()
);

CREATE TABLE concepts (
  id          uuid PRIMARY KEY,
  key         text UNIQUE,    -- e.g. go-to-market.1sg.present
  gloss_en    text,
  pos         text
);

CREATE TABLE contributions (
  id          uuid PRIMARY KEY,
  user_id     uuid REFERENCES users(id),
  concept_id  uuid REFERENCES concepts(id),
  type        text CHECK (type IN ('text','audio','correction')),
  content     text,
  audio_uri   text,
  source_type text,           -- freetype|rudaw|book|other
  region_country  char(2),
  region_city     text,
  region_locality text,
  geo         geography(Point, 4326),
  dialect_self_id text,
  status      text DEFAULT 'pending',
  embedding   vector(384),    -- pgvector for semantic clustering
  created_at  timestamptz DEFAULT now()
);

CREATE INDEX ON contributions USING GIST (geo);
CREATE INDEX ON contributions USING ivfflat (embedding vector_cosine_ops);

CREATE TABLE votes (
  contribution_id uuid REFERENCES contributions(id),
  user_id     uuid REFERENCES users(id),
  value       smallint CHECK (value IN (-1, 1)),
  PRIMARY KEY (contribution_id, user_id)
);

CREATE MATERIALIZED VIEW popular_per_region AS
SELECT concept_id, region_country, region_city,
       content,
       COUNT(*) AS support,
       RANK() OVER (PARTITION BY concept_id, region_country, region_city
                    ORDER BY COUNT(*) DESC) AS rk
FROM contributions
WHERE status = 'accepted'
GROUP BY 1,2,3,4;

Why geography + vector together? The geography column powers regional dashboards and "near me" tasks; the embedding powers semantic deduplication ("Ez bazarê diçim" vs "Ez diçim bazarê" cluster as variants of one concept).

AI is not the product. It is a processing layer that turns raw, messy crowdsourced data into structured, queryable patterns.

Stages

1. Pre-processing — KLPT (Kurdish Language Processing Toolkit) for tokenisation, stemming, lemmatisation. Critical for Sorani clitics and Kurmanji ergative alignment.
2. Embedding — multilingual sentence transformer; store vectors in pgvector.
3. Variant clustering — semantic dedupe; same concept across surface variants.
4. Pattern extraction (LLM) — given a cluster, the LLM names the morphological/lexical variation pattern.
5. Active learning — agent identifies low-coverage (concept × region) cells and surfaces tasks to fill them.

Sample LLM prompt template (variant naming)

# system
You are a Kurdish-language dialectology assistant. You do NOT propose
a "correct" form. You name the linguistic pattern that distinguishes
the given variants.

# user
Concept: go-to-market.1sg.present
Variants observed:
  - "ez diçim bazarê"   (n=312, region=TR/Diyarbakir)
  - "ez bazarê diçim"   (n=104, region=TR/Mardin)
  - "min diçim bo bazař" (n=88,  region=IQ/Erbil)

Return JSON:
{
  "pattern": "...",          // e.g. SOV vs SVO order
  "axes": ["word_order", "case_marking"],
  "confidence": 0.0-1.0,
  "minority_preserved": true
}

LLMs we use (and why we cross-check)

We treat any single model as a biased lens. Pattern-extraction prompts are run against a small ensemble (e.g. Claude, Grok, an OSS model like Llama or Qwen). Disagreement is logged and surfaced to a linguist for review.

Existing Kurdish NLP we leverage

• KLPT — tokenisation, lemmatisation, transliteration.
• AgaCKNER — ~64,563 annotated tokens, Sorani NER.
• KurdishMT, Hugging Face Kurdish corpora — seed data.
• Mozilla Common Voice (Kurdish) — bootstrap voice data.
• awesome-kurdish GitHub list — staying-current index.

A meaningful portion of "good" Kurdish text is locked in scanned books, Rudaw archives, government documents. We unlock it via a "Correct-a-Source" module.

Pipeline

1. Operator uploads a PDF / image batch into kk-workers.
2. Worker runs Tesseract 5 (fine-tuned for Kurdish scripts; we maintain our own model artefacts).
3. The MRWL (Max Rightmost White Line) segmentation algorithm handles cursive Arabic-script Kurdish, with reported CER ≈ 0.755% on print, much higher on handwriting.
4. Output is staged as candidate contributions with source_type=book.
5. App users see a side-by-side: scan + AI transcription, with three correction granularities — word, sentence, "quick confirm".

Legal: only ingest sources we have rights to (public domain, CC, partner agreements with publishers like Rudaw). Track provenance per row in contributions.source_type + a separate source_licenses table.

We measure data and model quality with standard error-rate metrics so the corpus can be benchmarked alongside other low-resource language datasets.

Character / Word Error Rate

Given S substitutions, D deletions, I insertions and N reference characters (or words):

ER = (S + D + I) / N

Reported separately as CER (character) and WER (word). Track per-script (Latin Hawar vs Perso-Arabic Sorani) — they don't behave the same.

Acceptance thresholds

Stage	Metric	Threshold
OCR (print)	CER	< 2%
OCR (handwritten)	CER	flagged for human review < 8%
Crowd contributions	community votes	≥ 3 net upvotes
LLM pattern extraction	cross-model agreement	≥ 2 / 3 ensemble agreement

Gamification levels that gate sensitive tasks (engagement + quality control) live on the docs page.

Environments

• local — docker-compose, mock LLM (Ollama), seeded Postgres.
• dev — auto-deployed from main on every merge.
• staging — promoted by tag v*-rc.*; full PII-safe seed data.
• prod — promoted by tag v*; behind change-management.

CI/CD

• GitHub Actions per repo. Reusable workflows in kk-infra/.github.
• Required checks before merge: lint, unit, integration, security scan (Trivy), license scan.
• Coverage gate: 80% minimum per repo, 90% for kk-api/domain.
• Container images pushed to GHCR; signed with cosign.

Observability

• OpenTelemetry traces → Tempo / Grafana.
• Structured JSON logs → Loki.
• Metrics → Prometheus; dashboards in Grafana (per-region contribution rate is a first-class SLI).
• Sentry for client + server errors.

Secrets

Never commit. Local: .env via direnv. Cloud: 1Password Connect → external-secrets-operator → k8s. Rotation policy: 90 days; LLM API keys: 30 days.

Project tracking lives in GitHub Projects (organisation-level). Two boards:

• Engineering — sprint board, two-week cadence, columns: Backlog → Ready → In progress → Review → Done.
• Roadmap — quarterly view, milestones grouped by phase (see the Roadmap on the docs page).

Definition of Ready

• Acceptance criteria written.
• Linked to a milestone.
• Estimated (S / M / L).
• Has area + type labels.

Definition of Done

• Tests green; coverage threshold respected.
• Docs updated where behaviour changed (this file or a sibling).
• Telemetry / metric added if a new flow.
• Linguistic sanity-check signed off if touching kk-nlp or contribution flow.
• Migrations applied to dev; rollback documented.

Code style

• Python: ruff + black; type-checked with mypy --strict in domain/.
• TS: ESLint + Prettier; strict: true in tsconfig.
• SQL: migrations only via alembic; no autogenerate straight to prod.

Data licensing, privacy & ethics standards (descriptive-only principle) live on the docs page.

Tick as you go. The whole list should be done by end of week 2.

Day 1 — accounts & orientation

• GitHub access to the korpusakurdi org · 2FA on
• 1Password vault provisioned
• Slack / Discord channels joined: #eng, #nlp, #ops, #standup
• Read sections 2–4 of the docs page (linguistic context, scientific approach, data analysis)
• 30-minute pairing session with your buddy

Week 1 — first commits

• Local dev environment up (docker compose up) — seeded Postgres, mock LLM
• Run the test suite for your area; everything green
• Pick a good-first-issue and ship it through the full PR loop
• Submit a contribution end-to-end on dev from the mobile app (or Postman)
• Open one improvement PR to these pages

Month 1 — own a slice

• Lead at least one feature from issue → merged → deployed to dev
• Write or co-author one ADR or RFC
• Pair with the linguistic reviewer on a contribution-flow change
• Be on rotation as the primary reviewer for your area
• Demo your work in the bi-weekly all-hands