# Data governance How Tāne Mahuta handles the sound recordings it learns from and the clips people send it. Written for someone who cares about Indigenous data sovereignty and Māori data governance and wants to check, not trust. Every claim below names the code that makes it true and the test that proves it; the table at the end collects them. Where the project falls short, it says so. Governing body: KiwiData, an Indigenous-governed AI and data sovereignty venture. Lui Hellesoe built the prototype. Version of this document: 2026-10-17 (matches `CONSENT_VERSION` in `tane/constants.py`). ## 1. What the model learns from ### 1.1 Sources The training pool is a manifest, `data/manifest.csv`, one row per source recording, with the columns fixed in `DATA_MANIFEST_SCHEMA.json`. Two sources are allowed: - `xeno-canto`: recordings of the ten target species, shared by their recordists on [xeno-canto.org](https://xeno-canto.org) under Creative Commons licences and downloaded through the xeno-canto v3 API by `pipeline/run.py`. Nearly every recording in the pool comes from here. - `contribution`: clips sent through `POST /contribute` by people who ticked `agree_to_train`. None are in the pool yet. Per recording the manifest keeps the source id, the recordist's name exactly as the source gives it, the licence, the source page URL, species, scientific name, the free-text location and country the recordist chose to publish, date, quality grade, length, sound type, original sample rate, whether another target species is audible in the background, and the split (train / val / test). It does not keep GPS coordinates, uploader accounts, or anything about the people who use the app. The species are ten manu of Aotearoa, listed with Māori names first in `SPECIES.json`. They are taonga. This project does not claim any ownership of the birds' calls; it claims only the right the recordists granted under their licences to copy and learn from their recordings, and it credits them. ### 1.2 Attribution `governance/attribution.py` reads the manifest and writes `ATTRIBUTION.md`: a per-species table of every recording, linked to its source page, with recordist and licence, plus a licence summary and counts. It is generated, never hand-edited, so a recording cannot be in the training pool without being credited. The generator refuses to run if any row has an empty recordist or a licence outside the allowed set. `tests/test_governance_attribution.py` checks that every recording id, URL and recordist in the manifest appears in the output and that no table row has an empty recordist. The committed `ATTRIBUTION.md` names the manifest it was built from in its first line; the same test re-checks the committed file against that manifest. The out-of-distribution evaluation set (`eval/ood/manifest.csv`, fetched by `eval/fetch_ood.py`) is credited the same way with `--extra`, under a heading saying it is never trained on. ### 1.3 Licences, and the non-commercial consequence Accepted licences, fixed in `tane/constants.py` as `ALLOWED_LICENCES`: CC BY, CC BY-SA, CC BY-NC and CC BY-NC-SA, versions 3.0 and 4.0. Because the pool contains BY-NC and BY-NC-SA recordings, everything derived from it is non-commercial: the trained weights in `models/`, the manifest, the windows, any dataset that includes them, and this demo. They are released under CC BY-NC-SA 4.0 (DECISIONS.md 11 and 26). The code is MIT and may be reused commercially with a different, licence-clean dataset. A contributor's default licence is also CC BY-NC-SA 4.0 so that contributed clips sit beside the xeno-canto data without creating a mixed-licence pool. ### 1.4 What is excluded and why | Excluded | Why | Where in code | |---|---|---| | Any ND (no-derivatives) licence | Training a model is a derivative use | `pipeline/xc.py: map_licence` returns `None` for `by-nd` and `by-nc-nd`; `filter_records` drops the record and counts it under `excluded_licence` | | CC0, public-domain marks, unknown or missing licence strings, other CC versions | We only use what we can attribute under terms we understand | same function: anything not matching BY / BY-SA / BY-NC / BY-NC-SA at 3.0 or 4.0 returns `None` | | Recordings with no recordist name | Cannot be credited | `pipeline/xc.py: filter_records` (`excluded_no_recordist`) | | Recordings xeno-canto has redacted | xeno-canto redacts location and other fields for restricted or sensitive species; a record whose `_meta.redacted_fields` is set is skipped entirely rather than used with partial metadata | `pipeline/xc.py: is_redacted`; `eval/fetch_ood.py: choose` applies the same rule to the OOD set | | Rows failing the schema | Every manifest row must validate against `DATA_MANIFEST_SCHEMA.json` before it is written | `pipeline/manifest.py: validate_rows`; checked again by `governance/check_manifest.py` | | Recordings with another target species audible in the background, from val and test | They would make held-out scores dishonest; they stay in train only | `pipeline/manifest.py: assign_split`; asserted by `governance/check_manifest.py` | | Deleted contributions and contributions without `agree_to_train` | See section 3 | `tane/contributions.py: ContributionStore.exclusions` | `governance/check_manifest.py` re-validates the whole manifest after the fact: schema, allowed licences, no background-target rows in val/test, roughly 70/15/15 per species, no recording in two splits, no empty recordist. It exits non-zero on any violation. `tests/test_governance_manifest.py` feeds it a manifest with nine deliberate violations and checks each is reported. ## 2. What the app stores ### 2.1 `POST /predict`: nothing The clip is held in memory and in one temporary file for the length of the request, scored, and the temporary file is unlinked in a `finally` block (`app/main.py: predict`). No audio, no transcript of the result and no IP address is written anywhere by the application. `tests/test_governance_api.py::test_predict_leaves_no_files_behind` snapshots the repository tree, `data/`, the contribution store and the system temp directory before and after a `/predict` call and fails if anything resembling the upload is left. What we do not control: the hosting platform (Hugging Face Spaces) and any reverse proxy in front of it keep their own access logs, which include client IP addresses and request paths. The rate limiter (`app/limits.py`) keeps a per-IP counter in process memory for one minute and never writes it to disk. ### 2.2 `POST /contribute`: only with consent, only these fields Nothing is contributed by default. The frontend only shows the contribute form after a prediction, and the API refuses the request unless the consent record passes `app/main.py: _validate_consent`: - `consent_version` must equal the current version (`2026-10-17`); the text someone agreed to is therefore recorded by date; - `agree_to_store` must be `true` (422 otherwise); - `agree_to_train` and `agree_to_public_release` must be present booleans and may be `false`; - `licence` must be one of the four CC BY family 4.0 licences; - `attribution_name` (max 80 chars, `null` for anonymous), `location_text` (max 120 chars, free text; no GPS is collected or accepted), `recorded_at`, `species_guess`, `notes` (max 500 chars) are optional. What is written, by `tane/contributions.py: ContributionStore.store`, under the directory named by the `TANE_CONTRIBUTIONS_DIR` environment variable (default `data/contributions/`, git-ignored): | File | Content | |---|---| | `.` | the audio bytes exactly as uploaded | | `.json` | the consent record as submitted, plus `token_prefix` (first 8 characters), `created_at`, `status`, `duration_s`, `used_in_training`, `used_in_model_versions`, `published`, and the file's extension, SHA-256 and size | | `ledger.jsonl` | one appended line per event, see 2.3 | Not written: the token itself, the client IP, user agent, cookies, any account. The contributor gets a random token `ctb_` + 32 hex characters shown once; the server keeps only its SHA-256 (as the file name) and its first 8 characters (for the ledger). `tests/test_governance_contributions.py::test_no_file_contains_the_raw_token` stores, trains on and deletes contributions and then greps every file in the store for the token. `tests/test_governance_api.py::test_contribution_lifecycle_stored_then_gone` repeats the check through the HTTP API. ### 2.3 The ledger `ledger.jsonl` in the store directory is append-only: the code has no method that rewrites it, and a fresh process on the same directory only appends (`test_ledger_is_append_only_and_survives_restart`). Each line carries `event`, `at` (UTC), `token_prefix` and event-specific fields: - `store`: consent version, licence, `agree_to_train`, `agree_to_public_release`, audio SHA-256 and size; - `train_use`: `model_version` and the SHA-256 of the manifest used, written by `ContributionStore.mark_train_use` when a model is trained on contributed clips; - `delete`: `was_used_in_training`, the model versions it was in, and `excluded_from_future_manifests: true`. The ledger is the project's record of what was consented to, used, and withdrawn. It contains no audio, no names and no tokens, so it can be published or audited without exposing contributors. ## 3. Deletion, and its limits `DELETE /contributions/{token}` calls `ContributionStore.delete`, which: 1. unlinks the audio file and the consent JSON immediately; 2. writes a small tombstone (`.deleted`: prefix, timestamps, whether it was used) so that later requests get 410 Gone rather than 404, including after a restart; 3. appends the `delete` event to the ledger; 4. returns `was_used_in_training`. What deletion does not do: it does not retrain a model that was already trained on the clip. A model released before the deletion keeps whatever it learned. What it guarantees instead is that the clip cannot enter any future manifest: `ContributionStore.exclusions()` returns the prefixes of every deleted contribution (from the ledger and the tombstones) together with every live contribution whose consent has `agree_to_train: false`, and `trainable_records()` filters them out. The pipeline that builds a manifest with contributions must consult `exclusions()`; today no contributions are in the pool, so that join is a contract for the next phase rather than code that has run. `tests/test_governance_contributions.py::test_delete_after_training_is_recorded_and_excluded` covers the mark, delete, ledger and exclusion path. Contributions marked `agree_to_public_release: false` (the default in the frontend) are never copied to a dataset hub. There is no code path that publishes contributions at all yet; the flag is stored so that one can be built without re-asking. ## 4. The public demo is not durable storage The demo runs on a free Hugging Face Space whose disk is ephemeral: every rebuild, redeploy or restart starts from the Docker image, and `.dockerignore` excludes `data/`. A clip contributed on the public demo lives in the container's `data/contributions/` until the next restart and is then gone, ledger included. The demo therefore cannot honour `agree_to_train` and no model will be trained from demo contributions. The contribute flow exists on the demo so that the consent, token and deletion mechanics can be seen working; anyone who wants their clip in a future training set should say so through the removal and contact route below and we will arrange a durable path. A self-hosted deployment with a persistent volume mounted at `TANE_CONTRIBUTIONS_DIR` keeps everything described in section 2. ## 5. Who governs the data, and how to ask for removal KiwiData governs the training pool, the contribution store and the ledger. In practice today that means Lui Hellesoe runs the pipeline and holds the only copy of any durable contribution store. To remove a contribution: use the token you were shown, either through the Manage panel in the app or `DELETE /contributions/`. The token is the only key; there is no account to recover it from and the server cannot look a clip up by name or date. If the token is lost, contact KiwiData with the approximate time, the species guess and the attribution name you gave, and a human will search the ledger prefixes and consent JSONs; deletion is still by prefix and recorded in the ledger. Contact route: email hellesoelui@gmail.com (Lui Hellesoe, KiwiData) with the subject "Tāne Mahuta removal", or open an issue on the public repository (see README.md) if the request contains nothing private. To ask for a xeno-canto recording to be removed from the pool: the licence allows our use, but if a recordist objects, open an issue naming the XC id; the id is dropped from `data/manifest.csv`, the windows are regenerated, `ATTRIBUTION.md` is regenerated, and the next trained model excludes it. Models already released are not retrained on demand. ## 6. How this sits with Māori data governance principles This section is an honest mapping, not a certification. The project touches taonga species but not personal or whakapapa data about Māori people. Mapped against the Te Mana Raraunga principles: - Rangatiratanga (authority): the training data is third-party CC recordings; KiwiData's authority extends to the manifest, the derived dataset, the model and the contributions, all of which are under its control and non-commercial. - Whakapapa (relationships): every recording keeps its lineage, recordist, source page and licence, and the ledger keeps the lineage of every contribution from consent to use to deletion. - Whanaungatanga (obligations): attribution is generated, not optional; recordists' names are reproduced as they gave them. - Kotahitanga (collective benefit): the species list leads with te reo Māori names, the model and data are shared non-commercially, and the evaluation reports what the model gets wrong rather than only what it gets right. - Manaakitanga (reciprocity, care): `/predict` stores nothing; contributors choose what to allow, can withdraw, and are told plainly what withdrawal cannot undo. - Kaitiakitanga (guardianship): restricted-species records that xeno-canto redacts are left out entirely; no GPS is collected from contributors; location is only the free text they type. Gaps we know about: there has been no consultation with iwi or hapū about the use of these species' calls; the recordists are mostly not Māori and were not asked beyond their licence; the demo cannot keep contributions; and the exclusion list only protects future manifests, not released models. ## 7. Claim to code to test | Claim | Where in code | Which test proves it | |---|---|---| | Only CC BY / BY-SA / BY-NC / BY-NC-SA 3.0 and 4.0 recordings enter the pool; ND, CC0, unknown are refused | `tane/constants.py: ALLOWED_LICENCES`; `pipeline/xc.py: map_licence`, `filter_records`; `governance/check_manifest.py` | `tests/test_governance_manifest.py::test_deliberate_violations_are_each_reported`; `tests/test_eval_harness.py::test_ood_licence_rules_match_pipeline` (pipeline and OOD fetcher agree) | | Redacted (restricted species) records are skipped | `pipeline/xc.py: is_redacted`; `eval/fetch_ood.py: choose` | Agent A's `tests/test_pipeline_*.py` | | Every training recording is credited with recordist, link and licence | `governance/attribution.py` | `tests/test_governance_attribution.py` (synthetic, real manifest when present, committed `ATTRIBUTION.md`) | | No manifest row has an empty recordist | `governance/attribution.py: validate_rows`; `governance/check_manifest.py` | `test_refuses_empty_recordist_and_bad_licence`; `test_deliberate_violations_are_each_reported` | | No held-out recording has another target species in the background; no recording is in two splits | `pipeline/manifest.py: assign_split`; `governance/check_manifest.py` | `test_deliberate_violations_are_each_reported` | | Derived artefacts are non-commercial | stated in README.md, MODEL_CARD.md, ATTRIBUTION.md and here; `governance/attribution.py` prints the notice whenever an NC licence is present | `test_synthetic_manifest_everyone_credited` asserts the notice | | `/predict` leaves no file behind | `app/main.py: predict` (temp file unlinked in `finally`) | `tests/test_governance_api.py::test_predict_leaves_no_files_behind` | | `/contribute` requires `agree_to_store: true` and an allowed licence | `app/main.py: _validate_consent` | `test_contribute_requires_agree_to_store`, `test_contribute_rejects_unknown_licence` | | Stored fields are exactly the consent record plus the metadata listed in 2.2 | `tane/contributions.py: ContributionStore.store` | `test_store_writes_audio_json_and_ledger` (asserts the public field set) | | The raw token is never written to disk | `tane/contributions.py: token_key` (SHA-256 file names), `token_prefix` (8 chars in the ledger) | `test_no_file_contains_the_raw_token`; `test_contribution_lifecycle_stored_then_gone` | | Deletion removes audio and consent immediately and is logged | `ContributionStore.delete` | `test_delete_removes_files_and_logs` | | A deleted contribution answers 410, not 404, even after a restart | tombstone in `ContributionStore._load_full`; `GoneError` is not a `KeyError` | `test_gone_after_delete_is_distinct_from_unknown` | | A clip used in training and then deleted is recorded as such and excluded from future manifests | `ContributionStore.mark_train_use`, `delete`, `exclusions`, `trainable_records` | `test_delete_after_training_is_recorded_and_excluded` | | The ledger is append-only and survives restarts | `ContributionStore._append_ledger` (open in `a` mode, fsync; no rewrite method) | `test_ledger_is_append_only_and_survives_restart` | | The store is safe for one worker | `threading.RLock` in `ContributionStore` | `test_concurrent_stores_do_not_corrupt_ledger` | | Secrets never reach the image, the repo or the logs | `tane/config.py: require_env`; `.gitignore` and `.dockerignore` exclude `.env`; `pipeline/xc.py: redact`, `eval/fetch_ood.py: redact` | `tests/test_phase0.py::test_gitignore_excludes_env`, `test_require_env_fails_loudly`; `tests/test_eval_harness.py::test_ood_query_is_literal_not_percent_encoded` (redaction); REVIEW_CHECKLIST.md commands | | Evaluation never tunes on test | `eval/evaluate.py: run` refuses `--tune-threshold` unless `--split val` | `tests/test_eval_harness.py::test_tune_threshold_requires_val_and_sweeps` | Run them all with `python -m pytest tests/test_governance_*.py tests/test_eval_*.py -q`.