# Model card: Tāne Mahuta bird-sound classifier

Model version 0.2.0, trained 2026-10-17, evaluated on the held-out test split of manifest `2ac230b89d88dadda5d4b356114c66a8d56d04da3e0ffb9c772be4f1702cfded`. Every number here is copied from `eval/REPORT.md` / `eval/results.json` (test) and `eval/exp_mnv3_slow_TUNE_val.md` (validation sweep), produced by `eval/evaluate.py`. Re-running those commands reproduces them.

## Model details

- Developer: Lui Hellesoe, for KiwiData (Indigenous-governed AI and data sovereignty venture).
- Task: 10-way classification of 3 s audio windows into ten New Zealand bird species, plus an explicit "unknown" outcome.
- Architecture: ImageNet-pretrained MobileNetV3-Small (torchvision) with its first convolution collapsed to one input channel and a 10-way head, fine-tuned on log-mel spectrograms (`mobilenet_v3_small-logmel`; DECISIONS.md 19, details in `docs/decisions/agent-b.md`). 1.53 M parameters, 6 MB `state_dict`.
- Input: mono audio resampled to 32 kHz; 3 s windows (96,000 samples); 128-band log-mel spectrogram, n_fft 1024, hop 320, 150 to 15,000 Hz, dB relative to the window peak floored at -80 dB, then per-window zero-mean unit-variance. The values are read from `models/metadata.json` at inference so training and serving cannot drift.
- Output: softmax over the ten `species_id` values in `SPECIES.json` order (`models/labels.json` must equal that order or the model refuses to load).
- Training: AdamW, lr 2e-4, weight decay 1e-2, batch 64, mixup alpha 0.4, label smoothing 0.05, SpecAugment, class-balanced sampling, seed 20261017, 40 epochs run of 60 requested, best epoch 25 by validation window macro-F1, on a local RTX 4090. Inference is CPU-only.
- Unknown threshold: 0.45 on the aggregate top-1 confidence (`tane/constants.py: UNKNOWN_THRESHOLD`; DECISIONS.md 103).
- Interface: `tane/model.py: BirdModel`, contract in `MODEL_CONTRACT.md`. Weights are CC BY-NC-SA 4.0 (see Licence); code is MIT.

## Species

Ten manu, Māori name first, in model label order (`SPECIES.json`):

| species_id | Māori | English | Scientific |
|---|---|---|---|
| tui | tūī | Tūī | *Prosthemadera novaeseelandiae* |
| kereru | kererū | New Zealand pigeon | *Hemiphaga novaeseelandiae* |
| ruru | ruru | Morepork | *Ninox novaeseelandiae* |
| piwakawaka | pīwakawaka | New Zealand fantail | *Rhipidura fuliginosa* |
| korimako | korimako | Bellbird | *Anthornis melanura* |
| riroriro | riroriro | Grey warbler | *Gerygone igata* |
| kaka | kākā | New Zealand kākā | *Nestor meridionalis* |
| pukeko | pūkeko | Pūkeko | *Porphyrio melanotus* |
| tauhou | tauhou | Silvereye | *Zosterops lateralis* |
| kotare | kōtare | Sacred kingfisher | *Todiramphus sanctus* |

## Intended use

- A prototype for curiosity: helping a person in Aotearoa put a likely name to a bird they can hear, from a phone recording or an uploaded clip, with the top three guesses and their confidence, a spectrogram, and a plain "not confident, possibly none of these" when the model is unsure. Treat a confident answer as a good lead, not an identification.
- Demonstrating, in code that can be read and tested, how a small audio model can be built on openly licensed recordings with attribution, consent-based contribution and deletion.
- Education and non-commercial research.

## Out-of-scope use

- Any commercial use. The training pool includes CC BY-NC and BY-NC-SA recordings, so the weights, manifest, windows and any derived dataset are non-commercial (CC BY-NC-SA 4.0). This is a licence condition, not a preference.
- Survey, monitoring, compliance, consenting or conservation decisions. With 59% recording-level accuracy, a third of clips returned as unknown, no kererū test data at all, and no field validation, the output is not evidence of presence or absence of a species.
- Species other than the ten above. There is no "other bird" class; a bird the model has never heard is either forced onto one of the ten or returned as unknown, and loud non-bird noise is not reliably rejected (see Out-of-distribution).
- Identifying people, places or anything other than bird sound. The model sees 3 s log-mel windows and nothing else.
- Use outside Aotearoa New Zealand: see Limitations.

## Training data

- Source: xeno-canto recordings of the ten species, downloaded by `pipeline/run.py`. The query ladder starts with country New Zealand and quality better than C, drops the quality filter for species with under 40 usable recordings, and drops the country filter if still under 40. That last step fired for four species, so the pool includes recordings of the same species made outside New Zealand: ruru 4 of 41, pūkeko 17 of 30, tauhou 64 of 98, kōtare 35 of 55 (120 recordings in total, 101 of them from Australia, the rest from Vanuatu, Papua New Guinea, Indonesia, the United States and France).
- Pool: 693 recordings, 17,390 three-second windows (13,780 train, 1,963 val, 1,647 test), 90 distinct recordist names (89 after merging one case variant). Quality grades A 365, B 279, C 38, D 5, E 6.
- Licences: CC BY-NC-SA 4.0 (646), CC BY-NC-SA 3.0 (34), CC BY 4.0 (12), CC BY-NC 4.0 (1). ND, CC0 and unknown licences refused; every exclusion in this run was an ND licence. No redacted (restricted species) record was encountered. No recording without a recordist name.
- Manifest: `data/manifest.csv`, schema `DATA_MANIFEST_SCHEMA.json`, SHA-256 `2ac230b89d88dadda5d4b356114c66a8d56d04da3e0ffb9c772be4f1702cfded`, identical to `models/metadata.json: manifest_sha256`.
- Split: 70/15/15 per species by a seeded hash of `recording_id` (seed 20261017); no recording's windows appear in two splits. Recordings with another target species audible in the background are forced to train and never scored. Recordings per split (train / val / test): tūī 181/27/28, kererū 16/1/0, ruru 26/6/9, pīwakawaka 32/6/5, korimako 64/9/14, riroriro 37/3/3, kākā 32/3/8, pūkeko 23/4/3, tauhou 71/16/11, kōtare 36/9/10. Train windows per species range from 199 (kererū) to 7,134 (tūī).
- Windows: non-overlapping 3 s chunks for training, silent windows (RMS under -50 dBFS) dropped. Contributions through the app: none in the pool.
- Attribution: every recording is listed in `ATTRIBUTION.md`, generated by `governance/attribution.py`. Governance: `DATA_GOVERNANCE.md`.

## How a prediction is made

1. The clip is decoded with ffmpeg to mono 32 kHz float, truncated to 60 s.
2. It is cut into 3 s windows with a 1.5 s hop (50% overlap), the last partial window zero-padded if it holds at least 1 s of audio.
3. Windows with RMS under -50 dBFS are marked silent and skipped.
4. Each remaining window is scored; softmax gives a probability per species.
5. The aggregate per species is the mean of the per-window probabilities over non-silent windows; the top three aggregate values are returned, each with the full species card.
6. `unknown` is true with reason `all_silent` if no window was scored, or `low_confidence` if the top aggregate value is below 0.45.

The API (`API_CONTRACT.md`) also returns every window's top three so the interface can show where in the clip each guess came from.

## Evaluation method

`eval/evaluate.py` scores the held-out test split at two levels:

- window level: every 3 s test window on its own through `BirdModel.score_windows`, the same feature and forward path the API uses;
- recording level: the windows of each test recording combined with `tane.model.aggregate_windows`, the same rule as the API, so the recording score is what a person uploading that whole recording would see.

An `unknown` prediction counts as wrong for accuracy and as a missed detection (false negative) for the true species; it is a false positive for nobody. Macro-F1 averages over species with test support. The unknown rate is also measured on an out-of-distribution set fetched by `eval/fetch_ood.py`: 8 xeno-canto recordings each of Eurasian blackbird and Australian magpie from New Zealand, plus 5 synthetic noise clips (white, pink, brown, 2 to 8 kHz band, 50 Hz hum with clicks).

Model selection used validation only (DECISIONS.md 102). The test split was scored for every candidate only after the promoted model and the threshold had been chosen.

## Metrics

Promoted model 0.2.0, threshold 0.45, test split: 91 recordings, 1,647 windows (all scored, none silent), 9 species with test support. Kererū has no test recordings and one validation recording (DECISIONS.md 100): its test precision, recall and F1 are undefined, not zero, and the rows below show it as such. The window-level table shows 15 windows of other species predicted as kererū.

### Recording level (what a user sees for a whole clip)

- Accuracy with unknown counted as wrong: **59.3%** (54 of 91)
- Macro-F1 over the 9 species with support: **0.684**
- Unknown rate: **34.1%** (31 of 91)
- Of the 60 recordings on which the model committed to a species, 54 were right (90.0%).

| Species | Precision | Recall | F1 | Test recordings | Predicted |
|---|---|---|---|---|---|
| tūī | 0.917 | 0.786 | 0.846 | 28 | 24 |
| kererū | undefined | undefined | undefined | 0 | 0 |
| ruru | 0.833 | 0.556 | 0.667 | 9 | 6 |
| pīwakawaka | 0.500 | 0.200 | 0.286 | 5 | 2 |
| korimako | 1.000 | 0.286 | 0.444 | 14 | 4 |
| riroriro | 0.750 | 1.000 | 0.857 | 3 | 4 |
| kākā | 0.800 | 0.500 | 0.615 | 8 | 5 |
| pūkeko | 1.000 | 1.000 | 1.000 | 3 | 3 |
| tauhou | 1.000 | 0.636 | 0.778 | 11 | 7 |
| kōtare | 1.000 | 0.500 | 0.667 | 10 | 5 |

Confusion matrix (rows true, columns predicted):

| true \ predicted | tui | kereru | ruru | piwakawaka | korimako | riroriro | kaka | pukeko | tauhou | kotare | unknown |
|---|---|---|---|---|---|---|---|---|---|---|---|
| tui | 22 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 |
| kereru | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| ruru | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 |
| piwakawaka | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 3 |
| korimako | 2 | 0 | 0 | 0 | 4 | 0 | 0 | 0 | 0 | 0 | 8 |
| riroriro | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 |
| kaka | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0 | 0 | 0 | 4 |
| pukeko | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 |
| tauhou | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 7 | 0 | 2 |
| kotare | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | 4 |

Only 6 of 91 recordings were given the wrong species; 31 were given no species. Korimako is the clearest case: 8 of 14 bellbird recordings came back unknown and 2 were called tūī, whose song overlaps it.

### Window level (each 3 s window on its own)

- Accuracy with unknown counted as wrong: **60.0%**
- Macro-F1 over the 9 species with support: **0.656**
- Unknown rate: **25.7%**

| Species | Precision | Recall | F1 | Test windows | Predicted |
|---|---|---|---|---|---|
| tūī | 0.888 | 0.695 | 0.780 | 742 | 581 |
| kererū | undefined | undefined | undefined | 0 | 15 |
| ruru | 0.768 | 0.589 | 0.667 | 90 | 69 |
| pīwakawaka | 0.490 | 0.369 | 0.421 | 65 | 49 |
| korimako | 0.807 | 0.464 | 0.589 | 343 | 197 |
| riroriro | 0.559 | 0.702 | 0.623 | 47 | 59 |
| kākā | 0.741 | 0.645 | 0.690 | 62 | 54 |
| pūkeko | 1.000 | 0.667 | 0.800 | 15 | 10 |
| tauhou | 0.696 | 0.467 | 0.559 | 167 | 112 |
| kōtare | 0.962 | 0.647 | 0.773 | 116 | 78 |

The window-level confusion matrix is in `eval/REPORT.md` under "Window level". Its largest off-diagonal cells are korimako windows called tūī (54 of 343) and tūī windows called korimako (34 of 742); the rest of the confusion is spread thinly. Tūī is 45% of all test windows, so window-level accuracy leans on it.

### Threshold sweep (validation split, recording level, promoted model)

From `eval/exp_mnv3_slow_TUNE_val.md` (84 validation recordings; the OOD set is the same 21 clips as below). Raising the threshold trades in-domain accuracy for rejection of out-of-distribution clips; 0.45 maximises the mean of the two and was adopted (DECISIONS.md 103).

| Threshold | Val accuracy (unknown wrong) | Val unknown rate | OOD unknown rate | Mean |
|---|---|---|---|---|
| 0.30 | 79.5% | 8.4% | 19.1% | 0.493 |
| 0.35 | 73.5% | 19.3% | 33.3% | 0.534 |
| 0.40 | 66.3% | 31.3% | 52.4% | 0.593 |
| **0.45** | **65.1%** | **32.5%** | **61.9%** | **0.635** |
| 0.50 | 54.2% | 45.8% | 66.7% | 0.604 |
| 0.55 | 44.6% | 55.4% | 71.4% | 0.580 |
| 0.60 | 34.9% | 65.1% | 71.4% | 0.532 |
| 0.70 | 21.7% | 78.3% | 81.0% | 0.513 |
| 0.90 | 1.2% | 98.8% | 95.2% | 0.482 |

The plain "accuracy with unknown counted wrong" objective is non-increasing in the threshold and would always pick 0.30; it is reported in the sweep file but was not used.

### Out-of-distribution (test, threshold 0.45)

21 clips, 13 returned unknown (61.9%).

| Group | Clips | Unknown | Unknown rate | Mean top-1 confidence | Mistaken for |
|---|---|---|---|---|---|
| Eurasian blackbird (NZ recordings) | 8 | 7 | 87.5% | 0.404 | pīwakawaka 1 |
| Australian magpie (NZ recordings) | 8 | 6 | 75.0% | 0.456 | kōtare 1, riroriro 1 |
| Synthetic noise | 5 | 0 | 0.0% | 0.637 | ruru 4, tauhou 1 |

The noise result is a plain failure. White, pink and brown noise, band-limited "rain" and a 50 Hz hum with clicks were each confidently (mean top-1 0.64) called ruru or tauhou. Only true silence (RMS under -50 dBFS) is rejected by the silence rule; loud broadband noise with no bird in it is not rejected by the model. A recording of wind, traffic or a waterfall will get a species name. The two non-target birds are mostly rejected, which is the behaviour the threshold was tuned for.

### Candidate comparison

Three runs were trained on the same split and compared on validation window macro-F1 before any test scoring (DECISIONS.md 102). Test numbers below are at threshold 0.50 for all three because that was the harness default when the candidates were scored; the promoted model's final numbers at 0.45 are the ones above. The baseline's test report was overwritten by the promoted run, so only its validation and OOD figures survive.

| Run | Validation window macro-F1 (selection) | Test recording accuracy / macro-F1 / unknown at 0.50 | OOD unknown at 0.50 (of 21) |
|---|---|---|---|
| MobileNetV3-Small, lr 5e-4 (baseline) | 0.450 | not preserved | 16 (noise 4/5, blackbird 6/8, magpie 6/8) |
| small-cnn-logmel | 0.482 | 50.5% / 0.517 / 41.8% | 10 (noise 2/5, blackbird 5/8, magpie 3/8) |
| MobileNetV3-Small, lr 2e-4, mixup 0.4 (promoted, 0.2.0) | 0.483 | 48.4% / 0.585 / 46.2% | 14 (noise 1/5, blackbird 7/8, magpie 6/8) |

Validation window macro-F1 for the promoted model (0.483) is well below its test recording macro-F1 (0.684): per-window scores are noisier than the aggregate over a whole recording, and the validation set includes the single kererū recording, which scores zero at window level and pulls the macro average down.

## Character of the model

It is a cautious model. When it commits to a species it is usually right (recording-level precision is 0.75 or better for every species but pīwakawaka, 1.0 for four of them), and most of its errors are abstentions: a third of in-domain test recordings come back unknown, and recall for korimako, pīwakawaka, kākā and kōtare is 0.5 or lower. A user will see "not confident" often and a wrong name rarely. The cost is that the unknown signal is overloaded: it means "quiet or unfamiliar bird" far more often than "not one of the ten", and it does not fire for loud noise.

## Limitations and bias

- Recording-quality bias. 93% of the pool is xeno-canto quality A or B: close, clean recordings made with directional microphones. A phone held up in the wind at a distance is a different distribution; expect lower confidence and more unknowns on real phone clips than the test split suggests. No codec or phone-microphone augmentation was used in training.
- Non-NZ recordings. For ruru, pūkeko, tauhou and kōtare the country filter was relaxed, and 120 recordings (17% of the pool, mostly Australian) are of the same species recorded elsewhere. Regional dialects of those four may be represented by overseas birds rather than New Zealand ones; tauhou in particular is 65% overseas recordings.
- Kererū. 17 recordings, 9 of them with another target species in the background and so train-only; the seeded split left 1 in validation and 0 in test. Nothing in this card says how well kererū is recognised. The 15 test windows of other species that were called kererū suggest it is at least sometimes predicted.
- Small test sets. Riroriro and pūkeko have 3 test recordings each, pīwakawaka 5: their per-species numbers move by a third with one recording. Confidence intervals were not computed.
- Recordist and place bias. New Zealand xeno-canto uploads come from a small number of recordists and the places they visit, so sanctuaries and accessible bush are over-represented.
- Ten species only. Common introduced birds (blackbird, song thrush, starling, magpie, sparrow) are not in the label set. The OOD check measures how often two of them are rejected; it does not make the model able to name them, and 3 of 16 were given a native species' name.
- Noise is not rejected. See Out-of-distribution: loud broadband noise is confidently labelled ruru or tauhou.
- Label noise. Species labels come from the recordist. Background birds the recordist did not list can be present; those listed as another target species are kept out of val and test but remain in train (162 recordings).
- Window independence. Aggregation is a plain mean over windows; a long clip dominated by one bird will drown out a second species, and a clip with mostly empty windows will dilute a short call towards unknown.
- The unknown threshold is one number on the aggregate top-1, chosen on 84 validation recordings and 21 OOD clips. The softmax is not calibrated; 0.45 does not mean a 45% chance of being right.
- No consultation with iwi or hapū has taken place about the use of these species' calls; see DATA_GOVERNANCE.md section 6.

## Future work

Not done in this pass, in rough order of expected value:

1. Pretrained bird embeddings (BirdNET or Perch) as the front end, with a small head trained on the same windows: both have seen far more bird sound than 693 recordings and should raise recall without more data.
2. Codec and phone augmentation at training time: re-encode windows through Opus and AAC at phone bitrates, add wind and traffic noise, band-limit, to close the gap between xeno-canto quality A and a phone on a track.
3. Guaranteed validation and test minimums for small species: at least two background-free val and test recordings per species before splitting, recorded as a split change in DECISIONS.md because it changes every number here.
4. More kererū and pūkeko recordings, from New Zealand, including through the app's contribution path once a durable store exists.
5. A noise or "no bird" class, or an energy-based rejection, so broadband noise is returned as unknown.
6. Calibration (temperature scaling on validation) so the reported confidence means something, and bootstrap confidence intervals on the per-species numbers.

## Licence

Model weights, `data/manifest.csv`, derived windows and any dataset built from them: CC BY-NC-SA 4.0, because the training pool contains CC BY-NC and BY-NC-SA recordings. Code: MIT. Each training recording keeps its own licence, listed in `ATTRIBUTION.md`.

## Contact and changes

Governance questions and removal requests: see DATA_GOVERNANCE.md section 5. Changes to any constant in this card go through `tane/constants.py` and DECISIONS.md. Reproduce the numbers with:

```
TANE_MODEL=real python eval/evaluate.py --data data --models models --split test --ood eval/ood --out eval/results.json --report eval/REPORT.md
TANE_MODEL=real python eval/evaluate.py --data data --models models --split val --tune-threshold --ood eval/ood
```