The posting asks for someone who can hold annotation quality, run calibration, and be the language authority for a TTS model. Below: a working calibration console built for this application, the measurement trap that makes most TTS studies read wrong, how I would run the English workstreams, and one proposal about coverage you have not posted.
On a model that is mostly good, raters agree on nearly every clip and kappa still collapses towards zero. Read that as "my annotators are uncalibrated" and you rewrite a rubric that was fine. The console demonstrates it on real arithmetic and names the coefficient that survives it.
The clips with the widest spread are where the rubric has no answer yet. Those are the calibration agenda, and the resolution belongs back in the guideline as a worked example. An edge case that is not written down gets re-argued next month.
A rater sitting consistently a point below consensus needs a threshold conversation. A rater scattered around it needs the task explained again. One overall agreement score cannot tell you which, so the console reports signed bias separately from absolute deviation.
Paste the rating matrix a study produces. It returns agreement three ways, per-rater drift, the clips the rubric is failing on, and a draft written assessment. Built for this application, live, and the arithmetic runs in the browser because ratings are the customer's data.
| Coefficient | When it is the right one | Why it is in there |
|---|---|---|
| Raw agreement | Never on its own | Reported because it is what intuition tracks, and because the gap between it and the chance-corrected numbers is itself the diagnosis. |
| Cohen's κ | Two raters, and you want a per-pair view | Shown as a full pairwise matrix, so you can see which specific pair is failing rather than one pooled number. Weighted, so a 1 against 5 costs more than a 4 against 5. |
| Krippendorff's α | The default for an ordinal MOS scale | Any number of raters, missing cells allowed, takes a distance function. Built from the coincidence matrix so a clip only one person rated contributes nothing rather than being quietly imputed. |
| Gwet's AC1 / AC2 | When one rating category dominates | Its chance term does not assume raters guess from the observed marginals, so it does not collapse when 90% of clips are fine. This is the number to report on a mature model. |
The coefficients are implemented from their definitions, not from a library. Cohen's κ reproduces the textbook 2x2 case exactly. Krippendorff's α reproduces the published three-observer worked example with missing data at 0.691 nominal and 0.811 interval, and agrees with a separately written implementation across 300 randomly generated matrices to within 1e-9. The check ships with the source.
Load the second example in the console. Three raters, twenty clips, a pass/fail rubric, and a model good enough that almost everything passes.
Raw agreement is above 90%. Chance-corrected agreement is close to zero. Read naively, the pool looks broken and the obvious response is to rewrite the guideline and re-run the study.
Kappa and alpha estimate chance agreement from the observed marginals. When one category takes almost all the mass, expected agreement approaches observed agreement and the coefficient collapses regardless of how well the raters are actually doing. Gwet's AC1 reads high on the same data, and it is right.
The better the model gets, the worse a naive agreement metric will make the annotators look. On a model topping a public arena, that is the normal condition rather than an occasional edge case. Two fixes, and they compose: report AC1 alongside alpha and say why, and stratify the sample so hard clips are not swamped by easy ones. Sampling only the clips a model is likely to fail is also cheaper per unit of information.
| From the posting | How I would run it | Detailed in |
|---|---|---|
| Check whether annotator ratings and written feedback align with the rubric | Per-rater signed bias against the consensus of everyone else on shared clips, separated from mean absolute deviation, plus a leave-one-out agreement figure so it is visible whether one person is holding the study down. That is a conversation with a number attached rather than an impression. | §01 |
| Prepare written quality assessments with specific examples and recommendations | The console drafts one: headline finding, the raters who need a threshold conversation and in which direction, the specific clips to play, and the next three actions. It names clips and people, because an assessment that summarises cannot be acted on. | §01 |
| Prep and run calibration meetings | Agenda comes from the widest-spread clips. Every rater states their score and their reason before hearing the others, otherwise the first confident voice sets the anchor and you have measured conformity. The resolution goes into the guideline the same day. | §02 |
| Update or expand the annotation guidelines when you find gaps | Guidelines fail at boundaries, not at definitions. Every edge case resolved in a calibration meeting becomes a worked example with the audio attached. I have written the equivalent for machines: the source-grading vocabulary and completeness rules a crawler of mine is signed off against. | §04 |
| Ground-truth assessments of pronunciation, prosody and naturalness | This is where I am careful about what I can offer, and it is the subject of §05. | §05 |
| Investigate language-specific issues reported by customers, find root causes | Published TTS evaluations keep finding the same failure classes: symbol and notation expansion, names, numbers and units, and anything requiring the model to expand rather than read. A customer report is usually an instance of a class, so the first question is which class, and the second is whether the eval set covers it. Usually it does not, which is the real fix. | §04 |
| Audit datasets for linguistic accuracy, completeness and consistency | Completeness is the one that quietly does damage, and it is the discipline I bring from running a data pipeline: a partial audit reported as a full one produces confident wrong conclusions about everything it did not reach. Coverage gets carried as its own field and stated, never inferred. | §04 |
I am not a linguist and will not claim to be. What a dental degree and three years as a Dental Officer give you is the articulatory half, learned on the anatomy: the hard and soft palate, tongue position, dentition and occlusion, and what actually changes about a sibilant or a dental fricative when any of them changes. When a TTS sample sounds wrong on /s/, /θ/ or /ʃ/, I can say where in the mouth the problem is rather than only that it sounds off.
lil-eleven (x Firecrawl): a voice agent that takes a spoken topic, retrieves live headlines and performs them as generated vocals. Halo (x Cloudflare): a voice companion with persistent memory, driving a custom LLM endpoint from the ElevenLabs Conversational AI SDK with Vectorize for recall and spoken affirmations through the TTS API. VoiceCast (x Replit): a two-host podcast generator using Voice Design to build voices from text descriptions rather than presets, Text-to-Dialogue with per-segment timestamps for native multi-speaker audio, and Text-to-Sound-Effects for the jingle. All three are live. Between them they cover most of a voice API surface, and the useful part for this role is the hours spent judging output I was responsible for: names, numbers, and anything the model expands rather than reads are where it breaks first.
data.prompt20.com is a live index of AI leaderboards, benchmarks and arenas with voice, speech-to-text and music model categories, which is the same class of public evaluation Sonic is measured on. news.prompt20.com aggregates lab news daily, blog.prompt20.com carries long-form technical writing on how the systems work. I built and run all of it. It is the clearest evidence I can offer that following evaluation results closely is a habit rather than an interview posture.
At KIP Protocol I built a failure scorecard and evaluation harnesses over automated output where the hard part was defining what counted as a real failure rather than noise. That is the same measurement problem as a naturalness rubric: the scale is easy, the boundary between adjacent points is the entire job.
I run a public register-verification system whose whole discipline is written rules with the edge cases enumerated, coverage stated rather than assumed, and language careful enough to survive being wrong about a named person. Annotation guidelines are the same artefact for a different reader.
Office hours and calibration syncs are a speaking job. I have mentored a support and community team and, earlier, a team of community health advocates, so telling someone their ratings are drifting without losing them is familiar ground.
The posting asks for native-level English as spoken in the US. My English is native-level and Malaysian: non-rhotic, syllable-timed rather than stress-timed, with vowel mergers General American keeps distinct. For running annotation quality, calibration, rubric development, auditing and reporting on English, none of that matters and I would be strong. For being the final arbiter of whether a General American vowel is right, it does, and you should have someone who is. I would rather say that here than have it surface in a calibration sync.
Sonic ships 42 languages. Four language leads are posted. Independent evaluations in 2026 put Cartesia at the top of the speech arena while noting that quality is concentrated in the top tier and that long-tail prosody is where competitors are judged safer. That gap is the business case for this whole function, and it says the four posted leads are the beginning of a programme.
There is no Mandarin lead posted, and no Malay or Bahasa Indonesia lead. I am native-level in Mandarin, written and spoken, and fluent in Bahasa Malaysia, and Malay and Indonesian are close enough that one lead covers the region meaningfully. Southeast Asia is where voice agents are being deployed hardest right now, and it is a market where a wrong tone is not an accent quibble but a different word.
So the offer is specific: run English quality and calibration, where the console above is the work sample, and take Mandarin and Malay as ground truth, where I actually am the authority. I have not done Mandarin TTS evaluation before and will not pretend otherwise. Tone is where TTS fails in Chinese and I would expect to be tested on it.
Run the current English pool through the agreement view before changing anything. Establish whether low agreement is prevalence, one rater, or a genuine rubric gap, because those have three different fixes and the wrong one costs a re-run.
Take the highest-spread clips into a calibration session, resolve them, and write each resolution into the guideline as a worked example with audio. Re-measure on a fresh sample rather than on the clips we just discussed.
A standing QA report per study in a fixed shape, with the coefficient choice justified once so nobody re-argues it. Then onboarding material for the next annotator, written from the edge cases that actually caused disagreement rather than from first principles.
Role details from the public job posting, 2026, including the other three language-lead postings. Product and evaluation claims from Cartesia's own launch material and from independent 2026 TTS evaluations. The statistics are implemented from their definitions and checked against published worked examples and a separately written implementation. The example rating matrices in the console are synthetic, constructed to show two failure patterns, and no real annotator data appears anywhere.