Telling speakers apart in the browser
Minutes labels who said each line of a meeting without sending audio to a diarization service. A small voice model runs in a web worker instead.
Minutes records a meeting in a browser tab and transcribes it as people talk. The live transcript comes from the browser's Web Speech API, which returns text and nothing else. It has no idea who was speaking.
The usual answer is a diarization service: upload the audio, get back "speaker A, speaker B". But the reason Minutes exists is to avoid adding a third party to a private conversation. So the labels are worked out on the device.
The pipeline
- Tap the microphone. An AudioWorklet copies the raw samples at 16 kHz, the rate the voice model expects, while MediaRecorder saves the recording as usual.
- Cut a clip per line. Each time speech recognition finishes a line, everything heard since the previous line belongs to it. Only the last 20 seconds are kept.
- Drop the silence. The clip is split into 20 ms frames and the quiet ones are removed. If less than one second of speech is left, the clip says too little about a voice, and the line takes the previous speaker.
- Turn the voice into numbers. A web worker runs a WeSpeaker ResNet34 model through transformers.js, as an 8-bit ONNX file on WebAssembly. It returns a vector that describes how the voice sounds.
- Match or start a new speaker. The vector is compared with the centre of each voice heard so far.
The last step is the whole "who is this" decision, and it is short:
assign(voice: number[]): number {
let best = -1, score = -1;
this.voices.forEach((v, i) => {
const s = cosine(v.centre, voice);
if (s > score) { score = s; best = i; }
});
if (best < 0 || (score < SAME_VOICE && this.voices.length < MAX_SPEAKERS)) {
this.voices.push({ centre: voice, lines: 1 });
return this.voices.length;
}
const v = this.voices[best];
v.centre = v.centre.map((c, i) => (c * v.lines + voice[i]) / (v.lines + 1));
v.lines += 1;
return best + 1;
}
If the closest voice is similar enough (cosine similarity of 0.45 or more), the line joins it and that voice's centre moves a little toward the new sample. Otherwise a new speaker starts, up to eight.
Why a web worker
The model takes a noticeable moment per clip. On the main thread that would freeze the transcript and the level meter. In a worker, the line appears straight away and its label arrives when it is ready.
Where it goes wrong
- Lines follow the recogniser, not the speakers. If two people talk inside one recognised line, the line gets one label.
- Short replies. "Yes" and "okay" are under a second of speech, so they inherit the previous speaker, who is often the wrong one.
- Similar voices and bad microphones. The 0.45 threshold was tuned by hand for this model. Too low merges people; too high splits one person in two.
So the page treats labels as a first guess. Clicking a name on any line moves it to the next speaker, and each speaker can be renamed, which rewrites the transcript that goes to the notes.
What leaves the device
For this step, nothing. The model file is downloaded once from Hugging Face and cached by the browser. The audio is turned into a list of numbers locally, and only the finished label is kept. It also only runs in browsers that have live speech recognition, which today means Chrome and Edge.
You can see the result on the sample meeting, where every line of the transcript carries a speaker.