An automatic transcript can make recorded speech easier to search and review. For an under-resourced language, however, preparing the material for a model can require as much care as choosing the model itself. Recordings, transcription conventions, speaker coverage and evaluation all shape what the resulting system can reliably do.
That makes the pipeline a useful unit of explanation: not just audio in and text out, but a sequence of documented decisions.
Research context
ASR pipeline for low-resourced languages: A case study on Pomak, by Chara Tsoukala, Kosmas Kritsis, Ioannis Douros, Athanasios Katsamanis, Nikolaos Kokkas, Vasileios Arampatzakis, Vasileios Sevetlidis, Stella Markantonatou and George Pavlidis, appeared at FieldMatters on 6 May 2023, pages 40–45. The work presents data processing and ASR training for Pomak, using a language-family-based approach. This article develops an independent evaluation example. Its English toy sentences are not Pomak corpus samples or reported model outputs.
Begin with a transcription policy
A transcript is not a transparent copy of sound. It reflects choices about spelling, punctuation, filled pauses, word boundaries and uncertain passages. Those choices may be especially important when a writing convention is not uniformly established across the available material.
For a training corpus, write the convention down and retain the relationship between each audio segment and its transcript. If an annotator changes a word boundary or orthographic form, the change should be traceable. Otherwise a model can be penalized for following one valid convention while the reference uses another.
Linguistic expertise belongs in this loop. A fluent-looking model output should not override a community-informed transcription decision merely because it resembles a more highly resourced language.
Split speakers before counting utterances
If the intended evaluation concerns new speakers, speaker identity should define the split. Randomly distributing utterances from the same person across training and test sets answers a different question. Recording session, microphone and background can also supply cues that make a test easier than the intended deployment.
A useful corpus manifest links an utterance ID, speaker ID, recording source, segment boundaries, transcription version and split. Keep access and usage rights attached to the material. A public paper does not, by itself, grant permission to redistribute its underlying recordings or voices.
Word error rate is an alignment calculation
Word error rate compares a hypothesis with a reference using substitutions, deletions and insertions:
where N is the number of reference words. It can exceed 100% if there are enough insertions. An empty reference needs a defined handling policy because this ratio has no ordinary denominator in that case.
The companion code computes the minimum word-level edit distance with dynamic programming. The short examples are chosen to make the errors inspectable:
reference = "the old bridge is closed".split()
hypothesis_a = "the old bridge is open".split()
hypothesis_b = "the bridge is closed".split()
# Each differs from the five-word reference by one edit: WER = 0.2.
Both scores are 20%, but they do not have the same consequence. The first changes whether the bridge is open; the second omits a descriptive word. An aggregate score does not identify that semantic difference.
Normalization changes the evaluation
Lowercasing, punctuation removal and number normalization can make a comparison appropriate for a particular task. They can also hide distinctions if applied indiscriminately. Report the normalization policy and preserve the original strings for inspection.
The tutorial code uses whitespace-separated tokens and no language-specific normalization. That is sufficient to demonstrate edit distance; it is not a recommended tokenization policy for Pomak. A real evaluation should use the conventions justified for the language and the transcription task.
For corpus-level WER, sum the edit counts across utterances and divide by the total reference-word count. Averaging utterance-level WER gives each utterance equal weight, so a short utterance can influence the result as much as a long one. Both summaries can be informative, but they answer different aggregation questions.
Build an error review that helps improve the corpus
I would examine errors by speaker, recording condition, word type and transcription uncertainty. A concentration of apparent mistakes around one spelling convention may call for reference review rather than an acoustic-model change. Errors concentrated in a noisy session may suggest an acquisition or segmentation problem.
For an assistive transcription workflow, measure the human correction effort as well as recognition scores. The goal is not to declare the recording “solved,” but to make reliable language documentation easier while preserving expert and community oversight.
Complete reproducible example
This standalone example uses fictional data or prescribed mathematical objects; it is not a reproduction of the source paper’s experiments.
"""Whitespace-tokenized toy sentences; not a Pomak evaluation pipeline."""
import json
def edits(ref,hyp):
previous=list(range(len(hyp)+1))
for i,r in enumerate(ref,1):
current=[i]
for j,h in enumerate(hyp,1):
current.append(min(previous[j]+1,current[j-1]+1,previous[j-1]+(r!=h)))
previous=current
return previous[-1]
def wer(reference,hypothesis):
r,h=reference.split(),hypothesis.split()
if not r: raise ValueError("Empty reference needs an explicit evaluation policy")
return edits(r,h)/len(r)
r="the old bridge is closed"
a,b="the old bridge is open","the bridge is closed"
assert wer(r,a)==.2 and wer(r,b)==.2
assert wer("one","one two three four")==3.
assert edits([],[])==0
print(json.dumps({"reference":r,"hypothesis_a":a,"hypothesis_b":b,"wer_a":wer(r,a),"wer_b":wer(r,b),"insertion_example_wer":wer("one","one two three four")},indent=2))