The best audio annotation company for one project may be the wrong operating model for another. A team seeking finished, reviewed labels is making a different purchase from a team seeking software for its own annotators. This guide helps you compare the proposed work and the evidence behind it.
Start with one brief, inspect an export, and agree a representative pilot. The comparison below is a buyer checklist, not a vendor league table. It connects the practical questions to a free editable provider scorecard and specification worksheet you can use with your shortlist.
Decide which audio labels the project needs
Start with the downstream task and the files you need to receive. A transcript, word or utterance timestamps, speaker turns, non-speech events, and intent or emotion labels are separate requirements. Do not assume that a quote for “audio annotation” includes every layer or the same review process. Our audio annotation guide explains those label types.
Write down the source audio's languages, locales, channels and difficult conditions, then identify the required labels and export schema. Specify how the guideline handles overlapping speech, uncertain words, speaker changes and unknown labels. Ask each provider to confirm which parts it can undertake and which remain your team's responsibility. These decisions define the comparison before a provider name or headline price does.
Compare managed services, platforms and tools by responsibility
| Option | What to establish | Who must own the missing work? |
|---|---|---|
| Managed annotation service | A written outcome covering the agreed labeling, review and delivery scope. | Confirm who supplies the guideline, annotators, reviewers, adjudication and acceptance decision. |
| Self-serve annotation platform | Software access and the workflows, workforce or review features included in the actual plan. | Confirm which operations your team must configure, staff and monitor; do not assume software includes a managed workforce. |
| Standalone labeling tool | The interface, export and integration you will operate. | Identify who will supply qualified people, instructions, review and delivery management outside the tool. |
Some offers combine these models. Compare the written division of responsibilities rather than treating the category as a quality rating. Spirelight can assess a managed audio-annotation brief; it is not an instant self-serve labeling platform or annotator marketplace.
How to compare audio annotation companies
Give shortlisted providers the same versioned brief and the same evaluation instructions. Share recordings only when you have the necessary permission and an approved transfer process. Use the table below to record evidence, gaps and assumptions. A polished answer without a project-specific artifact is still an open question.
| Decision | Ask the provider for | Check before accepting the answer |
|---|---|---|
| Labels and guideline | Supported layers, definitions, edge-case rules and a marked-up example. | Are required labels, overlap and unknown cases explicitly covered? |
| Language and domain fit | Screening method and reviewer requirements for the actual locale and vocabulary. | Does the evidence address your variant, rather than only the parent-language name? |
| Export compatibility | A sample export in your proposed schema. | Can your pipeline read its field names, timestamp units, speaker IDs and missing values? |
| Quality evidence | The metric, reference-set or agreement method, sample design and results. | Do the method and sample cover each required label and difficult slice? |
| Acceptance and corrections | Review ownership, rejection triggers, correction scope and versioning. | What happens to the batch when a check fails, and who decides when it passes? |
| Source, rights and handling | Applicable source records, permissions, responsibilities, access and retention terms. | Can the people responsible for privacy, security and legal review inspect the proposed workflow? |
| Capacity and schedule | A plan tied to the actual volume, staffing and review requirements. | Are dependencies and limits written down, including changes to the guideline? |
| Comparable price | The pricing unit, inclusions, assumptions and treatment of rejected or corrected items. | Are both proposals pricing the same accepted deliverable? |
Use the existing speech data provider scorecard to keep those answers in one comparison. Its editable CSV includes quality, acceptance, rights and commercial criteria. Set the weights and any mandatory gates before scoring proposals. An overall score should not conceal an unresolved requirement that your project treats as essential.
Test the proposal with one representative pilot
Agree the pilot and review method before production. Include the conditions that matter to your use case, such as overlapping speakers or difficult channels, rather than testing only the easiest files. Record which guideline version and source files were used, who reviewed the labels, and which checks must pass. The pilot's size and thresholds should come from the project's risk and intended use, not a universal percentage in a guide.
Consider an invented comparison: two proposals both promise “timestamped speaker labels.” One export uses milliseconds and another uses seconds. Both conventions can be valid, but an unrecorded unit can break your import. A timestamp of 120 milliseconds means 0.12 seconds. The comparison is incomplete until the schema states the unit and your test import confirms it.
Apply the same discipline to speaker identity and overlap. Ask whether a speaker ID is stable within one recording or across the intended delivery, and how simultaneous speech is represented. Then inspect an agreed example in the exported files. This example describes buyer checks, not a real vendor result, a sample Spirelight delivery or a promise that every schema is supported.
Read quality metrics with their definitions
A quality percentage is incomplete without its metric, reference or agreement method, sample size, label definitions and treatment of difficult cases. Ask for results by relevant layer and slice, plus the rule for what happens when a threshold is missed. Transcript wording, timestamp boundaries and speaker attribution need checks suited to their different outputs.
Inter-annotator agreement describes consistency among annotators. Even the calculation needs to be named: HumanSignal's agreement documentation distinguishes pairwise and consensus methods, which can return different scores for the same annotations. Do not compare unexplained percentages as if they measured the same thing.
If a proposal uses a reference or gold set, ask how its answers were reviewed, which guideline they follow and how disagreements are adjudicated. If it uses batch sampling, ask how items and slices are selected, which results trigger broader review, and what is corrected before acceptance. Keep these requirements in the written plan. The speaker diarization guide discusses the separate problem of assigning speech to speakers.
Make the quote and handling requirements comparable
Ask each provider to itemize the same scope: audio volume and condition, label layers, locale and personnel requirements, review, export, security, rights, schedule and acceptance. Confirm whether the price unit is an audio hour, an item, a batch or another agreed unit. State whether the quote includes guideline preparation, a pilot, corrections and later changes. This guide does not set a market rate or recommend a universal review budget.
For sensitive or regulated material, specify the data type, access restrictions, confidentiality, retention, transfer requirements and any domain-personnel requirements before sharing files. Have the responsible legal and security reviewers assess the proposed process. A provider category or an unsupported compliance label cannot substitute for that project review.
Turn the comparison into a written annotation brief
Use the speech dataset specification worksheet to capture source audio, labels, formats, rights and acceptance requirements. Keep only the fields relevant to annotation and mark unresolved details as unknown. Send the same saved version to each supplier so that later scope changes remain visible.
If you want Spirelight to assess the managed work, the audio annotation service page explains the scope and quote process. Supported labels, languages, reviewers, QA, security, capacity, schedule, rights and price are confirmed for the particular brief before production. Catalogue speech datasets and software-only annotation are separate buying decisions.