The history of
speech recognition

Speech recognition at a glance

The first speech recognition system was invented in 1952

The first speech recognizer was built at Bell Labs in 1952 and it understood exactly ten words: the digits zero through nine.

Audrey, the Automatic Digit Recognizer

K. H. Davis, R. Biddulph and S. Balashek built Audrey from analog circuits that filled a six-foot relay rack. A speaker said a digit into a telephone handset, the machine measured the vowel formants, and a lamp lit beside the digit it heard. Tuned to its designers' voices it scored 97 to 99 percent; with strangers it fell apart. Audrey never left the laboratory, because a human operator dialing numbers was cheaper, but it settled the question of whether a machine could hear at all.

IBM's Shoebox and the 1960s

In 1962 IBM demonstrated Shoebox at the Seattle World's Fair. It recognized 16 spoken words, the ten digits and six arithmetic commands, and drove an adding machine with them. Through the decade laboratories in the United States, Britain, Japan and the Soviet Union built recognizers for vowels, consonants and small word lists, and in 1968 Taras Vintsyuk described dynamic time warping, the alignment method that let a machine compare speech spoken at different speeds. By 1969 the field had ambition but no theory: John Pierce of Bell Labs compared it to schemes for turning water into gasoline, and the lab stopped funding it for years.

DARPA's Speech Understanding Research program took vocabularies past 1,000 words

Between 1971 and 1976 DARPA funded the largest speech recognition effort to date, and Carnegie Mellon's Harpy reached its goal of 1,000 words.

The five-year goal

The Speech Understanding Research program asked for a system that could understand connected speech from several speakers over a 1,000-word vocabulary with under 10 percent error, and it paid Carnegie Mellon, BBN, SRI and others to compete. Three systems reached the finish line: Hearsay-II and Harpy at Carnegie Mellon and HWIM at BBN. Harpy, finished in 1976, understood 1,011 words, roughly a three-year-old's vocabulary, and was the only one to meet the target. Its trick was a search that pruned unlikely word sequences early instead of weighing every possibility, an idea still at the heart of every decoder.

The statistical turn

The same years produced the idea that beat everything else for the next 40 years. At Carnegie Mellon, James Baker's 1975 thesis described the DRAGON system, which treated speech as a hidden Markov model and let probabilities, not hand-written rules, decide which words were said. At IBM, Fred Jelinek's group built the same approach into a research dictation system and added n-gram language models, so the recognizer could guess the next word from the last two. Bell Labs, meanwhile, moved from one voice to many and built recognizers that worked across speakers, which telephone applications required.

Hidden Markov models made speech recognition statistical during the 1980s

The 1980s replaced pattern matching with probability, and vocabularies grew from a few hundred words to 20,000 in a decade.

How a hidden Markov model hears

A hidden Markov model does not try to match a sound to a template. It asks which sequence of speech sounds most probably produced the audio it received, then which sequence of words most probably produced those sounds. Training on recorded speech sets the probabilities, so more data means a better recognizer, a property that would matter enormously once the internet arrived. Combined with n-gram language models, the HMM became the standard architecture in every research lab and stayed there until deep learning displaced it in 2012.

Tangora, Sphinx and the first products

IBM's Tangora recognized a 20,000-word vocabulary by the mid-1980s, though it needed a pause between words and training on each speaker. In 1988 Kai-Fu Lee's Sphinx system at Carnegie Mellon was the first to combine a large vocabulary, continuous speech and speaker independence in one recognizer. Dragon Systems, founded by James and Janet Baker in 1982, began selling recognition software for personal computers. Voice recognition reached consumers for the first time too: Worlds of Wonder's Julie doll answered a handful of spoken phrases in 1987, and telephone companies trialed voice dialing. The toys were crude, but they put the phrase "voice recognition" into ordinary homes.

The 1990s put speech recognition on personal computers

Faster processors turned dictation from a laboratory demonstration into shrink-wrapped software, and a shared benchmark made progress easy to measure.

Dragon Dictate to NaturallySpeaking

Dragon Dictate shipped in 1990 for about $9,000. It was the first speech recognition product a member of the public could buy, and it demanded a pause after every word. In 1997 Dragon NaturallySpeaking removed the pause: it recognized continuous speech at around 100 words a minute on a home PC, and IBM answered with ViaVoice the same year. For the first time a lawyer, a doctor or a writer could talk to a computer and get usable text, provided they trained it and corrected it.

Phone trees and the first benchmarks

In 1996 BellSouth launched VAL, a dial-in voice portal that answered spoken questions, and every automated phone system you have argued with since descends from it. Behind the scenes, DARPA and the National Institute of Standards and Technology ran yearly evaluations on shared recordings such as the Switchboard telephone corpus, and word error rate became the number every lab reported. From then on, the history of speech recognition is largely the history of that number falling.

The 2000s moved speech recognition into the cloud

Accuracy stalled near 80 percent for most of the decade, until Google moved recognition off the device and onto its servers.

The plateau

By 2001 the best systems transcribed dictated speech at roughly 80 percent accuracy, and the following years brought refinements rather than leaps. Desktop dictation stayed a niche, the phone-tree systems stayed frustrating, and research funding thinned. The hidden Markov model was near its ceiling, and the models that would replace it did not yet have the computing power or the data they needed.

Google Voice Search

Google's GOOG-411 directory service, launched in 2007, was a way to collect millions of spoken queries. In November 2008 the company released Google Voice Search as an iPhone app, and speech recognition changed shape: the phone only recorded, and the recognition ran in Google's data centers on models far larger than any device could hold. Google's English system was trained on 230 billion words of search queries, so it could predict what people were likely to say, and every new query fed the next model. Recognition became a service rather than a program, the form it still takes.

Deep learning cut error rates by more than half in the 2010s

Between 2011 and 2017 the word error rate on conversational telephone speech fell from the low teens to about 5 percent, and voice assistants moved into the home.

Siri and the assistants

Apple shipped Siri with the iPhone 4S in October 2011, and for the first time a mainstream phone came with a voice assistant behind the home button. Amazon followed with Alexa on the Echo in 2014 and Google with Google Home in 2016. The assistants mattered less for their recognition, which ran in the cloud much like Google Voice Search, than for the habit they formed: hundreds of millions of people started talking to machines every day, and every utterance became training data.

The neural network step

In 2012 researchers from Microsoft, Google, IBM and the University of Toronto, Geoffrey Hinton among them, published a joint paper showing that deep neural networks trained on the acoustic side of the problem cut error rates by up to a third compared with the Gaussian mixture models that HMM systems had used for 25 years. Baidu's Deep Speech went further in 2014 and trained a single recurrent network end to end, from audio to characters, with no pronunciation dictionary and no hand-built pipeline. Recognition became a deep learning problem, and the labs with the most GPUs and data pulled ahead.

The race to human parity

The Switchboard benchmark, a set of recorded telephone conversations, became the scoreboard. In October 2016 Microsoft reported a 5.9 percent word error rate, the same as the professional transcribers it hired to work the same calls, and called it human parity. IBM answered with 5.5 percent in March 2017, Google announced 4.9 percent on its own test sets that May, and in August 2017 Microsoft reached 5.1 percent against a more careful measurement of the human baseline. The chart below, from Mary Meeker's 2017 Internet Trends report, shows Google's word accuracy crossing the 95 percent mark the industry treated as the human threshold. The same year the Transformer architecture was published for machine translation, and within three years it had taken over speech as well.

Whisper and the Transformer made transcription a commodity

Self-supervised training and one open model turned accurate multilingual transcription from a Big Tech capability into something any product can offer.

Learning from unlabeled audio

The bottleneck of the 2010s was labeled data: every training hour needed a human transcript. In 2020 Facebook AI's wav2vec 2.0 learned speech representations from raw, untranscribed audio first and needed as little as ten minutes of labeled speech to fine-tune a usable recognizer. Google's Conformer, published the same year, combined convolution with Transformer attention and set new records on the standard benchmarks. Together they made the Transformer the default architecture for speech and cut the cost of adding a new language.

Whisper

In September 2022 OpenAI released Whisper, trained on 680,000 hours of audio collected from the web, covering nearly 100 languages, with the model weights published for anyone to run. It was not the most accurate system on every benchmark, but it was robust to accents, background noise and technical vocabulary in a way earlier academic models were not, and it was free. Within months it was inside thousands of products, and the question for transcription companies shifted from whether they could recognize speech to what they built around the transcript.

What that means for you today

A one-hour recording uploaded to Sonix comes back as a transcript in about 5 to 6 minutes, with speaker labels, timestamps and 99% accuracy on clear audio, in any of 54+ languages. Sonix was founded in 2017, the year of the human parity race, and has ridden every step since: the models improve each year, and the work now goes into the editor, the summaries, the translations and the exports that make a transcript useful. The seven decades above are why a first transcript costs nothing today: your first 30 minutes are free.

Automatic language identification has its own history

Knowing which language is being spoken is a separate problem from knowing which words, and it took its own 50 years to solve.

From phoneme statistics to i-vectors

The first language identification experiments in the 1970s tried to tell languages apart by the statistics of their sounds, on the theory that each language uses a different mix of phonemes at different rates. In the 1990s that became the phonotactic approach: run a phoneme recognizer, then ask which language's phoneme patterns fit best. The National Institute of Standards and Technology began formal Language Recognition Evaluations in 1996, and the benchmark culture that drove speech recognition drove language identification too. In the 2010s the i-vector, a compact fixed-length summary of a recording borrowed from speaker recognition, produced systems that could name a language from a few seconds of audio.

Built into the model

Modern multilingual recognizers fold the two problems together. Whisper and its successors predict the language as the first token of the transcript, so identification is a by-product of recognition rather than a separate step. That is what lets one product handle 54+ languages without a separate recognizer for each, and why the language you speak has stopped being a limit on what you can transcribe.

What comes next: voice becomes the interface

The technology needed for voice applications is now cheap and accurate, and the question has moved from whether machines can hear to what they should do with what they hear.

Voice-first products

Voice-first devices, from smart speakers to earbuds and cars, have made talking to a machine unremarkable, and the accuracy that took 70 years to reach is now an assumption rather than a feature. Language is the one interface almost every person already has, which is why a small improvement in recognition reaches more people than a new screen ever could. The companies that build on voice early tend to hold that ground, as the mobile-first companies of the 2010s did.

Beyond the transcript

The transcript itself has become the raw material. Large language models summarize meetings, answer questions about an interview, translate a lecture and draft the follow-up email from it, and they do those things well only because the transcript underneath is accurate. At Sonix that is the work: recognition is the foundation laid by everyone named above, and the product is what you do with the words once they exist.