Continuous speech processing is a fundamentally different problem from uploading a single audio file and transcribing it all at once. While the user is speaking, audio keeps arriving. The system has to segment an ongoing, incomplete stream into workable units, run inference without falling behind, and stitch each result to what came before.
The naive approach — just feed microphone input continuously into an STT model — doesn't hold up in practice. Models typically expect fixed-length audio chunks, human speech doesn't break cleanly at sentence boundaries, and silence, noise, and mid-sentence pauses are woven throughout. The core challenge in continuous speech processing is keeping that flow from falling apart.
This post covers the basic architecture for processing continuous speech sequentially. If the first post in this series was about "making incoming audio intelligible," this one is about "deciding how to slice a continuous stream and hand it off to the model."
Continuous speech processing is a pipeline: audio input → preprocessing → feature extraction → recognition model → text output, in that order. Source: NVIDIA Technical Blog
Continuous Speech Processing in One Pass
Continuous speech processing generally follows this flow:
- Continuously receive short audio frames from a microphone or audio stream.
- Accumulate frames in a buffer.
- Use VAD to distinguish speech from silence.
- Open a segment when speech starts; close it when speech is judged to have ended.
- Send the closed segment — or an in-progress one — to the STT model.
- Manage model output as either provisional or final results.
- Combine finalized text with surrounding context to form complete utterances.
The key terms here are frame, buffer, segment, provisional result, and final result. Understand these five and the overall structure of continuous speech processing becomes clear.
File STT vs. Streaming STT Are Different Problems
In file-based STT, the recording is already complete. The total audio duration is known, and the model can see the full context at once, which leads to more stable output.
In streaming STT, the input is still arriving. You don't know what the user will say next, and you can't tell whether the audio you just received is the middle or the end of a sentence. Google Cloud Speech-to-Text's streaming recognition is built around exactly this model: the client sends audio in small chunks continuously, and the server returns both interim and final results.
This distinction means streaming STT has to manage latency alongside accuracy. Waiting longer can improve accuracy, but the user experience suffers. Committing too early risks finalizing a result before enough of the sentence has arrived.
Both file and streaming STT convert speech to text, but in streaming, chunk-by-chunk intermediate processing and result updates are critical. Source: MobiDev
Frame: The Smallest Unit of Processing
Audio from a microphone is typically sliced into very short chunks for processing. These are called frames — commonly 10 ms, 20 ms, or 30 ms long.
A frame is not a human-readable unit. It's not a word or a phrase; it's a sliver of sound far smaller than any meaningful utterance. But it's the right granularity for a computer: you can measure energy per frame, decide whether it contains speech, and choose whether to buffer it or discard it.
This is the first thing that trips up newcomers. The STT model doesn't receive "sentences." In practice, very short audio chunks arrive first, and those chunks are assembled into speech segments before anything gets transcribed.
Buffer: Temporary Holding Memory
Because continuous speech never arrives all at once, the system needs intermediate storage. That's the buffer — a space that holds incoming audio frames temporarily.
There are three reasons a buffer is necessary:
- You need to preserve the brief audio just before speech begins.
- Determining whether speech has ended requires watching a few more frames.
- Input length needs to be adjusted before passing it to the STT model — neither too short nor too long.
For example, if a user starts saying "오늘 날씨" ("today's weather") and the VAD detects the first syllable a moment late, the front of the word gets clipped. To avoid this, implementations commonly attach a short pre-roll — a few frames buffered before the speech detection event — to the beginning of each segment.
The opposite problem also exists. Cutting the segment the instant speech seems to stop is also wrong, because people pause mid-sentence. Treating a brief pause as an utterance boundary can split "오늘" and "날씨 어때" into separate requests. That's why systems add an end-padding or hangover window and wait a bit before closing the segment.
The buffer holds short audio frames temporarily, reducing both front-end clipping and premature end-of-speech detection.
VAD: The Gatekeeper That Distinguishes Speech from Silence
VAD stands for Voice Activity Detection. It decides, frame by frame, whether the incoming audio is likely speech or background noise and silence. In a continuous speech processing pipeline, VAD acts as the gatekeeper.
Without VAD, the system would send everything to the STT model: keyboard clicks, air conditioning hum, silence, ambient noise — all of it. That drives up cost, increases latency, and raises the likelihood of spurious output.
Two widely used implementations are WebRTC VAD, a classical VAD designed for real-time communication, and Silero VAD, a neural-network-based open-source option. Their internals differ, but their role in the pipeline is the same: identifying where speech starts and stops.
VAD output is not ground truth. It signals that a given window is likely speech. Aggressively cutting on VAD decisions alone will break utterances. Real systems layer additional rules on top of VAD: buffer windows, minimum and maximum segment durations, and silence duration thresholds.
VAD classifies each frame as speech or non-speech, producing the trigger signals that open and close segments. Source: Introduction to Speech Processing
Segment: The Speech Span Handed Off to the STT Model
A segment is a single span of speech sent to the STT model. The most important design decision in continuous speech processing is how to open and close segments.
The basic logic is straightforward:
- VAD detects speech → open a segment.
- Keep appending incoming frames to the segment.
- Silence persists beyond a threshold → close the segment.
- Send the closed segment to the STT model.
Real speech is messier. Users think while talking, breathe, self-correct, and repeat words. Segment policy has to thread a needle — too short, and context is lost; too long, and results come back late, hurting responsiveness in real-time applications.
Deictic and anaphoric words like "that," "earlier," and "here" are nearly meaningless without surrounding context, which argues for longer segments. But real-time captioning needs short segments so text appears promptly. A good segmentation policy is always a tradeoff between accuracy and latency — short segments for systems where responsiveness matters, longer segments with post-processing for systems where transcript accuracy is paramount.
Separating Provisional and Final Results
Streaming STT output is typically split into two categories: provisional (interim) results that may still change, and final results that are stable.
For example, when the user has said "오늘 날씨가" ("today's weather"), the model might emit that as an interim result. When the user continues with "어때" ("how is it"), the final output becomes "오늘 날씨가 어때." Word choices and spacing may shift in the interim.
Without understanding this, streaming STT output looks unstable. It isn't a bug — it's expected behavior. The model is showing its best estimate given the speech received so far.
A common UI pattern is to display interim results in a lighter color and final results in normal text. On the server side, it's safer to keep interim results out of the database and only write finalized results to the conversation history.
In streaming STT, partial results keep updating until a sentence boundary is detected, at which point a final result is committed.
Streaming STT keeps provisional results — which can still change — separate from final results, where utterance boundaries have been determined.
Overlap: Handling Words That Straddle Chunk Boundaries
Slicing continuous audio into fixed-length chunks creates a boundary problem: a word can be split across two consecutive chunks. If "transformer" is divided between the end of one chunk and the beginning of the next, the model may not recognize it correctly.
Overlap addresses this. When constructing the next chunk, you include a short portion of the previous chunk's tail. For example, you might process audio in 5-second windows with a 0.5-second overlap on each side.
Overlap reduces boundary losses but introduces duplicate text. A post-processing step is needed to compare the overlapping portions of consecutive results and deduplicate them.
This matters most when processing long audio files in smaller chunks. It's also relevant in streaming when the model can't handle long contexts reliably, or when the architecture requires fresh inference at regular intervals — both scenarios call for overlap and deduplication.
With overlap between chunks, a word at a boundary can be re-recognized in the next chunk.
Overlap includes the tail of the previous chunk in the next one, reducing word loss at chunk boundaries.
How Models Understand Sequence
STT model architectures have evolved considerably. Older systems typically combined a separate acoustic model, pronunciation lexicon, and language model. Deep learning-based end-to-end ASR later became dominant, predicting text directly from audio features.
One concept that comes up often in continuous speech processing is CTC. Connectionist Temporal Classification enables training when the alignment between input frames and output characters isn't known in advance — which is nearly always the case in speech, since labeling exactly which frame corresponds to which character is impractical.
Models like Whisper use an encoder-decoder Transformer trained on large-scale weakly supervised data. The Whisper paper describes an approach that handles diverse speech processing tasks under a unified sequence-to-sequence formulation. Even when using a model this capable for continuous processing, you still need to decide how to chunk the input audio and how to carry context across chunks.
In other words, stronger models don't eliminate the pipeline. Continuous speech processing involves two distinct problems: managing audio flow outside the model, and converting speech to text inside it. Both have to be designed.
The Sequential Processing Pipeline as a State Machine
The flow of continuous speech processing maps naturally to a state machine:
- IDLE: No speech detected yet.
- SPEECH: Speech is arriving.
- MAYBE_END: Silence detected — waiting to confirm it's a real endpoint.
- FINALIZE: Close the segment and send it to the STT model.
The state machine exists to avoid snap judgments. Silence in one frame doesn't immediately end the segment — you wait to see if it persists. Speech in one frame doesn't immediately open a segment — you confirm some minimum continuity first.
A simple pseudocode sketch:
state = "IDLE"
segment = []
pre_roll = RingBuffer(max_ms=300)
for frame in audio_stream:
is_speech = vad(frame)
pre_roll.push(frame)
if state == "IDLE":
if is_speech:
segment = pre_roll.dump() + [frame]
state = "SPEECH"
elif state == "SPEECH":
segment.append(frame)
if not is_speech:
silence_ms = frame.duration_ms
state = "MAYBE_END"
elif state == "MAYBE_END":
segment.append(frame)
if is_speech:
state = "SPEECH"
else:
silence_ms += frame.duration_ms
if silence_ms >= 700:
send_to_stt(segment)
segment = []
state = "IDLE"
This is illustrative, not production code. The key point is that frames are processed one at a time, state transitions happen based on VAD decisions, and segments are opened and closed accordingly.
State management in continuous STT chains together VAD decisions, filtering, and segment open/close logic in sequence. Source: VOCAL Technologies
Queues Are Essential in Real-Time Conversational Systems
When STT is connected to a chatbot or voice agent, another problem emerges. The user keeps talking, STT produces text, the LLM generates a response, and TTS converts that response back to audio. None of these stages finish at the same rate.
That's why queues are needed between each stage. By separating concerns into an audio input queue, an STT task queue, a text event queue, and a response generation queue, a temporary slowdown in one stage won't immediately bring down the whole system.
Without queues, audio frames can be dropped when the STT model is busy, or state can become corrupted when new audio arrives while the LLM is still generating a response. Queues preserve ordering, absorb throughput differences, and provide control over retries and cancellations.
In a real-time system, the goal isn't just to make every stage faster. What matters more is deciding how much each stage can tolerate when things slow down, which data can be dropped, and which results must be preserved.
Barge-In and Overlapping Speech
In voice agents, users can start speaking again while the system is still delivering a response. This is called barge-in. When the user interrupts, TTS output must stop and the new user utterance must be prioritized.
At this point, the STT pipeline goes beyond simply converting speech to text — it becomes a signal for controlling conversational state. When VAD detects new speech, the system can stop the current TTS output, cancel the in-progress LLM generation, and start a new segment.
In environments where multiple people speak, such as meetings or call centers, speaker diarization also becomes important. pyannote.audio provides an audio processing pipeline that includes speaker diarization. Diarization is needed in systems that must record not just what was said, but who said it.
That said, adding diarization increases latency and architectural complexity. For real-time conversational agents, the typical safe approach is to first stabilize VAD and single-speaker STT, then layer in diarization only when the use case actually requires it.
Post-Processing: Turning Raw Text into Coherent Sentences
The text returned by an STT model is rarely ready to be used as final output. It may be missing punctuation, have awkward spacing, or contain overlap with the previous segment. Post-processing is therefore necessary.
Common post-processing steps include:
- Removing duplicate text between adjacent segments
- Correcting punctuation and sentence boundaries
- Normalizing numbers, dates, and units
- Masking profanity or personally identifiable information
- Merging interim and final results
- Deciding how to chunk output for conversation history storage
One thing beginners often miss: STT output is not immediately a "conversation message." Real-time STT results are an event stream that can be revised multiple times. A separate step is needed to convert that event stream into human-readable messages.
Summary: Continuous Speech Processing Is About Segmenting Well and Waiting Wisely
Continuous speech processing isn't a problem you solve with a single model. It's an end-to-end pipeline: buffering incoming frames from the microphone, using VAD to detect speech regions, constructing appropriate segments, distinguishing interim from final results, and reducing boundary artifacts through overlap and post-processing.
Ordering is the central concern. Audio arrives in chronological order, and the system must preserve that order throughout. At the same time, the system needs to make a judgment call — has the user stopped speaking, paused briefly, or are they about to continue?
That's why continuous speech processing is less about recognizing speech quickly and more about waiting long enough to get it right without finalizing too late. Viewed through that lens, it becomes clear why each component — buffers, VAD, segments, partial/final results, and queues — exists and how they all fit together.