> ## Documentation Index
> Fetch the complete documentation index at: https://smallestai-ff1e543d.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Pulse (Realtime)
> The Pulse STT WebSocket API provides real-time speech-to-text transcription capabilities with streaming audio input. This API uses WebSocket to deliver transcription results as audio is processed, enabling low-latency transcription without waiting for the entire audio file to upload. Perfect for live transcription, voice assistants, and real-time communication systems that require immediate speech recognition. Supports multiple languages, word-level timestamps, sentence-level timestamps (utterances), PII and PCI redaction, cumulative transcripts, and more advanced features.
## Query Parameters
The WebSocket connection accepts the following query parameters:
### Audio Configuration
| Parameter | Type | Default | Description |
| ------------- | ------ | ---------- | ------------------------------------------------------------------------------------------- |
| `encoding` | string | `linear16` | Audio encoding format. Options: `linear16`, `linear32`, `alaw`, `mulaw`, `opus`, `ogg_opus` |
| `sample_rate` | string | `16000` | Audio sample rate in Hz. Options: `8000`, `16000`, `22050`, `24000`, `44100`, `48000` |
### Language & Detection
| Parameter | Type | Default | Description |
| ---------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `language` | string | `en` | Language code for transcription. Use `multi` for automatic language detection. Supported: `it`, `es`, `en`, `pt`, `hi`, `de`, `fr`, `uk`, `ru`, `kn`, `ml`, `pl`, `mr`, `gu`, `cs`, `sk`, `te`, `or`, `nl`, `bn`, `lv`, `et`, `ro`, `pa`, `fi`, `sv`, `bg`, `ta`, `hu`, `da`, `lt`, `mt`, `multi` |
### Feature Flags
| Parameter | Type | Default | Description |
| --------------------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `word_timestamps` | string | `true` | Include word-level timestamps in transcription. Options: `true`, `false` |
| `full_transcript` | string | `false` | Include cumulative transcript received till now in responses where `is_final` is `true`. Options: `true`, `false` |
| `sentence_timestamps` | string | `false` | Include sentence-level timestamps (utterances) in transcription. Options: `true`, `false` |
| `redact_pii` | string | `false` | Redact personally identifiable information (name, surname, address). Options: `true`, `false` |
| `redact_pci` | string | `false` | Redact payment card information (credit card, CVV, zip, account number). Options: `true`, `false` |
| `numerals` | string | `auto` | "Convert spoken numerals into digit form (e.g., 'twenty five' to '25') and `auto` enables automatic detection based on context. Options: `true`, `false`, `auto` |
| `diarize` | string | `false` | Enable speaker diarization to identify and label different speakers in the audio. When enabled, each word in the transcription includes `speaker` (integer ID) and `speaker_confidence` (float 0-1) fields. Options: `true`, `false` |
| `keywords` | string | — | Comma-separated list of words/phrases to boost, each optionally followed by `:INTENSIFIER` (e.g. `NVIDIA:5,Jensen`). Intensifier defaults to `1.0` if omitted. Max 100 keywords per session. See [Keyword Boosting](/v4.0.0/content/speech-to-text/features/keyword-boosting) |
### Webhook Configuration
## Connection Flow
### Example Connection URL
```javascript theme={null}
const url = new URL("wss://api.smallest.ai/waves/v1/pulse/get_text");
url.searchParams.append("language", "en");
url.searchParams.append("encoding", "linear16");
url.searchParams.append("sample_rate", "16000");
url.searchParams.append("word_timestamps", "true");
url.searchParams.append("full_transcript", "true");
url.searchParams.append("sentence_timestamps", "true");
url.searchParams.append("redact_pii", "true");
url.searchParams.append("redact_pci", "true");
url.searchParams.append("numerals", "true");
url.searchParams.append("diarize", "true");
url.searchParams.append("keywords", "NVIDIA:5,Jensen:4");
const ws = new WebSocket(url.toString(), {
headers: {
Authorization: `Bearer ${API_KEY}`,
},
});
```
## Input Messages
### Audio Data (Binary)
Send raw audio bytes as binary WebSocket messages:
```javascript theme={null}
const audioChunk = new Uint8Array(4096);
ws.send(audioChunk);
```
### End Signal (JSON)
Signal the end of audio stream. This is used to flush the transcription and receive the final response with `is_last=true`:
```json theme={null}
{
"type": "finalize"
}
```
## Response Format
The server responds with JSON messages containing transcription results:
```json theme={null}
{
"session_id": "sess_12345abcde",
"transcript": "Hello, how are you?",
"is_final": true,
"is_last": false,
"language": "en"
}
```
### Response Fields
| Field | Type | Description |
| ------------ | ------- | ---------------------------------------------------------------------------------------- |
| `session_id` | string | Unique identifier for the transcription session |
| `transcript` | string | Partial or complete transcription text for the current segment |
| `is_final` | boolean | Indicates if this is the final transcription for the current segment |
| `is_last` | boolean | Indicates if this is the last transcription in the session |
| `language` | string | Detected primary language code, returns only when `is_final=True` |
| `languages` | array | List of languages detected in the audio included in Responses where `is_final` is `true` |
### Optional Response Fields (Based on Query Parameters)
| Field | Type | When Included | Description |
| ------------------- | ------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `full_transcript` | string | `full_transcript=true` AND `is_final=true` | Complete transcription text accumulated till now. Only present in responses when `full_transcript=true` query parameter is set AND `is_final=true` |
| `words` | array | `word_timestamps=true` | Word-level timestamps with `word`, `start`, `end`, and `confidence` fields. When `diarize=true`, also includes `speaker` and `speaker_confidence` fields |
| `utterances` | array | `sentence_timestamps=true` | Sentence-level timestamps with `text`, `start`, and `end` fields |
| `redacted_entities` | array | `redact_pii=true` or `redact_pci=true` | List of redacted entity placeholders (e.g., `[FIRSTNAME_1]`, `[CREDITCARDCVV_1]`) |
### Example Response with All Features
```json theme={null}
{
"session_id": "sess_12345abcde",
"transcript": "[CREDITCARDCVV_1] and expiry [TIME_2].",
"is_final": true,
"is_last": true,
"full_transcript": "Hi, my name is [FIRSTNAME_1] [FIRSTNAME_2] You can reach me at [PHONENUMBER_1] and I paid using my Visa card [ZIPCODE_1] [ACCOUNTNUMBER_1] with [CREDITCARDCVV_1] and expiry [TIME_1].",
"language": "en",
"languages": ["en"],
"words": [
{
"word": "[creditcardcvv_1]",
"start": 15.44,
"end": 17.36,
"confidence": 0.97,
"speaker": 0,
"speaker_confidence": 0.67
},
{
"word": "and",
"start": 18.0,
"end": 18.32,
"confidence": 0.94,
"speaker": 0,
"speaker_confidence": 0.76
},
{
"word": "expiry",
"start": 18.32,
"end": 19.2,
"confidence": 1.0,
"speaker": 0,
"speaker_confidence": 0.91
},
{
"word": "[time_2]",
"start": 19.2,
"end": 19.92,
"confidence": 0.91,
"speaker": 0,
"speaker_confidence": 0.82
},
],
"utterances": [
{
"text": "Hi, my name is Hans Miller.",
"start": 0.0,
"end": 2.64,
"speaker": 0
},
{
"text": "You can reach me at [PHONENUMBER_1], and I paid using my Visa card 4242 42424242 with CVV123 and expiry [TIME_1].",
"start": 2.64,
"end": 21.04,
"speaker": 0
}
],
"redacted_entities": [
"[CREDITCARDCVV_1]",
"[TIME_2]"
]
}
```
## Code Examples
```python python theme={null}
import asyncio
import json
import argparse
import numpy as np
import websockets
import librosa
from urllib.parse import urlencode
BASE_WS_URL = "wss://api.smallest.ai/waves/v1/pulse/get_text"
async def stream_audio(audio_file, api_key, language="en", encoding="linear16", sample_rate=16000, word_timestamps="true", full_transcript="false", sentence_timestamps="false", redact_pii="false", redact_pci="false", numerals="auto", diarize="false"):
params = {
"language": language,
"encoding": encoding,
"sample_rate": sample_rate,
"word_timestamps": word_timestamps,
"full_transcript": full_transcript,
"sentence_timestamps": sentence_timestamps,
"redact_pii": redact_pii,
"redact_pci": redact_pci,
"numerals": numerals,
"diarize": diarize
}
ws_url = f"{BASE_WS_URL}?{urlencode(params)}"
async with websockets.connect(ws_url, additional_headers={"Authorization": f"Bearer {api_key}"}) as ws:
print(f"Connected: {ws_url}")
async def send():
audio, _ = librosa.load(audio_file, sr=sample_rate, mono=True)
chunk_size = int(0.160 * sample_rate)
for i in range(0, len(audio), chunk_size):
chunk = audio[i:i + chunk_size]
await ws.send((chunk * 32768.0).astype(np.int16).tobytes())
await asyncio.sleep(len(chunk) / sample_rate)
await ws.send(json.dumps({"type": "finalize"}))
sender = asyncio.create_task(send())
async for message in ws:
data = json.loads(message)
print("Received:", json.dumps(data, indent=2))
if data.get("is_last"):
break
await sender
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("audio_file", nargs="?", default="path/to/audio.wav")
parser.add_argument("--api-key", "-k", default="your_api_key_here")
parser.add_argument("--language", "-l", default="en")
parser.add_argument("--encoding", "-e", default="linear16")
parser.add_argument("--sample-rate", "-sr", type=int, default=16000)
parser.add_argument("--word-timestamps", "-wt", default="true")
parser.add_argument("--full-transcript", "-ft", default="false")
parser.add_argument("--sentence-timestamps", "-st", default="false")
parser.add_argument("--redact-pii", default="false")
parser.add_argument("--redact-pci", default="false")
parser.add_argument("--numerals", default="auto")
parser.add_argument("--diarize", default="false")
args = parser.parse_args()
asyncio.run(stream_audio(
args.audio_file,
args.api_key,
args.language,
args.encoding,
args.sample_rate,
args.word_timestamps,
args.full_transcript,
args.sentence_timestamps,
args.redact_pii,
args.redact_pci,
args.numerals,
args.diarize
))
```
## AsyncAPI
````yaml asyncapi-spec/pulse-stt-ws.json /waves/v1/pulse/get_text
id: /waves/v1/pulse/get_text
title: /waves/v1/pulse/get_text
description: ''
servers:
- id: production
protocol: wss
host: api.smallest.ai
bindings:
- protocol: ws
version: latest
value:
query:
type: object
properties:
language:
type: string
enum:
- it
- es
- en
- pt
- hi
- de
- fr
- uk
- ru
- kn
- ml
- pl
- mr
- gu
- cs
- sk
- te
- or
- nl
- bn
- lv
- et
- ro
- pa
- fi
- sv
- bg
- ta
- hu
- da
- lt
- mt
- multi
default: en
description: >-
Language code for transcription. Use 'multi' for automatic
language detection
encoding:
type: string
enum:
- linear16
- linear32
- alaw
- mulaw
- opus
- ogg_opus
default: linear16
description: Audio encoding format
sample_rate:
type: string
enum:
- '8000'
- '16000'
- '22050'
- '24000'
- '44100'
- '48000'
default: '16000'
description: Audio sample rate in Hz
word_timestamps:
type: string
enum:
- 'true'
- 'false'
default: 'true'
description: Include word-level timestamps in transcription
full_transcript:
type: string
enum:
- 'true'
- 'false'
default: 'false'
description: >-
Include cumulative transcript received till now in responses
where is_final is true
sentence_timestamps:
type: string
enum:
- 'true'
- 'false'
default: 'false'
description: >-
Include sentence-level timestamps (utterances) in
transcription
redact_pii:
type: string
enum:
- 'true'
- 'false'
default: 'false'
description: >-
Redact personally identifiable information (name, surname,
address, etc)
redact_pci:
type: string
enum:
- 'true'
- 'false'
default: 'false'
description: >-
Redact payment card information (credit card, CVV, zip,
account number, etc)
numerals:
type: string
enum:
- 'true'
- 'false'
- auto
default: auto
description: >-
Convert spoken numerals into digit form (e.g., 'twenty five'
to '25') and `auto` enables automatic detection based on
context.
diarize:
type: string
enum:
- 'true'
- 'false'
default: 'false'
description: >-
Enable speaker diarization to identify different speakers in
the audio.
keywords:
type: string
description: >-
Comma-separated list of words/phrases to boost, each
optionally followed by :INTENSIFIER (e.g. NVIDIA:5,Jensen).
Intensifier defaults to 1.0 if omitted. Max 100 keywords per
session.
schemaProperties:
- name: query
type: object
required: false
properties:
- name: language
type: string
description: >-
Language code for transcription. Use 'multi' for automatic
language detection
enumValues:
- it
- es
- en
- pt
- hi
- de
- fr
- uk
- ru
- kn
- ml
- pl
- mr
- gu
- cs
- sk
- te
- or
- nl
- bn
- lv
- et
- ro
- pa
- fi
- sv
- bg
- ta
- hu
- da
- lt
- mt
- multi
required: false
- name: encoding
type: string
description: Audio encoding format
enumValues:
- linear16
- linear32
- alaw
- mulaw
- opus
- ogg_opus
required: false
- name: sample_rate
type: string
description: Audio sample rate in Hz
enumValues:
- '8000'
- '16000'
- '22050'
- '24000'
- '44100'
- '48000'
required: false
- name: word_timestamps
type: string
description: Include word-level timestamps in transcription
enumValues:
- 'true'
- 'false'
required: false
- name: full_transcript
type: string
description: >-
Include cumulative transcript received till now in responses
where is_final is true
enumValues:
- 'true'
- 'false'
required: false
- name: sentence_timestamps
type: string
description: >-
Include sentence-level timestamps (utterances) in
transcription
enumValues:
- 'true'
- 'false'
required: false
- name: redact_pii
type: string
description: >-
Redact personally identifiable information (name, surname,
address, etc)
enumValues:
- 'true'
- 'false'
required: false
- name: redact_pci
type: string
description: >-
Redact payment card information (credit card, CVV, zip,
account number, etc)
enumValues:
- 'true'
- 'false'
required: false
- name: numerals
type: string
description: >-
Convert spoken numerals into digit form (e.g., 'twenty five'
to '25') and `auto` enables automatic detection based on
context.
enumValues:
- 'true'
- 'false'
- auto
required: false
- name: diarize
type: string
description: >-
Enable speaker diarization to identify different speakers in
the audio.
enumValues:
- 'true'
- 'false'
required: false
- name: keywords
type: string
description: >-
Comma-separated list of words/phrases to boost, each
optionally followed by :INTENSIFIER (e.g. NVIDIA:5,Jensen).
Intensifier defaults to 1.0 if omitted. Max 100 keywords per
session.
required: false
variables: []
address: /waves/v1/pulse/get_text
parameters: []
bindings: []
operations:
- &ref_2
id: sendAudioData
title: Send audio data
description: Send audio data for transcription
type: receive
messages:
- &ref_5
id: audioData.message
contentType: application/octet-stream
payload:
- type: string
format: binary
examples: &ref_0
- Binary audio data chunk (4096 bytes recommended)
x-parser-schema-id:
name: AudioData
headers: []
jsonPayloadSchema:
type: string
format: binary
description: >-
Raw audio data chunk, transmitted in binary format using the
selected encoding
examples: *ref_0
x-parser-schema-id:
title: AudioData
example: '"Binary audio data chunk (4096 bytes recommended)"'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: audioData.message
bindings: []
extensions: &ref_1
- id: x-parser-unique-object-id
value: /waves/v1/pulse/get_text
- &ref_3
id: sendEndSignal
title: Send end signal
description: Send end of stream signal
type: receive
messages:
- &ref_6
id: endSignal.message
contentType: application/json
payload:
- name: EndSignal
type: object
properties:
- name: type
type: string
description: Signal to indicate end of audio stream
enumValues:
- finalize
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
properties:
type:
type: string
enum:
- finalize
description: Signal to indicate end of audio stream
x-parser-schema-id:
examples:
- type: finalize
x-parser-schema-id:
title: EndSignal
example: |-
{
"type": "finalize"
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: endSignal.message
bindings: []
extensions: *ref_1
- &ref_4
id: receiveTranscription
title: Receive transcription
description: Receive transcription results
type: send
messages:
- &ref_7
id: transcriptionResponse.message
contentType: application/json
payload:
- name: TranscriptionResponse
type: object
properties:
- name: session_id
type: string
description: Unique identifier for the transcription session
required: true
- name: transcript
type: string
description: Partial or complete transcription text for the current segment
required: false
- name: full_transcript
type: string
description: >-
Complete transcription text accumulated so far. Only included
when `full_transcript` query parameter is set to true AND
is_final=true
required: false
- name: is_final
type: boolean
description: >-
Indicates if this is the final transcription for the current
segment
required: false
- name: is_last
type: boolean
description: Indicates if this is the last transcription in the session
required: false
- name: words
type: array
description: Word-level timestamps (when word_timestamps=true)
required: false
properties:
- name: word
type: string
description: The transcribed word
required: false
- name: start
type: number
description: Start time in seconds
required: false
- name: end
type: number
description: End time in seconds
required: false
- name: confidence
type: number
description: Confidence score for the word (0.0 to 1.0)
required: false
- name: speaker
type: integer
description: Speaker label (when diarization is enabled)
required: false
- name: speaker_confidence
type: number
description: Confidence score for the speaker assignment (0.0 to 1.0)
required: false
- name: utterances
type: array
description: Sentence-level timestamps (when sentence_timestamps=true)
required: false
properties:
- name: text
type: string
description: The transcribed sentence
required: false
- name: start
type: number
description: Start time in seconds
required: false
- name: end
type: number
description: End time in seconds
required: false
- name: speaker
type: integer
description: Speaker label (when diarization is enabled)
required: false
- name: language
type: string
description: >-
Detected primary language code, only returned when
`is_final=True`
required: false
- name: languages
type: array
description: >-
List of codes of languages detected in the audio, only
returned when `is_final=True`
required: false
properties:
- name: item
type: string
required: false
- name: redacted_entities
type: array
description: >-
List of redacted entity placeholders (when redact_pii or
redact_pci is enabled)
required: false
properties:
- name: item
type: string
required: false
headers: []
jsonPayloadSchema:
type: object
required:
- session_id
properties:
session_id:
type: string
description: Unique identifier for the transcription session
x-parser-schema-id:
transcript:
type: string
description: Partial or complete transcription text for the current segment
x-parser-schema-id:
full_transcript:
type: string
description: >-
Complete transcription text accumulated so far. Only included
when `full_transcript` query parameter is set to true AND
is_final=true
x-parser-schema-id:
is_final:
type: boolean
default: false
description: >-
Indicates if this is the final transcription for the current
segment
x-parser-schema-id:
is_last:
type: boolean
default: false
description: Indicates if this is the last transcription in the session
x-parser-schema-id:
words:
type: array
description: Word-level timestamps (when word_timestamps=true)
items:
type: object
properties:
word:
type: string
description: The transcribed word
x-parser-schema-id:
start:
type: number
description: Start time in seconds
x-parser-schema-id:
end:
type: number
description: End time in seconds
x-parser-schema-id:
confidence:
type: number
description: Confidence score for the word (0.0 to 1.0)
x-parser-schema-id:
speaker:
type: integer
description: Speaker label (when diarization is enabled)
x-parser-schema-id:
speaker_confidence:
type: number
description: Confidence score for the speaker assignment (0.0 to 1.0)
x-parser-schema-id:
x-parser-schema-id:
x-parser-schema-id:
utterances:
type: array
description: Sentence-level timestamps (when sentence_timestamps=true)
items:
type: object
properties:
text:
type: string
description: The transcribed sentence
x-parser-schema-id:
start:
type: number
description: Start time in seconds
x-parser-schema-id:
end:
type: number
description: End time in seconds
x-parser-schema-id:
speaker:
type: integer
description: Speaker label (when diarization is enabled)
x-parser-schema-id:
x-parser-schema-id:
x-parser-schema-id:
language:
type: string
description: >-
Detected primary language code, only returned when
`is_final=True`
x-parser-schema-id:
languages:
type: array
description: >-
List of codes of languages detected in the audio, only returned
when `is_final=True`
items:
type: string
x-parser-schema-id:
x-parser-schema-id:
redacted_entities:
type: array
description: >-
List of redacted entity placeholders (when redact_pii or
redact_pci is enabled)
items:
type: string
x-parser-schema-id:
x-parser-schema-id:
examples:
- session_id: sess_12345abcde
transcript: Hello, how are you?
is_final: true
is_last: false
language: en
- session_id: session_id_12346abcde
transcript: I paid using my card [CREDITCARDCVV_1] with expiry [TIME_2].
is_final: true
is_last: true
full_transcript: >-
Hello, my name is [FIRSTNAME_1] [FIRSTNAME_2]. I paid using my
card [CREDITCARDCVV_1] with expiry [TIME_2].
language: en
languages:
- en
words:
- word: I
start: 0
end: 0.2
confidence: 0.98
speaker: 0
speaker_confidence: 0.95
- word: paid
start: 0.2
end: 0.5
confidence: 1
speaker: 0
speaker_confidence: 0.59
- word: using
start: 0.5
end: 0.8
confidence: 0.73
speaker: 0
speaker_confidence: 0.74
- word: my
start: 0.8
end: 1
confidence: 0.88
speaker: 0
speaker_confidence: 0.92
- word: card
start: 1
end: 1.3
confidence: 0.95
speaker: 0
speaker_confidence: 0.91
- word: '[creditcardcvv_1]'
start: 1.3
end: 1.8
confidence: 0.99
speaker: 0
speaker_confidence: 0.85
- word: with
start: 1.8
end: 2
confidence: 0.85
speaker: 0
speaker_confidence: 0.87
- word: expiry
start: 2
end: 2.4
confidence: 0.9
speaker: 0
speaker_confidence: 0.67
- word: '[time_2]'
start: 2.4
end: 2.7
confidence: 0.97
speaker: 0
speaker_confidence: 0.76
utterances:
- text: Hello, my name is [FIRSTNAME_1] [FIRSTNAME_2].
start: 0
end: 2.2
speaker: 0
- text: I paid using my card [CREDITCARDCVV_1] with expiry [TIME_2].
start: 2.2
end: 5
speaker: 0
redacted_entities:
- '[FIRSTNAME_1]'
- '[FIRSTNAME_2]'
- '[CREDITCARDCVV_1]'
- '[TIME_2]'
x-parser-schema-id:
title: TranscriptionResponse
example: |-
{
"session_id": "sess_12345abcde",
"transcript": "Hello, how are you?",
"is_final": true,
"is_last": false,
"language": "en"
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: transcriptionResponse.message
bindings: []
extensions: *ref_1
sendOperations:
- *ref_2
- *ref_3
receiveOperations:
- *ref_4
sendMessages:
- *ref_5
- *ref_6
receiveMessages:
- *ref_7
extensions:
- id: x-parser-unique-object-id
value: /waves/v1/pulse/get_text
securitySchemes:
- id: bearerAuth
name: bearerAuth
type: http
description: Bearer token authentication using Smallest AI API key
scheme: bearer
bearerFormat: JWT
extensions: []
````