Dictate Transcription WebSocket API 

Dictate Transcription WebSocket API is designed specifically for healthcare professionals to dictate clinical notes efficiently and accurately. It takes audio from your provider's microphone and returns a live transcription.

By receiving audio directly captured from the provider's microphone, this API delivers live transcription services tailored for the medical field. It differs from our general transcription service (/transcribe-ws) by offering specific configuration options catered to the unique needs of clinical documentation dictation, such as precise control over punctuation.

General specifications

• All messages sent and received via websockets are encoded as UTF-8 JSON text frames.
• We don't keep any of your data beyond the websocket lifecycle. So to be network-resilient, we recommend you store what is relevant for you to be back on track in case of untimely closure.

Authentication

• Get an access token, details in Authentication. It can be a server access token or a user's one, depending on whether you are calling the Server API or the User API.
• We support two different ways of specifying the authentication token:

  • The recommended way: You pass your bearer authentication token as an Authorization header when initiating the websocket.

    Example:

    url: 'wss://us.api.nabla.com/v1/core/server/dictate-ws',
    protocol: 'dictate-protocol',
    extra_headers: { 'Authorization': 'Bearer <YOUR_TOKEN>' }
    

  • An alternative way, especially when specifying extra headers is not supported by your websocket client (e.g. from web browsers), is to pass the token as a second websocket protocol prefixed with jwt-.

    Example:

    url: 'wss://us.api.nabla.com/v1/core/user/dictate-ws',
    protocols: ['dictate-protocol', 'jwt-<YOUR_TOKEN>']
    

Servers

• wss://us.api.nabla.com/v1/core/serverwssServer API

  Called from your servers, authenticated with a server key.
• wss://us.api.nabla.com/v1/core/userwssUser API

  Called from front-end apps, authenticated with a user-scoped access token.

Operations

• PUB /dictate-ws

  Transcription Dictate WebSocket API takes audio from your provider's microphone and returns a live transcription.

  Dictation API expects audio chunks to flow continuously (even if silent), if no audio chunk is received for 10 seconds the websocket will fail with a timeout error (code: 83011).

  Example communication:

  • You: Open the websocket specifying an authorization token.
  • You: Send a first message setting the configuration.
  • You: continuously send small audio chunks.
  • Core API: continuously computes and sends dictation items.
  • You: stop streaming audio and immediately send a { "type": "END" }.
  • Core API: finishes processing audio and pushes any ongoing dictation item to a final state.
  • Core API: closes the websocket.

  Stream the audio of the note being dictated by sending it in small, defined chunks (e.g., segments of 100 milliseconds each).

  object

  Some websocket clients only support the Sec-WebSocket-Protocol header, so we suggest an alternative way of specifying your authentication header. See Authentication section above.

  Sec-WebSocket-Protocol

  string

  WebSocket Sub-protocol header, as per the RFC 6455.

  Allowed values:
  • "dictate-protocol"

  Authorization

  string

  Your authentication token (server key or user access token) prefixed by Bearer . To get one, check Authentication.

  Additional properties are allowed.

  Accepts one of the following messages:

  • #0dictate_config

    Initiates the dictation feature with the given configuration.

    This should be your first message in the websocket.

    object

    uid: dictate_config

    First message to configure dictation (audio format, locale, etc).

    type

    required

    string

    Allowed values:
    • "CONFIG"

    encoding

    required

    string

    Encoding of the streamed audio.

    Allowed values:
    • "PCM_S16LE"

    sample_rate

    required

    integer

    [ 8000 .. 192000 ]

    Sample rate of submitted streaming audio, in hertz.

    speech_locale

    required

    string

    Allowed values:
    • "ENGLISH_US"
    • "ENGLISH_UK"
    • "SPANISH_ES"
    • "SPANISH_MX"
    • "FRENCH_FR"
    • "ARABIC_EG"
    • "ARABIC_LB"
    • "ARABIC_MA"
    • "ARABIC_SA"
    • "ARMENIAN_AM"
    • "BENGALI_IN"
    • "CANTONESE_CN"
    • "CROATIAN_HR"
    • "FILIPINO_PH"
    • "GERMAN_DE"
    • "GREEK_GR"
    • "GUJARATI_IN"
    • "HEBREW_IL"
    • "HINDI_IN"
    • "ITALIAN_IT"
    • "JAPANESE_JP"
    • "KHMER_KH"
    • "KOREAN_KR"
    • "MANDARIN_CN"
    • "PERSIAN_IR"
    • "POLISH_PL"
    • "PORTUGUESE_PT"
    • "PUNJABI_IN"
    • "RUSSIAN_RU"
    • "SERBIAN_RS"
    • "TAMIL_IN"
    • "TELUGU_IN"
    • "THAI_TH"
    • "URDU_IN"
    • "VIETNAMESE_VN"

    dictate_punctuation

    required

    boolean

    When enabled, the speaker dictating would explicitly dictate the punctuation. For instance, when the dictating provider says:

    The patient inquired quotation mark is this an allergy question mark close quotation mark displaying significant distress.

    the transcription would output:

    The patient inquired "is this an allergy?" displaying significant distress..

    enable_audio_chunk_ack

    boolean

    If enabled, audio chunks should contain a seq_id, and the server will periodically send acknowledgements up to the specified seq_id.

    Read more in Make transcription and dictation resilient to network interruptions.

    Default value:false

    Additional properties are allowed.

    Examples

    {
      "type": "CONFIG",
      "encoding": "PCM_S16LE",
      "sample_rate": 16000,
      "speech_locale": "ENGLISH_US",
      "dictate_punctuation": true,
      "enable_audio_chunk_ack": false
    }
    

    This example has been generated automatically.

  • #1audio_chunk

    A chunk of audio from the provider's microphone.

    Chunk (little portion) of audio from the provider's microphone on which they dictate. Maximum allowed duration is 1 second, recommended is 100ms.

    object

    uid: audio_chunk

    type

    required

    string

    Allowed values:
    • "AUDIO_CHUNK"

    payload

    required

    string

    Raw audio chunk in base64 string.

    seq_id

    integer

    A unique identifier for this audio chunk within its stream, expected to be sequential (n+1) from one audio chunk to the next. First value (i.e. for the first audio chunk of a stream) can start at any arbitrary value. This field is required if you enabled the audio chunk acknowledgement protocol in the configuration.

    Read more in Make transcription and dictation resilient to network interruptions.

    Additional properties are allowed.

    Examples

    {
      "type": "AUDIO_CHUNK",
      "payload": "ZXhhbXBsZQ==",
      "seq_id": 0
    }
    

    This example has been generated automatically.

  • #2end

    End the streaming.

    Signal the end of streaming and ask the speech-to-text engine to finish what is still in progress (e.g. gives a final state to the latest dictation item).

    object

    uid: end

    type

    required

    string

    Allowed values:
    • "END"

    Additional properties are allowed.

    Examples

    {
      "type": "END"
    }
    

    This example has been generated automatically.

• SUB /dictate-ws

  Transcription Dictate WebSocket API takes audio from your provider's microphone and returns a live transcription.

  Dictation API expects audio chunks to flow continuously (even if silent), if no audio chunk is received for 10 seconds the websocket will fail with a timeout error (code: 83011).

  Example communication:

  • You: Open the websocket specifying an authorization token.
  • You: Send a first message setting the configuration.
  • You: continuously send small audio chunks.
  • Core API: continuously computes and sends dictation items.
  • You: stop streaming audio and immediately send a { "type": "END" }.
  • Core API: finishes processing audio and pushes any ongoing dictation item to a final state.
  • Core API: closes the websocket.

  Receive the live transcription.

  object

  Some websocket clients only support the Sec-WebSocket-Protocol header, so we suggest an alternative way of specifying your authentication header. See Authentication section above.

  Sec-WebSocket-Protocol

  string

  WebSocket Sub-protocol header, as per the RFC 6455.

  Allowed values:
  • "dictate-protocol"

  Authorization

  string

  Your authentication token (server key or user access token) prefixed by Bearer . To get one, check Authentication.

  Additional properties are allowed.

  Accepts one of the following messages:

  • #0dictation_item

    A dictation item.

    A portion of the dictation being generated. Typically, the currently being spoken sentence transcribed from the last transmitted audio chunks. This might be an incomplete sentence since we keep transcribing as audio chunks are received. You should patch the previously received dictation item with the same id until is_final is true.

    object

    uid: dictation_item

    type

    required

    string

    Allowed values:
    • "DICTATION_ITEM"

    id

    required

    string

    format: uuid

    A unique identifier.

    text

    required

    string

    The transcribed text.

    start_offset_ms

    required

    integer

    The initial point of the audio segment that this dictation entry represents, measured in milliseconds from the start of the audio stream.

    end_offset_ms

    required

    integer

    The concluding point of the audio segment that this dictation entry represents, measured in milliseconds from the start of the audio stream.

    In other words: this basically equals start_time_ms plus the duration of the audio segment that this dictation entry represents (regardless of how it was chunked).

    is_final

    required

    boolean

    Indicates if this is the final version of the dictation item.

    Additional properties are allowed.

    Examples

    {
      "type": "DICTATION_ITEM",
      "id": "98FCE1EF-DBCA-41EF-8BC7-4D1621AC07C6",
      "text": "Patient showed signs of rapid improvement.",
      "start_offset_ms": 65100,
      "end_offset_ms": 69300,
      "is_final": true
    }
    

    This example has been generated automatically.

  • #1audio_chunk_ack

    Acknowledgement for audio chunks up to the specified sequential id.

    When enabled in the configuration, server will regularly send audio chunks acknowledgement to signal receipt and imminent processing of audio.

    Clients should consider acknowledged audio as processed and delete it from their buffer.

    Moreover, audio acknowledgement is intended to set the pace for streaming speed: Clients should refrain from sending new audio chunks until acknowledgement is received for previous ones. Server will accept up to 10 seconds of not-yet-acknowledged audio: clients going further will face an "audio chunks buffer overflow" error.

    Read more in Make transcription and dictation resilient to network interruptions.

    object

    uid: audio_chunk_ack

    Acknowledgement of audio receipt by the server up to the audio chunk with the sequential id ack_id.

    Read more in Make transcription and dictation resilient to network interruptions.

    type

    required

    string

    Allowed values:
    • "AUDIO_CHUNK_ACK"

    ack_id

    required

    integer

    The sequential id of the audio chunk up to which this acknowledgment applies.

    Additional properties are allowed.

    Examples

    {
      "type": "AUDIO_CHUNK_ACK",
      "ack_id": 42
    }
    

    This example has been generated automatically.

  • #2error_message

    An error message.

    An error message sent right before closing the websocket due to a fatal error. It explains shortly what went wrong.

    object

    uid: error_message

    type

    required

    string

    Allowed values:
    • "ERROR_MESSAGE"

    message

    required

    string

    The error message.

    Additional properties are allowed.

    Examples

    {
      "type": "ERROR_MESSAGE",
      "message": "Unable to parse JSON at path $.speech_locale"
    }
    

    This example has been generated automatically.

Messages

• #1dictate_config

  Initiates the dictation feature with the given configuration.

  This should be your first message in the websocket.

  object

  uid: dictate_config

  First message to configure dictation (audio format, locale, etc).

  type

  required

  string

  Allowed values:
  • "CONFIG"

  encoding

  required

  string

  Encoding of the streamed audio.

  Allowed values:
  • "PCM_S16LE"

  sample_rate

  required

  integer

  [ 8000 .. 192000 ]

  Sample rate of submitted streaming audio, in hertz.

  speech_locale

  required

  string

  Allowed values:
  • "ENGLISH_US"
  • "ENGLISH_UK"
  • "SPANISH_ES"
  • "SPANISH_MX"
  • "FRENCH_FR"
  • "ARABIC_EG"
  • "ARABIC_LB"
  • "ARABIC_MA"
  • "ARABIC_SA"
  • "ARMENIAN_AM"
  • "BENGALI_IN"
  • "CANTONESE_CN"
  • "CROATIAN_HR"
  • "FILIPINO_PH"
  • "GERMAN_DE"
  • "GREEK_GR"
  • "GUJARATI_IN"
  • "HEBREW_IL"
  • "HINDI_IN"
  • "ITALIAN_IT"
  • "JAPANESE_JP"
  • "KHMER_KH"
  • "KOREAN_KR"
  • "MANDARIN_CN"
  • "PERSIAN_IR"
  • "POLISH_PL"
  • "PORTUGUESE_PT"
  • "PUNJABI_IN"
  • "RUSSIAN_RU"
  • "SERBIAN_RS"
  • "TAMIL_IN"
  • "TELUGU_IN"
  • "THAI_TH"
  • "URDU_IN"
  • "VIETNAMESE_VN"

  dictate_punctuation

  required

  boolean

  When enabled, the speaker dictating would explicitly dictate the punctuation. For instance, when the dictating provider says:

  The patient inquired quotation mark is this an allergy question mark close quotation mark displaying significant distress.

  the transcription would output:

  The patient inquired "is this an allergy?" displaying significant distress..

  enable_audio_chunk_ack

  boolean

  If enabled, audio chunks should contain a seq_id, and the server will periodically send acknowledgements up to the specified seq_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  Default value:false

  Additional properties are allowed.

• #2audio_chunk

  A chunk of audio from the provider's microphone.

  Chunk (little portion) of audio from the provider's microphone on which they dictate. Maximum allowed duration is 1 second, recommended is 100ms.

  object

  uid: audio_chunk

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK"

  payload

  required

  string

  Raw audio chunk in base64 string.

  seq_id

  integer

  A unique identifier for this audio chunk within its stream, expected to be sequential (n+1) from one audio chunk to the next. First value (i.e. for the first audio chunk of a stream) can start at any arbitrary value. This field is required if you enabled the audio chunk acknowledgement protocol in the configuration.

  Read more in Make transcription and dictation resilient to network interruptions.

  Additional properties are allowed.

• #3end

  End the streaming.

  Signal the end of streaming and ask the speech-to-text engine to finish what is still in progress (e.g. gives a final state to the latest dictation item).

  object

  uid: end

  type

  required

  string

  Allowed values:
  • "END"

  Additional properties are allowed.

• #4dictation_item

  A dictation item.

  A portion of the dictation being generated. Typically, the currently being spoken sentence transcribed from the last transmitted audio chunks. This might be an incomplete sentence since we keep transcribing as audio chunks are received. You should patch the previously received dictation item with the same id until is_final is true.

  object

  uid: dictation_item

  type

  required

  string

  Allowed values:
  • "DICTATION_ITEM"

  id

  required

  string

  format: uuid

  A unique identifier.

  text

  required

  string

  The transcribed text.

  start_offset_ms

  required

  integer

  The initial point of the audio segment that this dictation entry represents, measured in milliseconds from the start of the audio stream.

  end_offset_ms

  required

  integer

  The concluding point of the audio segment that this dictation entry represents, measured in milliseconds from the start of the audio stream.

  In other words: this basically equals start_time_ms plus the duration of the audio segment that this dictation entry represents (regardless of how it was chunked).

  is_final

  required

  boolean

  Indicates if this is the final version of the dictation item.

  Additional properties are allowed.

• #5audio_chunk_ack

  Acknowledgement for audio chunks up to the specified sequential id.

  When enabled in the configuration, server will regularly send audio chunks acknowledgement to signal receipt and imminent processing of audio.

  Clients should consider acknowledged audio as processed and delete it from their buffer.

  Moreover, audio acknowledgement is intended to set the pace for streaming speed: Clients should refrain from sending new audio chunks until acknowledgement is received for previous ones. Server will accept up to 10 seconds of not-yet-acknowledged audio: clients going further will face an "audio chunks buffer overflow" error.

  Read more in Make transcription and dictation resilient to network interruptions.

  object

  uid: audio_chunk_ack

  Acknowledgement of audio receipt by the server up to the audio chunk with the sequential id ack_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK_ACK"

  ack_id

  required

  integer

  The sequential id of the audio chunk up to which this acknowledgment applies.

  Additional properties are allowed.

• #6error_message

  An error message.

  An error message sent right before closing the websocket due to a fatal error. It explains shortly what went wrong.

  object

  uid: error_message

  type

  required

  string

  Allowed values:
  • "ERROR_MESSAGE"

  message

  required

  string

  The error message.

  Additional properties are allowed.

Schemas

• object

  uid: dictate_config

  First message to configure dictation (audio format, locale, etc).

  type

  required

  string

  Allowed values:
  • "CONFIG"

  encoding

  required

  string

  Encoding of the streamed audio.

  Allowed values:
  • "PCM_S16LE"

  sample_rate

  required

  integer

  [ 8000 .. 192000 ]

  Sample rate of submitted streaming audio, in hertz.

  speech_locale

  required

  string

  Allowed values:
  • "ENGLISH_US"
  • "ENGLISH_UK"
  • "SPANISH_ES"
  • "SPANISH_MX"
  • "FRENCH_FR"
  • "ARABIC_EG"
  • "ARABIC_LB"
  • "ARABIC_MA"
  • "ARABIC_SA"
  • "ARMENIAN_AM"
  • "BENGALI_IN"
  • "CANTONESE_CN"
  • "CROATIAN_HR"
  • "FILIPINO_PH"
  • "GERMAN_DE"
  • "GREEK_GR"
  • "GUJARATI_IN"
  • "HEBREW_IL"
  • "HINDI_IN"
  • "ITALIAN_IT"
  • "JAPANESE_JP"
  • "KHMER_KH"
  • "KOREAN_KR"
  • "MANDARIN_CN"
  • "PERSIAN_IR"
  • "POLISH_PL"
  • "PORTUGUESE_PT"
  • "PUNJABI_IN"
  • "RUSSIAN_RU"
  • "SERBIAN_RS"
  • "TAMIL_IN"
  • "TELUGU_IN"
  • "THAI_TH"
  • "URDU_IN"
  • "VIETNAMESE_VN"

  dictate_punctuation

  required

  boolean

  When enabled, the speaker dictating would explicitly dictate the punctuation. For instance, when the dictating provider says:

  The patient inquired quotation mark is this an allergy question mark close quotation mark displaying significant distress.

  the transcription would output:

  The patient inquired "is this an allergy?" displaying significant distress..

  enable_audio_chunk_ack

  boolean

  If enabled, audio chunks should contain a seq_id, and the server will periodically send acknowledgements up to the specified seq_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  Default value:false

  Additional properties are allowed.
• object

  uid: audio_chunk

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK"

  payload

  required

  string

  Raw audio chunk in base64 string.

  seq_id

  integer

  A unique identifier for this audio chunk within its stream, expected to be sequential (n+1) from one audio chunk to the next. First value (i.e. for the first audio chunk of a stream) can start at any arbitrary value. This field is required if you enabled the audio chunk acknowledgement protocol in the configuration.

  Read more in Make transcription and dictation resilient to network interruptions.

  Additional properties are allowed.
• object

  uid: end

  type

  required

  string

  Allowed values:
  • "END"

  Additional properties are allowed.
• object

  uid: dictation_item

  type

  required

  string

  Allowed values:
  • "DICTATION_ITEM"

  id

  required

  string

  format: uuid

  A unique identifier.

  text

  required

  string

  The transcribed text.

  start_offset_ms

  required

  integer

  The initial point of the audio segment that this dictation entry represents, measured in milliseconds from the start of the audio stream.

  end_offset_ms

  required

  integer

  The concluding point of the audio segment that this dictation entry represents, measured in milliseconds from the start of the audio stream.

  In other words: this basically equals start_time_ms plus the duration of the audio segment that this dictation entry represents (regardless of how it was chunked).

  is_final

  required

  boolean

  Indicates if this is the final version of the dictation item.

  Additional properties are allowed.
• object

  uid: audio_chunk_ack

  Acknowledgement of audio receipt by the server up to the audio chunk with the sequential id ack_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK_ACK"

  ack_id

  required

  integer

  The sequential id of the audio chunk up to which this acknowledgment applies.

  Additional properties are allowed.
• object

  uid: error_message

  type

  required

  string

  Allowed values:
  • "ERROR_MESSAGE"

  message

  required

  string

  The error message.

  Additional properties are allowed.