Transcribe WebSocket API 

Transcribe WebSocket API takes audio streams of your encounters and returns a live transcription.

🏇Show me some code

We've published a JavaScript example of how you can integrate the /transcribe-ws API on a web page.

General specifications

• All messages sent and received via websockets are encoded as UTF-8 JSON text frames.
• We don't keep any of your data beyond the websocket lifecycle. So to be network-resilient, we recommend you store what is relevant for you to be back on track in case of untimely closure.

Authentication

• Get an access token, details in Authentication. It can be a server access token or a user's one, depending on whether you are calling the Server API or the User API.
• We support two different ways of specifying the authentication token:

  • The recommended way: You pass your bearer authentication token as an Authorization header when initiating the websocket.

    Example:

    url: 'wss://us.api.nabla.com/v1/core/server/transcribe-ws',
    protocol: 'transcribe-protocol',
    extra_headers: { 'Authorization': 'Bearer <YOUR_TOKEN>' }
    

  • An alternative way, especially when specifying extra headers is not supported by your websocket client (e.g. from web browsers), is to pass the token as a second websocket protocol prefixed with jwt-.

    Example:

    url: 'wss://us.api.nabla.com/v1/core/user/transcribe-ws',
    protocols: ['transcribe-protocol', 'jwt-<YOUR_TOKEN>']
    

Servers

• wss://{region}.api.nabla.com/v1/core/serverwssServer API

  Called from your servers, authenticated with a server key.

  region

  required

  string

  Geographical region where your Nabla Core API organization exists.

  Allowed values:
  • "us"
  • "eu"
• wss://{region}.api.nabla.com/v1/core/userwssUser API

  Called from front-end apps, authenticated with a user-scoped access token.

  region

  required

  string

  Geographical region where your Nabla Core API organization exists.

  Allowed values:
  • "us"
  • "eu"

Operations

• PUB /transcribe-ws

  Transcribe WebSocket API takes audio streams of your encounters and returns a live transcription.

  Each of the audio streams you declare in the configuration will expect audio chunks to flow continuously (even if silent), if a stream does not receive any audio chunk for 10 seconds the websocket will fail with a timeout error (code: 83011).

  Example communication:

  • You: Open the websocket specifying an authorization token.
  • You: Send a first message setting the configuration.
  • You: continuously send small audio chunks for each stream.
  • Nabla: continuously computes and sends transcript items.
  • You: stop streaming audio and immediately send a { "type": "END" }.
  • Nabla: finishes processing audio and pushes any ongoing transcript item to a final state.
  • Nabla: closes the websocket.
  • You: filter-out non-final transcript items and sort them by start_offset_ms before calling the note-generation API.

  Stream your encounter's audio by sending small chunks from each speaker.

  object

  Some websocket clients only support the Sec-WebSocket-Protocol header, so we suggest an alternative way of specifying your authentication header. See Authentication section above.

  Sec-WebSocket-Protocol

  string

  WebSocket Sub-protocol header, as per the RFC 6455.

  Allowed values:
  • "transcribe-protocol"

  Authorization

  string

  Your authentication token (server key or user access token) prefixed by Bearer . To get one, check Authentication.

  Additional properties are allowed.

  Accepts one of the following messages:

  • #0transcribe_config

    Initiates the transcribeing feature with the given configuration.

    This should be your first message in the websocket.

    object

    uid: transcribe_config

    First message to configure transcription (audio format, locale, etc).

    type

    required

    string

    Allowed values:
    • "CONFIG"

    required

    array<object>

    Describe the audio streams you intend to stream as input of the Transcribe API. Typically, if you have separate doctor/patient audio tracks (or do diarization yourself) you will configure two streams for transcription. Remember that a stream will expect audio to flow continuously, otherwise a timeout error will be thrown.

    id

    required

    string

    Give an identifier to this stream.

    speaker_type

    required

    string

    Who said the text in this transcript item.

    Allowed values:
    • "DOCTOR"
    • "PATIENT"
    • "UNSPECIFIED"

    encoding

    required

    string

    Encoding of the streamed audio.

    Allowed values:
    • "PCM_S16LE"

    sample_rate

    required

    integer

    [ 8000 .. 192000 ]

    Sample rate of submitted streaming audio, in hertz.

    speech_locale

    required

    string

    Allowed values:
    • "ENGLISH_US"
    • "ENGLISH_UK"
    • "SPANISH_ES"
    • "SPANISH_MX"
    • "FRENCH_FR"
    • "ARABIC_EG"
    • "ARABIC_LB"
    • "ARABIC_MA"
    • "ARABIC_SA"
    • "ARMENIAN_AM"
    • "BENGALI_IN"
    • "CANTONESE_CN"
    • "CROATIAN_HR"
    • "FILIPINO_PH"
    • "GERMAN_DE"
    • "GREEK_GR"
    • "GUJARATI_IN"
    • "HEBREW_IL"
    • "HINDI_IN"
    • "ITALIAN_IT"
    • "JAPANESE_JP"
    • "KHMER_KH"
    • "KOREAN_KR"
    • "MANDARIN_CN"
    • "PERSIAN_IR"
    • "POLISH_PL"
    • "PORTUGUESE_PT"
    • "PUNJABI_IN"
    • "RUSSIAN_RU"
    • "SERBIAN_RS"
    • "TAMIL_IN"
    • "TELUGU_IN"
    • "THAI_TH"
    • "URDU_IN"
    • "VIETNAMESE_VN"

    split_by_sentence

    boolean

    Indicates whether to segment transcription results at sentence boundaries. Default is false, meaning that a single transcript item may encompass multiple sentences, provided they are not delineated by pauses (silence) in the audio.

    Default value:false

    enable_audio_chunk_ack

    boolean

    If enabled, audio chunks should contain a seq_id, and the server will periodically send acknowledgements up to the specified seq_id.

    Read more in Make transcription and dictation resilient to network interruptions.

    Default value:false

    Additional properties are allowed.

    Examples

    {
      "type": "config",
      "encoding": "pcm_s16le",
      "sample_rate": 16000,
      "speech_locales": [
        "ENGLISH_US",
        "SPANISH_ES"
      ],
      "streams": [
        {
          "id": "doctor_stream",
          "speaker_type": "doctor"
        },
        {
          "id": "patient_stream",
          "speaker_type": "patient"
        }
      ],
      "split_by_sentence": true
    }
    

    This example has been generated automatically.

  • #1audio_chunk

    A chunk of an audio track from the encounter.

    Chunk (little portion) of a single audio track from the encounter. Maximum allowed duration is 1 second, recommended is 100ms.

    object

    uid: audio_chunk

    type

    required

    string

    Allowed values:
    • "AUDIO_CHUNK"

    payload

    required

    string

    Raw audio chunk in base64 string.

    stream_id

    required

    string

    Identifier of one of the streams you defined in streams in the configuration.

    seq_id

    integer

    A unique identifier for this audio chunk within its stream, expected to be sequential (n+1) from one audio chunk to the next. First value (i.e. for the first audio chunk of a stream) can start at any arbitrary value. This field is required if you enabled the audio chunk acknowledgement protocol in the configuration.

    Read more in Make transcription and dictation resilient to network interruptions.

    Additional properties are allowed.

    Examples

    {
      "type": "AUDIO_CHUNK",
      "payload": "ZXhhbXBsZQ==",
      "stream_id": "doctor_stream",
      "seq_id": 0
    }
    

    This example has been generated automatically.

  • #2end

    End the streaming.

    Signal the end of streaming and ask the Nabla Core API to finish what is still in progress (e.g. gives a final state to the latest transcript item).

    object

    uid: end

    type

    required

    string

    Allowed values:
    • "END"

    Additional properties are allowed.

    Examples

    {
      "type": "END"
    }
    

    This example has been generated automatically.

• SUB /transcribe-ws

  Transcribe WebSocket API takes audio streams of your encounters and returns a live transcription.

  Each of the audio streams you declare in the configuration will expect audio chunks to flow continuously (even if silent), if a stream does not receive any audio chunk for 10 seconds the websocket will fail with a timeout error (code: 83011).

  Example communication:

  • You: Open the websocket specifying an authorization token.
  • You: Send a first message setting the configuration.
  • You: continuously send small audio chunks for each stream.
  • Nabla: continuously computes and sends transcript items.
  • You: stop streaming audio and immediately send a { "type": "END" }.
  • Nabla: finishes processing audio and pushes any ongoing transcript item to a final state.
  • Nabla: closes the websocket.
  • You: filter-out non-final transcript items and sort them by start_offset_ms before calling the note-generation API.

  Receive the live transcription.

  object

  Some websocket clients only support the Sec-WebSocket-Protocol header, so we suggest an alternative way of specifying your authentication header. See Authentication section above.

  Sec-WebSocket-Protocol

  string

  WebSocket Sub-protocol header, as per the RFC 6455.

  Allowed values:
  • "transcribe-protocol"

  Authorization

  string

  Your authentication token (server key or user access token) prefixed by Bearer . To get one, check Authentication.

  Additional properties are allowed.

  Accepts one of the following messages:

  • #0transcript_item

    A transcript item.

    A portion of the transcript being generated. Typically, the currently being spoken sentence transcribed from the last transmitted audio chunks. This might be an incomplete sentence since we keep transcribing as audio chunks are received. You should patch the previously received transcript item with the same id until is_final is true.

    object

    uid: transcript_item

    type

    required

    string

    Allowed values:
    • "TRANSCRIPT_ITEM"

    id

    required

    string

    format: uuid

    A unique identifier.

    text

    required

    string

    The transcribed text.

    speaker_type

    required

    string

    Who said the text in this transcript item.

    Allowed values:
    • "DOCTOR"
    • "PATIENT"
    • "UNSPECIFIED"

    start_offset_ms

    required

    integer

    The initial point of the audio segment that this transcript entry represents, measured in milliseconds from the start of the audio stream.

    end_offset_ms

    required

    integer

    The concluding point of the audio segment that this transcript entry represents, measured in milliseconds from the start of the audio stream.

    In other words: this basically equals start_time_ms plus the duration of the audio segment that this transcript entry represents (regardless of how it was chunked).

    is_final

    required

    boolean

    Indicates if this is the final version of the transcript item.

    Additional properties are allowed.

    Examples

    {
      "type": "TRANSCRIPT_ITEM",
      "id": "98FCE1EF-DBCA-41EF-8BC7-4D1621AC07C6",
      "text": "Also, I’m allergic to peanuts.",
      "speaker_type": "DOCTOR",
      "start_offset_ms": 65100,
      "end_offset_ms": 69300,
      "is_final": true
    }
    

    This example has been generated automatically.

  • #1audio_chunk_ack

    Acknowledgement for audio chunks up to the specified sequential id.

    When enabled in the configuration, server will regularly send audio chunks acknowledgement to signal receipt and imminent transcription of audio.

    Clients should consider acknowledged audio as processed and delete it from their buffer.

    Moreover, audio acknowledgement is intended to set the pace for streaming speed: Clients should refrain from sending new audio chunks until acknowledgement is received for previous ones. Server will accept up to 10 seconds of not-yet-acknowledged audio: clients going further will face an "audio chunks buffer overflow" error.

    Read more in Make transcription and dictation resilient to network interruptions.

    object

    uid: audio_chunk_ack

    Acknowledgement of audio receipt by the server up to the audio chunk with the sequential id ack_id.

    Read more in Make transcription and dictation resilient to network interruptions.

    type

    required

    string

    Allowed values:
    • "AUDIO_CHUNK_ACK"

    stream_id

    required

    string

    ID of the stream to which the ack-ed chunk belongs.

    ack_id

    required

    integer

    The sequential id of the audio chunk up to which this acknowledgment applies.

    Additional properties are allowed.

    Examples

    {
      "type": "AUDIO_CHUNK_ACK",
      "stream_id": "patient_stream",
      "ack_id": 42
    }
    

    This example has been generated automatically.

  • #2error_message

    An error message.

    An error message sent right before closing the websocket due to a fatal error. It explains shortly what went wrong.

    object

    uid: error_message

    type

    required

    string

    Allowed values:
    • "ERROR_MESSAGE"

    message

    required

    string

    The error message.

    Additional properties are allowed.

    Examples

    {
      "type": "ERROR_MESSAGE",
      "message": "Unable to parse JSON at path $.speech_locale"
    }
    

    This example has been generated automatically.

Messages

• #1transcribe_config

  Initiates the transcribeing feature with the given configuration.

  This should be your first message in the websocket.

  object

  uid: transcribe_config

  First message to configure transcription (audio format, locale, etc).

  type

  required

  string

  Allowed values:
  • "CONFIG"

  required

  array<object>

  Describe the audio streams you intend to stream as input of the Transcribe API. Typically, if you have separate doctor/patient audio tracks (or do diarization yourself) you will configure two streams for transcription. Remember that a stream will expect audio to flow continuously, otherwise a timeout error will be thrown.

  id

  required

  string

  Give an identifier to this stream.

  speaker_type

  required

  string

  Who said the text in this transcript item.

  Allowed values:
  • "DOCTOR"
  • "PATIENT"
  • "UNSPECIFIED"

  encoding

  required

  string

  Encoding of the streamed audio.

  Allowed values:
  • "PCM_S16LE"

  sample_rate

  required

  integer

  [ 8000 .. 192000 ]

  Sample rate of submitted streaming audio, in hertz.

  speech_locale

  required

  string

  Allowed values:
  • "ENGLISH_US"
  • "ENGLISH_UK"
  • "SPANISH_ES"
  • "SPANISH_MX"
  • "FRENCH_FR"
  • "ARABIC_EG"
  • "ARABIC_LB"
  • "ARABIC_MA"
  • "ARABIC_SA"
  • "ARMENIAN_AM"
  • "BENGALI_IN"
  • "CANTONESE_CN"
  • "CROATIAN_HR"
  • "FILIPINO_PH"
  • "GERMAN_DE"
  • "GREEK_GR"
  • "GUJARATI_IN"
  • "HEBREW_IL"
  • "HINDI_IN"
  • "ITALIAN_IT"
  • "JAPANESE_JP"
  • "KHMER_KH"
  • "KOREAN_KR"
  • "MANDARIN_CN"
  • "PERSIAN_IR"
  • "POLISH_PL"
  • "PORTUGUESE_PT"
  • "PUNJABI_IN"
  • "RUSSIAN_RU"
  • "SERBIAN_RS"
  • "TAMIL_IN"
  • "TELUGU_IN"
  • "THAI_TH"
  • "URDU_IN"
  • "VIETNAMESE_VN"

  split_by_sentence

  boolean

  Indicates whether to segment transcription results at sentence boundaries. Default is false, meaning that a single transcript item may encompass multiple sentences, provided they are not delineated by pauses (silence) in the audio.

  Default value:false

  enable_audio_chunk_ack

  boolean

  If enabled, audio chunks should contain a seq_id, and the server will periodically send acknowledgements up to the specified seq_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  Default value:false

  Additional properties are allowed.

• #2audio_chunk

  A chunk of an audio track from the encounter.

  Chunk (little portion) of a single audio track from the encounter. Maximum allowed duration is 1 second, recommended is 100ms.

  object

  uid: audio_chunk

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK"

  payload

  required

  string

  Raw audio chunk in base64 string.

  stream_id

  required

  string

  Identifier of one of the streams you defined in streams in the configuration.

  seq_id

  integer

  A unique identifier for this audio chunk within its stream, expected to be sequential (n+1) from one audio chunk to the next. First value (i.e. for the first audio chunk of a stream) can start at any arbitrary value. This field is required if you enabled the audio chunk acknowledgement protocol in the configuration.

  Read more in Make transcription and dictation resilient to network interruptions.

  Additional properties are allowed.

• #3end

  End the streaming.

  Signal the end of streaming and ask the Nabla Core API to finish what is still in progress (e.g. gives a final state to the latest transcript item).

  object

  uid: end

  type

  required

  string

  Allowed values:
  • "END"

  Additional properties are allowed.

• #4transcript_item

  A transcript item.

  A portion of the transcript being generated. Typically, the currently being spoken sentence transcribed from the last transmitted audio chunks. This might be an incomplete sentence since we keep transcribing as audio chunks are received. You should patch the previously received transcript item with the same id until is_final is true.

  object

  uid: transcript_item

  type

  required

  string

  Allowed values:
  • "TRANSCRIPT_ITEM"

  id

  required

  string

  format: uuid

  A unique identifier.

  text

  required

  string

  The transcribed text.

  speaker_type

  required

  string

  Who said the text in this transcript item.

  Allowed values:
  • "DOCTOR"
  • "PATIENT"
  • "UNSPECIFIED"

  start_offset_ms

  required

  integer

  The initial point of the audio segment that this transcript entry represents, measured in milliseconds from the start of the audio stream.

  end_offset_ms

  required

  integer

  The concluding point of the audio segment that this transcript entry represents, measured in milliseconds from the start of the audio stream.

  In other words: this basically equals start_time_ms plus the duration of the audio segment that this transcript entry represents (regardless of how it was chunked).

  is_final

  required

  boolean

  Indicates if this is the final version of the transcript item.

  Additional properties are allowed.

• #5audio_chunk_ack

  Acknowledgement for audio chunks up to the specified sequential id.

  When enabled in the configuration, server will regularly send audio chunks acknowledgement to signal receipt and imminent transcription of audio.

  Clients should consider acknowledged audio as processed and delete it from their buffer.

  Moreover, audio acknowledgement is intended to set the pace for streaming speed: Clients should refrain from sending new audio chunks until acknowledgement is received for previous ones. Server will accept up to 10 seconds of not-yet-acknowledged audio: clients going further will face an "audio chunks buffer overflow" error.

  Read more in Make transcription and dictation resilient to network interruptions.

  object

  uid: audio_chunk_ack

  Acknowledgement of audio receipt by the server up to the audio chunk with the sequential id ack_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK_ACK"

  stream_id

  required

  string

  ID of the stream to which the ack-ed chunk belongs.

  ack_id

  required

  integer

  The sequential id of the audio chunk up to which this acknowledgment applies.

  Additional properties are allowed.

• #6error_message

  An error message.

  An error message sent right before closing the websocket due to a fatal error. It explains shortly what went wrong.

  object

  uid: error_message

  type

  required

  string

  Allowed values:
  • "ERROR_MESSAGE"

  message

  required

  string

  The error message.

  Additional properties are allowed.

Schemas

• object

  uid: transcribe_config

  First message to configure transcription (audio format, locale, etc).

  type

  required

  string

  Allowed values:
  • "CONFIG"

  required

  array<object>

  Describe the audio streams you intend to stream as input of the Transcribe API. Typically, if you have separate doctor/patient audio tracks (or do diarization yourself) you will configure two streams for transcription. Remember that a stream will expect audio to flow continuously, otherwise a timeout error will be thrown.

  id

  required

  string

  Give an identifier to this stream.

  speaker_type

  required

  string

  Who said the text in this transcript item.

  Allowed values:
  • "DOCTOR"
  • "PATIENT"
  • "UNSPECIFIED"

  encoding

  required

  string

  Encoding of the streamed audio.

  Allowed values:
  • "PCM_S16LE"

  sample_rate

  required

  integer

  [ 8000 .. 192000 ]

  Sample rate of submitted streaming audio, in hertz.

  speech_locale

  required

  string

  Allowed values:
  • "ENGLISH_US"
  • "ENGLISH_UK"
  • "SPANISH_ES"
  • "SPANISH_MX"
  • "FRENCH_FR"
  • "ARABIC_EG"
  • "ARABIC_LB"
  • "ARABIC_MA"
  • "ARABIC_SA"
  • "ARMENIAN_AM"
  • "BENGALI_IN"
  • "CANTONESE_CN"
  • "CROATIAN_HR"
  • "FILIPINO_PH"
  • "GERMAN_DE"
  • "GREEK_GR"
  • "GUJARATI_IN"
  • "HEBREW_IL"
  • "HINDI_IN"
  • "ITALIAN_IT"
  • "JAPANESE_JP"
  • "KHMER_KH"
  • "KOREAN_KR"
  • "MANDARIN_CN"
  • "PERSIAN_IR"
  • "POLISH_PL"
  • "PORTUGUESE_PT"
  • "PUNJABI_IN"
  • "RUSSIAN_RU"
  • "SERBIAN_RS"
  • "TAMIL_IN"
  • "TELUGU_IN"
  • "THAI_TH"
  • "URDU_IN"
  • "VIETNAMESE_VN"

  split_by_sentence

  boolean

  Indicates whether to segment transcription results at sentence boundaries. Default is false, meaning that a single transcript item may encompass multiple sentences, provided they are not delineated by pauses (silence) in the audio.

  Default value:false

  enable_audio_chunk_ack

  boolean

  If enabled, audio chunks should contain a seq_id, and the server will periodically send acknowledgements up to the specified seq_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  Default value:false

  Additional properties are allowed.
• object

  uid: audio_chunk

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK"

  payload

  required

  string

  Raw audio chunk in base64 string.

  stream_id

  required

  string

  Identifier of one of the streams you defined in streams in the configuration.

  seq_id

  integer

  A unique identifier for this audio chunk within its stream, expected to be sequential (n+1) from one audio chunk to the next. First value (i.e. for the first audio chunk of a stream) can start at any arbitrary value. This field is required if you enabled the audio chunk acknowledgement protocol in the configuration.

  Read more in Make transcription and dictation resilient to network interruptions.

  Additional properties are allowed.
• object

  uid: end

  type

  required

  string

  Allowed values:
  • "END"

  Additional properties are allowed.
• object

  uid: transcript_item

  type

  required

  string

  Allowed values:
  • "TRANSCRIPT_ITEM"

  id

  required

  string

  format: uuid

  A unique identifier.

  text

  required

  string

  The transcribed text.

  speaker_type

  required

  string

  Who said the text in this transcript item.

  Allowed values:
  • "DOCTOR"
  • "PATIENT"
  • "UNSPECIFIED"

  start_offset_ms

  required

  integer

  The initial point of the audio segment that this transcript entry represents, measured in milliseconds from the start of the audio stream.

  end_offset_ms

  required

  integer

  The concluding point of the audio segment that this transcript entry represents, measured in milliseconds from the start of the audio stream.

  In other words: this basically equals start_time_ms plus the duration of the audio segment that this transcript entry represents (regardless of how it was chunked).

  is_final

  required

  boolean

  Indicates if this is the final version of the transcript item.

  Additional properties are allowed.
• object

  uid: audio_chunk_ack

  Acknowledgement of audio receipt by the server up to the audio chunk with the sequential id ack_id.

  Read more in Make transcription and dictation resilient to network interruptions.

  type

  required

  string

  Allowed values:
  • "AUDIO_CHUNK_ACK"

  stream_id

  required

  string

  ID of the stream to which the ack-ed chunk belongs.

  ack_id

  required

  integer

  The sequential id of the audio chunk up to which this acknowledgment applies.

  Additional properties are allowed.
• object

  uid: error_message

  type

  required

  string

  Allowed values:
  • "ERROR_MESSAGE"

  message

  required

  string

  The error message.

  Additional properties are allowed.