Copilot Listen v0.2 documentation

Copilot Listen WebSocket API takes audio streams of your consultations
and returns a live transcription and a generated clinical note.

General specifications

• All messages sent and received via websockets are encoded as UTF-8 JSON text frames.
• We don't keep any of your data beyond the websocket lifecycle.
  So to be network-resilient, we recommend you store what is relevant for you to be back on track in case of untimely closure.
• You should pass your bearer authentication token as an Authorization header when initiating the websocket.
  To get one, contact us.
  Example: url: 'wss://api.nabla.com/v1/server/copilot/listen', protocol: 'copilot-listen-protocol', extra_headers: { 'Authorization': 'Bearer <YOUR_TOKEN> }`.

Servers

nabla Server

• URL: wss://api.nabla.com/v1/server/copilot
• Protocol: wss

Operations

PUB /listen Operation

Copilot Listen WebSocket API takes audio streams of your consultations
and returns a live transcription and a generated clinical note.

Note generation, depending on the length of the transcript, might take from 10 seconds up to few minutes.

Example communication:

• You: Open the websocket specifying an authorization token.
• You: Send a first message setting the configuration.
• You: continuously send small audio chunks for each stream.
• Copilot: continuously computes and sends transcript items.
• You: stop streaming audio and immediately send a { "object": "end" }.
• Copilot: generates the summarizing note and sends it within few seconds.
• Copilot: closes the websocket.

Stream your consultation audio by sending small chunks from each speaker.

wss Channel specific information

Accepts one of the following messages:

Message listen_config

Initiates the listening feature with the given configuration.

This should be your first message in the websocket.

Payload

Examples of payload (generated)

{
  "object": "listen_config",
  "output_objects": [
    "transcript_item"
  ],
  "streams": [
    {
      "id": "stream1",
      "speaker_type": "doctor"
    }
  ],
  "encoding": "pcm_s16le",
  "sample_rate": 48000,
  "language": "en"
}

Message audio_chunk

A chunk of an audio track from the consultation.

Chunk (little portion) of a single audio track from the consultation. Maximum allowed duration is 1 second, recommended is 50ms.

Payload

Examples of payload (generated)

{
  "object": "audio_chunk",
  "payload": "ZXhhbXBsZQ==",
  "stream_id": "stream1"
}

Message end

End the streaming.

Signal the end of streaming and ask the Copilot to finish what is still in progress (e.g. generate the final note).

Payload

Examples of payload

{
  "object": "end"
}

SUB /listen Operation

• Operation ID: receiveResults

Copilot Listen WebSocket API takes audio streams of your consultations
and returns a live transcription and a generated clinical note.

Note generation, depending on the length of the transcript, might take from 10 seconds up to few minutes.

Example communication:

• You: Open the websocket specifying an authorization token.
• You: Send a first message setting the configuration.
• You: continuously send small audio chunks for each stream.
• Copilot: continuously computes and sends transcript items.
• You: stop streaming audio and immediately send a { "object": "end" }.
• Copilot: generates the summarizing note and sends it within few seconds.
• Copilot: closes the websocket.

Receive the live transcription then a summarizing note.

wss Channel specific information

Accepts one of the following messages:

Message transcript_item

A transcript item.

A portion of the transcript being generated. Typically, the currently being spoken sentence transcribed from the last transmitted audio chunks. This might be an incomplete sentence since we keep transcribing as audio chunks are received. You should patch the previously received transcript item with the same id until is_final is true.

Payload

Examples of payload (generated)

{
  "object": "transcript_item",
  "id": "ba50e302-413d-480d-a880-c173624707c2",
  "text": "Also, I’m allergic to peanuts.",
  "speaker": "doctor",
  "start_offset_ms": 65100,
  "end_offset_ms": 69300,
  "is_final": true
}

Message note

The note summarizing the whole transcribed consultation so far.

An AI-generated summary of the consultation, presented as bullet points and separated into relevant sections (chief complaint, allergies, medication, etc).

Payload

Examples of payload (generated)

{
  "object": "note",
  "id": "ba50e302-413d-480d-a880-c173624707c2",
  "sections": [
    {
      "title": "Chief complaint",
      "content": [
        "Sleep disorder"
      ]
    },
    {
      "title": "Observations",
      "content": [
        "32 years old",
        "Chronic insomnia"
      ]
    }
  ],
  "is_final": true
}

Message duration_limit

Signals that the streamed audio has reached (or is close to reaching) the maximum duration of 1 hour.

Payload

Examples of payload (generated)

{
  "object": "duration_limit",
  "remaining_seconds": 0
}