Voice & Video

Voice media uses WebRTC. The normal protocol connection provides a voice endpoint and issues a short-lived token; a second WebSocket connection then coordinates the WebRTC session.

Connecting to a voice channel

The client completes these steps when you join a channel:

1. Request the room

Send voice.requestRoom over the normal Osmium API connection. The request contains the channel’s refs.ChatRef.

The response is voice.RoomInfo:

2. Open the voice connection

Create a voice engine with endpoint, token, the current user’s ID, and roomId. The engine opens the voice WebSocket and creates a WebRTC peer connection.

The WebRTC peer connection starts with audio and video transceivers. The client creates an SDP offer and collects the codecs it can send and receive. It sends those details in a prepare packet:

{
  "type": "prepare",
  "data": {
    "protocol": "webrtc",
    "sdp": "<client SDP offer>",
    "codecs": [
      {
        "name": "opus",
        "kind": "audio",
        "payloadType": 111,
        "parameters": {
          "minptime": "10",
          "useinbandfec": "1",
          "usedtx": "1"
        }
      }
    ]
  }
}

The SDP in this packet is deliberately small and non-standard from the voice server’s point of view. It is used to carry the WebRTC connection identity, not to negotiate the media layout. The server reads the DTLS fingerprint from the SDP, then uses the JSON codecs list and later updateAudio/updateVideo packets for codecs and SSRCs.

At the protocol level, the minimum is:

The server does not use the client SDP’s codec lines, payload lists, SSRC lines, ICE credentials, or ICE candidates to build the media room. Those values are carried out of band: codecs in prepare.codecs and roomConfig.codecs, and SSRCs in updateAudio/updateVideo. A browser still needs to create a normal RTCPeerConnection offer with its usual ICE, DTLS, and media boilerplate so it can apply the server’s answer; do not hand-write a reduced SDP unless your WebRTC implementation can still use it.

3. Authenticate the session

After the WebSocket is open, the engine sends a connect packet for the main room:

{
  "type": "connect",
  "data": {
    "roomId": "<roomId>",
    "userId": "<userId>",
    "token": "<token>"
  }
}

The server answers with ready. It assigns a sessionId to this connection and includes the server’s media address (ip and port). Keep the sessionId: participant and stream updates use it to identify this connection.

4. Finish WebRTC negotiation

The voice server responds to prepare with roomConfig. It contains the server’s SDP answer and the codecs enabled for the room:

{
  "type": "roomConfig",
  "data": {
    "sdp": "<server SDP answer>",
    "codecs": [
      {
        "name": "opus",
        "kind": "audio",
        "payloadType": 111,
        "parameters": {}
      }
    ]
  }
}

The client applies the answer, selects the enabled codecs, and completes the peer connection. Audio and video now travel over WebRTC; the WebSocket remains the signaling and control channel. The server’s SDP answer supplies the connection details (ICE candidates, ICE credentials, and its DTLS fingerprint). The codecs array supplies the negotiated media capabilities separately.

Codec objects

Each codec entry has this shape:

The server accepts Opus for audio and VP8, VP9, H.264, H.265, and AV1 for video. It matches video parameters as follows:

Missing video parameters use the server defaults: H.264 42e01f, packetization mode 1, and level asymmetry 1; VP9 profile 0; H.265 profile 1; and AV1 profile 0, level index 5, and tier 0. The server may rewrite payloadType, parameters, and rtxPayloadType to the values shared by the room. Use the values from the latest roomConfig or roomConfigUpdate when sending updateVideo.

Signaling messages

Voice signaling packets are JSON objects with a type and a data object. The client and server use these messages after the initial connection:

subRoomId is optional on these packets. The main room omits it. A subroom has its own WebRTC transport and codec intersection, but it uses the same signaling WebSocket.

Screen-share subrooms

Camera and microphone streams use the main room. Screen sharing uses a subroom whose ID is the sharing user’s main-room sessionId:

1. The sharing client creates the subroom by sending prepare with subRoomId set to its own sessionId, then sends join for that same ID.
2. Other clients send prepare for that subroom and then join the sharer’s sessionId.
3. The subroom sends its own clientConnected, roomConfig, roomConfigUpdate, and media update packets. Route each packet by subRoomId.
4. When sharing stops, the owner sends updateVideo without ssrc (or leaves the subroom). The server closes an owner-created subroom when its owner stops the video stream.

Screen-share audio, when present, is an updateAudio in the same subroom. A client can join another user’s subroom to receive that user’s screen and shared audio without moving its main voice connection.

Participants and media

clientConnected gives the client one or more { userId, sessionId } pairs. When a participant publishes media, the server sends an updateAudio or updateVideo packet with that session and its SSRC. The client maps the SSRC to a WebRTC receive transceiver and waits for the corresponding track.

updateAudio

The client publishes its local audio state without a session field. Omit ssrc to stop publishing audio:

{
  "type": "updateAudio",
  "data": {
    "subRoomId": "<optional subroom>",
    "ssrc": 123456,
    "producerPaused": false,
    "consumerPaused": false
  }
}

The server adds the owning participant when it forwards the update:

{
  "type": "updateAudio",
  "data": {
    "subRoomId": "<optional subroom>",
    "session": { "userId": "42", "sessionId": "99" },
    "ssrc": 654321,
    "producerPaused": false,
    "consumerPaused": false
  }
}

The forwarded SSRC identifies the receiving client’s consumer stream. producerPaused is the source participant’s mute state: the server pauses that participant’s audio producer. consumerPaused is the receiving participant’s pause state: when it is true, the server pauses all consumers for that participant, so the participant receives no audio. This is enforced server-side for privacy and security; muting a local player is not an equivalent substitute.

updateVideo

The client publishes its local video or screen stream with the negotiated codec. Omit ssrc to stop the stream:

{
  "type": "updateVideo",
  "data": {
    "subRoomId": "<optional subroom>",
    "ssrc": 234567,
    "rtxSsrc": 234568,
    "activeCodec": {
      "name": "vp8",
      "kind": "video",
      "payloadType": 106,
      "parameters": {}
    }
  }
}

The server forwards the same shape with a session field and a recipient-specific ssrc/rtxSsrc. activeCodec.payloadType must be one of the video codecs in the current room configuration.

For local media, the client sends its own SSRC in the matching update packet. producerPaused pauses the local audio producer (the microphone). consumerPaused pauses every incoming consumer on the server, including audio, camera video, and screen sharing. Use it when the participant is deafened or needs a server-enforced receive pause.

When the user leaves, send leave for any sub-room, stop the local tracks, and close the voice engine. The engine sends heartbeats while connected and can reconnect the WebSocket; a resumed connection sends connect with the previous sessionId and receives fresh participant state.