Realtime (WebSocket)

Bidirectional, low-latency synthesis over a WebSocket connection.

For interactive, realtime use cases the Voice API exposes a WebSocket endpoint that bridges your connection to the synthesis engine. Authenticate by passing your API key as the api_key query parameter (browsers and most WS clients can’t set custom headers during the handshake).

GET wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream?api_key=flv_...

A valid key upgrades the connection (101 Switching Protocols); an invalid key fails the handshake with 401. On the JSON control frames you send, the service replaces voice_id with your enrolled engine voice for that id — so you always reference your own Flam voice ids (the 24-char ids from List voices), never the engine’s. Everything else is forwarded verbatim in both directions.

Session model

One connection is one session; a session can run up to 4 concurrent contexts (independent synthesis streams), each keyed by a client-chosen context_id. You start a context, optionally keep appending text to it (ideal for LLM output arriving in chunks), and finalize it when the turn is done. Audio for each context streams back as it’s synthesized.

• You send — JSON text frames. The type field dispatches the message (see below).
• You receive — JSON text frames (status/control) interleaved with binary frames (raw audio).
• Limits — text frames over 1 MB close the socket (1009); a session idle for 600 s is closed by the server (session_ended, reason idle); per-key connection caps close with 4429.

Messages you send

task_start — begin a context

Field	Required	Default	Notes
type	yes	—	"task_start"
context_id	yes	—	client-unique per session; duplicate → context_already_exists
voice_id	yes	—	your Flam voice id; unknown/unenrolled → voice_not_found
text	no	""	text to synthesize; ≥ 10,000 chars → text_too_long; empty with finalize: true → text_required
finalize	no	false	true = no more text for this context; flush and finish
language	no	"en"	invalid → invalid_language
output_format	no	f32le	one of the audio formats; invalid is coerced to the default
sample_rate	no	24000	native rate is 24000; only upsampling above it
num_step	no	32	1–32; quality vs latency
expressiveness	no	—	0–1; overrides guidance_scale
guidance_scale	no	2.0	0–10
length_scale	no	1.0	(0, 5]; speaking-rate control
idle_flush_ms	no	800	clamped 200–5000; auto-flush a pending sentence after this much silence from you

task_continue — append text to an open context

{ "type": "task_continue", "context_id": "ctx-1", "text": "And another sentence.", "finalize": false }

Send as many as you like; set finalize: true on the last one to flush and finish the context.

task_cancel — abort a context

{ "type": "task_cancel", "context_id": "ctx-1" }

session_end — graceful close

{ "type": "session_end" }

The server replies with session_ended and closes the socket.

Messages you receive

task_started · text frame

{context_id, sample_rate, audio_format} — the effective rate and format for the context. Use these (not what you requested) to configure playback.

audio_chunk · text frame + binary frame

{context_id, sentence_index, chunk_index, audio_duration_s, byte_length} — immediately followed by one binary frame of raw PCM (or opus) bytes. Pair them in order: each audio_chunk announces the binary frame that follows it.

sentence_done · text frame

{context_id, sentence_index, text, audio_duration_s} — one sentence fully synthesized.

task_done · text frame

{context_id, num_sentences, total_audio_duration_s, profile, trace_id} — the context is complete. profile carries timing detail (ttfb_ms, generate_ms, overall_rtf, …).

task_cancelled · text frame

{context_id, sentences_completed, sentences_dropped, trace_id} — reply to task_cancel.

task_failed · text frame

{context_id, sentences_completed, code, trace_id} — terminal failure for one context.

error · text frame

{context_id?, code, message, fatal} — validation or billing errors. When fatal is true the session is closing; otherwise only the offending message was rejected.

session_ended · text frame

{reason} — idle or client_request. The socket closes after this frame.

Audio formats

Binary frames carry raw PCM (no container) in the context’s output_format:

Format	Bytes/sample	Notes
f32le	4	default; float32 in [-1, 1)
s16le	2	best choice for web playback and WAV wrapping
s24le / s32le / f64le	3 / 4 / 8	niche pipelines
opus	—	48 kHz, 20 ms frames; can stall under heavy chunking — prefer PCM on WS

Audio is mono at the effective sample_rate from task_started (native 24000 Hz). Wrap or play it client-side; nothing on this path returns WAV.

Billing

WebSocket synthesis is metered exactly like /speak and SSE streaming: you’re billed per character of text on every task_start / task_continue frame, against your plan’s monthly character allowance.

• The handshake checks your plan before upgrading: streaming not included in your plan → 403; character quota exhausted → 402 (when billing enforcement is active). The handshake response carries X-Billing-Remaining with your remaining characters.
• If your workspace runs out of credits mid-session (strict enforcement), the text frame that exceeded the balance is not forwarded — you receive a fatal error frame and the session closes:

{ "type": "error", "code": "insufficient_credits", "message": "You're out of credits - top up to continue.", "fatal": true }

Usage appears in your dashboard analytics and on the monthly invoice like any other synthesis.

Close codes

Code	Meaning
4401	unauthorized (bad/missing key at the engine layer)
4429	per-key connection cap exceeded
1009	text frame over 1 MB

End-to-end example

JavaScript (browser)

const url =
  "wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream" +
  `?api_key=${process.env.FLAM_API_KEY}`;

const ws = new WebSocket(url);
ws.binaryType = "arraybuffer";

// Gapless playback: schedule each PCM chunk on a running Web Audio playhead.
const audioCtx = new AudioContext();
let playhead = 0;
let sampleRate = 24000;

ws.addEventListener("open", () => {
  ws.send(JSON.stringify({
    type: "task_start",
    context_id: "ctx-1",
    voice_id: "665f1a2b3c4d5e6f7a8b9c0d", // your Flam voice id
    text: "Hello from Finesse, streaming in realtime.",
    finalize: true,                        // one-shot turn; omit to keep appending
    output_format: "s16le",
    sample_rate: 24000,
  }));
});

ws.addEventListener("message", (event) => {
  if (typeof event.data === "string") {
    const msg = JSON.parse(event.data);
    if (msg.type === "task_started") sampleRate = msg.sample_rate;
    if (msg.type === "task_done") ws.close();
    if (msg.type === "task_failed" || (msg.type === "error" && msg.fatal)) {
      console.error("synthesis failed", msg);
      ws.close();
    }
    return;
  }
  // Binary frame: raw s16le mono PCM for the most recent audio_chunk.
  const i16 = new Int16Array(event.data);
  const f32 = Float32Array.from(i16, (s) => s / 32768);
  const buf = audioCtx.createBuffer(1, f32.length, sampleRate);
  buf.getChannelData(0).set(f32);
  const src = audioCtx.createBufferSource();
  src.buffer = buf;
  src.connect(audioCtx.destination);
  playhead = Math.max(playhead, audioCtx.currentTime + 0.05);
  src.start(playhead);
  playhead += buf.duration;
});

The first connection after a long idle period may be slow while the synthesis engine warms up. For latency-sensitive apps, open the socket ahead of the first turn and keep it alive.

Need a one-way, fully self-described stream (e.g. for server-side playback)? Stream speech (SSE) uses a fixed start / chunk / done / error event shape. Use the WebSocket when you need a bidirectional, turn-based session.

Prompt

Copy this into your coding agent (Claude Code, Cursor, Codex…) to implement this endpoint.

You are implementing a typed client for one Finesse Voice API endpoint: "Realtime (WebSocket)".
Everything you need is in this prompt. Do not invent anything that is not specified here.

WHAT TO BUILD
- A RealtimeSession client (name it per the project's conventions) for bidirectional, low-latency
  synthesis over one WebSocket:
    connect()                           open the socket
    startContext(options)               send task_start; returns a context handle
    context.append(text, finalize?)     send task_continue (finalize: true on the last text)
    context.cancel()                    send task_cancel
    end()                               send session_end and wait for session_ended
  plus typed events per context (started, audio, sentence done, done, cancelled, failed, error)
  via an async iterator or callbacks, whichever fits the stack.
- Use the stack's standard WebSocket client: the global WebSocket in browsers and Node 22+, the
  "ws" package on older Node, "websockets" in Python. Receive binary frames as ArrayBuffer/bytes.
- Detect the stack first: look for package.json, tsconfig.json, pyproject.toml,
  requirements.txt, go.mod, Gemfile, composer.json and similar. Use the project's language, HTTP
  client, module layout, error-handling conventions and test runner. If there is no project yet or
  the stack is unclear, use TypeScript on Node 18+ with the built-in fetch and no new dependencies.
- If the project already has a Finesse API client module, add this to it (reuse its auth, base
  URL, error type and retry helper) instead of creating a parallel one.
- Keep the base URL (https://fi.production.flamapis.com) in one constant or config value.

ENDPOINT AND AUTH
- URL: wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream?api_key=<key>
- The API key goes in the api_key QUERY PARAMETER, because browsers and most WebSocket clients
  cannot set headers during the handshake. URL-encode it.
- Read the key from the environment variable FLAM_API_KEY (or the project's secrets mechanism).
  Never hard-code, log or commit it, and redact it from any logged URL.
- Keep this client server-side. The key can synthesize with every voice in the workspace, so it
  must not be shipped in a public browser bundle. If the app needs browser playback, relay audio
  from your server, or stop and ask me how the key should reach the browser.
- A valid key upgrades the connection (101 Switching Protocols). An invalid key fails the
  handshake with HTTP 401.
- The handshake also checks the plan BEFORE upgrading: 403 if streaming is not included in the
  plan, 402 if the character quota is exhausted (when billing enforcement is active). The
  handshake response carries X-Billing-Remaining (remaining characters). Surface these as distinct
  connect errors; do not auto-reconnect on 401, 402 or 403.

VOICE IDS
- Always send your own Flam voice ids (the 24-character hex ids from
  GET https://fi.production.flamapis.com/flam-voice-svc/api/v1/public/voices, whose voices are in data.voices). The service replaces voice_id with
  the enrolled engine voice on the frames you send; everything else is forwarded verbatim.
- An unknown or unenrolled voice returns an error frame with code "voice_not_found".

SESSION MODEL
- One connection is one session. A session can run up to 4 CONCURRENT contexts (independent
  synthesis streams), each keyed by a client-chosen context_id that is unique within the session.
  Enforce the limit of 4 open contexts client-side and generate unique ids (e.g. a counter or UUID).
- Start a context, optionally keep appending text (ideal for LLM output arriving in chunks), and
  finalize it when the turn is done. Audio for each context streams back as it is synthesized.
- You send JSON text frames; the "type" field dispatches the message.
- You receive JSON text frames (status/control) interleaved with binary frames (raw audio).

MESSAGES YOU SEND
task_start (begin a context):
  field           required  default  notes
  type            yes       -        "task_start"
  context_id      yes       -        unique per session; a duplicate gives context_already_exists
  voice_id        yes       -        your Flam voice id; unknown/unenrolled gives voice_not_found
  text            no        ""       text to synthesize; 10,000 chars or more gives text_too_long;
                                     empty text with finalize: true gives text_required
  finalize        no        false    true = no more text for this context; flush and finish
  language        no        "en"     invalid gives invalid_language
  output_format   no        f32le    one of the audio formats below; invalid is coerced to default
  sample_rate     no        24000    native rate is 24000; only upsampling above it
  num_step        no        32       1-32; quality vs latency
  expressiveness  no        -        0-1; overrides guidance_scale
  guidance_scale  no        2.0      0-10
  length_scale    no        1.0      greater than 0, up to 5; speaking-rate control
  idle_flush_ms   no        800      clamped to 200-5000; auto-flush a pending sentence after this
                                     much silence from you

    {
      "type": "task_start",
      "context_id": "ctx-1",
      "voice_id": "665f1a2b3c4d5e6f7a8b9c0d",
      "text": "Hello from Finesse, streaming in realtime.",
      "finalize": true,
      "output_format": "s16le",
      "sample_rate": 24000
    }

task_continue (append text to an open context; send as many as you like, and set finalize: true
on the last one to flush and finish):

    { "type": "task_continue", "context_id": "ctx-1", "text": "And another sentence.", "finalize": false }

task_cancel (abort a context):

    { "type": "task_cancel", "context_id": "ctx-1" }

session_end (graceful close; the server replies with session_ended and closes the socket):

    { "type": "session_end" }

Validate locally before sending: required fields present, context_id unique and at most 4 open,
task_start text under 10,000 characters, num_step 1-32, expressiveness 0-1, guidance_scale 0-10,
length_scale in (0, 5], and every text frame well under 1 MB.

MESSAGES YOU RECEIVE (JSON text frames, each with a "type")
- task_started: {context_id, sample_rate, audio_format}. The EFFECTIVE rate and format for the
  context. Use these, not what you requested, to configure playback.
- audio_chunk: {context_id, sentence_index, chunk_index, audio_duration_s, byte_length},
  IMMEDIATELY FOLLOWED BY ONE BINARY FRAME of raw PCM (or opus) bytes. Pair them in order: each
  audio_chunk announces the binary frame that follows it. Route that binary frame to the announced
  context_id, and check its length against byte_length (log a warning on mismatch).
- sentence_done: {context_id, sentence_index, text, audio_duration_s}. One sentence is fully
  synthesized.
- task_done: {context_id, num_sentences, total_audio_duration_s, profile, trace_id}. The context
  is complete; profile carries timing detail (ttfb_ms, generate_ms, overall_rtf, and more; type it
  as an open record).
- task_cancelled: {context_id, sentences_completed, sentences_dropped, trace_id}. Reply to
  task_cancel.
- task_failed: {context_id, sentences_completed, code, trace_id}. Terminal failure for one
  context.
- error: {context_id?, code, message, fatal}. Validation or billing errors. When fatal is true the
  session is closing; otherwise only the offending message was rejected (fail just that context
  if context_id is set).
- session_ended: {reason}, where reason is "idle" or "client_request". The socket closes after it.
Documented error codes: context_already_exists, voice_not_found, text_too_long, text_required,
invalid_language, insufficient_credits. Keep code typed as string; unknown codes must not crash.
Ignore unknown message types.

AUDIO FORMATS (binary frames carry raw PCM, no container, in the context's output_format)
  format                bytes/sample  notes
  f32le                 4             default; float32 in [-1, 1)
  s16le                 2             best choice for web playback and WAV wrapping
  s24le / s32le / f64le 3 / 4 / 8     niche pipelines
  opus                  -             48 kHz, 20 ms frames; can stall under heavy chunking; prefer
                                      PCM on WebSocket
Audio is MONO at the effective sample_rate from task_started (native 24000 Hz). Nothing on this
path returns WAV: provide a small helper that wraps mono s16le PCM in a WAV header when needed.
Default the client to output_format "s16le" unless the caller chooses otherwise, and state it.

LIMITS AND CLOSE CODES
- Text frames over 1 MB close the socket with code 1009.
- A session idle for 600 s is closed by the server (session_ended with reason "idle"). Surface
  that as a normal end, not an error.
- Close codes:
    4401  unauthorized (bad or missing key at the engine layer): do not reconnect
    4429  per-key connection cap exceeded: back off before reconnecting and reuse one socket for
          many contexts instead of opening more connections
    1009  text frame over 1 MB: a client bug; do not reconnect with the same payload
- On any close, fail every still-open context with a typed error carrying the close code/reason.
- Reconnect policy: never reconnect automatically on 401/402/403/4401. For other abnormal closes,
  reconnect only if the caller opted in, with exponential backoff, and never replay contexts that
  already sent text (replaying bills the text again).

BILLING
- Billed per character of text on every task_start and task_continue frame, against the plan's
  monthly character allowance, exactly like the /speak and SSE endpoints.
- If the workspace runs out of credits mid-session (strict enforcement), the frame that exceeded
  the balance is not forwarded; you receive a fatal error frame and the session closes:

    { "type": "error", "code": "insufficient_credits", "message": "You're out of credits - top up to continue.", "fatal": true }

  Map it to a distinct out-of-credits error.

LATENCY
- The first connection after a long idle period may be slow while the synthesis engine warms up.
  Allow a generous, configurable connect/first-audio timeout, and for latency-sensitive apps
  connect ahead of the first turn and keep the socket alive (reuse it across turns).

ACCEPTANCE CRITERIA
- Connects with the key in the api_key query parameter (never logged) and surfaces 401/402/403
  handshake failures distinctly.
- Sends only the documented message types and fields; enforces at most 4 open contexts and unique
  context ids.
- Pairs every audio_chunk with the next binary frame and delivers the bytes to the right context,
  using the effective sample_rate/audio_format from task_started.
- Handles task_done, task_cancelled, task_failed, non-fatal and fatal error frames, session_ended,
  and close codes 4401/4429/1009 as described.
- Supports multi-frame turns (task_start, then task_continue ..., finalize: true), e.g. feeding LLM
  output.

TEST PLAN
- Unit tests against a local fake WebSocket server: single-shot turn (task_start with finalize:
  true, s16le) collects PCM and resolves on task_done; audio_chunk/binary pairing with two
  interleaved contexts; a fifth concurrent context is rejected locally; non-fatal voice_not_found
  fails only that context; fatal insufficient_credits fails all contexts and closes; task_cancel
  resolves with task_cancelled; close 4429 and 4401 map to the right errors without reconnect loops;
  session_end resolves on session_ended.
- Smoke script:
- Get a ready voice id from FLAM_VOICE_ID, or by calling GET https://fi.production.flamapis.com/flam-voice-svc/api/v1/public/voices (same
  X-API-Key header; voices are in data.voices) and picking the first one whose status is "ready".
  - Connect, send one task_start with finalize: true, output_format "s16le", sample_rate 24000 and
    the text "Hello from Finesse, streaming in realtime.", collect PCM until task_done, wrap it as
    a mono WAV at the task_started sample_rate, write realtime.wav, and print time to first audio
    chunk, total bytes and task_done.profile.
  - Then run a second context as a multi-frame turn: task_start with "The quick answer is yes.",
    task_continue " The longer answer involves a few tradeoffs worth knowing.", and task_continue
    " Ask me about any of them." with finalize: true. Send session_end and exit 0 on
    session_ended; exit non-zero on any error.

AMBIGUITIES TO ASK ABOUT, NOT GUESS
- Whether a finalize-only task_continue with empty text is accepted (the spec only says empty text
  with finalize: true is rejected on task_start). Until confirmed, finalize by setting
  finalize: true on the last task_continue that carries text.
- Any per-context or per-session total text limit beyond the documented per-frame limits.

RULES
- Implement only what this spec describes. Do not invent parameters, fields, headers, endpoints or
  error codes. If something is ambiguous, missing, or conflicts with the existing codebase, stop
  and ask me before guessing.
- Match the project's existing style and conventions; do not refactor unrelated code.