Neural codecs, full-duplex voice, and the expressive face

Build the classic pipeline and turn on metrics so you can read the additive latency floor you are trying to beat.

1 · Install

# one venv; the extras pull VAD, transports, providers, smart-turn
pip install "pipecat-ai[silero,webrtc,openai,deepgram,cartesia,runner,local-smart-turn-v3]" \
            pipecat-ai-small-webrtc-prebuilt python-dotenv loguru

2 · Build the pipeline (transport.input → STT → LLM → TTS → transport.output)

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask, PipelineParams
from pipecat.audio.vad.silero import SileroVADAnalyzer
# services: DeepgramSTTService, OpenAILLMService, CartesiaTTSService, + a smart-turn analyzer

task = PipelineTask(
    Pipeline([
        transport.input(),     # Silero VAD + smart-turn-v3 on the transport
        stt, llm, tts,
        transport.output(),
    ]),
    params=PipelineParams(
        enable_metrics=True,        # ← the whole point of this lab
        enable_usage_metrics=True,
    ),
)

3 · Run, talk, read the metrics

# Pipecat logs TTFB / processing time per service when metrics are on.
# Sum: turn-detection wait + STT + LLM TTFT + TTS TTFB = your floor.

Expected output: per-stage TTFB lines in the log; add them up for a single end-to-end "mouth-closes → voice-starts" number (typically ~600–900 ms on commodity cloud services).

• The cascade runs and you can hold a (turn-based) conversation through the browser transport.
• You can read off the additive latency floor from the metrics — write the number down; it is your baseline for §C.

Path A — MLX on Mac / phone-class hardware

pip install moshi_mlx
python -m moshi_mlx.local_web -q 4   # 4-bit quantized; opens a local web client

Path B — PyTorch on the GPU host (~24 GB VRAM)

pip install -U moshi
python -m moshi.server                # serves the full-precision model

Build the intent watcher on the Inner-Monologue text stream

1. Subscribe to Moshi's text token stream (the inner-monologue output), not the audio.
2. Accumulate tokens into rolling phrases; run a lightweight intent check (keyword/regex first, a small classifier later).
3. On a physical-task intent ("pick up the…", "bring me…"), emit a do_physical_task tool call to the orchestrator. cross-link The full tool schema lives on page 6 · The Creature.
4. The voice stream keeps running uninterrupted — the watcher is a passive tap, never a blocking gate.

def on_text_token(tok):
    buf.append(tok)
    if intent := detect_physical_task("".join(buf)):
        emit_tool_call("do_physical_task", intent)   # schema: page 6

Optional (lowest priority): persona finetune via github.com/kyutai-labs/moshi-finetune — skip if time-poor.

• A sub-~500 ms-feel, interruptible spoken conversation runs on the phone (Path A) or host (Path B).
• The intent watcher fires do_physical_task from the text stream without interrupting speech.

1. Create track2_voice/face/face.html: a self-contained canvas/SVG face (eyes, pupils, mouth) with idle blink + sway as CSS keyframes — start from the live demo above.
2. Open a WebSocket to the orchestrator; parse messages {emotion, mouth, speaking} as JSON.
3. On each message: set mouth-openness from mouth (RMS), switch the expression preset from emotion, and toggle backchannel idle behaviour when speaking===false.
4. Compute mouth upstream: per Moshi output frame, RMS of the samples, normalized to 0–1 (clamp + smooth).
5. Add gaze: request the phone camera, run lightweight face-tracking, map the detected face offset to a small pupil bias.
6. Open face.html fullscreen on the phone; point Moshi's audio-out at the RMS computer feeding the socket.

const ws = new WebSocket(WS_URL);
ws.onmessage = (e) => {
  const {emotion, mouth, speaking} = JSON.parse(e.data);
  setMouthOpen(mouth);          // 0..1 RMS → mouth ry
  setExpression(emotion);       // preset from page-6 affect map
  setBackchannel(!speaking);    // nod/“mm-hm” while listening
};

• The phone face lip-syncs to audio RMS (mouth tracks Moshi's voice).
• The face shifts affect on emotion messages and blinks/sways when idle.
• Gaze biases toward the user and backchannels appear while listening.