Streaming Detections

Use an appendable detection source when predictions arrive after playback has started. This fits uploaded videos, live inference, webcam sessions, and server-sent inference results.

import {
  createMediaSession,
  DetectionFrameRetentionMode,
  type DetectionFrame,
} from "supervision";

const session = await createMediaSession({
  container,
  media: file,
  detections: {
    appendable: {
      datasetId: "upload-1",
      retention: {
        mode: DetectionFrameRetentionMode.PersistAll,
      },
    },
  },
  normalize: { stream: true },
  renderer: {
    autoPlay: true,
  },
});

for await (const frames of streamInferenceFrames(file)) {
  await session.appendDetectionFrames(frames);
}
await session.finalizeDetectionCoverage();
Copy

Gates are on by default, but opening does not wait for predictions that will arrive later. The initial media frame can be bare. Subsequent playback waits for detection coverage and prepared annotations, subject to each gate's wait limit. Pass playbackGate: false to createMediaSession when you would rather the picture keep moving and the annotations land as they arrive: a frame the source does not cover yet then presents without annotations and draws them when the append covering it lands, so inference falling behind slows annotations rather than the video. That one switch answers for detections.playbackGate and renderer.renderPreparation.playbackGate together; set either one's enabled to answer for that gate alone. Detection requiredAheadSeconds is the coverage needed ahead of playback. Preparation requiredAheadSeconds caps the prepared lead a stop may accumulate; it is not a minimum needed to resume.

Which Sources The Gate Reaches

After opening, both gates apply while playback runs. For the media inputs above, a URL, a File, or a Blob, the renderer pulls a decoded sample and holds it before drawing.

A media source that presents its own frames owns the playhead, and the renderer follows it rather than pacing it. createWebVideoEngineMediaRendererSource and openWebVideoEngineMediaSource return that kind of source, and they are what most hosts render video through. The renderer stops that producer when detection coverage or prepared artifacts are missing and starts it again when the wait settles. Wait on coverage yourself when you would rather decide where playback stops:

await session.detectionSource?.waitForRange?.({ startTime: 0, endTime: 2 });
await session.play();
Copy

Appending Results

Append semantic frames in batches. The batch can contain one frame or many frames:

async function appendInferenceResult(frame: DetectionFrame) {
  await session.appendDetectionFrames([frame]);
}
Copy

Appending detections outside the current hot window does not force the renderer to reload the active window. The session tracks source ranges and refreshes when the relevant playback range changes.

Retention Modes

For finite files, use PersistAll so seeking and replay do not require recomputing predictions.

For long-running streams, use a rolling retention window:

retention: {
  mode: DetectionFrameRetentionMode.PersistWindow,
  windowSeconds: 300,
}
Copy

Use MemoryOnly for fully ephemeral streams:

retention: {
  mode: DetectionFrameRetentionMode.MemoryOnly,
  windowSeconds: 60,
}
Copy

Loading State

Subscribe to session state to show buffering or processing UI:

session.subscribe((state) => {
  playButton.disabled = state.playbackBlocked;

  for (const activity of state.activities) {
    console.log(activity.kind, activity.label, activity.progress);
  }
});
Copy

playbackBlocked means playback should wait; media buffering and session errors raise it. Detection coverage raises it only through an enabled detection gate, so playbackGate: false keeps coverage from blocking playback at all. presentationBlocked means the visual frame is still preparing an artifact while playback continues.