CVCodeVibers Telegram

Technical branch for developers Assemble manually

Local voice, video, and editing pipeline

We’ll show how to combine Whisper, Silero, OpenCV, and FFmpeg into a single local pipeline: recording → transcription → analysis → editing → export—all without external APIs.

Discuss implementation All Academy Topics

What is a local voice, video, and editing pipeline?

This is an audio/video stream processing pipeline where each stage—recording, transcription, emotion analysis, segment trimming, and editing—is performed locally on your machine using open-source models and tools. Goal: full control, privacy, no latency, and no cloud-based cost variables.

Pipeline architecture

flowchart TD  
    A[Input: microphone/camera] --> B[Capture: PyAudio/OpenCV]  
    B --> C[Preprocessing: normalization, VAD]  
    C --> D[Transcription: Whisper Tiny/Small]  
    D --> E[Analysis: emotions (Wav2Vec2), key phrases]  
    E --> F[Segmentation: speech/pause timestamps]  
    F --> G[Editing: FFmpeg + OpenCV]  
    G --> H[Export: MP4/MP3 + JSON metadata]

Minimalist example: capture + transcription

import pyaudio
import wave
import whisper
import numpy as np
from silero_vad import load_silero_vad, get_speech_timestamps

# 1. Audio capture (16kHz, mono, 10 sec)
CHUNK, FORMAT, CHANNELS, RATE = 1024, pyaudio.paInt16, 1, 16000
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK)
frames = []
for _ in range(0, int(RATE / CHUNK * 10)):
    data = stream.read(CHUNK)
    frames.append(np.frombuffer(data, dtype=np.int16).astype(np.float32) / 32768.0)
stream.stop_stream()
p.terminate()
audio = np.concatenate(frames)

# 2. VAD (Silero) → keep only speech segments
vad_model = load_silero_vad()
speech_ts = get_speech_timestamps(audio, vad_model, sampling_rate=RATE)
if speech_ts:
    voice_audio = np.concatenate([audio[ts['start']:ts['end']] for ts in speech_ts])
else:
    voice_audio = np.array([], dtype=np.float32)

# 3. Transcription (Whisper)
model = whisper.load_model("tiny")
result = model.transcribe(voice_audio, language="ru")
print("Text:", result["text"])

Common mistakes and how to avoid them

  • Whisper is too large → On CPU: “quiet” processor kills: use tiny or small, quantize via GGUF.
  • No VAD → Model wastes tokens on noise/pauses → increased latency and errors.
  • Sample rate incompatibility → Whisper requires 16 kHz, but the camera provides 48 kHz → always resample via resampy or FFmpeg.
  • Assembly without sync timecodes → Video and audio are “drifting” → Save `frame_timestamps` and `audio_timestamps` in JSON.

Practical Case Study: “Auto-Director” for Local Lectures

You’re recording a lecture using a webcam and microphone. Pipeline:

  1. Capture video (OpenCV) and audio (PyAudio) in separate threads.
  2. Transcribe every second (Whisper + sliding window).
  3. Detect pauses and “uninteresting” segments (based on emotion: neutral tone + absence of keywords).
  4. Install the “clean” version: remove pauses >2s, cut out “um” and “uh.”
  5. Export MP4 + JSON with timestamps of remote fragments.

Result: 15-minute lecture → 9 minutes, no pauses or filler. Everything—locally—in 2 minutes.

Pre-Launch Checklist

What to checkWhy
FFmpeg version with libvpx/libaom supportCodecs for high-quality video without cloud
Whisper in FP16 or INT8 (via llama.cpp)CPU acceleration without GPU
Synchronization of frame_pts and audio_timeAvoid desynchronization during editing
VAD check on your noise (office/street)Too aggressive VAD cuts speech
RAM limit (Whisper + OpenCV + FFmpeg ≈ 3–6 GB)Avoid OOM on weak machines

Next step

Go to “How to Assemble an Agent with Tool-Calling” — Learn how to turn editing results (text, timestamps) into AI agent actions: “cut pauses”, “create preview”, “send to Telegram”.

Related Topics