A Lightweight Real-Time Karaoke System with Improved Singing Voice Separation via Fundamental Frequency Estimation Integration

Abstract

With the fast-paced progress of deep learning, music source separation models have achieved impressive results. However, high separation quality often comes at the cost of complex architectures, limiting real-time performance. This study enhances singing voice separation by improving MMDenseNet and developing a real-time karaoke system that captures and separates accompaniment from played songs. We integrate traditional fundamental frequency (F0) estimation into complex ratio mask estimation, forming an integrated complex mask framework. On MUSDB18, we raise SDR from 3.87 dB to 6.44 dB over the original MMDenseNet (+2.57 dB absolute; +61% relative) while using only 0.33 M parameters—the smallest among compared methods. To enable real-time applications, we redesign the model with a shallow neural network that learns from traditional F0 estimation. The final system achieves real-time throughput (RTF < 1), e.g., 0.80 on an Intel i5-10500 PC and 0.88 on an NVIDIA Jetson AGX Xavier, demonstrating its potential for karaoke and related applications.

Architecture

The architecture of our proposed system can be divided into three main stages: F0-integrated MMDenseNet, complex mask estimation, and real-time streaming processing. The core separation model, MMDenseNet, adopts a Multi-band Dense architecture composed of three parallel branches—high band, full band, and low band—each built from an encoder-decoder structure of stacked DenseBlocks, as illustrated in Fig. 1. Within each DenseBlock, an Efficient Channel Attention (ECA) module is inserted after every convolution block to capture channel-wise dependencies and emphasize features critical for vocal separation. In parallel, a lightweight full-band F0 estimation module, distilled from the Melodia algorithm, predicts a salience map that represents the vocal fundamental frequency information. This F0 salience map is injected into an integrated complex mask framework, where the model jointly estimates both magnitude and phase components of the target spectrogram rather than relying on magnitude-only masking. The overall system combines the separation model's output with the F0-derived salience map to construct the integrated complex mask, which is then applied to the input spectrogram to produce the estimated spectrogram for vocal and accompaniment separation. To support real-time deployment, the system further incorporates a streaming processing pipeline with pending, processing, and overlap buffers, enabling continuous audio input to be processed in fixed-size blocks with minimized latency for practical karaoke applications on both PC and edge devices such as the NVIDIA Jetson AGX Xavier.