Beyond Single-Direction Steering: Layer-Adaptive and SVD-Based Activation Steering for Speaker-Identity Suppression in Zero-Shot TTS

Research Overview

Zero-shot text-to-speech (TTS) systems synthesize personalized voices from short acoustic prompts, but pose severe misuse risks when synthesizing individuals who never consented. Existing speaker unlearning requires compute-intensive model retraining. Activation steering offers training-free inference-time intervention by perturbing representations with speaker difference vectors. This work investigates two fundamental extensions beyond canonical 1D steering: (1) multi-direction orthonormal subspace steering via thin Singular Value Decomposition (SVD, ranks \(K \in \{1, 2, 4, 8\}\)), and (2) layer-adaptive steering schedules (manual depth-differentiated splits and activation-divergence Dynamic-\(\alpha\)). We evaluate across 7,010 generated audio files on F5-TTS (Flow-Matching) and X-Voice Stage-1 (0.4B Diffusion Transformer) over LibriSpeech, VCTK, and LibriTTS.

Section A: SVD Subspace Rank Scaling & Dynamic-α

N=322 Matched Utterances

Loading description...

Empirical Finding
Loading findings...
Aggregate Cohort Evaluation Measurements

SIM: Speaker similarity. Lower values indicate stronger identity suppression. WER: Word error rate. PESQ: Objective speech quality proxy. Scores reflect aggregate cohort evaluations.

1. Select Utterance for Audition

Ground-Truth Reference Transcript Sample: — Speaker: — Duration: — s
Select an utterance to inspect reference text.
Ready

2. Matched Audio Outputs • Utterance —

Compare the exact same speech utterance synthesized under baseline unsteered vs. experimental steering conditions.

Condition / Steering Method Description Audio Playback