Speaker Recognition with Microsoft Wav2Vec2 Pre-Trained Model

This code showcases a practical implementation of speaker recognition using Microsoft's Wav2Vec2 pre-trained model. It outlines the essential steps for building a robust speaker recognition system.

Code Breakdown

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, Dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

# Load pre-trained model and processor
model = Wav2Vec2ForCTC.from_pretrained('microsoft/wav2vec2-large-960h-lv60-self')
processor = Wav2Vec2Processor.from_pretrained('microsoft/wav2vec2-large-960h-lv60-self')

# Define custom dataset for speaker recognition
class SpeakerDataset(Dataset):
    def __init__(self, audio_files, labels):
        self.audio_files = audio_files
        self.labels = labels
    
    def __len__(self):
        return len(self.audio_files)
    
    def __getitem__(self, index):
        audio_file = self.audio_files[index]
        label = self.labels[index]
        
        # Load audio file and apply processor
        audio_input, sample_rate = torchaudio.load(audio_file)
        input_values = self.processor(audio_input, sampling_rate=sample_rate, return_tensors='pt').input_values
        
        return input_values, label

# Define model architecture for speaker recognition
class SpeakerRecognitionModel(nn.Module):
    def __init__(self, num_classes):
        super(SpeakerRecognitionModel, self).__init__()
        self.num_classes = num_classes
        self.wav2vec2 = Wav2Vec2ForCTC.from_pretrained('microsoft/wav2vec2-large-960h-lv60-self')
        self.dropout = nn.Dropout(0.3)
        self.fc1 = nn.Linear(1024, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, num_classes)
    
    def forward(self, input_values):
        features = self.wav2vec2(input_values).last_hidden_state
        features = self.dropout(features)
        features = F.relu(self.fc1(features))
        features = self.dropout(features)
        features = F.relu(self.fc2(features))
        logits = self.fc3(features)
        
        return logits

# Instantiate model and define training parameters
num_classes = 10
model = SpeakerRecognitionModel(num_classes)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)

# Define training and validation data loaders
train_dataset = SpeakerDataset(train_audio_files, train_labels)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_dataset = SpeakerDataset(val_audio_files, val_labels)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)

# Train the model
for epoch in range(10):
    model.train()
    train_loss = 0.0
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        train_loss += loss.item() * inputs.size(0)
    train_loss = train_loss / len(train_loader.dataset)

    model.eval()
    val_loss = 0.0
    val_corrects = 0
    with torch.no_grad():
        for inputs, labels in val_loader:
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            val_loss += loss.item() * inputs.size(0)
            _, preds = torch.max(outputs, 1)
            val_corrects += torch.sum(preds == labels)
    val_loss = val_loss / len(val_loader.dataset)
    val_acc = val_corrects.double() / len(val_loader.dataset)

    print(f'Epoch {epoch + 1}: Train Loss {train_loss:.4f} Val Loss {val_loss:.4f} Val Acc {val_acc:.4f}')

Code Explanation

1. Pre-trained Model & Processor: The code starts by loading the pre-trained Wav2Vec2 model and its associated processor from the Hugging Face Transformers library. These are essential for extracting features from audio inputs.
2. Custom Dataset: A SpeakerDataset class is defined to handle loading audio files and labels specific to speaker recognition. The __getitem__ method loads the audio data, applies the processor, and returns the processed input values along with the corresponding label.
3. Speaker Recognition Model: A custom SpeakerRecognitionModel is constructed using the pre-trained Wav2Vec2 as a feature extractor. The model's architecture further includes dropout layers for regularization and fully connected layers for classification.
4. Training Loop: The code defines a training loop that iterates over epochs, processes data in batches, calculates loss, and updates model parameters using the Adam optimizer. Validation is performed periodically to track the model's performance.

Getting Started

1. Install Dependencies: Ensure you have the required libraries installed: pip install torch transformers torchaudio.
2. Data Preparation: Create a dataset consisting of audio files and corresponding speaker labels. The SpeakerDataset class can be modified to handle your specific dataset format.
3. Training: Modify the code to load your dataset, adjust hyperparameters (e.g., batch size, learning rate), and train the model.

Further Exploration

This code serves as a foundation. You can explore enhancements such as:

• Data Augmentation: Apply techniques like noise addition, time stretching, and pitch shifting to increase the robustness of your model.
• Different Model Architectures: Experiment with other architectures like convolutional neural networks (CNNs) or recurrent neural networks (RNNs).
• Evaluation Metrics: Implement additional evaluation metrics like Equal Error Rate (EER) or Receiver Operating Characteristic (ROC) curves for more comprehensive performance analysis.

Happy coding! Let me know if you have any questions.