Fusion and Orthogonal Projection for Improved Face-Voice Association

by   Muhammad Saad Saeed, et al.

We study the problem of learning association between face and voice, which is gaining interest in the computer vision community lately. Prior works adopt pairwise or triplet loss formulations to learn an embedding space amenable for associated matching and verification tasks. Albeit showing some progress, such loss formulations are, however, restrictive due to dependency on distance-dependent margin parameter, poor run-time training complexity, and reliance on carefully crafted negative mining procedures. In this work, we hypothesize that enriched feature representation coupled with an effective yet efficient supervision is necessary in realizing a discriminative joint embedding space for improved face-voice association. To this end, we propose a light-weight, plug-and-play mechanism that exploits the complementary cues in both modalities to form enriched fused embeddings and clusters them based on their identity labels via orthogonality constraints. We coin our proposed mechanism as fusion and orthogonal projection (FOP) and instantiate in a two-stream pipeline. The overall resulting framework is evaluated on a large-scale VoxCeleb dataset with a multitude of tasks, including cross-modal verification and matching. Results show that our method performs favourably against the current state-of-the-art methods and our proposed supervision formulation is more effective and efficient than the ones employed by the contemporary methods.


Learning Branched Fusion and Orthogonal Projection for Face-Voice Association

Recent years have seen an increased interest in establishing association...

Learnable PINs: Cross-Modal Embeddings for Person Identity

We propose and investigate an identity sensitive joint embedding of face...

Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

Nowadays, we have witnessed the early progress on learning the associati...

Voice-Face Cross-modal Matching and Retrieval: A Benchmark

Cross-modal associations between voice and face from a person can be lea...

Single-branch Network for Multimodal Training

With the rapid growth of social media platforms, users are sharing billi...

Triplet Similarity Embedding for Face Verification

In this work, we present an unconstrained face verification algorithm an...

Please sign up or login with your details

Forgot password? Click here to reset