Bi-level Feature Alignment for Versatile Image Translation and Manipulation

by   Fangneng Zhan, et al.

Generative adversarial networks (GANs) have achieved great success in image translation and manipulation. However, high-fidelity image generation with faithful style control remains a grand challenge in computer vision. This paper presents a versatile image translation and manipulation framework that achieves accurate semantic and style guidance in image generation by explicitly building a correspondence. To handle the quadratic complexity incurred by building the dense correspondences, we introduce a bi-level feature alignment strategy that adopts a top-$k$ operation to rank block-wise features followed by dense attention between block features which reduces memory cost substantially. As the top-$k$ operation involves index swapping which precludes the gradient propagation, we propose to approximate the non-differentiable top-$k$ operation with a regularized earth mover's problem so that its gradient can be effectively back-propagated. In addition, we design a novel semantic position encoding mechanism that builds up coordinate for each individual semantic region to preserve texture structures while building correspondences. Further, we design a novel confidence feature injection module which mitigates mismatch problem by fusing features adaptively according to the reliability of built correspondences. Extensive experiments show that our method achieves superior performance qualitatively and quantitatively as compared with the state-of-the-art. The code is available at \href{}{}.


page 2

page 4

page 5

page 8

page 9

page 10

page 11


Unbalanced Feature Transport for Exemplar-based Image Translation

Despite the great success of GANs in images translation with different c...

DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation

One key challenge of exemplar-guided image generation lies in establishi...

From Design Draft to Real Attire: Unaligned Fashion Image Translation

Fashion manipulation has attracted growing interest due to its great app...

Marginal Contrastive Correspondence for Guided Image Generation

Exemplar-based image translation establishes dense correspondences betwe...

SAMScore: A Semantic Structural Similarity Metric for Image Translation Evaluation

Image translation has wide applications, such as style transfer and moda...

Remote Sensing Image Translation via Style-Based Recalibration Module and Improved Style Discriminator

Existing remote sensing change detection methods are heavily affected by...

Screentone-Preserved Manga Retargeting

As a popular comic style, manga offers a unique impression by utilizing ...

Please sign up or login with your details

Forgot password? Click here to reset