DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities

by   Elias Eulig, et al.

Common deep neural networks (DNNs) for image classification have been shown to rely on shortcut opportunities (SO) in the form of predictive and easy-to-represent visual factors. This is known as shortcut learning and leads to impaired generalization. In this work, we show that common DNNs also suffer from shortcut learning when predicting only basic visual object factors of variation (FoV) such as shape, color, or texture. We argue that besides shortcut opportunities, generalization opportunities (GO) are also an inherent part of real-world vision data and arise from partial independence between predicted classes and FoVs. We also argue that it is necessary for DNNs to exploit GO to overcome shortcut learning. Our core contribution is to introduce the Diagnostic Vision Benchmark suite DiagViB-6, which includes datasets and metrics to study a network's shortcut vulnerability and generalization capability for six independent FoV. In particular, DiagViB-6 allows controlling the type and degree of SO and GO in a dataset. We benchmark a wide range of popular vision architectures and show that they can exploit GO only to a limited extent.


page 8

page 13

page 14

page 16

page 17

page 19

page 20

page 22


Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?

Deep neural networks (DNNs) are machine learning algorithms that have re...

Impact of Colour Variation on Robustness of Deep Neural Networks

Deep neural networks (DNNs) have have shown state-of-the-art performance...

Impact of Light and Shadow on Robustness of Deep Neural Networks

Deep neural networks (DNNs) have made remarkable strides in various comp...

Explainability and Robustness of Deep Visual Classification Models

In the computer vision community, Convolutional Neural Networks (CNNs), ...

Domain Generalization Emerges from Dreaming

Recent studies have proven that DNNs, unlike human vision, tend to explo...

ROBIN : A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts

Enhancing the robustness in real-world scenarios has been proven very ch...

Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective

Deep neural networks (DNNs) often rely on easy-to-learn discriminatory f...

Please sign up or login with your details

Forgot password? Click here to reset