Computing the Testing Error without a Testing Set

by   Ciprian Corneanu, et al.

Deep Neural Networks (DNNs) have revolutionized computer vision. We now have DNNs that achieve top (performance) results in many problems, including object recognition, facial expression analysis, and semantic segmentation, to name but a few. The design of the DNNs that achieve top results is, however, non-trivial and mostly done by trail-and-error. That is, typically, researchers will derive many DNN architectures (i.e., topologies) and then test them on multiple datasets. However, there are no guarantees that the selected DNN will perform well in the real world. One can use a testing set to estimate the performance gap between the training and testing sets, but avoiding overfitting-to-the-testing-data is almost impossible. Using a sequestered testing dataset may address this problem, but this requires a constant update of the dataset, a very expensive venture. Here, we derive an algorithm to estimate the performance gap between training and testing that does not require any testing dataset. Specifically, we derive a number of persistent topology measures that identify when a DNN is learning to generalize to unseen samples. This allows us to compute the DNN's testing error on unseen samples, even when we do not have access to them. We provide extensive experimental validation on multiple networks and datasets to demonstrate the feasibility of the proposed approach.


Reducing Flipping Errors in Deep Neural Networks

Deep neural networks (DNNs) have been widely applied in various domains ...

Use of Metamorphic Relations as Knowledge Carriers to Train Deep Neural Networks

Training multiple-layered deep neural networks (DNNs) is difficult. The ...

A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding

Over the past few years, deep neural networks (DNNs) have been continuou...

Generalizing Neural Networks by Reflecting Deviating Data in Production

Trained with a sufficiently large training and testing dataset, Deep Neu...

Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study

There is a growing body of research on developing testing techniques for...

POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks

Deep Neural Networks (DNNs) outshine alternative function approximators ...

Learning Credible Deep Neural Networks with Rationale Regularization

Recent explainability related studies have shown that state-of-the-art D...

Please sign up or login with your details

Forgot password? Click here to reset