RMT: Rule-based Metamorphic Testing for Autonomous Driving Models
share
Deep neural network models are widely used for perception and control in autonomous driving. Recent work uses metamorphic testing but is limited to using equality-based metamorphic relations and does not provide expressiveness for defining inequality-based metamorphic relations. To encode real world traffic rules, domain experts must be able to express higher order relations e.g., a vehicle should decrease speed in certain ratio, when there is a vehicle x meters ahead and compositionality e.g., a vehicle must have a larger deceleration, when there is a vehicle ahead and when the weather is rainy and proportional compounding effect to the test outcome. We design RMT, a declarative rule-based metamorphic testing framework. It provides three components that work in concert:(1) a domain specific language that enables an expert to express higher-order, compositional metamorphic relations, (2) pluggable transformation engines built on a variety of image and graphics processing techniques, and (3) automated test generation that translates a human-written rule to a corresponding executable, metamorphic relation and synthesizes meaningful inputs.Our evaluation using three driving models shows that RMT can generate meaningful test cases on which 89 predictions are found by enabling higher-order metamorphic relations. Compositionality provides further aids for generating meaningful, synthesized inputs-3012 new images are generated by compositional rules. These detected erroneous predictions are manually examined and confirmed by six human judges as meaningful traffic rule violations. RMT is the first to expand automated testing capability for autonomous vehicles by enabling easy mapping of traffic regulations to executable metamorphic relations and to demonstrate the benefits of expressivity, customization, and pluggability.
READ FULL TEXT