Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

08/13/2019
by   Tom Everitt, et al.
3

Can an arbitrarily intelligent reinforcement learning agent be kept under control by a human user? Or do agents with sufficient intelligence inevitably find ways to shortcut their reward signal? This question impacts how far reinforcement learning can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we use an intuitive yet precise graphical model called causal influence diagrams to formalize reward tampering problems. We also describe a number of tweaks to the reinforcement learning objective that prevent incentives for reward tampering. We verify the solutions using recently developed graphical criteria for inferring agent incentives from causal influence diagrams.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset