Key Papers in Safe RL

What follows is a list of papers in Safe RL that are worth reading. This is far from comprehensive, but should provide a useful starting point for someone looking to do research in the field.

Table of Contents

• Key Papers in Safe RL

  • 1. General Review

  • 2. Model-free RL

  • 3. Model-based RL

  • 4. Transfer Learning

  • 5. Ensemble Learning

  • 6. Human in The Loop

  • 7. Curriculum Learning

  • 8. Risk-sensitive RL

  • 9. Formal Methods

1. General Review

1. Unsolved Problems in ML Safety, Hendrycks et al, 2022.

2. Concrete Problems in AI Safety, Amodei et al, 2016.

3. A Comprehensive Survey on Safe Reinforcement Learning, García et al, 2015.

2. Model-free RL

1. Constrained Policy Optimization, Achiam et al, 2017.

2. Lyapunov-based Safe Policy Optimization for Continuous Control, Chow et al, 2019

3. Batch Policy Learning under Constraints, Le et al, 2019

4. Reward Constrained Policy Optimization, Tessler et all, 2019

5. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, Stooke et al, 2020

6. Projection-based Constrained Policy Optimization, Yang et al, 2020.

3. Model-based RL

1. Safe Model-based Reinforcement Learning with Stability Guarantees, Berkenkamp et al, 2017

2. Constrained model predictive control: Stability and optimality, Mayne et al, 2000

3. Constrained Policy Optimization via Bayesian World Models, As et al, 2022

4. Transfer Learning

1. Learning to be Safe: Deep RL with a Safety Critic, Srinivasan et al, 2020

5. Ensemble Learning

1. Gerneralzieing from a Few Environments in Safety-critical Reinforcement Learning, Kenton et al, 2019

2. Leave No Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning, Eysenbach et al, 2018

6. Human in The Loop

1. Trial without Error: Towards Safe Reinforcement Learning via Human Intervention, Saunders et al, 2017

7. Curriculum Learning

1. Safe Reinforcement Learning via Curriculum Induction, Turchetta et al, 2020

8. Risk-sensitive RL

1. Risk-Sensitive Reinforcement Learning Applied to Control under Constraints, Geibel et al, 2005

2. Risk-Aware Transfer in Reinforcement Learning using Successor Features, Gimelfarb et al, 2021

3. Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning, Fei et al, 2021

9. Formal Methods

1. Verifiably Safe Exploration for End-to-End Reinforcement Learning, Hunt et al, 2021. See also Guarantee Safety in Training and Testing for related work.