Learning Robot Safety from Sparse Human Feedback using Conformal Prediction

Aaron O. Feldman1, Joseph A. Vincent1, Maximilian Adang1, Jun En Low1, Mac Schwager1,
1Stanford University
arXiv Code

method overview
Mathematically defining safety for robotics tasks can be difficult. In our paper we show how sparse human feedback can be used to efficiently learn unsafe (latent) states. We use conformal prediction to calibrate the learned notion of safety resulting in an interpretable warning system and a lightweight recovery policy.

Abstract

Ensuring robot safety can be challenging; user-defined constraints can miss edge cases, policies can become unsafe even when trained from safe data, and safety can be subjective. Thus, we learn about robot safety by showing policy trajectories to a human who flags unsafe behavior. From this binary feedback, we use the statistical method of conformal prediction to identify a region of states, potentially in learned latent space, guaranteed to contain a user-specified fraction of future policy errors. Our method is sample-efficient, as it builds on nearest neighbor classification and avoids withholding data as is common with conformal prediction. By alerting if the robot reaches the suspected unsafe region, we obtain a warning system that mimics the human's safety preferences with guaranteed miss rate. From video labeling, our system can detect when a quadcopter visuomotor policy will fail to steer through a designated gate. We present an approach for policy improvement by avoiding the suspected unsafe region. With it we improve a model predictive controller's safety, as shown in experimental testing with 30 quadcopter flights across 6 navigation tasks.

Learning Safety

We construct unsafe sets using conformal prediction from either using only the unsafe data (balls, left) or from using both safe and unsafe data (polyhedra, right).

unsafe-only geometry unsafe-safe geometry
We can also probabilistically assess whether a new point belongs to the unsafe distribution using the conformal p-value. Constructing the unsafe sets from both safe and unsafe data, we can better distinguish safe and unsafe points.
unsafe-only likelihood unsafe-safe likelihood

Warning System

We provide a warning system by alerting when new states fall in the learned unsafe set. Because the set is calibrated, the user can dictate the miss rate, i.e., how often the warning system will fail to alarm during unsafe trajectories. More generally, we can track the conformal p-values for runtime safety monitoring. With this approach, we detect when a visuomotor quadcopter policy will fail to fly through a designated gate.

Labeled safe (goes through gate)

Labeled unsafe (goes over gate)

Test Safe (p remains low)

Test Unsafe (p increases)

Policy Modification

We can improve the robot policy by using a backup safety mode which is triggered upon a warning system alert. In the backup safety mode, we steer to historical safe data while constraining to avoid the suspected unsafe region. Using this approach, we teach a quadcopter to more safely navigate and avoid original unsafe behaviors such as passing too near the table’s surface, getting stuck near the gate, or going under the ladder.

chair
gate
ladder

Warning triggered, backup results in higher ascent over table

No warning triggered, so nominal policy throughout

Warning triggered, goes around not through ladder

Warning triggered, avoids getting stuck near gate

BibTeX

@misc{feldman2025learningrobotsafetysparse,
        title={Learning Robot Safety from Sparse Human Feedback using Conformal Prediction}, 
        author={Aaron O. Feldman and Joseph A. Vincent and Maximilian Adang and Jun En Low and Mac Schwager},
        year={2025},
        eprint={2501.04823},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2501.04823}, 
  }