ROS Rescue: Fault Tolerance System for Robot Operating System

Abstract

In this chapter we discuss the problem of master failure in ROS 1.0 and its impact on robotic deployments in the real world. We address this issue in this tutorial chapter where we outline, design and demonstrate a fault tolerant mechanism associated with a ROS master failure. Unlike previous solutions which use primary backup replication and external checkpointing libraries which are resource demanding, our mechanism adds a lightweight functionality to the ROS master to enable it to recover from failure. We present a modified version of the ROS master which is equipped with a logging mechanism to record the meta information and network state of ROS nodes as well as a recovery mechanism to go back to the previous state without having to abort or restart all the nodes. We also implement an additional master monitor node responsible for failure detection on the master by polling it for its availability. Our code is implemented in Python and preliminary tests were conducted successfully on a variety of land, aerial and underwater robots and a teleoperated computer running ROS Kinetic on Ubuntu 16.04. The code is publicly available under a Creative Commons license on Github at https://github.com/PushyamiKaveti/fault-tolerant-ros-master.