The importance of system characterization for safe cyber-physical system behavior
One of the difficult realities of developing robotic systems is the fundamental opposition between robustness and performance. A robust system is able to handle a wide variety of potential scenarios, but at the cost of performance in any specific scenario. For example, in bad weather conditions, you slow your vehicle down to account for things like sliding and limited visibility. In perfect track conditions, you can push cars to their absolute limits, since you don’t have to worry about these edge cases.
In order to develop a reliable cyber-physical system that sacrifices as little performance as possible, it is critical to develop rigorous characterizations of the physical capabilities of both the robot hardware and its sensors.
Hardware characterization: Detecting Abnormal Behavior Early
The first step one must take towards reliability is detection of undesired behaviors. How do you detect such behaviors in your systems? You can either instrument every single component of your system (which is very expensive and can lead to more problems – detecting failures or the sensors themselves with more sensors is impractical). The other way to go about it is to infer failures from behaviors. by looking at deviations from a nominal behavior with a limited set of onboard sensors. If a differential drive robot is constantly turning to the right when commanded to go forwards,the left motor certainly has a problem.
Early failure detection therefore requires the ability to detect small deviations in behavior between the current behavior in operation and a nominal behavior determined when the robot is known to operate properly, before those deviations become large enough to be critical. To do this, a precise hardware characterization of the system is required.
Such a characterization is called a dynamical model of the cyber-physical system. It captures via various mathematical representations the relation between the commands sent to the resulting system and the system behavior. In practice, it can be hard to measure physical properties of a system like inertia tensors (trust us, we built a machine to do it here). One way to circumvent this challenge is to characterize “low order models” of the system. These are models that lump together the robot and its controller, and look at the relationship between controller inputs and behavior. Such behavior is usually simpler to characterize and although it is less information rich (doesn’t give information useful to improve the controller for example), it still captures enough info to detect some behavior variations.
Once the model is built, deviations from nominal behaviors should be evaluated regularly as things can go wrong at any time. When deviations outside of the normal variance occur, developers and support engineers should be alerted to investigate. In this case, one of two things should happen:
- If nothing is wrong, the model of the system should be updated to include the deviation.
- If the system has failed in some way, the event should be recorded and countermeasures should be put in place for future errors of this type.
Sensing Characterization: Navigating Uncertainties and Failure Modes
Much like the hardware characterization, a proper understanding of the uncertainties and failure modes of the sensing and perception is vital for safe cyber-physical system behavior.
For any sensor characterization, it is necessary to test the system in an environment with ground truth. For mobile robots, this typically involves something like a Vicon or Optitrack system when indoors or a high-quality GPS system when outdoors. For manipulators, the robot itself can essentially provide ground-truth data due to the accuracy of the encoders onboard the robot. This can be used to touch points in the environment that are later localized with perception systems.
Once the ground truth is established, it is compared by the state estimates from the onboard sensors.
Vision-based perception systems can be the hardest system to characterize. Typically, errors are unbounded and abnormal lighting conditions can throw off even the most robust setups. Therefore, it is essential to pair such a system with more reliable sensing modalities, or craft a reliable set of failsafe maneuvers for use in the event of failure.
When fusing multiple signals, via something like a Kalman filter, it is not always necessary to characterize the sensors individually. A characterization of the fused state estimate itself can often be used to the same effect.
Unified System Characterization: Bridging Hardware and Sensing
Propagating the sensing characteristics through the hardware characterization is no small feat. In other words, it is difficult to quantify by how much uncertainty in a measurement of the system behavior should be compensated for via safety margins at the control level.
This problem has been at the core of robust control for decades.
There is no perfect solution that works in all settings. Interval arithmetic is a powerful and computationally efficient method for handling deterministic uncertainty bounds, but blows up numerically with several sequential operations. This is a large issue when integration is necessary, such as in a Model Predictive Control (MPC) framework.
Chance constraints can be used to handle stochastic uncertainty, but suffers from frequent constraint violations. Risk measures offer a more robust way to handle stochastic uncertainty, but suffers from significant computational complexity.
Ensuring Reliability Without Restraining Performance
In this post, we covered the importance of hardware and sensing characterizations for both developing and maintaining a cyber-physical system. Knowledge of the uncertainties in these characterizations is essential for ensuring reliability without restraining the performance of the system.
Just knowledge is not enough, however. It is critical to incorporate these characterizations in the planning and control frameworks. In future blog posts, we will cover how these values can be efficiently propagated through these stacks.