Visibility in Polygonal Environments with Holes: Finding Best Spots for Hiding and Surveillance

Visibility plays an important role for decision making in cluttered, uncertain environments. This paper considers the problem of identifying optimal hiding spots for an agent against line-of-sight detection by an adversary whose location is unknown. We consider environments modeled as polygons with holes. We develop a set of mathematical tools for reasoning about visibility as a function of position and rely on non-smooth analysis to formally characterize the regularity properties of various visibility-based metrics. These metrics are non-smooth and non-convex, so off-the-shelf optimization algorithms can only guarantee convergence to Clarke critical points. To address this, the proposed Normalized Descent algorithm leverages the structure of non-smooth points in visibility problems and introduces randomness to escape saddle points. Our technical analysis allows for the non-monotonic decrease in the visibility metric and strengthens the algorithm guarantees, ensuring convergence to local minima with high probability. Simulations on two hide-and-seek scenarios showcase the effectiveness of the proposed approach.