Intra-operative measurements of tissue shape and multi/ hyperspectral information have the potential to provide surgical guidance and decision making support. We report an optical probe based system to combine sparse hyperspectral measurements and spectrally-encoded structured lighting (SL) for surface measurements. The system provides informative signals for navigation with a surgical interface. By rapidly switching between SL and white light (WL) modes, SL information is combined with structure-from-motion (SfM) from white light images, based on SURF feature detection and Lucas-Kanade (LK) optical flow to provide quasi-dense surface shape reconstruction with known scale in real-time. Furthermore, "super-spectral-resolution" was realized, whereby the RGB images and sparse hyperspectral data were integrated to recover dense pixel-level hyperspectral stacks, by using convolutional neural networks to upscale the wavelength dimension. Validation and demonstration of this system is reported on ex vivo/in vivo animal/ human experiments.
@article{arxiv.1706.06081,
title = {Endoscopic Depth Measurement and Super-Spectral-Resolution Imaging},
author = {Jianyu Lin and Neil T. Clancy and Yang Hu and Ji Qi and Taran Tatla and Danail Stoyanov and Lena Maier-Hein and Daniel S. Elson},
journal= {arXiv preprint arXiv:1706.06081},
year = {2017}
}