Investigations on Multi-Sensor Image System and its Surveillance Applications

This thesis investigates the issues of implementing multi-sensor imaging system for surveillance applications. Three topics for the fusion of infrared and electro-optic images are studied, i.e. registration, fusion, and evaluation. The first topic is the image registration or alignment, which is to associate corresponding pixels in multiple images to the same physical point in the scene. A trajectory-based method for registering infrared and electro-optic video sequences is proposed in this study. The initial registration parameters are derived from matching the trajectories across the consecutive video frames. Further refinement can be carried out by applying a maximum mutual information approach. The frame difference, from which the feature point is detected, is found with an image structural similarity measurement. The second topic is the implementation of pixel-level fusion. Two applications are considered in this study. Motivated by the adaptive enhancement, a modified pixel-level fusion scheme is proposed to implement the context enhancement. A visual image is first enhanced with the corresponding infrared image. Then, the enhanced image is fused with the visual image again to highlight the background features. This achieves a context enhancement most suitable for human perception. As the application of multi-sensor concealed weapon detection (CWD) is concerned, this thesis clarifies the requirements and concepts for CWD. How the CWD application can benefit from multi-sensor fusion is identified and a framework of multi-sensor CWD is proposed. A solution to synthesize a composite image from infrared and visual image is presented with experimental results. The synthesized image, on one hand provides both the information of personal identification and the suspicious region of concealed weapons; on the other hand implements the privacy protection, which appears to be an important aspect of the CWD process. The third topic is about the fusion performance assessment. In this study, the evaluation metrics are developed for reference-based assessment and blind assessment respectively. An absolute measurement of image features, namely phase congruency, is employed. Future work should include the reliability and optimization study of multiple image sensors from applications' and human perception-related perspectives. This thesis is a contribution to such research.