Audio Cameras for Audio-Visual Scene Analysis

Опубликовано 7 сентября 2016, 17:11

Computer vision has been able to use images to reason about scenes. We develop a new device, the audio camera, that displays images of the auditory energy arriving at a particular point, much in the same way that a visual camera displays this information for light. This device is based on the spherical microphone array, and describe how we were able to achieve frame-rate operation using graphical processing. Next, we consider the combination of the audio-camera with visual cameras. Based on the observation that audio cameras also produce central projection images, we are able to jointly calibrate audio and visual cameras. This allows applications such as 1) Using the epipolar constraint to guide beamforming in noisy environments 2) Do audio-visual image transfers 3) Image room and concert-hall acoustics 4) Image-based matched filter beamforming and dereverberation The audio camera design, has also been iterated to make the design robust, and easy to deploy. Time permitting, we will also briefly touch on other research topics in our research group, including audio scene rendering; the fast multipole method, and fast kernel algorithms; scientific computing on GPUs.