Semantic Segmentation using Higher-Order Markov Random Fields

Опубликовано 17 августа 2016, 2:38

Many scene understanding tasks are formulated as a labelling problem that tries to assign a label to each pixel of an image, that may correspond to different object classes such as car, grass or sky, or to depths or to intensity after denoising. These labelling problems are typically formulated as a pairwise Random Field, modelling the dependencies of labels of pairs of variables in the local neighborhoods. However, these pairwise models are very restricted in their expressivity and can not induce desired complex structures in the output labelling. In this work we propose global structured formulation beyond pairwise models, that can combine features over multiple scales and enforce globally consistent label-set in the resulting labelling. We show that this complex model can still be learnt and optimised efficiently. Proposed method has been tested on several semantic segmentation data sets, where it gets (close to) state-of-the-art results. We then show how multiple random field models can be formulated jointly across different domains. We demonstrate the usefulness of this model on the problem of joint semantic segmentation and dense 3D stereo reconstruction and show that this approach significantly outperforms existing methods for street scenes. The complete source code is publicly available at cms.brookes.ac.uk/staff/Philip...