Concept Lexicon Construction and Affective Analysis: From Photos to MTV

Опубликовано 6 сентября 2016, 17:21

Concept-based photo search has become more and more popular in Multimedia Information Retrieval (MIR). However, which semantic concepts should be used for data collection and model construction is still an open question. Currently, there is very little research found on automatically choosing multimedia concepts with small semantic gaps. On the other hand, MTV has become an important favorite pastime to people because of its conciseness, convenience and the ability to bring both audio and visual experiences to audiences. It is an important and significant task to develop new techniques for automatically analyzing, retrieving, and managing the increasing amount of MTVs. Affective analysis could be a potential and promising solution to a natural, user-friendly, and effective MTV retrieval and management system. To address the above issues, in this talk, I will present our recent work in photo concept lexicon construction and MTV affective analysis. (i) For photos, we propose a novel framework to develop a lexicon of high-level concepts with small semantic gaps (LCSS) from a large-scale web photo dataset. By defining a confidence map and content-context similarity matrix, photos with small semantic gaps are selected and clustered. The final concept lexicon is mined from the surrounding descriptions (titles, categories and comments) of these photos. This lexicon offers a set of high-level concepts with small semantic gaps, which is very helpful for people to focus for data collection, annotation and modeling. (ii) For MTV affective analysis, we propose an integrated system (i.MTV) for MTV affective analysis, visualization, retrieval, and user profile analysis. In this system, we not only perform the effective MTV affective analysis, but also propose novel Affective Visualization techniques to make the abstract affective states intuitive and friendly to users. Based on the affective analysis and visualization, MTV affective retrieval is achieved. Furthermore, through user profile analysis, userΓÇÖs affective preferences are captured. Finally, novel methods are proposed for MTV recommendation based on userΓÇÖs affective preferences.