Discovering Collective Narratives of Theme Parks from Visitors’ Photo Streams
People
Leonid Sigal (Disney Research)
Publication
Gunhee Kim, Seungwhan Moon, and Leonid Sigal
Discovering Collective Narratives of Theme Parks from Large Collections of Visitors’ Photo Streams
21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2015), Sydney, Australia, August 10-13, 2015
[Paper (PDF)] [Presentation (PPTX)] [Poster (PDF)]
Description
Motivation of Research
Current technical advances, including widespread availability of mobile photo taking devices and ubiquitous network connectivity, are changing the way we tell our stories. Personal storytelling is becoming more data-driven and vision-oriented; people can simply capture their memorable moments by a stream of images, and then spontaneously reuse them to deliver their own stories. For example, even in a single day, tens of thousands of people visit Disneyland, and much of them take and are willing to share large streams of photos that record their experiences with families or friends. Each photo stream tells a slightly different story from its own point of view, but by aggregating them, it is likely that common storylines of Disneyland experience emerge.
The objective of this research is to develop an approach for creating and exploring spatio-temporal storylines from large collections of photo streams contributed by visitors to Disneyland, along with its public information like visitor's map. Taking advantage of computer vision techniques, we represent the visual contents of images in the form of story elements (e.g. human faces, supporting objects, and locations), and automatically extract shared key moments and put them together to create a story graph, which is a structural summary of branching narratives that visualize various events and/or activities recurring across the input photo sets.
 Figure 1. (a) Given large sets of online photo streams shared by visitors to a theme park (e.g. Disneyland), and its public information (e.g. visitor's map),we create story graphs as a structural summary visualizing various events or activities recurring across the visitors' photo sets.The story graphs are hierarchically created to enable users to explore the stories between attractions in (b), and sub-stories inside the individualattractions in (c)-(d). In (c), we show the connectivity between story subgraphs at the Enchanted Tiki Room, and their details are given in (d). |
Method and Experiments
The output of our algorithm is two-fold. First, we extract story elements from all images of the photo streams using computer vision techniques. Along with two low-level image features, we define four types of story elements as story-related, high-level descriptors of images in the context of theme parks: faces, time, location, supporting objects. Based on the estimated story elements over photo streams, the second output is the spatio-temporal story graph via an inference of sparse time-varying directed graphs. The vertices correspond to dominant image clusters of the photo streams, and the edge set links the vertices that sequentially recur in many photo streams.
Through quantitative evaluation and crowdsourcing-based user studies via Amazon Mechanical Turk, we show that the story graphs serve as a more convenient mid-level data structure to perform photo-based recommendation tasks than other alternatives. We also present storybook-like demo examples regarding exploration, recommendation, and temporal analysis, which may be most beneficial uses of the story graphs to visitors.
Funding
This research is supported by Disney Research.