Large Scale Video Analysis

(Right)A walkthrough of the LSVA’s shot segmentation and image-based search capabilities

Large-Scale Video Analysis seeks to explore the use of real-time supercomputing use as a means to explore large collections of moving image data. The research is a collaboration with Virginia Kuhn of the University of Southern California, who is the lead investigator of the project. She is joined by co-investigators Michael Simeone and Alan Craig of I-CHASS, as well as David Bock, Liana Diesendruck, and Luigi Marini of NCSA.

Project description as appears in Kuhn, et al., “Large Scale Video Analytics: On-demand, iterative inquiry for moving image research”:

The process of understanding and using large databases of video data has remained both time-consuming and laborious. Aside from the massive size of contemporary archives and the challenges that have faced semantically-sensitive image retrieval for the last 20 years,other key challenges to effectively analyzing video archives with existing methods include limited metadata and the lack of a precise understanding of the actual content of the archive. A final difficulty lies in the incompleteness of translation across semiotic registers – words can never fully represent sounds and images, leaving a gap in meaning when labels alone are employed to describe and search for content.

The real-time, interactive and iterative analysis of large video archives can be both compute-intensive and memory-intensive. High Performance Computing (HPC) platforms and storage resources are therefore needed to handle the large volume, velocity and variety associated with such video archives. Given that about 72 hours of video are uploaded toYouTube alone every minute (volume and velocity), and the videos come in diverse formats and codecs (variety), large-scale video analytics is actually a BigData problem [1] where data is semi-structured or unstructured.Though HPC is indispensable for analyzing such large databases of videos, for a humanities researcher, one of the obstacles associated with working in an open-science HPC environment is the long wait-time associated with job- processing when a job is submitted to a regular queue. The nature of the humanities research, especially video analysis,necessitates that the researcher is able to quickly get results from one query in order to formulate the next one. Therefore, a truly interactive system for video analysis that can function in an HPC environment is required to support researchers’ goals.

The Large Scale Video Analytics (LSVA) research project explores new possibilities offered by both an innovative use of the Gordon supercomputer at the San Diego SupercomputingCenter (SDSC), and the conjoined interests of HPC and the cultural and historical study of moving images. We aim to facilitate humanities research on moving images at a scale heretofore unthinkable, demonstrating the possibility for humanists to productively inform policies and infrastructure at the supercomputing centers, even as the affordances of HPC enlivens and extends humanities research.

Our aim in this project is to address obstacles in both image retrieval and research that uses extreme-scale archives of video data. The searching, tagging, and analysis enabled by image retrieval faces the semantic gap problem of satisfactorily using low-level image features and actions to retrieve user-identified objects. This gap is only exaggerated as queries by historians and cinema and media scholars demand a high degree of precision and nuance in their study of moving images. To solve this problem we propose a two pronged approach that 1) places more interpretive power in the hands of the human user through novel visualizations of video data, and 2) uses a customized on-demand configuration of Gordon that enables iterative queries over a short period of time.

This project has also been covered by the National Center for Supercomputing Applications in their article “Visual Literacy.”

For more information, contact Virginia Kuhn ( or Michael Simeone (