Project Director Andrew deWaard presented on ClipNotes at the Digital Humanities 2014 conference, the annual international conference of the Alliance of Digital Humanities Organizations (ADHO), in Lausanne, Switzerland. He presented as part of the Innovative Teaching Methods and Practices in Digital Humanities workshop, sponsored by Digital Research Infrastructures for the Arts and Humanities (DARIAH), and during the conference short paper sessions (abstract here). Below is an edited version of the talk. Download the pdf here.

ClipNotes in the Classroom: Video Annotation for Film & Television Analysis

Andrew deWaard

Today I’ll be talking about an instructional software project called ClipNotes, which facilitates video annotation and opens up teaching opportunities in terms of collaboration and student knowledge production. The progress we’ve made and the setbacks we’ve encountered in enabling simple annotation of copyrighted film material will hopefully provide insights for the broader field of digital humanities and audio-visual annotation.

There are a wide variety of video annotation platforms under development, and we’ve developed a resources page on our website to keep track of similar projects. These platforms all differ from ClipNotes in some essential way: they are either purely web-based, requiring you to upload video files, making them untenable for full-length copyrighted material, or they are designed with a vast, complex metadata schema for wide-ranging fields and applications, making them unappealing for our specific goal of increasing usage of annotation and digital humanities methods in the field of cinema and media studies.

One significant way cinema and media studies differs from many of the fields in DH is that our data is not only under copyright, but under copyright by an industry that is highly litigious and actively monitors copyrighted material online. Because our cultural material is film, Hollywood or otherwise, cinema and media studies scholars face certain restrictions in how we can pursue the kind of open, collaborative, data-driven research advocated by the Digital Humanities. Tight copyright control may in fact be one of the reasons cinema and media studies has been so slow to adopt digital methods, along with the fact that sound and image are inherently more difficult to catalogue and quantify than the written word. ClipNotes is an attempt to mitigate both of these thorny issues.

There are just 3 elements to the interface of ClipNotes: the video with controls, the annotation, and the scrollable menu. ClipNotes thus facilitates quick segmentation, annotation and presentation of film clips. The XML file that organizes the annotations only requires three elements as well: start/stop times, clip annotations, and clip titles, and is produced separately from the app. The video file must also be encoded separately from a DVD or other source, and then when ClipNotes launches, the XML file is linked to the video file and precise, granular analysis is easily presented and disseminated.

We currently have a little over 60 films annotated, the majority of which trace recurring stylistic and thematic patterns throughout a film, and 10 files which were designed for instructional purposes. This Citizen Kane file is an example of an instructional usage of ClipNotes, teaching the terminology of cinematography through quick and comprehensive access to Citizen Kane’s landmark visual style. Salient examples of deep focus, deep space, triangular compositions, door and window framings, and graphic matches are all catalogued, as are the visual motifs of light and shadow, bars and fences, and mirrors. The ability to quickly but briefly demonstrate a series of scenes is a tremendous resource in a lecture or presentation situation, particularly non-static examples, such as sound or camera and character movement. One of the distinct strengths of film as a medium is the symphonic arrangement of audiovisual patterns, something that is lost in merely showing screen grabs or a few extended scenes.

Now let’s take a look at the simple XML file that ClipNotes uses:

One of the tensions the ClipNotes team struggles with is simplicity versus complexity in metadata. To a crowd familiar with the Text Encoding Initiative, it would appear we have clearly sided with simplicity. To a crowd of cinema and media studies scholars, however, this is potentially overwhelming and complex, as mark-up language and digital methods are not a common part of cinema and media studies methodology. But we are also hoping to reach an audience beyond scholars. We want undergraduate students to be able to learn how to annotate with ClipNotes and XML as a course assignment, and we hope that ClipNotes might appeal to the casual cinephile as well, adding their non-academic insights to the public database. We do plan to develop a more detailed schema for film-focused categories and tags of analysis, and we hope to adhere to community standards such as the W3C Open Annotation Coalition, though again, we hesitate to add too much complexity so as to not scare away new users.

A preliminary public repository of XML files is now available at clipnotes.org, which users are encouraged to submit to and which allows for widespread sharing of textual analysis. We hope it will become an increasingly useful teaching resource for instructors, as well as a place for scholars to share their analytical research, a practice that barely exists in the field of cinema and media studies. For obvious copyright reasons, the films are not included, but guides are available to demonstrate how DVDs can be easily and legally encoded into digital files under fair use exemptions.

By separating the two files but maintaining a central, public database of XML files, ClipNotes can facilitate collaborative work without needing to host the actual filmic text. Although skeptical at first, UCLA library’s intellectual property lawyers were convinced that ClipNotes was a viable solution and that library iPads containing encoded DVDs were acceptable under educational fair use as long as the original films were already available in the library. This leads us to believe that the two-pronged approach we have employed is a good workaround for the challenge of collaborating on copyrighted material.

In addition to assisting annotation and presentation, ClipNotes facilitates unique pedagogical opportunities as well. At UCLA this past semester, we integrated ClipNotes into a first-year History of American Film course, partnering with a campus library who provided customized iPads with ClipNotes to all of the teaching assistants, who were then able to collaboratively standardize instructional material among their many discussion sections. In class, they were more able to quickly and efficiently teach film form and style, a difficult topic to teach because of its many, diverse incarnations. The library also made a number of iPads available to students in the library to access on their own time, and though we expected only minimal use, the students made ample use of this opportunity to review terminology and clip examples for their assignments and final exam. This pilot project generated a wealth of positive feedback and we are now in the process of expanding our collection of annotated films in preparation for more classes across campus and a number of partner universities, including the University of Wisconsin and a few universities in Germany.

Another teaching opportunity with ClipNotes arises with the possibility of teaching students how to make their own annotation files, either individually or as a collaborative assignment. Graduate level classes at UCLA have been implementing this assignment for a few years already, which is where many of our database XML files originate. By requiring students to markup their film analysis into an XML file that will be made available online, students are encouraged to think of their work as knowledge production that might prove beneficial to others. Not only do students learn the skill of granular film analysis, but they learn the responsibilities of knowledge production and the opportunities of open-access resources. In addition, students become an essential part of the long term effort to generate large data sets of audiovisual analysis and metadata, opening up potential quantification and data-mining possibilities in the future.

As for future development of ClipNotes, the continued expansion of our database and the assembly of large data sets of analytic material about film is one key goal. We also hope to branch out into other fields and disciplines that incorporate film and television representation and analysis, such as language classes which often make extensive use of video in introductory classes. As another example, Associate Dean of Humanities and English professor at UCLA, Robert N. Watson, recently published a book on Throne of Blood, about the Kurosawa film, and he has produced an annotation XML file that exhaustively catalogues the film with regards to its adaptation of MacBeth. You can watch the film while reading his annotations, which include comparing the original text from Shakespeare’s play to how Kurosawa has adapted it in an audiovisual medium. We’re thrilled to have this file as a very high quality example of annotation in ClipNotes that also functions as a media supplement to his book.

In conclusion, ClipNotes is based on the idea that the collaborative generation of a public repository of film and television annotations will allow future scholars to study much larger cultural dynamics than traditional analysis has allowed. Though the app itself specializes in solitary texts, discovering and annotating micro-relationships, the larger database project has the potential to produce broad, macro-cultural insights into film and television as a result of this wide-ranging data. In order to achieve a large database, we need a larger user base, and so ClipNotes is designed to be easy to use, for students and instructors and cinephiles alike.

Please visit clipnotes.org for more information or to download the app. Thank you so much for your time.