Since the fall of 2013 I’ve had the opportunity to collaborate with the Frick Art Reference Library Photoarchive, a venerable art history research institution here in New York City. We’ve been especially interested in finding ways of applying computer vision technology to improve art history research.

Art history photo archives are an interesting tool used by scholars: they’re a collection of thousands, if not millions, of photos of art. These are not the artworks themselves but photos of the artworks – oftentimes from all over the world – taken by individuals or institutions and aggregated into this single archive. This is an invaluable resource for researchers as oftentimes you can find more information about the history of an artwork (before/after conservation) or even find artworks that are lost (stolen, destroyed, or moved into unknown private collections).

The Frick Photoarchive has 1.2 million photos of art which they are in the process of digitizing and bringing online. They’re part of a larger photo archive consortium, called PHAROS, which has many other photo archives from all over the world. In total they’re likely going to have over 10 million photos of art once all of their collections are pooled.

This poses an interesting problem however as combining this much data from all of these institutions is especially challenging. In the world of digital humanities the most-common tool to utilize is that of Linked Open Data. Making sure all the records have consistent attributes being used, along with consistent data, then it’ll become easy to match up data from multiple institutions. In practice, however, this is quite difficult. Moving massive databases over to another schema can be very challenging, in fact very few institutions have done it. The British Museum is an especially notable example that has made the switch to this format and has served as an example to other institutions around the world.

Madonna and Child, 13th century, La Chiesa di S. Eufrasia, Pisa.

I’ve been proposing an alternative means of merging archives, namely using computer vision techniques to find the missing connections between images of artworks, to avoid the process of having to manually identify images and making sure that their data is pre-aligned. I published a paper on my research looking in to this topic (which has since been re-published in the Journal of Digital Humanities). The results are extremely promising. I’ve been using the MatchEngine service graciously provided by Idee Inc. to great effect. The algorithm they use is able to match duplicate images of the same artwork, even if there are differences in the photo (e.g. color vs. black-and-white, skewed photo angles, etc.). I highly recommend reading my paper for more information on the algorithm’s capabilities.

Building off of that work last year the Frick Art Reference Library and I received a digital resources grant from the Kress Foundation to continue exploring these computer vision techniques. Specifically I looked in to using the technology to merge multiple art history photo archives together. I attempted to merge the Frick Photoarchive’s Anonymous Italian Art collection and the Zeri Foundation’s 15th and 16th century Italian art collections using only the images and no additional metadata.

As it turns out this technique proved to be very successful – so successful that we’re now looking to use this technology as a primary means of merging records between all the archives in the PHAROS consortium. You can learn more about the results of this process in a talk that I gave at last year’s PHAROS (then IDPAC) annual meeting:

Now that we’ve proven that inter-institutional merging of photo archives is possible the next step becomes obvious: building a new database, in many ways like the Japanese print database I built previously, but focused on the digital photo collections of these archives.

I’m happy to announced that the Frick Art Reference Library and I have just received another grant from the Kress foundation to fund the development of this new database. At the start it’ll primarily focus on Italian art, as that’s the majority what has already been digitized by PHAROS archives. However even just focusing on Italian art it’ll immediately become one of the largest and most-important online collections for studying Italian art in the world.

A few key features of this new database:

• The database, images, and data contained within it will all be publicly accessible. This was an extremely important point for me as I wanted to ensure that we were providing a resource for everyone to learn from.
• Computer vision techniques will be used to merge the records from the archives together and will also be provided as a tool for research: you’ll be able to upload an image of an artwork and find other copies of it in the archive. This feature alone will save researchers so much time and energy when attempting to learn about an artwork.
• Metadata about the images will be provided by the institutions and will be made searchable. In the first version of this database we will not be doing any work to rectify the metadata between the institutions (many of them will be in different languages) but that should serve as a good “next step” for the project after launch.
• The interface for the database will be provided in multiple languages. Initially in English, German, and Italian.

More information about this project can be found on the project proposal page. I will be working on this project, outside of my work at Khan Academy, over the next year. I’m extremely excited to have this opportunity to work on this project. Being able to improve art history scholarship is a passion of mine and being able to provide these tools for others is a fantastic opportunity. I’ll be sure to write more about my work as it progresses, especially the interesting technical innovations that come in to play.