Since the fall of 2013 I’ve had the opportunity to collaborate with the Frick Art Reference Library Photoarchive, a venerable art history research institution here in New York City. We’ve been especially interested in finding ways of applying computer vision technology to improve art history research.
Art history photo archives are an interesting tool used by scholars: they’re a collection of thousands, if not millions, of photos of art. These are not the artworks themselves but photos of the artworks – oftentimes from all over the world – taken by individuals or institutions and aggregated into this single archive. This is an invaluable resource for researchers as oftentimes you can find more information about the history of an artwork (before/after conservation) or even find artworks that are lost (stolen, destroyed, or moved into unknown private collections).
The Frick Photoarchive has 1.2 million photos of art which they are in the process of digitizing and bringing online. They’re part of a larger photo archive consortium, called PHAROS, which has many other photo archives from all over the world. In total they’re likely going to have over 10 million photos of art once all of their collections are pooled.
This poses an interesting problem however as combining this much data from all of these institutions is especially challenging. In the world of digital humanities the most-common tool to utilize is that of Linked Open Data. Making sure all the records have consistent attributes being used, along with consistent data, then it’ll become easy to match up data from multiple institutions. In practice, however, this is quite difficult. Moving massive databases over to another schema can be very challenging, in fact very few institutions have done it. The British Museum is an especially notable example that has made the switch to this format and has served as an example to other institutions around the world.
|Madonna and Child, 13th century, La Chiesa di S. Eufrasia, Pisa.|
I’ve been proposing an alternative means of merging archives, namely using computer vision techniques to find the missing connections between images of artworks, to avoid the process of having to manually identify images and making sure that their data is pre-aligned. I published a paper on my research looking in to this topic (which has since been re-published in the Journal of Digital Humanities). The results are extremely promising. I’ve been using the MatchEngine service graciously provided by Idee Inc. to great effect. The algorithm they use is able to match duplicate images of the same artwork, even if there are differences in the photo (e.g. color vs. black-and-white, skewed photo angles, etc.). I highly recommend reading my paper for more information on the algorithm’s capabilities.
Building off of that work last year the Frick Art Reference Library and I received a digital resources grant from the Kress Foundation to continue exploring these computer vision techniques. Specifically I looked in to using the technology to merge multiple art history photo archives together. I attempted to merge the Frick Photoarchive’s Anonymous Italian Art collection and the Zeri Foundation’s 15th and 16th century Italian art collections using only the images and no additional metadata.
As it turns out this technique proved to be very successful – so successful that we’re now looking to use this technology as a primary means of merging records between all the archives in the PHAROS consortium. You can learn more about the results of this process in a talk that I gave at last year’s PHAROS (then IDPAC) annual meeting:
Now that we’ve proven that inter-institutional merging of photo archives is possible the next step becomes obvious: building a new database, in many ways like the Japanese print database I built previously, but focused on the digital photo collections of these archives.
I’m happy to announced that the Frick Art Reference Library and I have just received another grant from the Kress foundation to fund the development of this new database. At the start it’ll primarily focus on Italian art, as that’s the majority what has already been digitized by PHAROS archives. However even just focusing on Italian art it’ll immediately become one of the largest and most-important online collections for studying Italian art in the world.
A few key features of this new database:
- The database, images, and data contained within it will all be publicly accessible. This was an extremely important point for me as I wanted to ensure that we were providing a resource for everyone to learn from.
- Computer vision techniques will be used to merge the records from the archives together and will also be provided as a tool for research: you’ll be able to upload an image of an artwork and find other copies of it in the archive. This feature alone will save researchers so much time and energy when attempting to learn about an artwork.
- Metadata about the images will be provided by the institutions and will be made searchable. In the first version of this database we will not be doing any work to rectify the metadata between the institutions (many of them will be in different languages) but that should serve as a good “next step” for the project after launch.
- The interface for the database will be provided in multiple languages. Initially in English, German, and Italian.
More information about this project can be found on the project proposal page. I will be working on this project, outside of my work at Khan Academy, over the next year. I’m extremely excited to have this opportunity to work on this project. Being able to improve art history scholarship is a passion of mine and being able to provide these tools for others is a fantastic opportunity. I’ll be sure to write more about my work as it progresses, especially the interesting technical innovations that come in to play.
Tony (May 28, 2015 at 10:08 am)
David Bruchmann (May 28, 2015 at 10:28 am)
I’m sure I can spend a whole day or more on all the links in the article, so I’ve to read it again when I’ve more time!
Jajang (May 28, 2015 at 9:57 pm)
wow perfect, congrats sir.
Stuart (May 29, 2015 at 11:16 am)
Any change of open sourcing this project on GitHub for others to use on their own image projects?
John Resig (May 29, 2015 at 11:02 pm)
@Stuart: Yes! All of the code for this will be made open source! I’ll have much more to announce around that as the project progresses. I’m excited to have this be open for other institutions to use!
David Haskiya (May 31, 2015 at 6:46 am)
Congratulations! Very interesting work and very happy to see this gradually become open sourced.
If you’re looking to extend your test corpus don’t hesitate to give us at Europeana a ping and we’ll se if we can’t find some suitable collections in our vaults!
Krzysztof Szafranek (June 3, 2015 at 3:25 am)
I’m very glad to see your talents applied to a project so interesting and useful to a wider audience, not only tech community. Congratulations!
Marc Gauthier (June 4, 2015 at 1:20 pm)
Thank you for sharing your experience. As an art historian and a computer programmer, I find your results very interesting.
Tom (June 8, 2015 at 9:51 am)
What I can say – Congrats!
anizone (June 9, 2015 at 12:06 am)
Hi teman, saya tidak begitu paham soal ini, tapi kamu sungguh luar biasa.
Animer (June 9, 2015 at 12:15 am)
Naik naek ka puncak gunung luhur luhur abot kenca atas bwah ges ah poho dei
handha (June 9, 2015 at 12:17 am)
dan terjadi lagi, Kisah lama yang terulang kembali.
Kalian Luar biasa
mangakun (June 9, 2015 at 12:19 am)
salam kenal sobat