deep learning for conservators, archaeologists, and art historians -凯发k8网页登录
using deep learning and image processing to restore and preserve artwork
when carola-bibiane schönlieb started her ph.d. studies in mathematics in 2005, one of her first projects was to help restore a medieval fresco in vienna. once hidden by the walls of an old apartment, the remains of the painting were pockmarked with white holes, damage from the removal of the walls a few years prior. instead of paint, solvents, or resin, schönlieb’s restoration tools were algorithms. “there were some conservators from the university of vienna who had started physically restoring this,” says schönlieb. “then we got to doing this digitally.”
the mach initiative combines the expertise of art historians, conservators, classicists, and medievalists with that of mathematicians to advance restoration and archaeology.
after taking photographs of the damaged fresco, schönlieb researched algorithms that could use those photos to fill in the missing, damaged pieces of the painting, creating a digital mockup of what the original fresco could have looked like. at the time, there were just a handful of papers describing how mathematicians and conservators could work together to restore artwork. conservators and art historians were just beginning to digitize their archives, preserving paintings, manuscripts, and pottery on machines.
section of a medieval fresco, once hidden behind the apartment walls in vienna. image credit: mach.
schönlieb, now professor of applied mathematics at the university of cambridge, uses image analysis and processing for art restoration and conservation. currently, she works with other mathematicians and humanities experts on the (mach) initiative. the initiative combines the expertise of art historians, conservators, archaeologists, and medievalists with that of mathematicians to advance restoration and material culture studies.
in their current project, “unveiling the invisible,” the mach team uses matlab® algorithms to catalogue roman pottery, analyze paint cross-sections to allow scholars to see new relationships between artifacts, and digitally restore illuminated manuscripts too delicate for physical restoration. these three areas of project focus have been years in the making.
re-illuminating manuscripts
the mach collaboration began in 2013, when schönlieb gave a talk about digital image restoration at the university. staff from cambridge’s fitzwilliam museum, which features art from ancient times through the present, were in attendance. they thought that schönlieb’s methods could be applied to their work.
the keeper of manuscripts and printed books at the fitzwilliam museum at the time had been pursuing noninvasive approaches for manuscript restoration when she heard schönlieb’s image restoration talk. she approached schönlieb and suggested they work together. that’s when the mach project was born.
when confronted with damaged paintings and artifacts, conservators face a dilemma. while they have the option of restoring a piece to what it might have looked like in its original form, the damage itself can chronicle an object’s history. illuminated manuscripts from the middle ages, for example, are handwritten books, written on parchment, with painted decoration that often includes precious metals such as gold or silver. in some cases, the manuscripts might have had paint intentionally removed or added to eliminate marks of ownership, or cover up an offensive image.
“using virtual and mathematical methods allows you to keep the [illuminated manuscript] as it is and retain that history, and offers a restored, pristine version of what it might have looked like. it gives you the opportunity to have the best of both worlds.”
“what do we gain by restoring, and what do we lose by restoring?” asks suzanne reynolds, curator at the fitzwilliam museum in the department of manuscripts and printed books. reynolds is a member of the mach group and works with medieval illuminated manuscripts.
detail.
inpainting domain.
final result.
manuscript with large damaged area is digitally restored. image: © the fitzwilliam museum, cambridge.
illuminated manuscripts are especially problematic when it comes to restoration. they’re rarely physically restored compared with other forms of painting. working with reynolds, schönlieb and simone parisotto, a research associate in cambridge’s department of applied mathematics and theoretical physics and the fitzwilliam museum, are developing an app to address these challenges in one arm of the larger “unveiling the invisible” project.
designed for conservators and developed using matlab, the app uses image processing techniques to identify damage and virtually reconstruct the images in manuscripts. it relies on inpainting, a term that originally referred to physically reconstructing a painting. in mathematical circles, inpainting means digitally restoring images.
using deep learning and partial differential equations, the mach group’s program can fill in the blanks of a damaged manuscript and predict the results of different avenues of restoration. a user trains the algorithm with examples from the same or related manuscript(s)—the more the better—and the algorithm then reconstructs the missing contents of the image that needs to be restored.
“using virtual and mathematical methods allows you to keep the object as it is and retain that history, and offers a restored, pristine version of what it might have looked like,” reynolds says. “it gives you the opportunity to have the best of both worlds.”
in addition to restoration, mathematical methods can not only digitize archives, but also leverage artificial intelligence to make the archival data even more useful to conservators, art historians, and archaeologists.
clustering roman pottery
the idea for the second focus of “unveiling the invisible” originated in 2015, when alessandro launaro, senior lecturer in classics at cambridge, came to schönlieb with a problem. an archaeologist, launaro focuses on the roman period and had been excavating sites in western italy. as is almost invariably the case, he had found a great amount of commonware pottery, the kind used for day-to-day tasks like cooking, but faced the daunting task of analyzing several thousand pottery shapes, pieces of pot rims, and bases.
“i had an archaeological problem, the analysis of a great body of evidence,” says launaro. although there are systematic and comprehensive catalogues that help archaeologists analyze more refined classes of roman pottery (fineware), nothing zeroed in on the day-to-day pottery he wanted to learn more about.
“someone could just go through all these [pottery fragment] shapes, but that’s super tedious. and since we are human, we make mistakes, we are tired sometimes, and we might miss something. an algorithm will not tire.”
commonware pottery, as the name suggests, accounts for a majority of pottery found at archeological sites. but given the very diverse shapes this pottery can take, and the sheer number of these finds, figuring out the relationships between different pottery types at different sites has been challenging.
“because commonware represents everyday objects, it allows us to see a larger proportion of the ancient population than statues, mosaics, or the nice, painted pots in museums do,” launaro says. creating a catalogue to record these objects and their relationships to one another could offer further valuable insights into the daily life of past civilizations. “but that’s not something one person can easily do,” he says.
schönlieb and parisotto thought they could help launaro address this problem. “someone could just go through all these shapes, but that’s super tedious,” says schönlieb. “and since we are human, we make mistakes, we are tired sometimes, and we might miss something. an algorithm will not tire.”
parisotto and schönlieb turned to matlab to create the catalogue that launaro envisioned. in 2016, they piloted a system meant to match a drawing of a piece of pottery in profile to a similar image in a database. archaeologists classify ancient pottery by their profile shape and think that similar shapes signify chronological as well as functional relationships.
but it didn’t work well. the reference pottery images weren’t well organized. the team needed to take a step back and do that organization themselves. “given that it would’ve involved thousands of unique shapes of pottery, we needed to tap into the processing powers of a computer,” says launaro.
to fill out and organize their own pottery database, the team has added thousands of images, black-and-white profiles of pieces of commonware. by the end of 2020, around 6,000 pottery profiles will be included. parisotto is using unsupervised deep learning algorithms to group or cluster related pottery shapes. the program creates hierarchical dendrograms grouping the pottery shards to better show archaeologists the relationships between different types. “the idea is to extract the relevant features from the available objects,” says parisotto, “and to find the relationships uniting different features.”
by determining the relationships between commonware types, archaeologists could better map their development and distribution over space and time. these relationships may then offer clues about important developments in trade, settlement patterns, or eating habits. the mach group is still developing and testing the app, but, “the idea in the end is to create a tool that enables archaeologists to more effectively interpret the sites they excavate and study,” says launaro.
art lessons from paint chips
kasia targonska-hadzibabic, a research associate with the mach team and physicist by training, is working on an “unveiling the invisible” subproject that draws on principles similar to the roman pottery database. but instead of pottery, targonska-hadzibabic is working with parisotto to build a platform for digital images of paint chip cross-sections that enables their sorting and comparison.
in art conservation, studying the cross-sections of small chips from a painting can reveal how the artist created the work. “it gives you information on the techniques the artist used, how the painting was painted,” says targonska-hadzibabic.
the mach team is also creating a system that could identify connections between paint chip cross-sections from different paintings, artists, or periods in time to figure out what similarities could mean.
traditionally, conservators take these cross-sections, preserve them in resin, and examine the different paint layers under a microscope. up close, a resin-covered paint chip looks like a colorful, many-layered sandwich. targonska-hadzibabic’s colleague in art conservation had been digitizing his own archive of paint chip cross-sections and wanted to see what more, aside from an individual artist’s technique, these paint chips could reveal.
targonska-hadzibabic teamed up with schönlieb to create a system that could identify connections between cross-sections from different paintings, artists, or periods in time. the layers in a cross-section aren’t uniform, and vary not just in color, but in texture, mixture, and consistency in shooting conditions. as with the roman pottery project, the team is using machine learning techniques in matlab to cluster more than 10,000 digital images of cross-sections into meaningful groups based on their features.
according to targonska-hadzibabic, they’re not yet sure what these algorithms might reveal. “it’s an iterative process that relies on communication with conservators to find similarities important from the history of art perspective,” she says.
but with their resulting app, they hope that conservators will be able to compare layers in their source cross-section with similar patches in other cross-sections in the database. not only will app users see these results, but targonska-hadzibabic is working to make sure the conservators can easily modify the results according to their needs, too.
in the field
feedback from archaeological, art conservation, and art historical experts has been crucial to guiding these projects. “only experts can guide the people in data science in the correct path to follow,” says parisotto.
mach collaborators at the fitzwilliam are just beginning to test these apps, but the goal is to publish these resources for all scholars and conservators, expanding their existing toolboxes. for launaro, the commonware pottery reference catalogue will enable more detailed study of a previously overlooked side of archaeology.
according to reynolds, not only could mach’s digital manuscript restoration tool help conservators, but it could also expand options for teaching and public engagement. “there’s a hope that it will be very useful for teaching because it will enable you to work with images of objects at their best,” says reynolds. the tool could also expand museums’ virtual offerings, showing the members of the public both the actual artifact and the digital “original.”
targonska-hadzibabic says that compiling a paint cross-section database that can identify connections between samples could help experts identify new painting methods and reveal previously unknown connections between artists and artworks.
none of these tools, however, will replace the work of humanities professionals. “there is a level of interpretation where you need a human being,” says launaro. “but there are other things that would make our work so much easier and straightforward that a machine can achieve.”
header image: © the fitzwilliam museum, cambridge.
read other stories
green technology / control systems
removing millions of tons of co2 emissions at seaports each year
electrifying commercial vehicles with hydrogen fuel cells
stem / academia
building a future in stem
high school student discovers there is more to coding than a computer screen
academia
professor reengineers matlab course overnight in response to covid
online and interactive tools simplified the move to remote learning
