Blog Post #5 – Teaching Robots to Handle History: How AUTOMATA Is Developing Intelligent Manipulation for Cultural Heritage
Imagine working in a museum storage room filled with archaeological artefacts. Shelves contain pottery fragments, fragile artefacts, and objects whose surfaces have survived centuries of burial, excavation, and conservation. Each item is unique. Some are irregularly shaped, others delicate, and many are irreplaceable.
Now imagine asking a robot to pick one of them up.
For humans, this task already requires experience, patience, and a steady hand. For a robot, it is an even greater challenge. Unlike industrial environments, where objects are standardised and predictable, cultural heritage collections are characterised by variability and fragility. Teaching robots to interact safely with these artefacts requires a combination of advanced perception, intelligent planning, and carefully designed manipulation strategies.
This is precisely the challenge addressed by the AUTOMATA project. Researchers are developing a robotic system capable of detecting, analysing, and manipulating cultural heritage artefacts in a safe and reliable way. At the core of this system lies a set of algorithms designed to help the robot “understand” the objects it encounters and decide how to interact with them
Why manipulating artefacts is so complex
Archaeological artefacts rarely conform to neat geometries. A ceramic fragment may have sharp edges, curved surfaces, or missing parts. A stone tool might present irregular contours formed by ancient manufacturing techniques. Even objects belonging to the same category can differ greatly in size, shape, and preservation state.
For a robotic system, this variability creates a series of challenges. Before touching an artefact, the robot must first identify it within the scene, understand its geometry, estimate how it should be grasped, and plan a movement that minimises risk.
Each of these steps requires sophisticated algorithms working together in real time. In the AUTOMATA system, this process begins with object detection.
Teaching the robot to “see” artefacts
The first step in robotic manipulation is recognising the object in front of the robot. This might sound straightforward, but in practice it is far from trivial.
Artefacts may appear in cluttered environments such as museum workspaces or storage facilities. Their surfaces can be worn, textured, or partially hidden. Lighting conditions may vary, and the objects themselves rarely have the regular shapes typically found in industrial components.
To address this problem, the AUTOMATA system employs an advanced image segmentation method known as the Bilateral Reference Network (BiRefNet). This algorithm analyses high-resolution images captured by the robotic system and separates the artefact from the surrounding environment.
High resolution is particularly important in the context of cultural heritage. Many artefacts feature subtle details -small edges, decorative elements, or irregular contours- that must be recognised accurately. BiRefNet allows the system to preserve these fine details while maintaining computational efficiency, ensuring that the robot can operate in real time (fig. 1).
Fig. 1 – BiRefNet high-resolution segmentation of a video frame showing a ceramic fragment.
In simple terms, the algorithm enables the robot to distinguish the artefact from everything else in the scene, effectively allowing it to “see” the object it needs to interact with.
Understanding how to grasp an object
Once the artefact has been detected, the next challenge is determining how to pick it up safely. This involves estimating the object’s pose, that is, its exact position and orientation in three-dimensional space.
Pose estimation allows the robot to determine where its gripper should approach the artefact and how it should align itself before making contact. Accurate estimation is essential: even a small error in positioning could result in an unstable grasp or unwanted contact with fragile surfaces.
In the AUTOMATA system, this step is closely connected to the design of the robotic gripper itself. Instead of using a rigid industrial gripper, the project incorporates a soft end-effector. Soft robotic grippers are designed to adapt to the shape of the object they handle, applying gentle and distributed pressure rather than rigid force.
This adaptability is particularly valuable when dealing with irregular artefacts. The algorithms developed within the project estimate the optimal pose for the soft end-effector relative to the object, ensuring that the robot approaches from a safe direction and establishes a stable grasp.
The combination of pose estimation and soft robotics allows the system to interact with objects that would otherwise be extremely difficult to manipulate automatically (fig. 2).
Fig. 2 – from the output of the object detection algorithm to the soft end-effector grasping pose.
Planning safe robotic movements
Grasping the artefact is only part of the story. The robot must also move safely before and after the grasping action.
This is where motion planning becomes crucial. Motion planning algorithms calculate the path the robot should follow to reach the object, grasp it, and move it to its destination without collisions or sudden movements.
In the AUTOMATA system, these algorithms generate feasible trajectories that guide the robot arm through its environment. The planned movements must avoid obstacles, maintain stability during manipulation, and minimise mechanical stress on the artefact (fig. 3).
Fig. 3 – Top: trajectory planning showing waypoints A, B, and C in execution order; Bottom: robot movements during execution, stopping at waypoints A, B, and C.
Equally important is the ability to operate in real time. Museum environments can be dynamic, and unexpected changes may occur. The planning algorithms therefore continuously evaluate the robot’s actions, allowing the system to adapt if needed and ensuring that manipulation remains safe and reliable.
Supporting heritage professionals with robotics
The goal of the AUTOMATA project is not to replace human expertise. Archaeologists, conservators, and museum professionals possess invaluable knowledge about the artefacts they work with. Instead, the aim is to develop technologies that can support them in their daily activities.
Robotic systems capable of safe manipulation could assist in several areas of cultural heritage work. For example, they could help automate parts of the digitisation process, allowing artefacts to be scanned and documented more efficiently. They could also support the management of museum storage facilities, where large numbers of objects must be organised, moved, and monitored.
By handling repetitive or physically demanding tasks, robots could reduce the risk of accidental damage and allow experts to focus on research, conservation, and interpretation.
A step towards intelligent heritage robotics
Developing robots that can interact safely with cultural heritage objects is a complex interdisciplinary challenge. It requires expertise in robotics, computer vision, artificial intelligence, and heritage science.
Within the AUTOMATA project, these fields come together to create a system capable of perceiving artefacts, understanding their geometry, and planning safe interactions. Through advanced segmentation algorithms, pose estimation methods, soft robotic grippers, and intelligent motion planning, the project is pushing the boundaries of what robotic systems can achieve in heritage environments.
While this technology is still evolving, its potential is significant. In the future, intelligent robotic systems may become valuable partners in museums, laboratories, and conservation centres, helping us document, study, and preserve the objects that tell the story of our past.
In this way, innovation in robotics does not only advance technology. It also contributes to protecting and understanding the cultural heritage that connects us to human history.