The University of Kyoto in Japan lost approximately 77 TB of scientific data due to a bug in the backup system of its Hewlett-Packard supercomputer. The incident occurred between December 14 and 16, 2021, and resulted in the deletion of 34 million files belonging to 14 different scientific groups from the system and the backup media.
“Dear users of supercomputer services, today error in the storage system backup program caused an accident in which some files in / LARGE0 were lost. We stopped the problem, but we may have lost almost 100 TB of files. “ said the Information Infrastructure Division on Thursday, December 16. Five days later, it specified the total extent of the damage at 77 TB.
Lost scientific data
After the investigation, the university came to the unpleasant conclusion that the work of four of the 14 affected groups could not be resumed. All affected users were individually notified of the incident by e-mail, but no details of the type of work lost were published. Available data suggests that the university operates Hewlett Packard Cray computing systems and the DataDirect ExaScaler storage system.
The backup process has now been stopped. To prevent repeated data loss, the university has phased out the backup system, and plans to apply the adjustments and reintroduce it in January 2022. In addition to mirroring full backups, the plan is to keep incremental backups that include files that have changed since the last backup.
It should be noted that the total capacity of Kyoto University’s main repository is 24 PB. From this point of view, the loss of 77 TB is only a small fraction – specifically 0.32%. Of course, we don’t want to downplay the size of the problem – just to suggest that while 77 TB may seem like an unimaginable amount from a regular user’s perspective, it’s just “crumbs” on a supercomputer system scale.
The Kyoto supercomputer helps science
Kyoto University is considered one of Japan’s most important research institutions and enjoys the second largest investment in scientific research from national grants. Its scientific level and importance stand out especially in the field of chemistry, where it ranks fourth in the world, but it also contributes to the development of biology, pharmacology, immunology, materials science and physics.
Supercomputers differ from ordinary computers mainly in their speed and ability to use multiple computer systems to process complex mathematical calculations. Due to their advantages over conventional computers, they are a valuable tool for research in a wide range of fields, including climate and atmosphere modeling, physics, vaccine development and many other fields.