EUDAT cloud and data preservation solutions support the ICOS mission

The Integrated Carbon Observation System (ICOS) ERIC is an organisation of twelve member countries and over 130 greenhouse gas measuring stations aimed at quantifying and understanding the greenhouse gas balance of the European and neighbouring regions. The overall aim of ICOS is to support high-quality climate change research and provide access to research data.

The Carbon Portal is the data portal of ICOS ERIC and ICOS Research Infrastructure where all users can find and download the ICOS data and ancillary products. The Carbon Portal also provides virtual research environments where users can apply models, combined with ICOS and other data and visualize and/or publish the results again through the data portal. Carbon Portal is now in the process to apply for the CoreTrustSeal certificate, supported by the FAIRSFAIR Horizon2020 project.

The Carbon Portal requires trusted long term storage and distributed computing facilities to enable the portal functionality and to provide the virtual research environments with enough resources for the potential growing needs from its users. To address this challenge ICOS decided to leverage on the cloud and data analysis and preservation solutions offered by the EUDAT CDI.

The partners started to work on a proof of concept that should include transfering ICOS data to EUDAT B2SAFE in order to maintain a secure copy of the data, and to make the data available for data analysis on a cloud platform. As a result, the raw and processed measurement data of all ICOS RI stations is now copied and stored in the EUDAT distributed and secure policy based storage, B2SAFE, to maintain a secure copy of the data.

This data is then made available for processing and data analysis on the data portal and on the CSC Pouta cloud platform: in this way, researchers can easily and seamlessly access the ICOS data. To enable the above, EUDAT has developed interfaces (APIs) that allow transfer of data from ICOS RI to B2SAFE. The APIs also support streaming of data in near real-time to enable the data management at ICOS Carbon Portal.

The implementation of the streaming solution is an outstanding result achieved by the EUDAT and ICOS teams that over the past months have worked closely together to improve the performance of the streaming solution to enhance its reliability and performance. Currently, the ICOS data is managed in the B2SAFE installation located at CSC, the Finnish IT Center for Science where also the development work has mainly been done.

A replica of the data to a second EUDAT center (FZJ Julich, Germany) is planned. The data from B2SAFE is then successfully made available for researchers via the CSC’s Pouta open cloud platform. The proof of concept of the above system has been developed within the EOSC-hub ICOS competence center of which still some aspects have to be finalised to enhance the developed solution.

“ICOS strives to provide reliable data to inform science and society on the greatest environmental challenge of our time: global warming through greenhouse gases. Trust in our data also depends on robust and reliable data storage in real-time. Our checksums connected to persistent identifiers and recorded provenance metadata guarantees at any time that the data provided is the same as generated at measurement and/ or processing time and provides full transparency on our data, leaving no room for skepticism.” highlights Alex Vermeulen, Director of the ICOS ERIC Carbon Portal.

As the early results of the implementation were very promising, ICOS and EUDAT have decided to continue the development and deployment of services in mutual collaboration and after an open tender procedure for procuring storage and computing services for the ICOS ERIC Carbon Portal, the EUDAT CDI is now supporting ICOS also beyond the EOSC-hub project. The workflow implemented for ICOS sets the basis for a solution linking data portals with large scale computing facilities for data analysis and for distributing data to second location for trusted long-term preservation that can support communities from other thematic sectors in the future.

Photos by Konsta Punkka together with ICOS ERIC.