This is a website for an H2020 project which concluded in 2019 and established the core elements of EOSC. The project's results now live further in www.eosc-portal.eu and www.egi.eu

Data Movement

DataHub webinar

Training material used during the webinar: "The EGI Datahub to federate distributed data sets for data-intensive applications in the cloud"

Recording is also available here: https://indico.egi.eu/event/5089/attachments/12945/15280/zoom_0.mp4

Transparent data movement from EGI to EUDAT

The main goal of the EOSC-hub project is to synergize the services and infrastructures from major European providers including EGI and EUDAT, enabling researchers to freedom of choice of the services, data and infrastructure necessary for their work.  Users from many communities have reported interest in being able to use resources provisioned by these 2 infrastructures simultaneously in a transparent fashion, i.e.:

  • using single sign on mechanisms between infrastructures
  • being able to transparently transfer data between infrastructures

EOSC-hub WP6 has been endeavoring to enable transparent data access between the infrastructures. The main data management components in EGI and EUDAT are respectively, EGI-DataHub based on Onedata platform and B2SAFE/B2STAGE based on iRODS. Currently no transparent mechanisms allowing users to move data between these infrastructures exists, the only options are manual transfers using such tools as GridFTP.

These two demo videos present the managment of data transfer across EGI, EUDAT, INDIGO data and compute services. 

The first demo video shows the how data can be transparently moved from EGI to EUDAT. It divided into 4 sections:

  • Registration of B2STAGE storage in EGI-DataHub
  • Creation of dsata space in EGI-DataHub supported by the B2STAGE storage
  • Upload of some data to the data space
  • Data access from GUI and command line

The second demo video presents import of existing data from EUDAT to EGI. The demo is divided into the following 4 sections:

  • Registration of B2STAGE storage in EGI-DataHub
  • Creation of data space in EGI-DataHub supported by the B2STAGE storage from which existing data is imported to EGI-DataHub
  • Access of data from B2STAGE using EGI-DataHub user interface
  • Automatic discovery of new data added on the B2STAGE storage

EGI DataHub

This course introduces the EGI DataHub and Open Data Platform and how to exploit their main features such as:

  • creation of distributed filesystems;
  • access to distributed filesystems through POSIX and CDMI interfaces;
  • exploiting DataHub and Open Data Platform features from a cloud VM hosted in the EGI Federated Cloud;
  • publish data in open data catalogues such as OpenAIRE.