WP4 Iliad DTO – Ocean Interoperability Data Space

The NEXTGEOSS DataHub provides a federated data/service catalogue for GEO-oriented data sets and services, which will become interoperable with the GAIA-X/IDS Federated catalogue and the Federated Identity with the OAuth2 and X509 standards. The Iliad Marketplace will use this with the data/service marketplace extensions developed for store4EO.

Each existing platform or system exposes an interface to the Iliad interoperability space via a GAIA-X/IDSA Provider Connector. The Connector acts as a gateway that efficiently and securely transfers data from a data provider to a data consumer, while data sovereignty is technically enforced by usage constraints that govern data usage. The data owner remains in control of the data, including private or business critical information. The data sharing infrastructure will be decentralised, whereby all data is transferred directly between a data provider and a data consumer based on a virtual peer-to-peer network. The data sharing infrastructure itself only stores metadata that describes the data sources. Metadata describes the syntax and serialisation as well as the semantics of data sources. Each Connector contains an Information Model which is an essential agreement shared by the owners of existing systems and the Iliad framework, facilitating compatibility and interoperability. Interoperability will be enhanced by applied mappings to the Ocean Information Model (see Task 4.2). The primary purpose of this formal model is to enable (semi-)automated exchange of digital resources within a trusted ecosystem of distributed parties, while preserving data sovereignty of data owners. The Information Model is a generic model, with no commitment to any particular domain. Domain modelling is delegated to shared vocabularies and data schemata, as provided by domain-specific and other communities, mediated by the Definition Server developed in task 4.2 and applied to data according to new approaches developed in task 4.3. The user management with an authentication, authorisation and accounting (AAA) service is based on identity4EO by DEIMOS.

• Duration:  M1 - M18
• Task Lead:  Deimos Space, Sociedad Limitada Unipersonal (DEIMOS)

This task delivers the semantic interoperability mechanisms that enable different systems in the Ocean domain to provide and exchange data with unambiguous meaning. Using linked data approaches and Ocean Information Model, this task will link essential information required for integrated views on heterogeneous datasets and establishes the baseline for data processing, data science, and advanced data analytics. Starting with a thorough state of the art analysis, this task will design the Iliad Ocean Information Model (OIM), a common semantic data model to be used to ensure semantic interoperability among Ocean data related data sets and services. This common data model will not be built ab initio, but will incorporate and extend existing ontologies and vocabularies available for the Ocean domain (e.g., Ocean Data Ontology), as well as for related domains such as weather data (e.g., SWEET), IoT (OGC SensorThings API,SSN /SOSA Ontology, sensorML), statistical and multidimensional data (e.g., RDFdata cube vocabulary, QB4ST and SDMX), Citizen science/user generated data (e.g., RDF review vocabulary, MUTO ontology), as well as ocean transport/shipping data (e.g., OTM ontologies), while it will also make use of existing ontologies for data catalogues (DCAT/GeoDCAT-IP). OIM is going to be an agile artefact responding to the changing requirements of the pilots and the wider ecosystem. Enhanced interoperability for existing resources and services, even with concurrent underlying models and semantics, will be further provided through extending the OGC Definition Server. The Definition Server is a Web accessible source of information about things ("Concepts") that are defined by the broader oceans community or result from external but related standardisation and policy bodies. OGC uses stable web addresses (URIs) to unambiguously identify concepts in its specifications. The OGC Definitions Server makes those URIs "work" - i.e., makes them dereference to a definition that can be used, so that vocabularies with uniquely defined terms can be created. This approach allows handling co-exiting concurrent and non-consistent base semantics and information resources, which is an essential requirement for services and products that cannot be modified easily.

• Duration:  M4 - M30
• Task Lead:  Instytut Chemii Bioorganicznej Polskiej Akademii Nauk (PSNC)

The overall aim of this task is it to enhance input (meta)data needed by the DTOs for downstream enrichment (e.g. analytics), inter-linkage, and consumption by providing semantically enabled tools and software services. This task will result in new platform functionalities, incl. a web interface, to explore and discover datasets using a faceted, semantic search interface compliant with any metadata standard demanded by the Iliad Marketplace. The semantic basis for the search interface is that items (maps or datasets) are found by Terms which are organised into Facets which are organised into Taxa. Items can have some displayable properties and can be related by isPartOf or isReplacedBy; where terms can imply other terms. All this can be cleanly expressed in RDF, and as metadata standards (e.g. INSPIRE) are now all based on ontology representations any other metadata model need only be mapped into this RFDF model in order for it to be consistently exposed in the search interface. Also, selected classes from Schema.org will be inherited for Search Engine Optimisation (SEO) purposes. Thus, this is an expansion of the semantic API, which forms the basis for providing compliance to international standards for interoperability of structures and services of the IDSbased Iliad architecture. The UIX of the faceted search will allow for graphical discovery and matching between multiple Consumer/Producer offerings to allow competition at an asset (i.e. dataset) level, but also for fail-over and redundancy purposes. Metadata contributes to the discoverability of a dataset. Therefore, a service will be designed and deployed that provides recommendations for metadata attributes published through the DAP streamer to facilitate better discovery and interoperability in the Iliad Marketplace. This proposed webservice provides an easy-to-understand visual overview based on a completenessscoring per data discovery facet (e.g. Identification and Metadata, TextSearch, Extent Search, etc.). Given the proliferation of multiple metadata standards, a tool will be designed that can check for compliance to multiple metadata standards. The service is entitled “DRS-validator” as it validates a Data Service Provider (DSP)’s dataset, e.g. exposed through the OPeNDAP interface, by checking for compliance with the Data Reference Syntax (DRS) metadata. This will allow the interface to trigger an early alarm to a DSP that signals what metadata is still non-compliant. What is not there can also not be harvested. So, in case metadata remains missing or non-compliant a Metadata Content Management System (M-CMS) will be developed allowing Marketplace content managers to augment and manage the (meta)data of any datasets on behalf of “lazy” Data Service Providers (DSP).

• Duration:  M13 - M36
• Task Lead:  Ramani B.V (RAMANI)

The DTO will be supported by the open Ocean Data Platform of C4R Ocean – with the associated Contextualisation services powered by Cognite Fusion for time and space interoperability resolution mappings and harmonisation. The further extensions for the DTO will be supported through the modelling facilities provided by WP5 – which will enable various aspects of the DTO to be enhanced. Thistask will provide support for activities focusing on preparing, analysing, fusing and integrating the incoming datasets into the DTO as well as providing the mechanisms to access and share them. To achieve that, a big data strategy will be followed in order to “translate data into useful information”. To this end, all necessary tools and services for rendering the incoming datasets, performing (spatial) data fusion and providing an integrated data view, ready for more complex analysis, will be supported. This will require the use of the Ocean Information Model (OIM) for the representation of the different data elements and their relationships. Having many technologies, platforms, systems and devices in the Iliad ecosystem implies that there will be a multitude of data sources producing information that will be consumed, processed and stored by several interested consumers simultaneously through the harmonised DTO.

• Duration:  M7 - M30
• Task Lead:  HUB Ocean (HUB Ocean)

This task develops a set of standardised Web API building blocks that make the exchange of ocean data and information more efficient, reliable, and cost effective. These building blocks shall be aligned with international standardisation efforts to ensure reliability and interoperability with existing and emerging external systems. The task will provide Web API development expertise to all Iliad partners, align current API work with Iliad requirements, and ensure sustainability of Iliad results by bringing them back into the standardisation process. A robust starting point in this context is the Environmental Data Retrieval (EDR) API, defined by the OGC Oceans and Meteorology Working group. The Web API is optimised for many situations of ocean data discovery, access, and processing. The OGC Earth Observation Applications architecture is well aligned with specific requirements from ESA and Copernicus and allows the ad-hoc deployment of arbitrary applications at remote cloud platforms. OGC is happy to develop these concepts further as part of Iliad. Further on, the task shall develop guidance on API governance and consensus driven development

• Duration:  M13 - M36
• Task Lead:  Open Geospatial Consortium Europe (OGC)

This task addresses integration of live data streams into the DTO by further developing the StreamHandler platform by INTRA. Simulation and testing data sources will be seamlessly integrated with processing components by means of integration connectors (Connectors) efficiently interoperable with all modern data storage technologies (RDBMS, NoSQL, HDFS Hadoop, Apache HBASE). The communication platform will connect external data sources and make them available to the Digital Twin of the Ocean (DTO) by employing the streaming component and multiple workers of the connect component allowing the realisation of scalable and secure data stream pipelines. The deployment of the underlying infrastructure is also part of this Task, which spans multiple VMs and provides all necessary technologies and components enabling the storage and analysis of data and the usage of any technology agnostic algorithms by providing a distributed computing environment that enables needed functionalities (Apache Spark, Hadoop, Kafka Streams, Spark Streaming, etc., and across big data languages including Python, Java, R and Scala).

• Duration:  M7 - M33
• Task Lead:  Netcompany-Intrasoft (INTRA)