1. Introduction

The Open Geospatial Consortium (OGC®) is releasing this Call for Participation ("CFP") to solicit proposals for the OGC Earth Observation Applications Pilot (also called "Initiative" or just "Pilot"). The goal of the pilot is to evaluate the maturity of the Earth Observation Applications-to-the-Data specifications that has been developed over the last two years as part of various OGC Innovation Program (IP) initiatives in a real world environment. ‘Real world’ includes integration of the architecture in an environment requiring authenticated user identity, access controls and billing for resources consumed.

pilotLogo

The pilot consists of two phases:

Phase One (P1)

Invites application developers that work with Earth observation satellite data to join a requirements definition workshop. The objective of this workshop is to understand the exact requirements of application developers in terms of data discovery, data loading, data processing, and result delivery. The results from this requirements gathering will define the implementation evaluation and performance criteria.

The Goal of Phase 1 is understanding all the requirements that need to be implemented by the OGC Earth Observation Applications Pilot to support efficient app developments.

This phase includes:

  1. App developers to join workshop and explain their approach to allow functionality checks and identification of optimization options for the current architecture. The approach description should include aspects such as:

    1. finding data

    2. working with data

    3. making processes available

    4. result handling

  2. App developers get informed what is available from platform providers:

    1. data and existing processes and corresponding interfaces

    2. processing environments

    3. user identity and authentication, including the possibility to use an existing identity (depending on Identity Provider)

    4. quoting options

    5. billing mechanisms

  3. App developers to describe/document the use-cases for the analysis/applications that they wish to develop and deploy to the platforms. This should consider/agree the data to be made available in order to support the use cases of the developed apps.

Phase Two (P2)

Invites earth observation platform operators to implement the OGC Earth Observation Applications Pilot architecture as it has been defined in previous IP initiatives. The implementation shall take P1 results into account. In essence, this requires implementing two Web API endpoints to allow the registration, deployment and execution of new processes on the platform. Meanwhile, application developers implement their cases (defined in Phase 1) to provide test cases for the architecture.

The Goal of Phase 2 is for Platform Providers to develop the necessary user infrastructure and ADES/EMS services to allow app developers to deploy their apps, and to allow consumers to obtain quotes and execute processes, with associated billing. Simultaneously, app developers implement and deliver their Earth observation applications.

This phase includes:

  1. development and deployment of EMS/ADES on platforms

  2. development and deployment of apps

  3. registration of apps on enterprise platforms

  4. consumers using the apps

  5. consumers execute workflows across platform boundaries

  6. evaluation of architecture design and elements

  7. recording of evaluation results

  8. definition of future work items

Overall Goal

The overall goal of both phases is to achieve a solid understanding of the suitability of the current architecture. This includes better understanding which elements of the envisioned architecture work, and which parts require what type of modification. In this context, the questions shall be answered regarding what works and what does not, including:

  1. discovery and loading of available data

  2. discovery and invocation of existing services

  3. development and deployment of user defined services

  4. usage of external data

  5. making results available

  6. advertisement of new processes, discovery for consumers

  7. enforcement of access controls across platform boundaries (federated/delegated access)

It is expected that the pilot will identify several missing/sub-optimal elements. As is currently hard to predict what form, functionality, or architectural design will require modifications, it is expected that participants have some flexibility during the course of the project in terms of evaluation and analysis. Aspects that cannot be addressed or need further review shall be recorded in a future work definition. Some expected potential issues have been captured in section Architecture Evaluation Aspects.

1.1. Context

The availability of free and open data, such as that provided by the Copernicus Sentinel fleet, together with the availability of affordable computing resources, create an opportunity for the wide adoption and use of Earth Observation (EO) data in all fields of our society.

ESA’s “EO Exploitation Platforms” initiative aims at achieving a paradigm shift from “bring the data to the user” (i.e. user downloads data locally) to “bring the user to the data” (i.e. move user exploitation to hosted environments with collocated computing and storage). This leads to a platform-based ecosystem that provides infrastructure, data, computing and software as a service. The resulting Exploitation Platform is where scientific and value-adding activities are conducted, to generate targeted outputs for end-users.

In order to unite the existing and future resources into an ‘Network of EO Resources’ there is a motivation to define standard interfaces that facilitate the federation and interoperation of these scattered resources – allowing the user to efficiently access and consume the disparate services of different providers seamlessly.

The approach can be applied to different domains (other than Earth Observation) and ESA is thus working closely with OGC in the definition of these standard interfaces and related solutions.

1.2. Background

OGC Testbed activities in Testbed-13, Testbed-14, and the ongoing Testbed-15 have developed an architecture that allows the ad-hoc deployment and execution of applications close to the physical location of the source data. The goal is to minimize data transfer between data repositories and application processes. The following Engineering Reports describe the work accomplished in the Testbed-13/14:

  • OGC Testbed-14: Application Package Engineering Report (18-049r1)

  • OGC Testbed-14: ADES & EMS Results and Best Practices Engineering Report (18-050r1)

  • OGC Testbed-14: Authorisation, Authentication, & Billing Engineering Report (18-057)

  • OGC Testbed-14: Next Generation Web APIs - WFS 3.0 Engineering Report (18-045)

  • OGC Testbed-13: EP Application Package Engineering Report (17-023)

  • OGC Testbed-13: Application Deployment and Execution Service Engineering Report (17-024)

  • OGC Testbed-13: Cloud Engineering Report (17-035)

Testbed-13 reports are referenced as they allow to understand the history of this work and design decisions in context, but are mostly superseded by Testbed-14 reports.

At the same time, significant progress towards more Web-oriented interfaces has been made in OGC with the emerging OGC APIs -Core, -Features, -Coverages, and -Processes. All of these APIs are using OpenAPI. These changes have not fully explored in the current architecture yet, which provides additional ground for experimentation as part of the pilot.

1.2.1. OGC Innovation Program Initiative

This Initiative is being conducted under the OGC Innovation Program. The OGC Innovation Program provides a collaborative agile process for solving geospatial challenges. Organizations (sponsors and technology implementers) come together to solve problems, produce prototypes, develop demonstrations, provide best practices, and advance the future of standards. Since 1999 more than 100 initiatives have been taking place.

1.2.2. Benefits of Participation

This Initiative provides a unique opportunity towards implementing the "applications to the data" paradigm in the context of Earth observation satellite data processing. It allows to create new business opportunities, explores the market readiness of the own platform, and close interaction with application developers; while the provided cost sharing boosts your R&D budget. With the European Space Agency (ESA) as main funding organization, new contacts and relationships with ESA and other space agencies can be established.

The outcomes are expected to shape the landscape of cloud-based platforms developed to facilitate and standardize the access to Earth observation data and information. The sponsorship supports this vision with cost-sharing funds to partially offset the costs associated with development, engineering, and demonstration activities that are part of this pilot. The cost-sharing offers selected participants a unique opportunity to recoup a portion of their initiative expenses.

1.3. Acronyms and Abbreviations

The following acronyms and abbreviations have been used in this report.

Table 1. Acronyms and Abbreviations

Term

Definition

ADES

Application Deployment and Execution Service

CFP

Call for Participation

CR

Change Request

DER

Draft Engineering Report (OGC Document)

DWG

Domain Working Group

ER

Engineering Report (OGC Document)

ESA

European Space Agency

FDER

Final Developer Engineering Report

FPPER

Final Platform Provider Engineering Report

GPKG

GeoPackage

IDER

Initial Developer Engineering Report

IP

Innovation Program

IPPER

Initial Platform Provider Engineering Report

OGC

Open Geospatial Consortium

ORM

OGC Reference Model

OWS

OGC Web Services

PA

Participation Agreement

POC

Point of Contact

Q&A

Questions and Answers

RM-ODP

Reference Model for Open Distributed Processing

SOW

Statement of Work

SWG

Standards Working Group

TBD

To Be Determined

TC

OGC Technical Committee

TEM

Technical Evaluation Meeting

TIE

Technology Integration/Interoperability Experiments

URL

Uniform Resource Locator

WFS

Web Feature Service

WPS

Web Processing Service

WG

Working Group

1.4. Applicable and Reference Documents

The following is a list of Reference Documents with a direct bearing on the content of this document.

Table 2. Applicable Documents
Reference Document Details

[IPPP]

OGC Innovation Program Policies & Procedures
http://www.opengeospatial.org/ogc/policies/ippp

Table 3. Reference Documents
Reference Document Details

[TB13-ER]

OGC Testbed-13 EOC Thread Engineering Reports:

[TB14-ER]

OGC Testbed-14 EOC Thread Engineering Reports:

[OGC-ER-PROC]

[OGC-ER-TPL]

[OGC-ABS-SPEC]

OGC Abstract Specifications
http://www.opengeospatial.org/docs/as

[OGC-STD]

[OGC-PROF]

[OGC-COM-STD]

[OGC-STD-BASE]

[OPENEO]

[OPENEO-UDF]

[ECMWF-CDS]

[SEED]

[DOCKER-OBJ]

[OPENAPI]

1.5. Roles and Responsibilities

Within this document and activity, the following roles are defined.

Table 4. Roles and Responsibilities
Term Company / Entity Role

Agency

The European Space Agency

Sponsor of OGC Pilot described by this CFP

EO Exploitation Platform Prime Contractor

Telespazio VEGA UK Ltd

Prime Contractor to ESA and supervisor for this pilot

Subcontractor (Participant)

The Supplier

The organization that will take part in the OGC pilot task and whose activities are defined in this CFP. Subcontractors will have subcontract with OGC.

OGC

Open Geospatial Consortium

Overall responsibility for the Pilot activities and contractor to participants.


2. Procurement Context

2.1. Project Organisation and Responsibilities

This pilot is conducted under the OGC Innovation Program and implements the OGC Innovation Program "Pilot" initiative policies and procedures. In this specific context, additional policies and procedures apply: The pilot is open to all organizations. Cost sharing funds can only be requested by entities from ESA member states, Canada, and Slovenia.

Within the context of this procurement ESA are acting as a sponsor to the OGC Earth Observation Applications Pilot. Participants for ESA activities within the pilot will be selected jointly by ESA/TVUK and the OGC from responses to this Call for Participation, and will become Subcontractors to OGC. Subcontractors will be referred to as Participants.

Once initiated, the pilot activities will be managed by OGC. The Participants will interact directly with OGC for technical, management, and contractual matters. ESA and TVUK should be copied on correspondence and invited to meetings.

Cost sharing payments to Participants selected by this call are funded by ESA. During and on completion of the pilot activities, OGC evaluates achievement of each milestone in order to trigger payments due under this contract. Payments will be made directly from OGC to the Participant.

The contractual conditions, at project level, come from the prime contract between ESA and TVUK, incorporate OGC pilot terms & conditions and then down to the Participant.

2.2. Financial Model for the Pilot

The pilot initiative works on a model of Participant in-kind contributions of engineering resources, technology, and/or facilities and cost-share funding provided by the Sponsors such as ESA.

TVUK, ESA or OGC will not cover the costs of equipment or software required for the normal development of participant software products. Nor will TVUK, ESA or OGC fund the purchase of additional hardware or software. OGC on behalf of TVUK and ESA will support the unique costs associated with engineering labour and travel (limited to kick-off and demonstration events) - noting that travel is often provided as an in-kind contribution for OGC activities.

This would include, but not necessarily be limited to: labour to develop engineering requirements, engineering specification reports, prototypical software component development exercising OGC specifications, instantiating new services based on existing software that exercises relevant OGC specifications, documentation of specifications, demonstrations of prototypical or existing software components that exercise OGC specifications or draft specifications and travel to demonstration events.

Proposals submitted under this call shall include a full breakdown of in-kind contributions by participants and required cost sharing contributions from OGC on behalf of TVUK/ESA.


3. Technical Architecture

This section provides the technical architecture and identifies all requirements and corresponding work items. It references the OGC standards baseline, i.e. the complete set of member approved Abstract Specifications ([OGC-ABS-SPEC]), Standards ([OGC-STD]) including Profiles ([OGC-PROF]) and Extensions, and Community Standards ([OGC-COMM-STD]) where necessary. Further information on the OGC standards baseline ([OGC-STD-BASE]) can be found online.

The overall goal is to implement the "application to the data" paradigm for satellite data that is stored and distributed on independent platforms. The basic idea is that each platform provides a standardized interface that allows the deployment and parameterized execution of applications that are packaged in Docker containers. A second platform, called Exploitation Platform, allows chaining applications into workflows with full support for quoting and billing. The architecture is described in full detail in the Engineering Reports for Testbeds 13/14 referenced in [TB13-ER] and [TB14-ER].

3.1. Architecture Components

The OGC Earth Observation Applications Pilot defines a set of interface specifications and data models working on the HTTP layer. The architecture allows application developers and consumers to interact with service endpoints that abstract from the underlying complexity of data handling, scheduling, resource allocation, or infrastructure management.

components
Figure 1. Earth observation cloud application architecture components

It consists of the following logical components:

  • Application Developers that develop Earth observation data processing applications

  • Application Consumers requesting the execution of these applications on remote data and processing platforms

  • One (or more) Docker Hubs that allow storing containerized applications. The hub(s) need to be accessible for the data and processing platform(s)

  • One (or more) Exploitation Platform to register applications, to chain these into workflows, and to request the deployment and execution on data and processing platforms

  • One (or more) Data and Processing Platform, where applications are executed on local data

Partly, these logical components can be implemented jointly. For example, it is likely that a data and processing platform provides exploitation platform elements, in particular as the various APIs are based on the same underlying OGC specifications.

Together, the architecture components allow the execution of the following steps:

  1. Application developers can develop applications in their local environment and make the application available as a Docker container on a Docker Hub. A corresponding metadata file called Application Package describes the application and defines all necessary parameters, such as required input data, start-up parameterization, etc. The Application Package gets registered with the Execution Management Service (EMS), a RESTful Web service endpoint. The API of that Web service is defined by an OpenAPI document. It implements the currently emerging standard OGC API Process. The EMS backend contains the control logic as well as additional components such as a catalog to register all Application Packages, a catalog client to find appropriate data etc.

  2. Application Consumers discover available applications through the same EMS endpoint. For that reason, the EMS delivers a list of available processes that it can execute. The app consumer identifies a specific application and provides the necessary parameterization (e.g. values for area/time of interest). From this data, the EMS then builds the necessary calls to the Application Deployment and Execution Service (ADES). The ADES is basically a lightweight EMS with much reduced functionality. It only allows the (un-)deployment of applications and their parameterized execution.

3.1.1. Exploitation Platform

The Exploitation Platform is responsible for registration and management of application packages and the deployment and execution of applications on data and processing platforms. It further supports workflow creation based on registered applications, and aggregates quoting and billing elements that are part of these workflows. Ideally, the Exploitation Platform selects the best suited Data and Processing Platform based on application consumer’s needs.

exploitationPlatform
Figure 2. Components of the exploitation platform

The Exploitation Platform itself consists of the following components:

EMS API

The EMS (Execution Management Service) provides a RESTful interface to application developers to register their applications and to build workflows from registered applications.

App Registry

Registry implementation (i.e. application catalog) that allows managing registered applications (with create, read, update, and delete options)

Workflow Builder

This optional component supports the application developer to build workflows form the registered applications.

Workflow Runner

Workflow execution engine to execute workflows and to handle the necessary data transfers from on application to the other.

ADES Client

Client application to interact with the data and processing environments that expose the corresponding ADES (Application Deployment and Execution Service) API.

Billing & Quoting

A component that aggregates billing and quoting elements from the data and processing environments that are part of a workflow.

Identity & Access Management

User management

3.1.2. Data and Processing Platform

These platforms have the raw data the applications work upon available locally. They allow deployment and execution of applications through the Application Deployment and Execution API.

dataProcessing
Figure 3. Components of the data and processing environment

The Data and Processing Platform consists of the following components:

Data Repository

A repository of data that can be made available to Docker containers for local processing.

ADES API

Application Deployment and Execution Platform_ API to deploy, discovery, and execute applications or to perform quoting requests.

Docker Daemon

Docker environment to instantiate and run Docker containers.

Billing & Quoting

Component that allows obtaining quotes and final bills.

Workflow Runner

Workflow runner that can start the Docker container applications.

Identity & Access Management

User management

3.2. Architecture Interactions

The following figure illustrates all major components and the corresponding interactions.

BIDS2019 environment
Figure 4. Earth Observation Cloud Application Architecture

The Execution Management Service (EMS) represents the front-end to both application developers and consumers. It makes available an OGC Web Processing Service interface that implements the emerging OGC Web API principles, i.e. provides a Web API that follows REST principles. The API supports the registration of new applications. The applications themselves are made available by reference in the form of containerized Docker images that are uploaded to Docker Hubs. These hubs may be operated centrally by Docker itself, by the cloud providers, or as private instances that only serve a very limited set of applications. Details mostly depend on security considerations. Initially developed to deploy applications only, the EMS has been developed into a workflow environment that allows application developers to re-use existing applications and orchestrate them into sequential workflows that can be made available as new applications again. This process is transparent to the application consumer.

The Application Package (AP) serves as the application meta data container that describes all essential elements of an application, such as its functionality, required satellite data, other auxiliary data, and input parameters to be set at execution time. The application package describes the output data and defines mount points to allow the execution environment to serve data to an application that is actually executed in a secure memory space; and to allow for persistent storage of results before a container is terminated.

The execution platform, which offers EMS functionality to application developers and consumers, acts itself as a client to the Application Deployment and Execution Services (ADES) offered by the data storing cloud platforms. The cloud platforms support the ad-hoc deployment and execution of Docker images that are pulled from the Docker hubs using the references made available in the deployment request.

Once application consumers request the execution of an app, the exploitation platform forwards the execution request to the processing clouds and makes final results available at standardized interfaces again, e.g. at Web Feature Service (WFS) or Web Coverage Service (WCS) instances. In the case of workflows that execute a number of applications sequentially, the exploitation platform realizes the transport of data from one process to the other. Upon completion, the application consumer is provided a data access service endpoint to retrieve the final results. All communication is established in a web-friendly way implementing the emerging next generation of OGC services that provides the various OGC APIs -Features, -Coverages, or -Processes.

BILLING AND QUOTING

Currently, satellite image processing still happens to a good extent on the physical machine of the end-user. This approach allows the end-user to understand all processing costs upfront. The hardware is purchased, prices per satellite product are known in advance, and actual processing costs are defined by the user’s time required to supervise the process. The approach is even reflected in procurement rules and policies at most organizations that often require a number of quotes before an actual procurement is authorized.
The new approach featured in this pilot requires a complete change of thinking. No hardware other than any machine with a browser (could even be a cell phone) needs to be purchased. Satellite imagery is not purchased or downloaded anymore, but rented just for the time of processing, and the final processing costs are set by the computational resource requirements of the process. Thus, most of the cost factors are hidden from the end-user, who does not necessarily know if his request results in a single satellite image process that can run on a tiny virtual machine, or a massive amount of satellite images that are processed in parallel on a 1000+ machines cluster.

The currently ongoing efforts to store Earth Observation data in Datacubes adds to the complexity to estimate the actual data consumption. The reason is that the old unit ”satellite image” is blurred when data is stored in multidimensional structures not made transparent to the user. Often, it is even difficult for the cloud operator to calculate exact costs prior to the completed execution of a process. This leads to the challenging situation for both, cloud operators that have to calculate costs upfront, and end-users that do not want to be negatively surprised by the final invoice for their processing request.

To address this challenge, OGC has started the integration of quoting and billing features into the cloud processing architecture. Specific service endpoints support quote requests that are executed prior to the actual execution requests. These allow a user to understand what costs will occur for a given service call, and they allow execution platforms to identify the most cost-effective cloud platform for any given application execution request.

Quoting and Billing information has been added to the Execution Management Service (EMS) and the Application Deployment and Execution Service (ADES). Both services are implemented in a web-friendly way as a Resource Oriented Architecture (ROA) Web API that resembles the behavior of the current transactional OGC Web Processing Service v2.0 (this version, v.3.0, is not published yet by OGC). The API is described by an OpenAPI 3.0 specification (see [OPENAPI]) that allows deploying and executing new processes by sending HTTP POST request to the DeployProcess operation or Execute operation endpoints. Following the same pattern, it allows posting similar requests against the Quota endpoint, which causes a JSON response with all quote related data. The sequence diagram in Figure 5 illustrates the workflow.