

Data Lifecycle

We now explain the general evolution of data. This evolution spans the duration of events from the moment the data is sent to the computing resources for the execution of simulation tasks, to the moment it is stored as objects or structured data in the database to be read by the Web Interface. We refer to this evolution flow as the Data Lifecycle.

Flowchart

In the image below, we present a flowchart summarizing the steps involved in the data lifecycle.

The Data Lifecycle is composed primarily of the following three steps. The reader is referred to the same number labelling cited in the above image.

1. Entity Structured Data Creation

The actions performed by the user in the Web Interface lead to the creation of entity-related structured data in the corresponding account-owned collections, stored within the database. Jobs corresponding to the simulation tasks provide one such example.

2. Execution of Simulation

The structured data describing such Entities is then sent from the database to the computing clusters, the core component of the overall computational infrastructure. Here, the relevant simulations are executed, which creates non-structured data in the form of output files.

3. Data Retrieval

Following simulation completion, the output data produced by the simulation engine is stored on disk as unstructured files.

3a. Extraction of Properties as Structured Data

The properties (and/or other information) contained in the output data are subsequently extracted through the processing of simulation output.

This information is further stored as structured data in the database, and can be retrieved, for example, from the Web Interface as part of the account-owned collections of entities.

3b. Storage of Output Files as Objects

All simulation output files are additionally stored in the Object storage, in order to make them accessible to the Web Interface. This can be accomplished under the Files Tab of Job Viewer, for example.