Chapter 6. Managing the Application Data

The way data is managed by the application can be tailored to the project requirements in many ways before its deployment. For more details, refer to Section Deploying the Application. These customizations are described in greater details in the other Parts of this documentation, Understanding, Customizing, and Securing.

The data that the application allows to manage is contained in "scenarios". A scenario file is a file that contains data for a set of:

tables called "entities", which contain
columns called "attributes" or "fields", which themselves contain
"values" pertaining only to the scenario file, whereas entities and attributes are predefined for the whole data model.

The relationships between entities are defined in the .jdl file(s) describing the data model. For more details, refer to Chapter Defining the Data Model.

The scenarios are processed by the application through the Scenario Service. For more details, refer to Chapter Understanding the Scenario Service.

Scenarios are associated to workspaces, over which users have permissions. For more details, refer to Chapter Setting Permissions.

From the web client, users can access the workspaces and scenarios over which they have permissions using specific menus. For more details, refer to Sections Understanding the Sidenav Workspace Selector and Understanding the Scenario Selector.

Users can also display more details on the workspaces and scenarios over which they have permissions using dedicated widgets. For more details, refer to Sections Using the Workspace List Widget and Using the Scenario List Widget.


Scenario links can be created from the Scenario list widget using the option Create link. Creating a scenario link to a scenario adds said scenario to a workspace accessible to users that cannot access the current Workspace. Editing a scenario in a workspace also impacts its corresponding scenario links, even when they are in other workspaces. Deleting a scenario link does not delete the linked scenario. However, deleting a scenario does delete all related links. In a composite data model application, when adding a scenario that references other scenarios, it is possible to select a scenario link or a normal scenario. Once the scenario is added, a permission rule is also created to make the referenced scenario accessible to everyone. This is required, in case it is moved into another workspace for instance, to prevent losing ACCESS to the new scenario which, itself, also requires ACCESS to the referenced scenario. For more details, refer to Section Understanding Operations.

Scenario links can be created from the Scenario list widget using the option Create link.

Creating a scenario link to a scenario adds said scenario to a workspace accessible to users that cannot access the current Workspace.
Editing a scenario in a workspace also impacts its corresponding scenario links, even when they are in other workspaces.
Deleting a scenario link does not delete the linked scenario. However, deleting a scenario does delete all related links.
In a composite data model application, when adding a scenario that references other scenarios, it is possible to select a scenario link or a normal scenario. Once the scenario is added, a permission rule is also created to make the referenced scenario accessible to everyone. This is required, in case it is moved into another workspace for instance, to prevent losing ACCESS to the new scenario which, itself, also requires ACCESS to the referenced scenario. For more details, refer to Section Understanding Operations.

Finally, users can edit the content of a scenario either by:

Exporting a scenario or a scenario template in Excel format, manually editing it and reimporting it as a new scenario. For more details, refer to the sections in this chapter.
Using the Data Grid and Data Explorer widgets. For more details, refer to Sections Using the Data Grid Widget and Using the Data Explorer Widget.
Using tasks and jobs. For more details, refer to Chapter Understanding the Execution Service.
Implementing specific ways to import data to the application. For more details, refer to Chapter Customizing Data Integration.

1. Understanding the Scenario File Format

Scenario files should respect the data types specified in the data model for each entity and entity attributes. Each cell is evaluated at import and formatted upon export based on this definition. For more details, refer to Section Understanding Data Types.

A scenario can be stored in the following formats:

.xlsx file: This is the classic scenario import format of DOC. The Excel file has to be composed of one sheet per entity. Any file with a .xlsx extension is processed as an Excel file.
.dbrf file: This format is optimized for DOC. It is recommended to use the DBRF export and import as often as possible to reduce performance issues when processing large files. Any file with a .dbrf or a .gz extension is processed as a DBRF file.
.zip archive containing .csv files: This is the internal format of DOC. Files in this format are supposed to be used as is. This format is thus undocumented. Any file with a .zip extension is processed as an archive of .csv files (internal format).

When importing a file, the application tries to match entities and entity attributes with the existing table names and column headers:

First, by looking for an exact match;
Then, using case-insensitive matching;
Finally, by removing any white spaces, dashes or underscores.

For example, the following table names are possible:

Entity	Allowed names
`SolutionSummary`	SolutionSummary, Solution Summary, SOLUTION_SUMMARY, solution-summary, SoLuTioN SuMMaRy,...
`numberOfHours`	numberOfHours, Number of Hours, NUMBER_OF_HOURS, number-of-hours, ...
`country.id`	country.id, COUNTRY_ID, Country ID, country-id, ...

Within each sheet/entity/table, the first row has one entry per attribute and relation of the entity in the JDL model. The only required columns are the attributes and relations that are part of the business key of the entity, while the other columns may be omitted.

Each row after the first one represents a given instance of the entity. For each column, data should be provided in a format accepted for the corresponding data type.


Note that: For number types (Integer, Long, Double), the cell has to be parsable as a double. In other words, any values matching the following regular expression are accepted: `[0-9]+[.,]?[0-9]*`. If the attribute has type Integer or Long, the decimal part will be discarded on import. Integer and Long attributes are exported as `[0-9]+`. For the Boolean type, the following values are accepted: `true`, `t` or `1` for true and `false`, `f`, `0` for false. For the String and Binary types, any content is accepted. For date and time types, Excel cells formatted as a Date are supported. Alternatively columns can be formatted as strings following the ISO-8601 format, e.g. "2007-12-03T10:15:30Z", "2007-12-03T10:15:30", "2007-12-03", "10:15:30". For duration types: Duration can be an Excel date, representing a Duration as the interval between day 1899-12-31 0:00 and the cell value. Duration can be a string with the following format 150:20:10.123 where 150 are hours, 20 are minutes, 10 are seconds, and 123 are milliseconds. (Seconds and milliseconds are optional.) ISO-8601 format can be used as well to express a duration.

Note that:

For number types (Integer, Long, Double), the cell has to be parsable as a double. In other words, any values matching the following regular expression are accepted: [0-9]+[.,]?[0-9]*. If the attribute has type Integer or Long, the decimal part will be discarded on import. Integer and Long attributes are exported as [0-9]+.
For the Boolean type, the following values are accepted: true, t or 1 for true and false, f, 0 for false.
For the String and Binary types, any content is accepted.
For date and time types, Excel cells formatted as a Date are supported. Alternatively columns can be formatted as strings following the ISO-8601 format, e.g. "2007-12-03T10:15:30Z", "2007-12-03T10:15:30", "2007-12-03", "10:15:30".
For duration types:
- Duration can be an Excel date, representing a Duration as the interval between day 1899-12-31 0:00 and the cell value.
- Duration can be a string with the following format 150:20:10.123 where 150 are hours, 20 are minutes, 10 are seconds, and 123 are milliseconds. (Seconds and milliseconds are optional.)
- ISO-8601 format can be used as well to express a duration.

Here is a simple example.

entity Country {
  id String required
}
entity Plant {
  plantId String required
}
relationship ManyToOne {
  Plant{country} to Country{plants}
}

If the application is defined based on the JDL file above, an exported Excel template would have two sheets, one per table/entity:

A first one, named Country, with one column (id);
A second one, named Plant, with two columns (plantId and country.id).

For more details, refer to Section Using Scenario Data Templates.

2. Using Scenario Data Templates

To add data to the application through a scenario, the file must have a precise structure in terms of entity and attribute names.

Users can download an empty template of the data model structure using the Topbar Settings menu. For more details, refer to Section Understanding the Topbar Settings Menu.

Figure 6.1. Exporting the Scenario Data Tables to an Excel Template

The template is an Excel file that contains a sheet for each entity defined in the application. The first row of a sheet indicates the entity attribute names.

If the application is based on a composite data model, even an empty template can be reimported as such or such scenario type and displayed using the Data Explorer widget to spot which entities are related to the displayed scenario type. For more details, refer to Section Using the Data Explorer Widget.

3. Adding Scenarios and Folders

Scenarios can be:

Imported and exported using Docker commands to restore the Data Service PostgreSQL data volumes, and therefore, all the scenarios in the application database. For more details, refer to the official Docker documentation.


Restoring a Docker volume overwrites the database, i.e. the new elements are not added to the ones in the application but they replace them. One must have access to the corresponding files as, by default, the information about Docker containers and volumes is located in the `db-gene/deployment` folder. It is key that the backup/restore of both the Data and Scenario Services happens at the same time to ensure that the content databases remain synchronized, even though the two Docker volumes are processed separately.

Restoring a Docker volume overwrites the database, i.e. the new elements are not added to the ones in the application but they replace them.
One must have access to the corresponding files as, by default, the information about Docker containers and volumes is located in the db-gene/deployment folder.
It is key that the backup/restore of both the Data and Scenario Services happens at the same time to ensure that the content databases remain synchronized, even though the two Docker volumes are processed separately.

Added, imported and exported using the web client.

Procedure 6.1. To Add Scenarios and Folders from the Web Client

Connect to the web client as a user with sufficient permissions to proceed. For more details, refer to Section Accessing the Web Client and Part Securing.
Depending on the need:
- From the Topbar Tasks option Import from File. For more details, refer to Section Understanding the Topbar Tasks Menu;
- From the Scenario List widget toolbar option New Scenario. From the widget toolbar, users can also add folders using the option New Folder. Scenarios can then be dragged and dropped into the folders to be organized as needed. For more details, refer to Section Using the Scenario List Widget; and
- From the Scenario List widget, users can add folders using the option New Folder in the widget toolbar. Scenarios can then be dragged and dropped into the folders to be organized as needed.
- From the Scenario Selector option New Scenario. For more details, refer to Section Understanding the Scenario Selector;
A dedicated dialog allows importing a file to generate a single scenario for a simple data model or as many scenarios as needed for a composite data model. For more details, refer to Chapter Defining the Data Model.
Figure 6.2. Importing a Scenario in a Simple Data Model Application
When adding a scenario in a composite data model application, the dialog allows selecting the type of scenario, its name and potential referenced scenarios. Not selecting a file adds an empty scenario with the right entities.
An option also allows to Create a new scenario from the same file instead of selecting an existing one.
In a composite data model application, when adding a scenario that references other scenarios, it is possible to select a scenario link or a normal scenario. Once the scenario is added, a permission rule is also created to make the referenced scenario accessible to everyone. This is required, in case it is moved into another workspace for instance, to prevent losing ACCESS to the new scenario which, itself, also requires ACCESS to the referenced scenario. For more details, refer to Section Understanding Operations.
Figure 6.3. Importing a Scenario in a Composite Data Model Application
Indicate the required information and click on IMPORT. The scenario is added to the application database.

4. Deleting Scenarios and Folders

Scenarios and folders can be deleted from the database individually from the web client.

Procedure 6.2. To Delete Scenarios and Folders from the Web Client

Connect to the web client as a user with sufficient permissions to proceed. For more details, refer to Section Accessing the Web Client and Part Securing.
In the Scenario List widget, select the elements to delete and click on the toolbar option Move to Trash. For more details, refer to Section Using the Scenario List Widget. A dialog opens.
When using a composite data model, deleting a scenario can have consequences on other scenarios. For more details, refer to Section Defining a Composite Data Model (CDM).
- When deleting a CDM scenario referenced by other scenarios, they are displayed to select whether or not to proceed with the deletion.
  Figure 6.4. Deleting a Referenced Scenario
- In case any other scenarios or folders, that can be deleted inconsequentially, are present and selected for deletion at the same time, they will be mentioned both in the total count and the deletion button. These elements do not have to be selected and will be deleted automatically when clicking the Delete button, whether or not any items were selected from the table.
  Figure 6.5. Deleting Referenced and Unreferenced Scenarios
- If none of the items selected for permanent deletion are referenced, a simple confirmation popup is displayed.
  Figure 6.6. Deleting a Scenario
Indicate the required information and click on Import. The deleted scenarios and folders are placed in the Trash folder. From there, each element can be restored or permanently deleted using the dedicated button.
Figure 6.7. Understanding the Trash Folder in the Scenario List Widget
The elements in the Trash folder can be permanently deleted from the database at once using the option Empty Trash, from the widget toolbar menu .
Figure 6.8. Using the Empty Trash Option

5. Exporting Scenarios

Scenarios can be exported:

Using Docker commands to back up the PostgreSQL data volumes, and therefore, all the scenarios in the application database. For more details, refer to the official Docker documentation.


Restoring a Docker volume overwrites the database, i.e. the new elements are not added to the ones in the application but they replace them. One must have access to the corresponding files as, by default, the information about Docker containers and volumes is located in the `db-gene/deployment` folder. It is key that the backup/restore of both the Data and Scenario Services happens at the same time to ensure that the content databases remain synchronized, even though the two Docker volumes are processed separately.

Restoring a Docker volume overwrites the database, i.e. the new elements are not added to the ones in the application but they replace them.
One must have access to the corresponding files as, by default, the information about Docker containers and volumes is located in the db-gene/deployment folder.
It is key that the backup/restore of both the Data and Scenario Services happens at the same time to ensure that the content databases remain synchronized, even though the two Docker volumes are processed separately.

Using the web client.

Procedure 6.3. To Export a Scenario from the Web Client

Connect to the web client as a user with sufficient permissions to proceed. For more details, refer to Section Accessing the Web Client and Part Securing.
To export:
- The scenario currently present in the Scenario Selector, in the Topbar Menu Tasks, click on Jobs > Export Data > To Excel File or To DBRF File, depending on the need. A dialog opens.
- A scenario using a task, in the Topbar Menu Tasks, click on Jobs > Run New Job. A dialog opens.
Figure 6.9. Setting an Export Job
Indicate the required information and click on RUN. The files are exported and downloaded from the browser.