Chapter 11. Understanding the Data Service
![[Note]](images/note.svg) | |
Note that:
|
DOC stores scenario information through two different services, each one using its own database:
The Scenario Service relies on a MongoDB database to store scenario specific metadata and properties, the workspaces and folders hierarchy and application configuration settings. For more details, refer to Chapter Understanding the Scenario Service.
The Data Service relies on a relational database to store the actual data associated with each scenario. It can import, export scenarios in various formats, such as Excel ftemplates or DOM snapshots (
dbrf). For more details, refer to Chapter Managing the Application Data.
The Data Service allows to retrieve or edit subsets of scenario data and can serve paginated access on specified entities. It also allows comparing values between scenarios.
The data is stored in a PostgreSQL database which is run in its own Docker container. Additional relational databases will be supported in the future.
The default behavior of the database is to split the data by scenario using Table Partitioning. This guarantees a high level of performance when having a lot of scenarios in the application. Under some special cases, and with some specific PostgreSQL configurations, this feature can be disabled using the following application property in the data-service-extension module. This setting CANNOT be changed once a database has already been initialized.
spring.datasource.data-partitioning=false
1. Understanding the Data Service Checkers
The relational schema of the database does not implement the constraints as defined through the JDL model such as uniqueness, nullablility, max-length or bounds. This allows you to import data with inconsistencies and to clean them afterward within DOC.
To help with this process, DOC Data Service provides built-in data checkers that are automatically triggered upon any data modifications (e.g. importing a scenario, editing data), and that report all inconsistencies found in a Scenario based on the JDL model definitions.
Built-in data checkers can also be triggered manually from the Web client or through the service APIs. For more details, refer to Chapter Understanding the APIs.
The detected issues with data are stored in the GeneSchemaIssue collection and can be visualized in the Web client, either using the Data Grid component or using the Issue List component.
After running built-in data checkers, the scenario data status is updated to a value among VALID, WARNING or ERROR depending on the highest issue severity reported in both GeneSchemaIssue and GeneIssue collection. For more details, refer to Chapter Understanding the Execution Service.
Note: GeneSchemaIssue collection is reserved for DOC built-in data checkers, and should never be modified by project code. Each time the data checkers are run, the content of this collection is automatically replaced with the newly detected issues.
2. Understanding the Data Service Events Hooks
When a bean implements ScenarioDataEventsHandler (Javadoc), DOC framework will automatically call its methods when schema checkers are executed or when scenario data are modified using the GeneTransactionalQuery API (i.e. user data modifications in the web client)
This interface defines two methods onDataUpdated, and onSchemaCheckCompleted.
onDataUpdatedwill be called every time scenario data are modified by a GeneTransactionalQuery and will providethe modified scenario ID
the id of the transactional query that modified the data
the
GeneTransactionalQueryquery object
onSchemaCheckCompletedwill be called every time the data checkers have run and will providethe scenario ID
the new
ScenarioDataStatus
3. Understanding the Data Service Scenario Comparison
The Data Service allows allows comparing values between scenarios through:
A dedicated feature in the web client. For more details, refer to Section Understanding Scenario Comparison.
The Data Service GraphQL API. For more details, refer to Section Understanding GraphQL Scenario Comparison.
4. Understanding the Data Service Logs
Data Service can be configured to log data changes performed by users. There are two kinds of logs that can be activated.
4.1. Understanding the System Logs
The Data Service, similarly the other microservices outputs logs to different destinations depending on its log configuration. By enabling this property, a log with Info level will be emitted on every data modification.
To activate these system logs the following property needs to be set to true in the data-service-extension configuration file (namely application.yml).
Setting the enable property to true, will activate the logs.
Setting the verbose property to true, will also log the modification details. This property has no effect if enabled is false
Setting the anonymous property to true, will log the modifications without user details. This property has no effect if enable is false. Note that setting this property to false will perform some nominative (username, user-id) records they may require some legal agreements.
4.2. Understanding the Scenario Events Logs
Activating these logs will allow record Scenario Events that can be displayed through the Scenario Timeline widget. The recorded Scenario Event will store for each modification the query used, the author and a summary of modified tables and rows.
services:
data:
log-changes:
scenario-events:
enabled: true # When set to true a scenario event is stored on every time a user perform scenario data modifications
anonymous: true # When set to true the scenario event will not record details of the user who perform the changes5. Understanding the Data Service Conflicts
When performing a GeneTransactionalQuery, the optional parameter evaluateConflits activates conflict detection, and when a conflict between values is detected, the query changes are not persisted, the query returns a result containing the exhaustive list of conflicts.
To describe the conflict detection mechanism we have to define the different types of values field involved in a transactional query:
Remote Value: The remote value is the current value stored in the database, used as reference for conflict detection.
Local Value: The local value represents from the API consumer perspective, the current database value for a given field of an entity. The local value is considered outdated as soon as it is different from the Remote Value. Values conflict detection is only applied to provided field local value, so make sure to provide in the query all fields local value you want to evaluate for conflicts.
New Value: When performing a GeneTransactionalQuery, the new value is the value we want to set as the new remote value. As soon as the GeneTransactionalQuery is committed the New Value becomes the new Remote Value.
To enable conflict detection, the GeneTransactionalQuery has to provide local values of fields it is trying to update, which means that most of the conflict detections will only work when the local values are provided for conflict evaluation.
Depending on the changes that can be processed concurrently on any data, the type of conflict can be one of the following types:
FIELD_VALUE_CONFLICT: Field value conflict indicates that we are trying to update a field of an entity (table column) from a value that has changed to a new value.
UPDATING_DELETED_ROW: The GeneTransactionalQuery is trying to update one or more fields of an entity which has been deleted. This evaluation is based on the entity internal id.
DELETING_MODIFIED_ROW: The GeneTransactionalQuery is requesting to delete an entity (row) that has been modified.
When performing a GeneTransactionalQuery with conflicts evaluation through the web client, a Conflict Editor will automatically show up when a query triggers conflicts. The user will be able to resolve the conflicts and persist changes.
The Conflict editor shows conflicts by tables and allows the user to resolve conflicts by Keeping or Discarding changes. The following table describes the possible and default resolutions.
 |
| Conflict Type | Available Resolutions | Default | Comment |
|---|---|---|---|
| FIELD_VALUE_CONFLICT | Keep/Discard | Keep | ... |
| DELETE_MODIFIED_ROW_CONFLICT | Keep/Discard | Keep | ... |
| UPDATE_DELETED_ROW_CONFLICT | Discard | Discard | Query may not contain enough data to resolve by keeping changes |
5.1. Understanding a FIELD_VALUE_CONFLICT
Here is an example of two queries triggering a FIELD_VALUE_CONFLICT:
 |
User 1 query:
{
"creates" : [],
"updates" : [{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"fieldValues" : [
{
"fieldName" : "age",
"newValue" : "41",
"localValue": "40"
}
]
}],
"deletes" : []
}User 2 query:
{
"creates" : [],
"updates" : [{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"fieldValues" : [
{
"fieldName" : "age",
"newValue": "45",
"localValue": "40"
}
]
}],
"deletes" : []
}User 2 query response:
{
"created" : [],
"updated" : 0,
"deleted": 0,
"conflicts": [
{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"conflictType": "FIELD_VALUE_CONFLICT",
"fieldValues": [
{
"fieldName" : "age",
"newValue" : "45",
"localValue": "40",
"remoteValue": "41"
}
]
}
]
}5.2. Understanding an UPDATING_DELETED_ROW Conflict
 |
User 2 query:
{
"creates" : [],
"updates" : [],
"deletes" : [
{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"fieldValues": []
}
]
}User 1 query:
{
"creates" : [],
"updates" : [{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"fieldValues" : [
{
"fieldName" : "age",
"newValue" : "41",
"localValue": "40"
}
]
}],
"deletes" : []
}User 1 query response:
{
"created" : [],
"updated" : 0,
"deleted": 0,
"conflicts": [
{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"conflictType": "UPDATING_DELETED_ROW",
"fieldValues": [
{
"fieldName" : "age",
"newValue" : "45",
"localValue": "40"
}
]
}
]
}5.3. Understanding a DELETING_MODIFIED_ROW Conflict
 |
User 1 query:
{
"creates" : [],
"updates" : [{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"fieldValues" : [
{
"fieldName" : "age",
"newValue" : "41",
"localValue": "40"
}
]
}],
"deletes" : []
}User 2 query:
{
"creates" : [],
"updates" : [],
"deletes" : [
{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"fieldValues": [
{
"fieldName" : "age",
"localValue": "40"
}
]
}
]
}User 2 query response:
{
"created" : [],
"updated" : 0,
"deleted": 0,
"conflicts": [
{
"entityType" : "Employee",
"dbGeneInternalId": "john_smith_id",
"conflictType": "DELETING_MODIFIED_ROW",
"fieldValues": [
{
"fieldName" : "age",
"localValue": "40",
"remoteValue": "41"
}
]
}
]
}