Chapter 13. Understanding the Execution Service


	Note that: `DOC` features can be extended further depending on the project requirements. For more details, refer to Chapter Extending the Application Features. Access to this service requires authentication. For more details, refer to Chapter Managing Users.

The Execution Service receives all the requests pertaining to the execution of "tasks" and processes them through "jobs".

1. Understanding Tasks and Jobs

Tasks are the means by which one can give users access to the business logic. They differ from actions defined using the Action API which are constrained to the web client library. For more details, refer to Section Using the Action API.

From the web client, tasks declared to the Execution Service can be launched as jobs:

Using the Run New Job option in the Topbar Tasks menu. For more details, refer to Section Understanding the Topbar Tasks Menu.
Using the Button widget, by setting the action Run New Job. For more details, refer to Section Using the Button Widget.

It is possible to:

Display information on jobs from the web client. For more details, refer to Sections Using the Job List Widget and Using the Job Details Widget.
Limit parallel jobs and clean old ones. For more details, refer to Section Extending the Execution Service.

Some tasks are shipped with DOC. For more details, refer to Section Task Samples.

One can declare a task via a Spring bean in a configuration class that is scanned when the Execution Service starts. A typical way to achieve that is to add a bean to an existing or new configuration class in the extensions/execution-service-extension library.

In a newly generated application, the library contains such a configuration class named Tasks. This class declares four Spring beans as examples.

One can delete any of these methods from the class to remove the corresponding tasks from the application.

To add tasks to the application, new methods can be added to the class. They must:

be public;
be annotated with @Bean;
take no argument;
return an instance of ScriptedTaskDescription.

One of the sample beans, excelExport, declares a task for the Excel file import as follows:

@Configuration
public class Tasks {
    @Bean
    public ScriptedTaskDescription excelExport() {
        ScriptedTaskDescription task = new ScriptedTaskDescription("ExcelExportTask", "Export scenario to Excel");
        task.setDescription("Exports the selected scenario to an Excel file");

        VariableAccessExpression scenario = VariableAccessExpression.of("scenario");
        VariableAccessExpression baseFileName = VariableAccessExpression.of("base file name");
        VariableAccessExpression filter = VariableAccessExpression.of("filter");

        task.getScript()
            .addStatement(AskInputStatement.of(scenario.getVariableName(), true, JobInputType.SCENARIO_ID))
            .addStatement(AskInputStatement.of(baseFileName.getVariableName(), true, JobInputType.TEXT, "Name of the file to export to. Warning: a '.xlsx' extension will be appended."))
            .addStatement(AskInputStatement.of(filter.getVariableName(), false, JobInputType.ENTITIES, "Select the tables to export"))
            .addStatement(SetTaskOutputStatement.of("Excel file",
                FileExpression.of(
                    StringExpression.concat(baseFileName, StringExpression.of(".xlsx")),
                    BlobExpression.of(
                        ScenarioDataExpression.of(scenario)
                            .withFormat(ScenarioDataFormat.EXCEL)
                            .onlyTables(filter)),
                    StringExpression.of("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
                )));

        return task;
    }
    // ...
}

As illustrated in the example above, the logic of a task is described in its script, a collection of statements that handle expressions. An expression evaluates to a value and may, or may not, have a side effect. A statement executes, typically has a side effect, and terminates with an exit code.

Within the task, data can be processed depending on their type. For more details, refer to Section Using Core Expressions.

The task script is accessed with task.getScript() and statements are added to it with addStatement(). For more details, refer to Sections Understanding Flow Control Statements and Understanding Scenario-Related Statements

A specific statement indicates to the web client that a value should be provided by the user. For more details, refer to Sections Understanding Job Memory Statements and Understanding Job Input Statements.

A task can also be equipped with an exit script that is executed after the main script, whether the job is successful or not. The exit script is accessed with task.getExitScript() and statements are added to it with addStatement(), as for the main script.

The exit script inherits data from the job memory, with a few additional variables:

com.decisionbrain.gene.jobStatus contains the string representation of the job status at the end of the main script;
com.decisionbrain.gene.errorMessage contains null if the main script completed successfully, the message of the exception if it failed, and the alerting message if it was interrupted by a call to ExitTaskStatement.alerting().
com.decisionbrain.gene.latestStepLabel contains null if the main script completed successfully, otherwise the label of the step where the execution of the main script was interrupted. The label is the string that is displayed in the Job Details widget (please refer to Section Using the Job Details Widget).

For more details, refer to Section Understanding Job Termination Statements.

The Statement class is documented in DOC JavaDoc.

2. Understanding Core Expressions

Tasks scripts, use core expressions to handle data. For example, the following task takes two numeric arguments as input, and returns the addition of them as a result.

@Configuration
public class Tasks {
    @Bean
    public ScriptedTaskDescription additionTask() {
        ScriptedTaskDescription task = new ScriptedTaskDescription("add", "Addition");
        task.setDescription("Performs an addition");
        task.getScript()
            .addStatement(AskInputStatement.of("operand 1", true, JobInputType.REAL))
            .addStatement(AskInputStatement.of("operand 2", true, JobInputType.REAL))
            .addStatement(SetVariableStatement.of("sum", NumericExpression.plus(VariableAccessExpression.of("operand 1"), VariableAccessExpression.of("operand 2"))))
            .addStatement(SetTaskOutputStatement.of("result", VariableAccessExpression.of("sum")))
            .addStatement(LogStatement.info(StringExpression.concat(
                VariableAccessExpression.of("operand 1"),
                StringExpression.of(" + "),
                VariableAccessExpression.of("operand 2"),
                StringExpression.of(" = "),
                VariableAccessExpression.of("sum")
            )));
        return task;
    }
}

The generated web client component for this task is the following:

Figure 13.1. Configuring a Custom Addition Task

The following data types are compatible with DOC:

Boolean constants can be returned using the BooleanExpression.TRUE and BooleanExpression.FALSE expressions.
- The BooleanExpression class also provides the not(Expression), and(Expression...), and or(Expression...) methods that create Boolean expressions evaluating to the corresponding logical operations on the values of their operands, which must evaluate to Boolean values.
- The BooleanExpression.eq(Expression...) method creates an expression that, when evaluated, evaluates all provided expressions and returns true if, and only if, they all evaluate to the same value. Values are compared with Objects.equals(). Evaluation of the provided expressions stops as soon as the result can be determined. The BooleanExpression.neq(Expression...) method is the negation of the previous.
- The BooleanExpression.isNull(Expression) method tests whether the provided expression evaluates to the null value. The BooleanExpression.isNotNull(Expression) method is its negation.
String constants can be returned using the StringExpression.of(String constant) method.
- This method is heavily used in scripts, to provide constant string values to methods that expect an expression. The StringExpression.of(Expression expression) method creates an expression that, when evaluated, returns the string representation of the provided expression. This method is similar to Java Object.toString() and is automatically called on the arguments of LogStatement.info() and of StringExpression.concat().
- The StringExpression.concat(Expression... expressions) method creates an expression that, when evaluated, returns the concatenation of the values of the provided expressions, or their string representations if they do not evaluate to strings.
- The StringExpression.isEmpty(Expression) method creates an expression that evaluates to a Boolean value. It expects an expression evaluating to a string. The returned expression evaluates to true if the provided expression evaluates to an empty string. If the provided expression evaluates to the null value, the returned expression evaluates to false. The StringExpression.isEmptyOrNull(Expression) method creates an expression that evaluates to true if the provided expression evaluates to an empty string or to the null value.
- The StringExpression class also provides the following methods, whose semantics is the same as the equivalent Java methods of the String class: startsWith(Expression, Expression), endsWith(Expression, Expression), contains(Expression, Expression), matches(Expression, Expression), lengthOf(Expression), substringOf(Expression, Expression [, Expression]), indexOf(Expression, Expression), lastIndexOf(Expression, Expression), toLowerCase(Expression), toUpperCase(Expression).
Number constants can be returned using the NumericExpression class through the of(Number) method.
- The of(Expression) method parses string expressions for numeric values. Finally, the plus(Expression...), minus(Expression), minus(Expression, Expression), times(Expression...), divide(Expression, Expression), abs(Expression), max(Expression...), and min(Expression...) methods provide the eponymous operations.
- In addition, the class provides the lt(Expression, Expression), le(Expression, Expression), ge(Expression, Expression), and gt(Expression, Expression) methods, which compare the (expectedly numeric) values of the provided expressions. The expression created by close(Expression expression1, Expression expression2, Expression maxGap) evaluates to true if the absolute value of the difference between the values of expression1 and expression2 is strictly less than the value of maxGap.
Temporal constants can be returned using the TemporalExpression class which provides a significant number of methods to create expressions that evaluate to one of the Instant, ZonedDateTime, LocalDateTime, LocalDate, or LocalTime class of the java.time package. All these methods are close replicas of the methods on the respective Java types.
- In addition, the class provides the isBefore(Expression, Expression) and isAfter(Expression, Expression) methods, which compare the values of the provided expressions. The durationInMilliseconds(Expression, Expression) method returns an expression that evaluates to the duration in milliseconds between the provided expressions. The expressions must evaluate to comparable temporal value, in the sense that they either are both of the same Java type, or they can be converted to the other type, which boils down to one being an Instant and the other a ZonedDateTime.
- The epochSecondOf(Expression) and epochMilliOf(Expression) methods return expressions that evaluate to the number of seconds, resp. milliseconds, for the provided expressions, which must evaluate to an Instant or a ZonedDateTime.
Blob (Opaque Objects) constants can be returned using the BlobExpression.of(byte[] bytes) method which creates an expression that, when evaluated, returns the array of bytes provided. This is typically useful for passing arbitrary and opaque data structures between routines or worker tasks.
- The BlobExpression.of(Expression) method creates an expression that, when evaluated, returns the representation of the value or content of the provided expression as an array of bytes. For Boolean and numeric values, this will be the bytes of the string representation of the value. For string values, this will be its bytes. For file values, this will be the file content. For scenario data values, this will be the raw content of the scenario data. For list of strings, this will be the bytes of the comma-separated concatenation of the strings. Other kinds of lists are not supported.
- The BlobExpression.sizeOf(Expression) method creates an expression that, when evaluated, returns a number. It expects the provided expression to evaluate to an array of bytes. The returned expression then evaluates to the size, in bytes, of this array.
File constants can be returned using the FileExpression.of(String filename, byte[] fileContent [, String mimeType]) method which creates an expression that, when evaluated, returns a file value object that holds the provided byte array, associated with the provided file name, and optionally the provided MIME type.
- The FileExpression.of(Expression, Expression, Expression) method is analogous to the previous one, with its first and third arguments expected to evaluate to string values, and its second to an array of bytes. The third argument may also be passed null.
- The FileExpression.filenameOf(Expression) method creates an expression that evaluates to a string. It expects the provided expression to evaluate to a file value object. The returned expression then evaluates to the file name of this file value object. Similarly, the FileExpression.sizeOf(Expression) and FileExpression.mimeTypeOf(Expression) methods create expressions that evaluate to, respectively, the size (in bytes) of the content and the MIME type of the file value object that the provided expression is expected to evaluate to.

List constants can be returned using the ListExpression.of(Expression...) method which creates an expression that, when evaluated, returns the collection of the values to which the provided expressions evaluate. There must be at least one provided expression, and they must all evaluate to the same type. The ListExpression.empty(Type) method creates an expression that evaluates to an empty list, typed as specified.

The ListExpression.sizeOf(Expression) method creates an expression that evaluates to the number of elements in the list that the provided expression evaluates to.

The ListExpression.elementAt(Expression list, Expression index) method creates an expression that evaluates the expression list and return the element at the index provided by the expression index.


	Note that, following best practice for performances, it is preferable to use `elementAt` on an already created/evaluated list. Also, lists like `ListExpression.of(ScenarioDataExpression.of(...)` are created and load data on each call of `elementAt`.

3. Understanding Flow Control Statements

Tasks use general statements and expressions to handle specific data flows.

The IfStatement.of(Expression test, Statement thenBranch, Statement elseBranch) method creates a statement that, when executed, first evaluates the test expression. This expression must evaluate to a Boolean value. If it evaluates to true, then the thenBranch statement is executed; otherwise, the elseBranch statement is executed. The IfStatement.of(Expression test, Statement thenBranch) method is a variant for an empty "else" branch.

For example, the following sample task takes a number as input, then uses IfStatement to:

log a different message when the number is odd or even.
set the task output value to " odd" or "even".

@Configuration
public class Tasks {
    @Bean
    public ScriptedTaskDescription evenOrOddTask() {
        ScriptedTaskDescription task = new ScriptedTaskDescription("evenOrOddTask", "Is that number even or odd?");
        VariableAccessExpression n = VariableAccessExpression.of("A number");
        NumericExpression zero = NumericExpression.ZERO;
        NumericExpression two = NumericExpression.of(2);
        task.getScript()
            .addStatement(AskInputStatement.of(n.getVariableName(), true, JobInputType.INTEGER))
            .addStatement(LogStatement.info(StringExpression.concat(StringExpression.of("Is "), n, StringExpression.of(" even or odd?"))))
            .addStatement(IfStatement.of(BooleanExpression.eq(NumericExpression.minus(n, NumericExpression.times(NumericExpression.divide(n, two), two)), zero),
                Block.of(
                    LogStatement.info(StringExpression.concat(n, StringExpression.of(" is even"))),
                    SetTaskOutputStatement.of("Even or odd?", StringExpression.of("even"))
                ),
                Block.of(
                    LogStatement.info(StringExpression.concat(n, StringExpression.of(" is odd"))),
                    SetTaskOutputStatement.of("Even or odd?", StringExpression.of("odd"))
                )));
        return task;
    }
}

The generated web client component for this task is the following:

Figure 13.2. Configuring a Custom EvenOrOdd Task

The RepeatStatement.of(String loopVariableName, Expression nbLoops, Statement... statements) method creates a statement that, when executed, first evaluates the nbLoops expression. This expression must evaluate to a number. Then the statements are executed as many times. If a variable name is provided as loopVariableName, a variable is set with this name to an increasing value on each loop execution, starting with one. If you do not need a loop index variable, you can pass null as the first argument to the of method.

For example, the following task takes a number as input, then uses RepeatStatement to compute its factorial; finally, it sets the task output to the result.

@Configuration
public class Tasks {
    @Bean
    public ScriptedTaskDescription factorialTask() {
        ScriptedTaskDescription task = new ScriptedTaskDescription("FactTask", "Factorial");
        VariableAccessExpression n = VariableAccessExpression.of("n");
        VariableAccessExpression fact = VariableAccessExpression.of("n!");
        VariableAccessExpression index = VariableAccessExpression.of("index");

        task.getScript()
            .addStatement(AskInputStatement.of(n.getVariableName(), true, JobInputType.INTEGER, "The number to compute the factorial of"))
            .addStatement(SetVariableStatement.of(fact.getVariableName(), NumericExpression.ONE))
            .addStatement(RepeatStatement.of(index.getVariableName(), n,
                SetVariableStatement.of(fact.getVariableName(), NumericExpression.times(fact, index))))
            .addStatement(SetTaskOutputStatement.of("result", fact));
        return task;
    }
}

Here is an example of job computing 4! = 24.

Figure 13.3. Configuring a Custom 4! = 24 Task

The ForeachStatement.of(String elementVariableName, Expression elementsExpression, Statement... statements) method creates a statement that, when executed, first evaluates the elementsExpression expression. This expression must evaluate to a homogeneous list of values (they must all have the same type). Then for each element in that list, the statements are executed. If elementVariableName is not null, a variable with this name is set to the current list element during the execution of the statements. The breakOnError(boolean) method can be applied to a for-each statement to set the behavior of the loop if one of the statements in its body terminates with a non-zero exit code. When called with true, the loop will exit on the first (top-level) statement that exits/terminates with a non-zero exit code. When called with false (the default), exit codes do not influence the loop. In all cases, the exit code of the loop is the one of the last executed statement.

Anywhere a statement is expected, and typically as the "then" or "else" branch of an "if" statement, a block can be provided using the Block.of(Statement...) method.


	Note that variables have a global scope, even if first set in a block.

If the execution of a statement terminates normally, i.e. not interrupted by an exception, the statement has an exit code, which is an integer (a Java int) . As per Unix convention, "zero" means "everything is normal".

Other exit code values must be defined by each statement. In addition:

The NoopStatement.ok() method creates a statement that, when executed, terminates immediately with exit code zero.
The NoopStatement.withExitCode(Expression) method creates a statement that, when executed, terminates immediately with the value of the provided expression as its exit code. It terminates with an exception if this expression does not evaluate to a number.

If a statement terminates normally, i.e. not interrupted by an exception, and with a non-zero exit code, the execution of the rest of the script is not impacted and the next statement is executed.

However, the script itself can test the exit code of the last executed statement to determine its behavior, using the following expressions:

NumericExpression.LAST_STATEMENT_EXIT_CODE is a numeric expression that, when evaluated, provides the value of the exit code of the last executed statement.
Statement.OK_EXIT_CODE is a numeric expression that, when evaluated, yields the exit code of " everything ok", that is, zero.
BooleanExpression.LAST_STATEMENT_EXIT_CODE_OK and BooleanExpression.LAST_STATEMENT_EXIT_CODE_NOT_OK are Boolean expressions that, when evaluated, return respectively "true" or "false" if the exit code of the last executed statement is, respectively, zero or non-zero.

If a statement throws an exception, specific termination statements are executed. For more details, refer to Section Understanding Job Termination Statements.

4. Understanding Scenario-Related Statements

Tasks can use specific statements and expressions to handle scenario data. They are to be used in conjunction with the other types statements.

For instance, scenario-related statements and expressions can be used when:

Using the ID of a workspace or scenario. For example, the execution of a job is typically performed in the context of a scenario as it is launched from the actions menu of a row in the Scenario List widget. In this case, workspace and scenario IDs are handled in scripts as string expressions which are typically passed to a job through the inputs SCENARIO_ID or WORKSPACE_ID. For more details, refer to Section Understanding Job Input Statements.
Checking the schema of the data model using the CheckSchemaStatement.forScenario statement. Note that this statement returns an exit code 1 when the check ends with errors.
Creating a scenario using the StringExpression.idOfNewScenario(Expression scenarioName, Expression workspaceId) method. It creates an expression that, when evaluated, adds a scenario with the specified name and returns its ID within the specified workspace.
The workspaceId argument may consist of a workspace ID and a folder ID separated with a comma.
The variant method idOfNewScenarioWithMetadata will read any existing scenario file metadata if it exists. Then the created scenario will include the same properties that are present in the metadata.
Editing the content of a scenario as the content of a scenario is typically represented with a collector, that is, a subclass of DbDomCollector that was generated with the application. In the script of a task, this is handled through scenario data expressions. A scenario data object contains the ID of a scenario and its content, which can be converted into a collector.
The ScenarioDataExpression.of(Expression scenarioId) method creates an expression that, when evaluated, returns a scenario data object for the scenario with the specified ID. The scenario data object will contain the data stored by the Data Service for the scenario. The onlyTables(Expression... tables) method can be applied to a scenario data expression to restrict the scenario content to the specified tables.
If the data of a scenario is modified, for example, by the execution of an ExecuteRoutineStatement or an ExecuteOptimizationServerTaskStatement, its modified content is saved to the Data Service at the end of this statement execution.
Checking the data status of a scenario. The data status of a scenario indicates results from running a set of checkers that verify that the scenario data is consistent with respect to the constraints included in the data model. These schema checkers are typically run each time a scenario is saved.
The ScenarioStatusExpression.dataStatusOf(Expression) method creates an expression that, when evaluated, provides the current data status of the scenario with the ID to which the provided expression evaluates. The ScenarioStatusExpression class additionally provides the constant string expressions UNCHECKED, VALID, WARNING, and ERROR that can be used in comparisons. It also provides the dataStatusIs(Expression scenarioId, Expression status), which compares the data status of a scenario with a value; both arguments must evaluate to strings.
Schema checkers are automatically run when a scenario is saved except instructed otherwise. Schema checkers can also be triggered manually, using the statement created by CheckScenarioStatement.forScenario(Expression scenarioId).
When the checkers are run, you can react to the changes using scenario data. For more details, refer to Chapter Understanding the Data Service.
Importing data into a scenario using the ImportScenarioStatement.of(Expression scenarioId, Expression file) method which creates a statement that, when executed, evaluates the scenarioId expression into a string and the file expression into an instance of FileValue. This imports the content of the file into the scenario with that ID.
The storingIssuesInto(String issuesVariableName) method can be applied to such a statement to provide the name of a variable in the job memory where to store a JSON file containing the issues raised during the import.
The thenCheckSchema(Expression booleanVariable) method can also be applied to this statement to activate/deactivate running the schema checkers on the scenario after importing the data. By default, the schema checkers are always triggered after data import.
When executed, the statement exits with a non-zero code if at least one error-level issue was raised during data import or schema checking.
Deleting a scenario using the DeleteScenarioStatement.of(Expression scenarioId) method which creates a statement that, when executed, evaluates the scenarioId expression into a string, and delete the scenario with that ID.
By default, the scenario is moved to the trash. The moveToTrash(Expression moveToTrash) method can also be applied to this statement. Its argument must evaluate to a Boolean. If it evaluates to false, the scenario is permanently deleted instead.
Locking scenarios. As explained in Chapter Understanding the Scenario Service, there are two kinds of scenario lock:
- System locks are put by DOC on a scenario. The lock is owned by a job. By default, a system lock is put when the job is started, on each of the scenario passed as job inputs, as well as all scenarios accessed or created later in the task script.
  The scenario data expression created by ScenarioDataExpression.of(Expression scenarioId) locks the scenario when it is evaluated, except if the notLocked() method is applied to the scenario data expression before it is evaluated.
  In the StringExpression.idOfNewScenario(Expression scenarioName, Expression workspaceId, Expression lockIt) variant, if the lockIt expression evaluates to false, then the new scenario is not locked.
  The SystemLockStatement.lock(Expression scenarioId) method creates a statement that, when executed, puts a system lock on the scenario with the specified ID.
  The SystemLockStatement.unlock(Expression scenarioId) method creates a statement that, when executed, removes the system lock from the scenario with the specified ID.
  These two operations will succeed only if the user who executes the job has "modify" permission on the scenario. Also, the scenario will usually already be system-locked by the job; in case it is not, it must not be locked by another user or another job.
- User locks are put by users. This can be replicated by the UserLockStatement.lock(Expression scenarioId) method which creates a statement that, when executed, puts a user lock on the scenario with the specified ID.
  The UserLockStatement.unlock(Expression scenarioId) method creates a statement that, when executed, removes the user lock from the scenario with the specified ID.
  These operations will succeed only if the user who executes the job has "modify" permission on the scenario. Also, the scenario will usually already be system-locked by the job; in case it is not, it must not be locked by another user or another job.
Using the properties of a scenario. A property has a name and a value, both of which are strings.
The ScenarioPropertyExpression.of(Expression scenarioId, Expression propertyName) method creates an expression that, when evaluated, returns the value of property with the specified name on the scenario with the specified ID.
The SetScenarioPropertyStatement.of(Expression scenarioId, Expression name, Expression value) method creates a statement that, when executed, sets the value of the property with the specified name on the scenario with the specified ID.
The RemoveScenarioPropertyStatement.of(Expression scenarioId, Expression propertyName) method creates a statement that, when executed, removes the property with the specified name on the scenario with the specified ID.
Using the Scenario Timeline which contains most system- and user-events related to a scenario: lock/unlock, import, comment, etc.
It is possible to use PublishMessageToTimelineStatement.publishToScenario(Expression scenarioId, Expression message) to add a comment, which content is specified in the second argument, to the timeline of the scenario with the specified ID.

5. Understanding User-Defined Routine Statements

Tasks can rely on the Backend Service to execute user-defined routines.

The ExecuteRoutineStatement.of(Expression routineName) method creates a statement that, when executed, executes the given routine on the Backend Service.

The routine name is provided as an expression that evaluates to a string, which is then matched against the name provided to the @Routine annotation on the Java class that implements the routine.

@Bean
public ScriptedTaskDescription myFirstTask() {
    ScriptedTaskDescription task = new ScriptedTaskDescription("Task 1", "My first task");

    task.initI18nNameAndDescription("TASK_1");

    VariableAccessExpression scenarioData = VariableAccessExpression.of("scenarioData");
    VariableAccessExpression textOutput = VariableAccessExpression.of("textOutput");

    task.getScript()
        .addStatement(AskInputStatement.of(scenarioData.getVariableName(), true, scenarioId(WRITABLE), "Select a scenario to edit"))
        .addStatement(ExecuteRoutineStatement
            .of(StringExpression.of("SampleEditScenario"))
            .withInput(ScenarioDataExpression.of(scenarioData))
            .withOutput("message", textOutput.getVariableName()))
        .addStatement(SetTaskOutputStatement.of("my_task_output", textOutput));

    return task;
}

A user-defined routine is a class of the Backend Service annotated with @Routine(...). Annotating the routine class with @Routine automatically makes it a Spring component, the bean name of this component is the same as the routine name.

Here is an example of routine:

import ...

@Routine
public class SampleEditScenario {

    private static final Logger LOGGER = LoggerFactory.getLogger(SampleEditScenario.class);
    private final Random random = new Random();

    // Set execution parameters
    public void execute(RoutineExecutionContext executionContext, ScenarioData scenarioData) throws RoutineExecutionException {
        LOGGER.info("Routine started by {} with job {}" context.getExecutingUserId(), context.getJobId());
        CapacityPlanning collector = executionContext.getCollector(scenarioData);

        populate(collector);

        // Save only the modified entities
        executionContext.markModified(scenarioData, collector, "Country", "Plant");

        // Set return values
        executionContext.emit("message", "Scenario edited!");

    }

    /**
     * Populates the collector with a new country, plant.
     * @param collector the collector to populate
     */
    private void populate(CapacityPlanning collector) {
        Country country = collector.createCountry();
        country.setId(String.valueOf(random.nextInt(1000)));
        country.setName("Country " +  random.nextInt(1000));

        Plant plant = collector.createPlant();
        plant.setPlantId("Plant " + random.nextInt(1000));
        plant.setCountry(country);

    }
}

Additional information can be given for the execution of the routine with the following methods.

withInput(Expression value) provides a value for the next argument of the execute() method. See below details on the signature of this method.
withOutput(String outputName, String variableName) will store the value of the specified routine output into the script memory under the specified variable name.

These methods are to be used for all types of routine inputs and outputs (within the supported types, see below). They will typically be repeated as many times as needed to address all the inputs and outputs of the routine.

This class must provide an execute() method, the signature of which is discussed below.

    // Set execution parameters
    public void execute(RoutineExecutionContext executionContext, ScenarioData scenarioData) throws RoutineExecutionException {
        LOGGER.info("Routine started by {} with job {}" context.getExecutingUserId(), context.getJobId());
        CapacityPlanning collector = executionContext.getCollector(scenarioData);

As a routine is implemented by a class in the Backend Service, this class must have an execute() method.

The first argument of this method must be the RoutineExecutionContext described below. The remaining arguments must match, in number and in type, the calls to the ExecuteRoutineStatement.withInput() method in the task description. The type of an argument is determined as follows, depending on the expression type of the corresponding input.

An argument for a Boolean input should be typed with java.lang.Boolean or boolean.
An argument for a numeric input should be typed with java.lang.Number.
An argument for a text input should be typed with java.lang.String.
An argument for a blob input should be typed with byte[].
An argument for a file input should be typed with FileValue.
An argument for a temporal input should be typed with java.time.temporal.Temporal.
An argument for a scenario data input should be typed with ScenarioData.
An argument for a list input should be type with java.util.Collection<T>, where T is the type corresponding to the elements of the list, for example String if the list contains text elements.

The execute() method may have any return type, including void. If it returns an int, then this return value is used as the exit code of the ExecuteRoutineStatement statement. Otherwise, the statement has exit code zero.

As mentioned, the first argument of the execute() method is an execution context. It has the following methods.

getExecutingUserId() returns the ID of the user who has triggered the execution of the job. It can be set as follows:
```
        LOGGER.info("Routine started by {} with job {}" context.getExecutingUserId(), context.getJobId());
```

getJobId() returns the ID of the job that requested the execution of the current routine. It can be set as follows:

        LOGGER.info("Routine started by {} with job {}" context.getExecutingUserId(), context.getJobId());

getCollector(ScenarioData scenarioData) turns a scenario data object, such as the one passed in an argument that corresponds to a call of withInput() on a scenario data expression, into an instance of the collector class of your application. It can be set as follows:
```
        CapacityPlanning collector = executionContext.getCollector(scenarioData);
```
markModified(ScenarioData scenarioData, DbDomCollector collector, String... modifiedTables) records modifications made to the scenario data and saves them in a collector, optionally restricting the modifications to a list of entities. It can be set as follows:
```
        executionContext.markModified(scenarioData, collector, "Country", "Plant");
```
The markModified(ScenarioData scenarioData, DbDomCollector collector, boolean checkSchema, String... modifiedTables) variant has an additional Boolean argument to run the schema checkers on the scenario. In a CDM application, if no entity is indicated, it records the modifications on all visible scenario types of the declared scenario.
emit(String outputName, Object outputValue) sets the value of the routine output with the specified name. It can be set as follows:
```
        executionContext.emit("message", "Scenario edited!");
```

6. Understanding Optimization Server Statements

Optimization Server tasks, also referred to as worker tasks, are executed remotely on an Optimization Server worker service. Their implementation is described in Chapter Implementing Worker Tasks. Once you have implemented a worker task, you can include its execution in the script of one of your tasks.

The ExecuteOptimizationServerTaskStatement.forTaskId(Expression taskId) method creates a statement that, when executed, executes the worker task with the provided ID on the Optimization Server. Additional information can be given for the execution of the worker task with the following methods.

usingOnDemandContext(Expression onDemandExecutionContext) provides a value for configuring the On-Demand job Execution Context. The On-Demand feature is only available on optimization servers deployed on Kubernetes. The worker and the different On-Demand Execution Contexts available are defined using a ConfigMap. For more details on the deployment of an "on demand" worker, refer to the Optimization Server documentation, Chapter Workers.


	Note that, if the On-Demand Context value is defined for a job that is run on an Optimization Server instance that is not deployed on Kubernetes (e.g. Docker), the specified value for `onDemandExecutionContext` is ignored.

withJobDescription(Expression description) provides a free-text description of the purpose of the worker task execution.
withInput(String name, Expression value) provides a value for the worker task input with the specified name, in accordance with the description of the worker task in the worker.yml file associated with the worker that implements the task.
withOutput(String outputName, String variableName) will store the value of the specified worker task output into the script memory under the specified variable name. This method is to be used for numeric, text, and binary outputs, which are mapped to the NUMBER, TEXT, and BLOB expression types, respectively.
withOutputScenario(String outputName, Expression scenarioId, Expression... filterTables) is to be used when the worker task output contains scenario data. The method will associate the output data with the scenario having the provided ID. Table names can be provided to restrict a subsequent save operation. The resulting scenario data will be stored in the script memory under the specified variable name, and saved to the database. Schema checkers will be run, and the schema status of the scenario updated. The variant withOutputScenarioCheckSchema(String outputName, Expression scenarioId, Expression checkSchema, Expression... filterTables) has an additional argument that must evaluate to a Boolean value; this value controls whether the schema checkers should be run on the scenario or not.
withFileOutput(String outputName, String variableName, Expression filename [, Expression mimeType]) is to be used when the worker task output is the content of a file. This method will associate the output data with a file having the provided name. A MIME type (used by web browsers to know how to handle the file as a task output) can be specified. The resulting file value will be stored in the script memory under the specified variable name.
withTaskGroupLabel(Expression label, Expression value) is used when the worker task belongs to a group on which limits have been defined based on job labels, to provide a value for the specified label. The label argument receives an expression that must evaluate to a string. The value argument receives an expression that can evaluate to any type that can be converted to a string. Note that if the worker task belongs to a group on which limits have been defined, this method must be called once for each of the labels included in the limits definition.

Except for the withJobDescription method, the methods above are typically repeated as many times as needed to address all the inputs, outputs, and task group limits of the worker task.

When a worker task is launched by executing an ExecuteOptimizationServerTaskStatement, the Optimization Server master process looks for a worker to execute the task. For more details, refer to the Optimization Server documentation, Chapter Workers.

If all eligible workers are busy, the master process waits for certain amount of time. This amount can be configured in the Execution Service extension through the services.execution.dbos.scheduled-job-max-waiting-time Spring property, or the SERVICES_EXECUTION_DBOS_SCHEDULED_JOB_MAX_WAITING_TIME environment variable. Its default value is 30 minutes.

If the worker task emits a numeric output named com.decisionbrain.gene.exitCode, the value of this output is used as the exit code of the statement. Otherwise, the statement has exit code zero.

A worker task can receive an interruption signal as a result of user interaction, in two possible forms.

When the user selects the Stop Job command in the web client while the current step executes an Optimization Server task, the task is required to abort with the TERMINATE_WITH_NO_SOLUTION policy. It should then stop execution as soon as possible, with no requirement to emit any output. The job will then be stopped. For more details, refer to Section Understanding Job Termination Statements
When the user presses the next step button in the job details view, while the current step executes an Optimization Server task. In this case, the task is required to abort with the TERMINATE_WITH_CURRENT_SOLUTION policy. The intent for this action is that the user considers (for example by looking at the task KPIs in the Optimization Server console) that the task computation has reached an acceptable state. It should then stop execution as soon as possible, but is required to emit its current solution as output. The job execution will then proceed to the next statement in the task script.

6.1. Understanding the IBM Watson Statement

IBM Watson Machine Learning (IBM WML or WML for short) is an IBM product that is available as part of IBM Cloud Pac for Data. Within IBM WML, code is packaged as "services". One can invoke a WML service from a task script.


	Note that the WML service must already be deployed on an IBM Cloud Pak for Data deployment space. To invoke it, you will need: An API Key The URL of the WML server The ID of the WML deployment space The ID of the WML deployment job

The invocation of the WML service is performed through a dedicated Optimization Server worker task named "WatsonML Connector". For more details, refer to Chapter Implementing Worker Tasks.

As a consequence, the invocation of the WML service in a task script is done with an ExecuteOptimizationServerTaskStatement, as in this example:

    .addStatement(ExecuteOptimizationServerTaskStatement.forTaskId(StringExpression.of("WatsonML Connector"))
        .withInput("wmlUrl", StringExpression.of("The URL of the WML server"))
        .withInput("spaceId", StringExpression.of("The ID of the WML deployment space"))
        .withInput("deploymentId", StringExpression.of("The ID of the WML deployment job"))
        .withInput("inputCollector", ScenarioDataExpression.of(VariableAccessExpression.of("scenarioId")))
        .withInput("outputTables", ListExpression.of(StringExpression.of("Schedule"), StringExpression.of("SolutionSummary")))
        .withOutputScenario("outputCollector", VariableAccessExpression.of("scenarioId"),
            ListExpression.of(StringExpression.of("Schedule"), StringExpression.of("SolutionSummary")))
    )

In this example:

The Optimization Server task named "WatsonML Connector" is called.
The characteristics of the WML service is passed to this task though the wmlUrl, spaceId, and deploymentId inputs.
Two additional inputs are provided, namely inputCollector and outputTables, which contain the data of a scenario and the list of tables in the scenario that may be modified by the WML service, respectively.
The modified scenario is taken from the task outputCollector output.

For more details, refer to Section Understanding Optimization Server Statements.

When the task that contains the above statement is executed, an Optimization Server worker process that implements the WatsonML Connector worker task must be running. This can be done by running the corresponding Docker image using the file deployment/docker/app/docker-compose-wml-worker.yml.

This requires setting the value of your API Key in an environment variable named IBM_WML_API_KEY, which is typically achieved using the following commands:

$ export IBM_WML_API_KEY=your-api-key
$ cd deployment/docker/app
$ docker compose -f docker-compose-wml-worker.yml up -d

7. Understanding Job Statements

A job refers to the execution of a task, which can be customized through specific statements.

7.1. Understanding Job Memory Statements

When a job is executed, it is equipped with a memory. The initial content of the memory is provided by the user when launching the job from the web client. The job can then access and modify variables in its memory. All variables in a job memory are global to the job.

The VariableAccessExpression.of(String variable) method creates an expression that, when evaluated, returns the value of the variable with the provided name, as found in the job memory. It is an error if there is no variable with this name in the job memory.

The SetVariableStatement.of(String variable, Expression value) method creates a statement that, when executed, sets the value of the variable with the provided name in the job memory, to the value to which the provided expression evaluates. For example, the statement created by the following code will set variable bar to the value of variable foo:

SetVariableStatement.of("bar", VariableAccessExpression.of("foo"))

7.2. Understanding Job Input Statements

The AskInputStatement.of(String variable, boolean required, ParameterType inputType [, String description]) method creates a statement that indicates to the web client that a value of the specified input type should be provided by the user when launching the task, and should be stored under the specified variable name in the job memory.

The ParameterType argument provides indications on the type and bounds of the task input to the web client, so that it can assist the user in entering valid input. Note that this is an aid, not a validation tool. This argument can take the following values.

JobInputType.BOOLEAN: A Boolean value is expected. The web client will display a checkbox.
JobInputType.REAL: A number is expected. The web client will allow any number to be entered, unless one of the following forms is used instead.
- JobInputType.real(Double min, Double max): The web client will make its best effort to constrain the number entered to be greater than, or equal to, the min value, and lower than, or equal to, the max value. The null value can be provided for either argument, meaning that the corresponding bound is open.
JobInputType.INTEGER: The web client will make its best effort to constrain the number entered to be an integer.
- JobInputType.integer(Long min, Long max): The web client will make its best effort to constrain the number entered to be an integer between the provided bounds, both inclusive. Either can be provided with null to indicate an open bound.
JobInputType.TEXT: A text is expected. As for other inputs, whether it can be empty or not is controlled by the required argument to AskInputStatement.of().
Temporal data: Input types for dates and times are provided, as follows. If you need a date-time, use two inputs, one of each type.
- JobInputType.DATE: A date is expected, that is, a day, a month, and a year. The web client will display a date chooser. The job variable for this input will contain a java.time.LocalDate value.
- JobInputType.TIME: A time is expected, that is, an hour and minutes. The web client will display a time chooser. The job variable for this input will contain a java.time.LocalTime value.
JobInputType.FILE: A file is expected.
- JobInputType.file(String... allowedExtensions): The web client will make its best effort to constrain the extension of the provided file to be among the ones specified.
JobInputType.SCENARIO_ID: The ID of a scenario is expected. The web client will display a scenario picker.
- JobInputType.scenarioId(JobInputType.WorkspaceOrScenarioFilter... filters): The web client will make its best effort to constrain the scenario selected for this input to match the provided filters, which can be any of the following.
  - UNIQUE: The selected scenario should not be already selected as the value for another input to the job. This filter is only relevant when there are at least two job inputs with the type SCENARIO_ID.
  - OWNED: The owner of the selected scenario should be the user who executes the task.
  - WRITABLE: The user who executes the task should have the necessary writing permissions on the selected scenario.
JobInputType.WORKSPACE_ID: The ID of a workspace is expected. The web client will display a workspace picker.
- JobInputType.workspaceId(JobInputType.WorkspaceOrScenarioFilter... filters): The web client will make its best effort to constrain the workspace selected for this input to match the provided filters, which are the same as for scenarios above.
JobInputType.SCENARIO_CREATION_PARAMETERS: The parameters required to create a scenario. Allows to choose the scenario name and, in a composite data model, the scenario type and its scenario references. Instead of a scenario reference, it's possible to create an additional empty scenario just from a name.

7.3. Understanding Job Logging Statements

The LogStatement.info(Expression message [, Throwable throwable]) method creates a statement that, when executed, logs the string to which the provided expression evaluates, and possibly the provided exception, with the INFO level. Similar methods named trace, debug, warn, and error create statements that will log messages at different levels.

The log of a job can be accessed, or downloaded as a text file, from the Job Details view.

7.4. Understanding Job Termination Statements

Jobs can terminate:

Due to Statement Execution and Expression Evaluation, i.e. the evaluation of an expression or the execution of a statement terminates abruptly with an exception. In such a case, the execution of the job stops and the exception is reported to the web client. The final status of the job is FAILED.
If the execution of all the statements of the script, including the evaluations of the embedded expressions, terminates normally, the final status of the job is COMPLETED. The job may then have output values, as specified by its script.
The SetTaskOutputStatement.of(String outputName, Expression outputValue) method creates a statement that, when executed, sets the task output with the specified name with the value to which the provided expression evaluates.
Its variant SetTaskOutputStatement.of(Expression outputName, Expression outputValue) allows you to provide an expression for the output name; this expression must evaluate to a string.
Due to User Interruption, for instance, when a user selects the Stop Job command in the web client. The job execution stops, at the latest, after the current step. The final status of the job is STOPPED.
If the current step executes an interruptible statement, a signal is sent to the statement to request it to stop prematurely. When an interruptible statement receives an interruption request, it can choose to honor it and stop its execution, or not.
The interruptible statements are the ones executing optimization server tasks and user-defined routines. For more details, refer to Sections Understanding Optimization Server Statements and Understanding User-Defined Routine Statements.
Due to Programmatic Interruption. This can be configured, for example, using the ExitTaskStatement.ok() method. It creates a statement that, when executed, causes the job to stop immediately, that is, without executing the rest of its script. The status of the job is then COMPLETED. The job outputs that have been set so far are available to the user.
- The ExitTaskStatement.withTaskOutput(String, Expression) method can be called (possibly several times) on the exit statement to add outputs before exiting.
- The ExitTaskStatement.failing(Expression) method creates a statement that, when executed, causes the job to fail immediately, that is, to stop with an exception. The status of the job is then FAILED. No job outputs are available to the user. The ExitTaskStatement.withTaskOutput(String, Expression) method is thus irrelevant. If not null, the expression provided to the failing() method is evaluated and displayed as the error message in the web client in the results and log views.
- The ExitTaskStatement.alerting(Expression) method creates a statement that, when executed, causes the job to stop immediately. The status of the job is then ALERTING. Since the job did not end with an exception, the job outputs that have been set so far are available to the user. The ExitTaskStatement.withTaskOutput(String, Expression) method can be called (possibly several times) on the exit statement to add outputs before exiting. In addition, the expression provided to the failing() method is evaluated and displayed in the web client in the results and log views.
- The ExitTaskStatement.when(Expression) method creates a conditional statement with the provided expression as its condition, and the exit statement as its "then" branch.
For example, the following sample task takes two inputs, a boolean and a text, then programmatically interrupts the execution of a job and return the text message as a job output. According to the value of the boolean chosen by the user, the interruption can end the job as a success (boolean true), or as a failure (boolean false). Here, the last two statements, SetTaskOutputStatement and LogStatement, are never executed.
```
@Configuration
public class Tasks {
    @Bean
    public ScriptedTaskDescription exitJobWithOutputTask() {
        ScriptedTaskDescription task = new ScriptedTaskDescription("exitJobWithOutputTask", "A task that exits before its end (and emits an output)");
        VariableAccessExpression successful = VariableAccessExpression.of("Successful?");
        VariableAccessExpression message = VariableAccessExpression.of("Error Message");
        task.getScript()
            .addStatement(AskInputStatement.of(successful.getVariableName(), true, JobInputType.BOOLEAN))
            .addStatement(AskInputStatement.of(message.getVariableName(), false, JobInputType.TEXT))
            .addStatement(LogStatement.info(StringExpression.of("Task start")))
            .addStatement(SetTaskOutputStatement.of("output 1", NumericExpression.of(1)))
            .addStatement(IfStatement.of(successful,
                IfStatement.of(StringExpression.isEmptyOrNull(message),
                    ExitTaskStatement.ok().withTaskOutput("extra output", StringExpression.of("exit in COMPLETED")),
                    ExitTaskStatement.alerting(message).withTaskOutput("extra output", StringExpression.of("exit in ALERTING"))
                ),
                ExitTaskStatement.failing(message)))
            .addStatement(SetTaskOutputStatement.of("output 2", NumericExpression.of(2)))
            .addStatement(LogStatement.info(StringExpression.of("Task end")));
        return task;
    }
}
```
The generated web client component for this task is the following:
Figure 13.4. Configuring a Custom ExitJobWithOutput Task

If a task includes an exit script, it is executed whether the job terminates with the status:

COMPLETED because it executed all the statements of its main script without exception,
FAILED because of an exception, or
else due to an ExitTaskStatement.

However, if the job is manually stopped by the user via the Stop Job button, the exit script is not executed. For more details, refer to Section Using the Job Details Widget.

Unless the exit script itself includes an ExitTaskStatement, the final status of the job will be the one at the end of the main script execution. For example, if the main script was interrupted by an exception, the job will execute the exit script and end in status FAILED. For more details, refer to Section Understanding Tasks and Jobs.

8. Understanding Rule-Based Routine Statement

Rule-based programming is a coding paradigm where a program is expressed in terms of ‘if-then’ rules, where the ‘if’ part describes the conditions under which the rule should apply, and the ‘then’ part provides statements to execute when the rule is applied. Rules are written in the Drools rule language (DRL), which syntax can be found here. Basically, a DRL rule has the following form:

rule "Name of the rule, between quotes"
when
    $optionalVariable: ClassName(field == value, otherField < someValue)
    ...
then
    // some Java-like code using the $variables defined in the 'when' part
end

8.1. Examples of Rules

Detecting and reporting issues. The rule engine tries to match each condition in the 'when' part with an entity instance in the scenario. For each successful match, the 'then' part is executed. An example of a rule that adds an issue if two employees have the same first name is:

rule "First names of employees should be distinct"
when
    $empl1: Employee($first1: firstName)
    $empl2: Employee(this != $empl1, firstName == $first1)
then
    helper.addIssue($empl2, "Employee has same first name as employee #%s".formatted($empl1.getId()));
end

As this example shows:

There can be several conditions, matching the same or different entity types.
Variables can be introduced that are bound to the entity instance matched by a condition, or to (the value of) a field.
In the ‘when’ part, you can use the field names (firstName as opposed to getFirstName()). In the ‘then’ part, you must use getters.
Strings comparison is made with equals even if written with ==.
A helper global variable is available with utility methods. More about globals below.

Output logs. The following example is a very simple one and shows how to output some logs, here the number of employees in the scenario:

rule "Count employees"
when
    accumulate(Employee(); $n: count())
then
    logger.info("There are {} employees in the current scenario.", $n);
end

Modifying the scenario data. The following example gives a small raise to all employees who have more than two years of seniority:

rule "Raise some salaries"
when
    $e: Employee(yearsInCompany >= 2)
then
    $e.setSalary($e.getSalary() + 100);
end

Such a rule can typically be used to perform some bulk updates on the data of a scenario.

8.2. Globals

Two global variables are predefined, named helper and dataset. These variables can be used in the rules whenever useful.

The helper global is bound to an instance of the RulesCollectorHelper class, which is scaffolded in the backend service extensions module. It natively contains a few methods to create issues (instances of GeneIssue) and add them to the collector. You are free to add more utility methods that could be useful to you. You will then be able to call these methods in rules, in the ‘when’ part or in the ‘then’ part. Note that you must not call a method in the ‘when’ part if it has a side effect.

The dataset global is bound to the collector that reflects the scenario data.

8.3. Routine

The built-in routine named ExecuteRulesetOnScenario executes a set of rules (a.k.a. a ruleset) on a scenario. This routine takes two inputs:

A ScenarioData, expected to contain the data of the scenario on which to run the rules.
A string, which can either be the name of a GeneParameter in the scenario above, from which the rules are retrieved; or the text of the rules themselves.

Invoking this routine with the name of a GeneParameter typically corresponds to the use case where the rules are edited in a Code Editor widget, see Section 20, “Using the Rules Script Editor Widget”.

The routine first retrieves and compiles the ruleset. Unless compilation fails, the routine then creates the instance of the helper class, sets the globals, and inserts all the entities of the scenarios into the rule engine working memory. Finally, it triggers the execution of the rules and marks the scenario as modified.

All compilation and execution messages are added to the log, and emitted as an output of the routine, named executionLogs. If the ruleset was retrieved from a GeneParameter, the messages are also stored in another GeneParameter that has the same name, plus the -logs suffix.

The routine exits with a code that describes how things went:

0: The rules could be successfully compiled and executed.
1: The ruleset could not be retrieved.
2: The ruleset could not be compiled.
3: Executing the ruleset raised an error.
4: Some other error occurred in the process.

This routine is called from the built-in scripted task described below, but can also be called explicitly in any scripted task. It is enabled by the @AddExecuteRulesetOnScenarioRoutine annotation, which is added by default to a configuration of the backend service.

8.4. Scripted Task

The built-in scripted task named ExecuteRulesetOnScenarioTask is called from the Run button of the Rules Script Editor widget toolbar (see Section 20, “Using the Rules Script Editor Widget”), but can also be called from any UI Action. It is enabled by the @AddExecuteRulesetOnScenarioTask annotation, which is added by default to a configuration of the execution service.

The script task takes the following inputs:

In an input named scenario, the ID of the scenario on which to execute the rules.
In an input named scriptParameterName, the name of the GeneParameter that contains the text of the rules.

The task invokes the above routine, emits the log as a task output named executionLogs, and exits in Alerting status if the routine exit code is not zero.

9. Understanding JupyterLab Statement

JupyterLab is a web-based interactive development environment for notebooks. Using JupyterLab from an application is enabled by setting the application preference named TASK_MENU_SHOW_JUPYTERLAB to true, see Setting Application Preferences. This application preference activates a menu entry that opens a browser tab on a JupyterLab server. This server is password-protected, the password is gene.

A sample notebook is provided that gives further information on how to connect the notebook with the rest of the application, how to load data from a scenario, or how to save data into a scenario. Notebooks are stored under the processing/jupyterlab hierarchy in the source code of the application.

9.1. Statement

The ExecuteJupyterNotebookStatement.of(Expression notebookName) method creates a statement that, when executed, executes all the cells of the JupyterLab notebook whose name results from the evaluation of the expression passed in notebookName.

Additional inputs can be passed to the notebook using the ExecuteJupyterNotebookStatement.withInput(String, Expression) method.

9.2. Scripted Task

The built-in scripted task named ExecuteJupyterNotebookTask takes two inputs: a scenario and a notebook name. This task executes all the cells of the notebook. It is enabled by the @AddJupyterLabTask annotation, which is added by default to a configuration of the execution service.

The ID of the scenario is passed to the notebook as an input named inputScenario. The notebook name must contain the .ipynb extension.

10. Understanding ChatGPT Routine Statement

The ChatGPT integration provides access to OpenAI's GPT models. The conversational dimension is replicated in a text-only manner using a simple mark-up convention.

A conversation is composed of consecutive conversation turns. A conversation turn includes a question (from the user) and an answer (from the model). The last turn of a conversation may only include a question, in which case the turn and the conversation are said to be open; otherwise, they are said to be closed.

Conversation turns are separated with a line of hyphens (----------------------------------------). In a closed conversation turn, the question is introduced with ‘» ’; the answer is introduced with ‘— ’. A blank line separates the question and the answer. Note that the question and the answer may also contain blank lines. In an open conversation turn, the question is not introduced by anything special.

Below is an example of an open conversation.

» What is my name?

- I'm sorry, I don't have access to personal information about users.
----------------------------------------
» I'm John. What is your name?

- I am a virtual assistant and do not have a personal name. You can simply reter to me as Assistant or Chatbot. How can I assist you today, John?
----------------------------------------
If your had the same name as me, what would be your name?

10.1. Routine

The built-in routine named ProcessChatGptConversation completes an open conversation with an answer from a GPT model. This routine takes the following inputs:

A ScenarioData from where the conversation will be retrieved and stored back.
The name of a GeneParameter in the scenario above, where the conversation is stored.
An API key for OpenAI.
The name of a model, such as gpt-3.5-turbo.
The temperature to use, that is, a floating-point number between 0 and 1, where higher values means more “creativity” from the GPT model.

The last three inputs are optional. If omitted, defaults are used as explained below in the subsection on parameters.

This routine first retrieves the content of the provided GeneParameter and parses it according to the mark-up convention described above. If the conversation is already closed, the routine stops with exit code 2. Otherwise, it looks for a GeneParameter with the same name as the one containing the conversation, suffixed with -instructions. If this parameter is present, its value is used as general instructions to the GPT model—sometimes referred to as the “system” part of the prompt. The open conversation, together with the instructions, if any, are then submitted to the GPT model, and the answer is appended to the conversation, which is finally written back in the GeneParameter. Outputs of the routine include the name of the GPT model used—usually a variant of the model requested—and an estimate of the cost of the interaction with the GPT model.

This routine is called from the built-in scripted task described below, but can also be called explicitly in any scripted task. It is enabled by the @AddChatGptConversationRoutine annotation, which is added by default to a configuration of the backend service.

10.2. Scripted Task

The built-in scripted task named ProcessChatGptConversationTask takes the same five inputs as the routine above. These inputs are named scenario, scriptParameterName, OpenAI API key, GPT model, and Model temperature. The last three are optional. If omitted, defaults are used as explained below in the subsection on parameters.

The scripted task is enabled by the @AddChatGptConversationTask annotation, which is added by default to a configuration of the execution service.

10.3. OpenAI API Key and Other Parameters

The scripted task and routine above require an API key to work properly. This API key is to be obtained from OpenAI, and its usage is usually paid. Various options exists about how to provide the API key to the routine; below are some, from the least to the most preferred.

Hard-code the API key in your code.
Store the API key as a Spring property named openai.apiKey in the application.yml file of the backend service extension.
When running with Docker Compose, write the API key in the deployment/docker/app/.env file.
When running locally from your IDE or through Docker Compose, store the API key in an environment variable.
When deploying with Kubernetes, store the API key in the values of your deployment, possibly in a secret.

The application.yml file of the backend service extension contains other properties that serve as parameters and default values for the routine inputs. These are not sensitive and can be kept in this file if appropriate for your use case.

The openai.model and openai.temperature properties provide default values for the corresponding inputs of the routine.

The openai.pricing property is used to compute the cost estimate of requests to GPT. The cost computation is based on two parameters: the cost per million of input tokens, and the cost per millions of output tokens. These costs are provided by OpenAI in the Pricing section of their website. The openai.pricing property is a list of prices that depend on the model used. This list is processed in order, until the model used in the request matches the name patterns; the associated costs are used. This processing can be bypassed by providing values for the openai.costPerMillionInputTokens and openai.costPerMillionOutputTokens properties. If no cost estimate can be computed, no error is raised but the routine output is set to "unknown".