Skip to content

Commit

Permalink
[docs] Add documentation on DCS op behaviour
Browse files Browse the repository at this point in the history
  • Loading branch information
teo committed Jul 18, 2024
1 parent bcd2c1a commit d83667f
Showing 1 changed file with 113 additions and 1 deletion.
114 changes: 113 additions & 1 deletion docs/handbook/operation_order.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,116 @@ This is the order of actions happening at a healthy end of run.
- `"run_end_completion_time_ms"` is set using current time. It is considered as the EOEOR timestamp.
- `after_STOP_ACTIVITY` hooks with positive weights (incl. 0) are executed:
- `ccdb.RunStop()` at `0`
- `bookkeeping.UpdateRunStop()`, `bookkeeping.UpdateEnv()` at `+100`
- `bookkeeping.UpdateRunStop()`, `bookkeeping.UpdateEnv()` at `+100`

# Integrated service operations

## DCS

### DCS operations

The DCS integration plugin exposes to the workflow template (WFT) context the
following operations. Their associated transitions in this table refer
to the [readout-dataflow](https://github.com/AliceO2Group/ControlWorkflows/blob/master/workflows/readout-dataflow.yaml) workflow template.

| **DCS operation** | **WFT call** | **Call timing** | **Critical** | **Contingent on detector state** |
|-----------------------|---------------------|---------------------------|--------------|----------------------------------|
| Prepare For Run (PFR) | `dcs.PrepareForRun` | during `CONFIGURE` | `false` | yes |
| Start Of Run (SOR) | `dcs.StartOfRun` | early in `START_ACTIVITY` | `true` | yes |
| End Of Run (EOR) | `dcs.EndOfRun` | late in `STOP_ACTIVITY` | `true` | no |

The DCS integration plugin subscribes to the [DCS service](https://github.com/AliceO2Group/Control/blob/master/core/integration/dcs/protos/dcs.proto) and continually
receives information on operation-state compatibility for all
detectors.
When a given environment reaches a DCS call, the relevant DCS operation
will be called only if the DCS service reports that all detectors in that
environment are compatible with this operation, except EOR, which is
always called.

### DCS PrepareForRun behaviour

Unlike SOR and EOR, which are mandatory if `dcs_enabled` is set to `true`,
an impossibility to run PFR or a PFR failure will not prevent the
environment from transitioning forward.

#### DCS PFR incompatibility

When `dcs.PrepareForRun` is called, if at least one detector is in a
state that is incompatible with PFR as reported by the DCS service,
a grace period of 10 seconds is given for the detector(s) to become
compatible with PFR, with 1Hz polling frequency. As soon as all
detectors become compatible with PFR, the PFR operation is requested
to the DCS service.

If the grace period ends and at least one detector
included in the environment is still incompatible with PFR, the PFR
operation **will not run for any detector**.

However, the environment
can still transition forward towards the `RUNNING` state, and any DCS
activities that would have taken place in PFR will instead happen
during SOR. Only at that point, if at least one detector is not
compatible with SOR (or if it is but SOR fails), will the environment
declare a failure.

#### DCS PFR failure

When `dcs.PrepareForRun` is called, if all detectors are compatible
with PFR as reported by the DCS service (or become compatible during
the grace period), the PFR operation is immediately requested to the
DCS service.

If this operation fails for one or more detectors, the
`dcs.PrepareForRun` call as a whole is considered to have failed,
but since it is non-critical the environment may still reach the
`CONFIGURED` state and transition forward towards `RUNNING`.

As in the case of an impossibility to run PFR, any DCS activities that
would have taken place in PFR will instead be done during SOR.

### DCS StartOfRun behaviour

The SOR operation is mandatory if `dcs_enabled` is set to `true`
(AliECS GUI "DCS" switched on).

#### DCS SOR incompatibility

When `dcs.StartOfRun` is called, if at least one detector is in a
state that is incompatible with SOR as reported by the DCS service,
or if after a grace period of 10 seconds at least one detector is
still incompatible with SOR, the SOR operation **will not run for any
detector**.

The environment will then declare a **failure**, the
`START_ACTIVITY` transition will be blocked and the environment
will move to `ERROR`.

#### DCS SOR failure

When `dcs.StartOfRun` is called, if all detectors are compatible
with SOR as reported by the DCS service (or become compatible during
the grace period), the SOR operation is immediately requested to the
DCS service.

If this operation fails for one or more detectors, the
`dcs.StartOfRun` call as a whole is considered to have failed.

The environment will then declare a **failure**, the
`START_ACTIVITY` transition will be blocked and the environment
will move to `ERROR`

### DCS EndOfRun behaviour

The EOR operation is mandatory if `dcs_enabled` is set to `true`
(AliECS GUI "DCS" switched on). However, unlike with PFR and SOR, there
is **no check for compatibility** with the EOR operation. The EOR
request will always be sent to the DCS service during `STOP_ACTIVITY`.

#### DCS EOR failure

If this operation fails for one or more detectors, the
`dcs.EndOfRun` call as a whole is considered to have failed.

The environment will then declare a **failure**, the
`STOP_ACTIVITY` transition will be blocked and the environment
will move to `ERROR`.

0 comments on commit d83667f

Please sign in to comment.