[3827] Load and save documents in parallel

[pcdavid]

Bug: #3827
Signed-off-by: Pierre-Charles David [email protected]

Pull request template

General purpose

What is the main goal of this pull request?

• Bug fixes
• New features
• Documentation
• Cleanup
• Tests
• Build / releng

Project management

• Has the pull request been added to the relevant project and milestone? (Only if you know that your work is part of a specific iteration such as the current one)
• Have the priority: and pr: labels been added to the pull request? (In case of doubt, start with the labels priority: low and pr: to review later)
• Have the relevant issues been added to the pull request?
• Have the relevant labels been added to the issues? (area:, difficulty:, type:)
• Have the relevant issues been added to the same project and milestone as the pull request?
• Has the CHANGELOG.adoc been updated to reference the relevant issues?
• Have the relevant API breaks been described in the CHANGELOG.adoc? (Including changes in the GraphQL API)
• In case of a change with a visual impact, are there any screenshots in the CHANGELOG.adoc? For example in doc/screenshots/2022.5.0-my-new-feature.png

Architectural decision records (ADR)

• Does the title of the commit contributing the ADR start with [doc]?
• Are the ADRs mentioned in the relevant section of the CHANGELOG.adoc?

Dependencies

• Are the new / upgraded dependencies mentioned in the relevant section of the CHANGELOG.adoc?
• Are the new dependencies justified in the CHANGELOG.adoc?

Frontend

This section is not relevant if your contribution does not come with changes to the frontend.

General purpose

• Is the code properly tested? (Plain old JavaScript tests for business code and tests based on React Testing Library for the components)

Typing

We need to improve the typing of our code, as such, we require every contribution to come with proper TypeScript typing for both changes contributing new files and those modifying existing files.
Please ensure that the following statements are true for each file created or modified (this may require you to improve code outside of your contribution).

• Variables have a proper type
• Functions’ arguments have a proper type
• Functions’ return type are specified
• Hooks are properly typed:
  • useMutation<DATA_TYPE, VARIABLE_TYPE>(…)
  • useQuery<DATA_TYPE, VARIABLE_TYPE>(…)
  • useSubscription<DATA_TYPE, VARIABLE_TYPE>(…)
  • useMachine<CONTEXT_TYPE, EVENTS_TYPE>(…)
  • useState<STATE_TYPE>(…)
• All components have a proper typing for their props
• No useless optional chaining with ?. (if the GraphQL API specifies that a field cannot be null, do not treat it has potentially null for example)
• Nullable values have a proper type (for example let diagram: Diagram | null = null;)

Backend

This section is not relevant if your contribution does not come with changes to the backend.

General purpose

• Are all the event handlers tested?
• Are the event processor tested?
• Is the business code (services) tested?
• Are diagram layout changes tested?

Architecture

• Are data structure classes properly separated from behavioral classes?
• Are all the relevant fields final?
• Is any data structure mutable? If so, please write a comment indicating why
• Are behavioral classes either stateless or side effect free?

Review

How to test this PR?

Please describe here the various use cases to test this pull request

• Has the Kiwi TCMS test suite been updated with tests for this contribution?

[pcdavid]

With the new version, which uses parallel streams (and thus the underlying ForkJoinPool) for both loading & saving, I get the following results on my machine (12th Gen Intel® Core™ i7-12700H × 20, OpenJDK Runtime Environment Temurin-17.0.4+8).

Each time the scenario is the same:

• Open/load a "Sirius Web (1.3M)" project which is a synthetic project with the "Sirius Web" Papaya example copied 10 times (so it's a "good" use case here as the 1.3M elements are split across 230 resources).
• Note the time taken to load the whole editing context.
• Expand the first semantic element in the Explorer, edit its name in the Details view, and validate.
• Note the time taken to save the whole editing context.

In each of the 3 settings tested, the VM is restarted and the scenario repeated 3 times to allow for some warmup. Only the times of the last iteration is reported.

Sequential (warm):

• Loading: 13.12s (1309800 objects have been loaded in 13123 ms)
• Saving: 11.04s (Editing context d90b1ea7-2eaf-45db-8107-fe8699e993b1 saved in 11042 ms)

Parallel (java.util.concurrent.ForkJoinPool.common.parallelism=8; warm)

• Loading: 8.49s (1309800 objects have been loaded in 8490 ms)
• Saving: 4.96s (Editing context d90b1ea7-2eaf-45db-8107-fe8699e993b1 saved in 4966 ms)

Parallel (java.util.concurrent.ForkJoinPool.common.parallelism=20; warm)

• Loading: 7.25s (1309800 objects have been loaded in 7255 ms)
• Saving: 5.18s (Editing context d90b1ea7-2eaf-45db-8107-fe8699e993b1 saved in 5184 ms)