Checkpointing Example #164

satyaog · 2023-03-14T21:43:10Z

No description provided.

satyaog · 2023-03-14T23:08:01Z

quick urls :

btravouillon · 2023-03-22T22:25:30Z

@satyaog Is it expected to see an overlap with #163 in the changes? A lot of files include the same changes in both PR.

btravouillon · 2023-04-05T17:47:51Z

Waiting for merge of #161.

satyaog · 2023-04-18T15:44:25Z

There a lot of overlapping with #163. This one should be rebased on #163 once done

lebrice

Not completely done reviewing, moving to #163 before I complete this.

docs/examples/data/checkpointing/job.sh

lebrice · 2023-05-01T21:21:00Z

.github/workflows/tests.yml

+      run:  python3 -m pip install -r docs/requirements.txt
+
+    - name: Run files generation tests
+      run:  pre-commit run --all-files && [[ -z "$(git status -s)" ]]


What's && [[ -z "$(git status -s)" ]] for?

making sure that nothing changed :

&& : run the next command only if the previous succeeded

git status -s : only print the files that differs (including new and deleted)

[[ -z ... ]] : checks if the length of the output or variable is zero and returns true if it is zero.

Note to self, we should do pre-commit run --all-files preprocess-py to only run this hook in particular.
(I locally added other hooks in my pre-commit config. If we ever add some other hooks for writing the examples (e.g. ruff, black, etc, then we should probably only run the generate hook here)

docs/examples/preprocess.py

docs/examples/frameworks/pytorch_setup/_index.rst

.pre-commit-config.yaml

docs/examples/data/checkpointing/main.py

satyaog · 2023-05-03T19:29:37Z

Rebased on #182 which should be merged before this one

Signed-off-by: Fabrice Normandin <[email protected]>

This reverts commit 3b383ad.

Signed-off-by: Fabrice Normandin <[email protected]>

lebrice

(NOTE: J'approve ce qui est devenu ma propre PR, il me faudrait 1-2 autres reviews svp)

Signed-off-by: Fabrice Normandin <[email protected]>

obilaniu · 2023-06-29T20:44:00Z

docs/examples/checkpointing/job.sh

+# trap the signal to the main BATCH script here.
+sig_handler()
+{
+	echo "BATCH interrupted"
+	wait # wait for all children, this is important!
+}
+
+trap 'sig_handler' SIGINT SIGTERM SIGCONT


Useless after exec python main.py

Yeah you're right! Thanks @obilaniu. Should we encourage users to do srun python (...) (and trap?) or exec python (...) in your view?

This being an example only, I would adopt one of two strategies - either minimize the delta from the foundation mini-example to this one, or minimize the complexity of this one in the absolute.

I would prefer that you drop the trap business, and use the Python equivalent (the signal package).

However, I have no faith in asynchronous checkpointing at all and recommend only synchronous checkpointing. I would use the signal package only to demonstrate in the mini-example that Python can install signal handlers and execute Python code as a result.

Ok, here's my take: 779413b

Signed-off-by: Fabrice Normandin <[email protected]>

btravouillon

I'm not able to review the python code, but that changes looks good to me overall.

obilaniu · 2023-07-06T17:03:56Z

Overall this is fine, however I do not like the strategy for checkpointing. It's not safe.

Your implementation:

For writing:
1. If it exists, move aside the current checkpoint to a <PATH>.backup file
2. Writes a checkpoint to <PATH>
3. Unlinks <PATH>.backup.
For reading:
1. If <PATH>.backup exists, load that
2. If <PATH> exists, load that
3. Otherwise, start from scratch

You vulnerability is that if you are killed in step 2 during the first checkpoint's write, the reload will detect no <PATH>.backup but does detect the existence of the (unbeknownst to it, corrupt) checkpoint <PATH>, and fail to reload.

My preferred implementation:

For writing:
1. Writes a checkpoint to <PATH>.<tmpXXXXXXXX>. The name should be unique to avoid collisions.
2. Atomically move <PATH>.<tmpXXXXXXXX> into <PATH> using os.replace().
For reading:
1. If <PATH> exists, load that
2. Otherwise, start from scratch.

Signed-off-by: Fabrice Normandin <[email protected]>

lebrice · 2023-07-06T18:56:38Z

Hey @obilaniu , I implemented your suggested changes in 4d3fd15

satyaog force-pushed the checkpoint branch 2 times, most recently from 4d29c95 to 33627fe Compare March 14, 2023 22:55

satyaog force-pushed the checkpoint branch from 33627fe to 38e4650 Compare March 20, 2023 19:53

satyaog marked this pull request as ready for review March 21, 2023 20:47

satyaog requested a review from btravouillon as a code owner March 21, 2023 20:47

satyaog force-pushed the checkpoint branch 3 times, most recently from 0b75ac4 to 51ee228 Compare April 25, 2023 05:07

satyaog force-pushed the checkpoint branch from 51ee228 to 559c7e0 Compare May 1, 2023 04:30

lebrice requested changes May 1, 2023

View reviewed changes

satyaog force-pushed the checkpoint branch 2 times, most recently from 776bcd7 to ad75d92 Compare May 3, 2023 18:16

satyaog force-pushed the checkpoint branch from ad75d92 to f35c925 Compare May 25, 2023 16:45

satyaog marked this pull request as draft May 31, 2023 17:56

satyaog and others added 10 commits June 29, 2023 16:22

Add checkpointing example

27922cc

Minor changes (addressing some comments)

512f045

Signed-off-by: Fabrice Normandin <[email protected]>

Moved the files to docs/examples/checkpointing

398d0e7

Signed-off-by: Fabrice Normandin <[email protected]>

Improve the file, make run reproducible

62c9162

Signed-off-by: Fabrice Normandin <[email protected]>

Add changes from just the checkpointing to diffs

9756c98

Signed-off-by: Fabrice Normandin <[email protected]>

Add note in README for using cp --update

3a6075d

Signed-off-by: Fabrice Normandin <[email protected]>

Move the restart_count stuff to load_checkpoint

9ed5896

Signed-off-by: Fabrice Normandin <[email protected]>

Update the comment about type hints in the example

65314b8

Signed-off-by: Fabrice Normandin <[email protected]>

Remove unwanted modification to generate_diffs.sh

4cec515

Signed-off-by: Fabrice Normandin <[email protected]>

Remove unwanted change to docs/index.rst

3b383ad

Signed-off-by: Fabrice Normandin <[email protected]>

lebrice force-pushed the checkpoint branch from 642a8d9 to 3b383ad Compare June 29, 2023 20:22

lebrice added 2 commits June 29, 2023 16:23

Revert "Remove unwanted change to docs/index.rst"

d204cd2

This reverts commit 3b383ad.

Add missing newline to fix docs lint workflow

4737825

Signed-off-by: Fabrice Normandin <[email protected]>

lebrice marked this pull request as ready for review June 29, 2023 20:30

lebrice requested review from lebrice and obilaniu June 29, 2023 20:30

lebrice approved these changes Jun 29, 2023

View reviewed changes

Move example to a "Good Practices" section

5c66038

Signed-off-by: Fabrice Normandin <[email protected]>

obilaniu reviewed Jun 29, 2023

View reviewed changes

lebrice added 2 commits June 29, 2023 16:53

Remove change to training_epochs

65e3822

Signed-off-by: Fabrice Normandin <[email protected]>

Add a signal handler in Python

779413b

Signed-off-by: Fabrice Normandin <[email protected]>

lebrice force-pushed the checkpoint branch from fa48f2f to 779413b Compare July 3, 2023 19:01

lebrice requested review from obilaniu and bouthilx July 5, 2023 12:44

btravouillon approved these changes Jul 5, 2023

View reviewed changes

Fix issue with checkpointing logic

4d3fd15

Signed-off-by: Fabrice Normandin <[email protected]>

obilaniu approved these changes Jul 6, 2023

View reviewed changes

lebrice merged commit 71289ec into mila-iqia:master Jul 6, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Checkpointing Example #164

Checkpointing Example #164

satyaog commented Mar 14, 2023

satyaog commented Mar 14, 2023 •

edited

Loading

btravouillon commented Mar 22, 2023

btravouillon commented Apr 5, 2023

satyaog commented Apr 18, 2023

lebrice left a comment

lebrice May 1, 2023

satyaog May 2, 2023

lebrice Jun 29, 2023

satyaog commented May 3, 2023

lebrice left a comment

obilaniu Jun 29, 2023

lebrice Jun 29, 2023 •

edited

Loading

obilaniu Jul 1, 2023

lebrice Jul 3, 2023

btravouillon left a comment

obilaniu commented Jul 6, 2023

lebrice commented Jul 6, 2023

Checkpointing Example #164

Checkpointing Example #164

Conversation

satyaog commented Mar 14, 2023

satyaog commented Mar 14, 2023 • edited Loading

btravouillon commented Mar 22, 2023

btravouillon commented Apr 5, 2023

satyaog commented Apr 18, 2023

lebrice left a comment

Choose a reason for hiding this comment

lebrice May 1, 2023

Choose a reason for hiding this comment

satyaog May 2, 2023

Choose a reason for hiding this comment

lebrice Jun 29, 2023

Choose a reason for hiding this comment

satyaog commented May 3, 2023

lebrice left a comment

Choose a reason for hiding this comment

obilaniu Jun 29, 2023

Choose a reason for hiding this comment

lebrice Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

obilaniu Jul 1, 2023

Choose a reason for hiding this comment

lebrice Jul 3, 2023

Choose a reason for hiding this comment

btravouillon left a comment

Choose a reason for hiding this comment

obilaniu commented Jul 6, 2023

lebrice commented Jul 6, 2023

satyaog commented Mar 14, 2023 •

edited

Loading

lebrice Jun 29, 2023 •

edited

Loading