differences for PR #5

config.yaml

@@ -60,6 +60,9 @@ contact: '[email protected]'
 # Order of episodes in your lesson
 episodes: 
 - introduction.md
+- profiling-introduction.md
+- profiling-functions.md
+- profiling-lines.md
 
 # Information for Learners
 learners: 

index.md

@@ -2,8 +2,45 @@
 site: sandpaper::sandpaper_site
 ---
 
-This is a new lesson built with [The Carpentries Workbench][workbench]. 
+![Welcome to Performance Profiling & Optimisation (Python) Training!
+](episodes/fig/pando-python-hex-sticker.png){
+alt='Performance Profiling & Optimisation (Python) Training'
+style='padding: 2%'}
 
+The training curriculum for this course is designed for researchers that are writing Python and lack formal training. The curriculum covers how to assess where time is being spent during execution of a Python program, it also provides a high level understanding of how code executes and how this maps to the limiting factors of performance.
 
-[workbench]: https://carpentries.github.io/sandpaper-docs
+If you are now comfortable using Python, this course may be of interest to supplement and advance your programming knowledge. This course is particularly relevant if you are writing research code and desire greater confidence that your code is both performant and suitable for publication.
 
+<!-- TODO: course duration? -->
+<!-- TODO: confident code syllabus? -->
+
+
+## Learning Objectives
+<!-- Aim for 3-4 objectives for every 6 hours of training -->
+<!-- SMART Objectives
+    - Specific
+    - Measureable
+    - Attainable (within the span of the course)
+    - Relevant
+    - Time-bound (implicitly the length of the course)
+-->
+<!-- Evaluation tool: https://web.cs.manchester.ac.uk/iloadvisor/ -->
+After attending this training, participants will be able to:
+
+- identify the most expensive functions and lines of code using `cprofile` and `line_profiler`.
+- evaluate code to determine the limiting factors of it's performance.
+- recognise and implement optimisations for common limiting factors of performance.
+
+::::::::::::::::::::::::::::::::::::::::::  prereq
+
+## Prerequisites
+
+Before joining Performance Profiling & Optimisation (Python) Training, participants should be able to:
+
+- implement basic algorithms in Python
+- follow the control flow of Python code, and dry run the execution in their head or on paper.
+
+See the [Research Computing Training Hub](https://sites.google.com/sheffield.ac.uk/research-training/research-training) for other courses to help with learning these skills.
+<!-- TODO: could make a dedicated page (like https://carpentries.github.io/lesson-development-training/markdown-github-primer.html) that highlights specific courses/resources. -->
+
+::::::::::::::::::::::::::::::::::::::::::::::::::

md5sum.txt

@@ -1,11 +1,14 @@
 "file" "checksum" "built" "date"
 "CODE_OF_CONDUCT.md" "c93c83c630db2fe2462240bf72552548" "site/built/CODE_OF_CONDUCT.md" "2023-12-07"
 "LICENSE.md" "b24ebbb41b14ca25cf6b8216dda83e5f" "site/built/LICENSE.md" "2023-12-07"
-"config.yaml" "509085b79e6ec689b015216d87ddbeff" "site/built/config.yaml" "2023-12-08"
-"index.md" "a02c9c785ed98ddd84fe3d34ddb12fcd" "site/built/index.md" "2023-12-08"
+"config.yaml" "9086af5e5e979722dcad1ab925ec6412" "site/built/config.yaml" "2024-01-01"
+"index.md" "df8ef5258ba527e8fc3ca82f97fa27d8" "site/built/index.md" "2024-01-01"
 "links.md" "8184cf4149eafbf03ce8da8ff0778c14" "site/built/links.md" "2023-12-07"
 "episodes/introduction.md" "6c55d31b41d322729fb3276f8d4371fc" "site/built/introduction.md" "2023-12-07"
+"episodes/profiling-introduction.md" "21f5df1349d1e14ee6e1f1720cbce68b" "site/built/profiling-introduction.md" "2024-01-01"
+"episodes/profiling-functions.md" "6e3d4d42db22b5ea2d9a112c61940289" "site/built/profiling-functions.md" "2024-01-01"
+"episodes/profiling-lines.md" "f21cb8b587a238657ac5bdf28df59e50" "site/built/profiling-lines.md" "2024-01-01"
 "instructors/instructor-notes.md" "cae72b6712578d74a49fea7513099f8c" "site/built/instructor-notes.md" "2023-12-07"
 "learners/reference.md" "1c7cc4e229304d9806a13f69ca1b8ba4" "site/built/reference.md" "2023-12-07"
-"learners/setup.md" "61568b36c8b96363218c9736f6aee03a" "site/built/setup.md" "2023-12-07"
+"learners/setup.md" "b2304a5b62e01a2ae8beed609ab1b725" "site/built/setup.md" "2024-01-01"
 "profiles/learner-profiles.md" "60b93493cf1da06dfd63255d73854461" "site/built/learner-profiles.md" "2023-12-07"

profiling-functions.md

@@ -0,0 +1,17 @@
+---
+title: "Function Level Profiling"
+teaching: 0
+exercises: 0
+---
+
+:::::::::::::::::::::::::::::::::::::: questions
+
+- TODO
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::: objectives
+
+- TODO
+
+::::::::::::::::::::::::::::::::::::::::::::::::

profiling-introduction.md

@@ -0,0 +1,213 @@
+---
+title: "Introduction to Profiling"
+teaching: 0
+exercises: 0
+---
+
+:::::::::::::::::::::::::::::::::::::: questions
+
+- Why should you profile your code?
+- How should you choose which type of profiler to use?
+- Which test case should be profiled?
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::: objectives
+
+- explain the benefits of profiling code and different types of profiler
+- identify the appropriate Python profiler for a given scenario
+- explain how to select an appropriate test case for profiling and why
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+## Introduction
+
+<!-- Profiling is (what) -->
+Performance profiling is the process of analysing and measuring the performance of a program or script, to understand where time is being spent during execution.
+
+<!-- It can be used for (where) -->
+Profiling is useful when you have written any code that will be running for a substantial period of time.
+As your code grows in complexity, it becomes increasingly difficult to estimate where time is being spent during execution.
+Profiling allows you to narrow down where the time is being spent, to identify whether this is of concern or not.
+
+<!-- This allows enables faster/more (why) -->
+Profiling is a relatively quick process which can either provide you the peace of mind that your code is efficient, or highlight the performance bottleneck.
+Knowing the bottleneck allows you to optimise it (or more specifically request support in optimising it), potentially leading to significant speedups enabling faster research. In extreme cases, addressing bottlenecks has enabled programs to run hundreds or thousands of times faster!
+
+<!-- Increasingly, concern for green/eco compute and or cloud costs (why) -->
+Increasingly, particularly with relation to HPC, attention is being paid to the energy usage of software. Profiling your software will provide you the confidence that your software is an efficient use of resources.
+
+## When to Profile
+
+Profiling is most relevant to working code, when you have reached a stage that the code works and are considering deploying it.
+
+Any code that will run for more than a few minutes over it's lifetime, that isn't a quick one-shot script can benefit from profiling.
+
+Profiling should be a relatively quick and inexpensive process. If there are no significant bottlenecks in your code you can quickly be confident that your code is reasonably optimised. If you do identify a concerning bottleneck, further work to optimise your code and reduce the bottleneck could see significant improvements to the performance of your code and hence productivity.
+
+
+::::::::::::::::::::::::::::::::::::: callout
+
+## All Programmers Can Benefit
+
+<!-- Everyone benefits (why) -->
+Even professional programmers make oversights that can lead to poor performance, that can be identified through profiling.
+
+For example Grand Theft Auto Online, which has allegedly earned over $7bn since it's 2013 release, was notorious for it's slow loading times.
+8 years after it's release [a 'hacker'](https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/) had enough, they reverse engineered and profiled the code to enable a 70% speedup!
+
+*How much revenue did that unnecessary bottleneck cost, through user churn?*
+
+*How much time and energy was wasted, by unnecessarily slow loading screens?*
+
+:::::::::::::::::::::::::::::::::::::::::::::
+
+## Types of Profiler
+
+There are multiple approaches to profiling, most programming languages have one or more tools available covering these approaches.
+Whilst these tools differ, their core functionality can be grouped into four categories.
+
+### Manual Profiling
+
+Similar to using `print()` for debugging, manually timing sections of code can provide a rudimentary form of profiling.
+
+```Python
+import time
+
+t_a = time.monotonic()
+# A: Do something
+t_b = time.monotonic()
+# B: Do something else
+t_c = time.monotonic()
+# C: Do another thing
+t_d = time.monotonic()
+
+mainTimer_stop = time.monotonic()
+print(f"A: {t_b - t_a} seconds")
+print(f"B: {t_c - t_b} seconds")
+print(f"C: {t_d - t_c} seconds")
+```
+
+*Above is only one example of how you could manually profile your Python code, there are many similar techniques.*
+
+Whilst this can be appropriate for profiling narrow sections of code, it becomes increasingly impractical as a project grows in size and complexity.
+Furthermore, it's also unproductive to be routinely adding and removing these small changes if they interfere with the required outputs of a project.
+
+::::::::::::::::::::::::::::::::::::: callout
+
+## Benchmarking
+
+You may have previously used [`timeit`](https://docs.python.org/3/library/timeit.html) for timing Python code.
+
+This package returns the **total runtime** of an isolated block of code, without providing a more granular timing breakdown.
+Therefore, it is better described as a tool for **benchmarking**.
+
+:::::::::::::::::::::::::::::::::::::::::::::
+
+### Function-Level Profiling
+<!-- Context -->
+Software is typically comprised of a hierarchy of function calls, both functions written by the developer and those used from the core language and third party packages.
+
+<!-- What -->
+Function-level profiling analyses where time is being spent with respect to functions. Typically function-level profiling will calculate the number of times each function is called and the total time spent executing each function, inclusive and exclusive of child function calls.
+
+<!-- Why -->
+This allows functions that occupy a disproportionate amount of the total runtime to be quickly identified and investigated.
+
+<!-- We will be covering -->
+In this course we will cover the usage of the function-level profiler `cprofile` and how it's output can be visualised with `snakeviz`.
+
+### Line-Level Profiling
+<!-- Context -->
+Function-level profiling may not always be granular enough, perhaps your software is a single long script, or function-level profiling highlighted a particularly complex function.
+
+<!-- What -->
+Line-level profiling provides greater granularity, analysing where time is being spent with respect to individual lines of code. 
+
+<!-- Why -->
+This will identify individual lines of code that occupy an disproportionate amount of the total runtime.
+
+<!-- Caveat (too early to introduce this?) -->
+<!-- Typically, function-level profiling should be attempted first as it has a greater signal-to-noise ratio and is often significantly cheaper to perform. -->
+
+<!-- We will be covering -->
+In this course we will cover the usage of the line-level profiler `line_profiler`.
+
+### Timeline Profiling
+<!-- Context -->
+Timeline profiling takes a different approach to visualising where time is being spent during execution.
+
+<!-- What -->
+Typically a subset of function-level profiling, the execution of the profiled software is instead presented as a timeline highlighting the order of function execution in addition to the time spent in each individual function call.
+
+<!-- Why -->
+By highlighting individual functions calls, patterns relating to how performance scales over time can be identified. These would be hidden with the aforementioned aggregate approaches.
+
+<!-- We will be covering -->
+In this course we will cover the usage of the timeline profiler `viztracer`.
+
+### Hardware Metric Profiling
+
+Processor manufacturers typically release advanced profilers specific to their hardware with access to internal hardware metrics.
+These profilers can provide analysis of performance relative to theoretical hardware maximums (e.g. memory bandwidth or operations per second) and detail the utilisation of specific hardware features and operations.
+
+Using these advanced profilers requires a thorough understanding of the relevant processor architecture and may lead to hardware specific optimisations.
+
+Examples of these profilers include; Intel's VTune, AMD's uProf, and NVIDIA's Nsight Compute.
+
+Profiling of this nature is outside the scope of this course.
+
+
+## Selecting an Appropriate Test Case
+
+<!-- Profiling runs slower -->
+The act of profiling your code, collecting additional timing metrics during execution, will cause your program to execute slower. The slowdown is dependent on many variables, however the profiling covered by this course shouldn't more than double the runtime. <!-- Is this true? -->
+
+<!-- Profiling may generate large amounts of data -->
+Similarly, the longer your code runs, the more code that is being executed, the more data that will be collected. A profile that runs for hours could produce gigabytes of output data!
+
+<!-- Important to select appropriate test-case/s -->
+Therefore, it is important to select an appropriate test-case that is both representative of a typical workload and small enough that it can be quickly iterated.
+Ideally, it should take no more than a few minutes to run the profiled test-case from start to finish, however you may have circumstances where something that short is not possible.
+
+<!-- For example -->
+<!-- I don't really like this paragraph -->
+For example, you may have a model which normally simulates a year in hourly timesteps.
+It would be appropriate to begin by profiling the simulation of a single day.
+If the model scales over time, such as due to population growth, it may be pertinent to profile a single day later into a simulation if the model can be resumed or configured.
+A larger population is likely to amplify any bottlenecks that scale with the population, making them easier to identify.
+
+
+::::::::::::::::::::::::::::::::::::: discussion
+
+# Exercise (5 minutes)
+
+Think about a project where you've been working with Python.
+Do you know where the time during execution is being spent?
+
+Write a short plan of the approach you would take to investigate and confirm where the majority of time is being spent during it's execution.
+
+<!-- TODO should they share this anywhere, should it be discussed within the group? -->
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::: hint
+
+- What tools and techniques would be required?
+- Is there a clear priority to these approaches?
+- Which test-case/s would be appropriate?
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+
+::::::::::::::::::::::::::::::::::::: keypoints
+
+- Profiling is a relatively quick process to analyse where time is being spent and bottlenecks during a program's execution.
+- Code should be profiled when ready for deployment if it will be running for more than a few minutes during it's lifetime.
+- There are several types of profiler each with slightly different purposes.
+    - function-level: `cprofile` (visualised with `snakeviz`)
+    - line-level: `line_profiler`
+    - timeline: `viztracer`
+- A representative test-case should be profiled, that is large enough to amplify any bottlenecks whilst executing to completion quickly.
+
+::::::::::::::::::::::::::::::::::::::::::::::::

profiling-lines.md

@@ -0,0 +1,17 @@
+---
+title: "Line Level Profiling"
+teaching: 0
+exercises: 0
+---
+
+:::::::::::::::::::::::::::::::::::::: questions
+
+- TODO
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::: objectives
+
+- TODO
+
+::::::::::::::::::::::::::::::::::::::::::::::::

setup.md

@@ -2,53 +2,30 @@
 title: Setup
 ---
 
-FIXME: Setup instructions live in this document. Please specify the tools and
-the data sets the Learner needs to have installed.
-
+<!--
 ## Data Sets
 
-<!--
 FIXME: place any data you want learners to use in `episodes/data` and then use
        a relative link ( [data zip file](data/lesson-data.zip) ) to provide a
        link to it, replacing the example.com link.
--->
+       
 Download the [data zip file](https://example.com/FIXME) and unzip it to your Desktop
+-->
 
 ## Software Setup
 
 ::::::::::::::::::::::::::::::::::::::: discussion
 
 ### Details
 
-Setup for different systems can be presented in dropdown menus via a `solution`
-tag. They will join to this discussion block, so you can give a general overview
-of the software used in this lesson here and fill out the individual operating
-systems (and potentially add more, e.g. online setup) in the solutions blocks.
-
-:::::::::::::::::::::::::::::::::::::::::::::::::::
-
-:::::::::::::::: solution
-
-### Windows
-
-Use PuTTY
+This course uses Python and was developed using Python 3.11, therefore it is recommended that you have a Python 3.11 or newer environment.
 
-:::::::::::::::::::::::::
+<!-- Todo suggest using a venv?-->
 
-:::::::::::::::: solution
-
-### MacOS
-
-Use Terminal.app
-
-:::::::::::::::::::::::::
-
-
-:::::::::::::::: solution
-
-### Linux
-
-Use Terminal
-
-:::::::::::::::::::::::::
+The non-core Python packages required by the course are `snakeviz`, `line_profiler` and `viztracer` which can be installed via `pip`.
+
+```input
+pip install snakeviz line_profiler[all] viztracer[full]
+```
 
+:::::::::::::::::::::::::::::::::::::::::::::::::::