Author: Kelly Barnes, Director of Assessment
Created: October 2023 Last Updated: October 2023
The goal of this handbook is to provide a basic introduction to the purpose of assessment and how to do it.
It is specifically focused on the goals, tools, and workflows we use at The Carpentries. It is intended to ensure that everyone in the organisation is following a consistent approach to assessment.
At The Carpentries we are “Always Learning”. We embody the growth mindset we teach using data and community feedback to measure our impact and foster continuous improvement. Gathering community feedback and data, analyzing and synthesizing that information, and drawing conclusions is the domain of assessment.
- The reader should gain:
- an understanding of The Carpentries assessment philosophy
- resources to conduct assessment (e.g., list and description of the tools we use, workflows, tips and tricks, etc.)
- You are not meant to read this cover to cover. It is more intended to serve as a reference..
- Assessment, in the context of educational and organisational settings, refers to the systematic process of gathering, interpreting, and using information to understand and improve learning, performance, or organisational outcomes.
Why It's Critical Assessment is the bridge between educational or organisational intentions and outcomes. It allows stakeholders to understand if they're on the right path to achieving their goals or if course corrections are necessary.
Benefits
- Quality Improvement: Regular assessment ensures that programs and initiatives maintain high standards and continue to meet the changing needs of their audience.
- Accountability: Assessment results demonstrate to stakeholders, including donors, participants, and team members, that an organisation is responsible and outcome-driven.
- Informed Decision-making: Real-time data from assessments empower leaders to make evidence-based decisions.
- Formative Assessment: Ongoing assessment used to provide feedback and support during the learning or operational process. It informs adjustments in teaching strategies or organisational practices.
- Educational Example: In a Carpentries workshop, instructors might use quick polls or quizzes halfway through a session to gauge learner understanding and adjust the pace or content accordingly.
- Organisational Example: In a nonprofit's outreach campaign, feedback from an initial set of stakeholders can inform adjustments in messaging or strategy for subsequent phases.
- Summative Assessment: This evaluates learner or organisational achievement at the end of an instructional unit or project by comparing it against a benchmark or standard.
- Educational Example: A post-workshop survey at The Carpentries that measures participants' overall understanding and satisfaction.
- Organisational Example: At the end of a fiscal year, a non-profit might assess the overall success of a fundraising campaign by comparing the funds raised against its set target.
- Activity: Work performed to implement programs or services to our community.
- Impact: the measurable or observable effects, changes, or outcomes that occur as a result of specific actions, initiatives, or interventions. Impact can be positive or negative, intended or unintended, and can vary in scale from individual to systemic levels.
- Metric: Quantifiable piece of information used to measure change over time.
- Outcome: A specific result a program is intended to achieve.
- Output: Product or service that result from activities.
- Target: A clearly defined objective or planned result to be achived within a stated time, against which actual results can be compared.
- Theory of Change: A diagram that maps how our programmatic activities lead to change. It shows how the activities being conducted, the outputs produced, and the outcomes anticipated are all connected.
For a complete list of a assessment terms, please refer to the assessment glossary.
A Theory of Change (ToC) is a comprehensive description of how and why a desired change is expected to happen.
It states the activities, outcomes, and assumptions that connect an organisation's mission to its strategic goals. The ToC serves as a roadmap for both planning and assessment, helping organisations articulate the steps needed to achieve their objectives.
- Planning: A ToC provides a clear framework for developing metrics that measure an organisation's impact. It informs what kind of data should be collected, when, and how.
- Alignment: It ensures that everyone in the organisation is on the same page about what they are trying to achieve, the steps to get there, and how they'll know if they have.
- Accountability: By mapping out outcomes and the paths to reach them, a ToC serves as an accountability tool. Teams can regularly check whether they are on course to achieve the intended outcomes and take corrective action if needed.
- Communication: A well-articulated ToC can be used to communicate with external stakeholders like funders, partners, or volunteers to transparently share the organisation's goals and methods.
- Strategic Planning: A ToC informs the strategic plan by providing a detailed roadmap of actions and outcomes, helping to prioritize activities.
- Resource Allocation: Knowing what activities lead to which outcomes helps in better allocation of resources like time, money, and person time.
- Risk Assessment: By listing assumptions and external factors, a ToC can help in identifying risks that might derail the project, allowing for better risk management.
- Adaptability: A ToC isn’t set in stone; it's a dynamic tool that should be updated regularly. This makes it invaluable for planning, as it can adapt to new insights, challenges, or opportunities that arise.
A Theory of Change is an essential tool for organisations aiming for systemic, long-term impact, allowing for more effective planning, execution, and assessment.
- Identify Clear Objectives: Start by pinpointing the long-term goals you aim to achieve. These should align with your organisation's mission.
- Map Outcomes: Identify the outcomes that need to occur to achieve these objectives. Break them down into short-term, intermediate, and long-term outcomes.
- Specify Inputs and Activities: List the resources (inputs) and actions (activities) required to bring about these outcomes.
- Establish metrics: Decide on measurable metrics for each outcome, which will help in future assessments.
- Identify Assumptions: Note any assumptions that underlie your expectations and outcomes.
- Sequence the Steps: Arrange all elements (inputs, activities, outcomes) in a logical order to show causality and flow.
- Visualize: Create a visual representation (flowchart, diagram) to illustrate the Theory of Change.
- Alignment: Align your ToC with the organisation's missions and strategic objectives.
- SMART Goals: Make sure all objectives and outcomes are Specific, Measurable, Achievable, Relevant, and Time-bound.
- Accountability: Assign responsibilities for each activity and outcome.
- Review Periodically: Make it a habit to review and update the Theory of Change, especially after key milestones or evaluations.
- Documentation: Keep thorough records of how decisions were made during the creation process for future reference and adaptation.
Through community building, curriculum development, and workshops, The Carpentries aims to achieve specific, measurable outcomes: increased skill and confidence in data science and computational topics, high-quality co-created resources, and an engaged, inclusive community. These outcomes, in turn, contribute to broader impacts such as a positive reputation in open-source/open-science ecosystems, sustainable operations, and a network of organizations with shared values.
| Core Activities | Outcomes |
|---|---|
| Workshops & Instruction | Participation in Carpentries workshops increases individuals’ skills and confidence in data, computational, and reproducible research topics. |
| Community Building | The Carpentries has active and engaged members that sustain the core activities of the organisation. |
| Curriculum Development | Carpentries curricula are co-created, relevant, and high quality. |
| Supporting Activities | Outcomes |
| Business & Internal | The Carpentries has an organisational structure (management practices, policies, workflows, governance) that supports daily operations and allows us to pursue goals and priorities |
Qualitative Metrics These capture non-numerical information like feedback, observations, or subjective experiences.
- Advantages: Good for understanding context, behaviors, and opinions.
- Disadvantages: May be harder to analyze or quantify.
- Examples: Testimonials, case studies.
Quantitative Metrics Numerical data that can be easily measured and analyzed.
- Advantages: Easy to analyze and compare, good for establishing trends.
- Disadvantages: May lack nuance or context.
- Examples: Number of workshops held, participant attendance rates.
- Align with Goals: Ensure that your metrics directly contribute to achieving the organisation’s goals and objectives.
- Relevance to Theory of Change: Make sure your metrics align with the outcomes and indicators identified in your Theory of Change.
- Feasibility: Consider the resources needed to collect and analyze the metric. Is it doable?
- Sensitivity: The metric should be sensitive enough to reflect small but meaningful changes.
- Clarity: The metric should be easy to understand and interpret, both for team members and external stakeholders.
Checklist or Guidelines for Evaluating Potential Metrics
- Relevance: Does it align with the organisational goals and Theory of Change?
- Measurability: Can it be quantified or qualified in a meaningful way?
- Actionability: Will it drive decision-making or actions?
- Reliability: Is the data source consistent and trustworthy?
- Cost-effectiveness: Is it worth the resources required for data collection and analysis?
The Carpentries utilize a diverse range of data collection methods to understand, monitor, and enhance their educational programs and outreach initiatives.
Programmatic Data Collection (AMY Database)
The Carpentries employs the AMY database to programmatically gather and manage workshop data, instructor profiles, and other organisational details.
This central database streamlines administrative tasks, aids in reporting, and facilitates easy access to key metrics for The Carpentries' team.
AMY stands as the central repository for programmatic data within The Carpentries, capturing details such as workshop events, instructor certifications, and affiliated member organisations.
By consolidating this information, AMY offers a comprehensive overview of the organisation's activities, reach, and impact.
Please see the AMY documentation and database structure for more information.
Surveys
Direct feedback from participants is invaluable. The Carpentries conducts surveys before and after workshops to gauge participant expectations, satisfaction, and learning outcomes.
This continuous feedback loop enables The Carpentries to adapt content, identify areas for improvement, and measure workshop efficacy.
- Pre and Post Worshop Surveys
- Pre and Post Instructor Training Surveys
- Long-term Impact Survey
- Annual Community Survey
Please see the survey archive for the current (and past versions) of many of our surveys.
Survey Platforms
For feedback collection, the Carpentries uses Typeform and Google Forms. They facilitate the generation, distribution, and initial analysis of surveys.
These tools transform the process of gathering insights from participants and community members, making it efficient and structured.
- Typeform
- Google Forms
GitHub Metrics
The Carpentries uses GitHub not only to house our curricula. It's a collaborative space for project management, content development, and community discussions.
Metrics such as commit activity, issue and pull request creation, and community contributions offer insights into content development, collaboration dynamics, and community engagement.
With built-in analytics, the platform sheds light on how content evolves, how the community interacts, and which topics or issues gain traction.
Qualitative Methods and Tools
-
Focus Groups and Interviews
-
Taguette: A tool for qualitative data analysis
- Taguette is a free and open-source tool for doing qualitative data analysis.
- Ethical Data Collection
- Always seek consent before collecting personal data, especially from workshop participants.
- Provide clear information about how data will be used and ensure it's used only for its intended purpose.
- Data Privacy
- Respect participants' rights to their data. Avoid collecting unnecessary personal details and ensure data storage is secure.
- Regularly review stored data and purge outdated or irrelevant information.
- Ensuring Accuracy and Reliability
- Use tried-and-tested data collection tools and methods.
- Regularly audit data sources for errors or inconsistencies and rectify them promptly.
- Making Data Actionable
- Ensure that the data collected aligns with organisational goals.
- Data for the sake of data isn't beneficial. It should inform decisions, guide strategies, or provide measurable insights into The Carpentries' impact.
What is R? R is a popular open-source programming language designed specifically for statistical analysis and data visualization. Within the realm of data assessment, R has carved its niche due to its extensive library support, flexibility, and active community contributions.
Getting Started Please review the episode Introduction to R and RStudio from the R for Reproducible Scientific Analysis Software Carpentry Lesson for instructions on how to download, install, and get started with R and RStudio.
Common Uses in The Carpentries
- Data cleaning and pre-processing
- Data Visualization
- Statistical analysis to evaluate the impact of Carpentries' programs.
- What are some Basic Analysis Methods?
- These are foundational techniques to summarize, analyze, and interpret data.
- Overview of statistical methods relevant to assessment
- Methods such as mean, median, variance provide insights into data distribution, trends, and patterns.
- Importance of statistical analysis in interpreting data.
- Statistical analysis is crucial in interpreting data as it provides a structured framework to extract meaningful insights, identify patterns, make informed decisions, and validate hypotheses, ensuring that conclusions drawn are robust, valid, and not due to random chance.
- How to Conduct Statistical Analysis in R
- Using R, one can easily conduct a myriad of statistical analyses ranging from basic descriptive statistics to more complex inferential tests.
- Basic Operations in R:
- Filter: Isolate subsets of data based on specific criteria.
- Group by and Summarise: Aggregate data by certain variables and then derive summary metrics.
- Descriptive Statistics:
- Count: Tally of data points or specific occurrences.
- Mean: Arithmetic average of a data set.
- Median: Middle value in a sorted data set.
- Code snippets or example scripts.
- Interpretation and Reporting
- Understanding Statistical Outcomes:
- Results derived from statistical tests aren't mere numbers; they narrate the underlying story encapsulated within the data.
- When interpreting, it's essential to relate the outcome to the context, acknowledging the potential implications or consequences of the findings.
- Reporting in the Carpentries Context:
- Aim for clarity and conciseness, ensuring that findings are communicated in a manner that resonates with the audience's familiarity and interests.
- Whenever possible, contextualize the results within the larger Carpentries ecosystem, highlighting relevance and potential applications.
- Understanding Statistical Outcomes:
Introduction to interpreting raw data sets
Raw data, though abundant, often lacks context; understanding its nuances requires systematic interpretation techniques.
Recognizing patterns, outliers, and anomalies is pivotal for accurate insights.
Descriptive statistics for initial insights
Descriptive statistics like the mean, median, mode, variance, and standard deviation offer a summarized view of the data's main features.
These metrics help discern the central tendencies, variability, and dispersion in a data set.
Visual methods for data interpretation
- Graphs: Line or bar graphs can reveal trends and fluctuations over time or across categories.
- Histograms: Helpful in visualizing the distribution of a dataset, especially to discern its skewness.
Frequent mistakes during data interpretation
- Correlation vs. Causation: Just because two variables move in tandem doesn't mean one causes the other.
- Overfitting: Relying too heavily on your current dataset, without ensuring the model's general applicability.
- Ignoring Outliers: Disregarding anomalies can skew results.
Implications of these pitfalls
Misinterpretations can lead to flawed decisions, strategies, or policies, potentially causing financial or reputational harm.
Mistakes, once recognized, can undermine trust in data analytics.
Use a combination of text, tables, and visuals to convey findings. Opt for clarity over complexity; intricate charts or jargon-heavy text can confuse rather than clarify.
Results should be articulated in simple, straightforward language, avoiding ambiguities. Visuals should be self-explanatory, or accompanied by succinct legends or captions.
Internal Teams: Might appreciate more detail and technical jargon as they're familiar with the project's intricacies. External Stakeholders: Often prefer summaries, with a focus on results, implications, and actionable insights.
Report Look and Feel Keep in mind Carpentries Brand Identity, especially colors and fonts.
Report Structure Reports should follow a structured format to ensure clarity and easy comprehension.
Common structure:
- Introduction
- Methods
- Results
- Discussion
- Conclusion
Navigating the Structure
- Headings: Use distinct styles for headings and subheadings. This helps break the content and aids in scanning the document.
- Numbering: Use a clear numbering system, especially for large reports, so readers can refer to specific sections easily.
- Tables of Contents: Especially for lengthy reports, a clickable TOC can greatly improve navigability.
- The Executive Summary: A concise overview of the main findings and recommendations. Aimed at decision-makers who might not have the time to delve into the report's details.
RMarkdown for Reporting
RMarkdown integrates code, visuals, and text into a single document, making it an excellent tool for reproducible research and dynamic report generation. Steps:
- Install the rmarkdown package.
- Within RStudio, select File > New File > R Markdown.
- Choose your desired output format (HTML, PDF, Word).
- Embed R code chunks where needed.
- Knit the document to produce the report.
Consider the story you're trying to tell. For trends over time, line graphs are great. For comparing categories, bar graphs are often ideal. Creating Visuals:
R: The ggplot2 package in R is a powerful tool for creating sophisticated visuals. Other Tools: Google Sheets and Excel have built-in chart tools that are user-friendly for beginners.
Ensure every visual has a clear title and axis labels. Use contrasting colors, and always provide a legend. Consider colorblind readers by avoiding problematic color combos (e.g., red-green).
Work performed to implement programs or services to our community.
Process of assessing the success of our programs.
Level of result at a given time that provides a starting point for assessing changes.
the measurable or observable effects, changes, or outcomes that occur as a result of specific actions, initiatives, or interventions. Impact can be positive or negative, intended or unintended, and can vary in scale from individual to systemic levels.
Resources (employee time, funding, etc.) used to conduct activities and provide services.
Benefits, changes in behaviour, ultimate or long-term consequences for organisational or societal benefit (e.g., ).
Quantifiable piece of information used to measure change over time.
Process of collecting and analyzing information to track program outputs and progress toward desired outcomes.
A specific result a program is intended to achieve. See Short-Term, and Long-Term Outcomes.
Product or service that result from activities.
Immediate effect or response to the outputs.
A clearly defined objective or planned result to be achived within a stated time, against which actual results can be compared.
A diagram that maps how our programmatic activities lead to change. It shows how the activities being conducted, the outputs produced, and the outcomes anticipated are all connected.