Sampling Distribution Dance Party!_copy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sampling Distributions Dance Party!\n",
    "Get your dancing shoes ready! You are a DJ trying to make sure you are ready for a big party. You don’t have time to go through all the songs you can work with. Instead, you want to make sure that any sample of 30 songs from your playlist will get the party started. To do this, you will use the power of sampling distributions!\n",
    "\n",
    "The dataset we are using for this project can be found here. For simplicity, we have removed some unnecessary columns.\n",
    "\n",
    "Note that a **helper_function**.py file is loaded for you in the workspace. This file contains functions that you will use throughout this project. A **solution.py** file is also loaded for you in the workspace, which contains solution code for this project. We highly recommend that you complete the project on your own without checking the solution, but feel free to take a look if you get stuck or want to check your answers when you’re done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helper_functions import choose_statistic, population_distribution, sampling_distribution\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy import stats\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Tasks\n",
    "\n",
    "### Loading in the Data\n",
    "1. You will be working with a dataset called **spotify_data.csv**. In **script.py**, use the `read_csv()` pandas function to load in **spotify_data.csv** into a variable called `spotify_data`.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To load in the data, write the following line of code:\n",
    "\n",
    "```py\n",
    "spotify_data = pd.read_csv('spotify_data.csv')\n",
    "```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 1: load in the spotify dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Use the pandas `.head()` function to preview the `spotify_data`. If you need a \n",
    "reminder of how to use this function, click the hint below.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To preview a dataframe using the `.head()` function, do the following:\n",
    "\n",
    "```py\n",
    "print(dataframe_variable.head())\n",
    "```\n",
    "In **script.py**, our `dataframe_variable` is `spotify_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 2: preview the dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. For this project, we are going to focus on the `tempo` variable. This column gives the beats per minute (bpm) of each song in **spotify.csv**. The other columns in our dataset are:\n",
    "\n",
    "    - `danceability`\n",
    "    - `energy`\n",
    "    - `instrumentalness`\n",
    "    - `liveness`\n",
    "    - `valences`\n",
    "    \n",
    "    For now, we are going to ignore these other columns.\n",
    "\n",
    "    Create a variable called `song_tempos` that contains the `tempo` column data.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To save the `tempo` column data in a variable called `population`, do the following:\n",
    "\n",
    "```py\n",
    "population = spotify_data['tempo']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 3: select the relevant column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper Functions\n",
    "4. Let’s investigate the helper functions we will use in the following sections. A file called **helper_functions.py** should be opened in the workspace for you. It contains three functions: `choose_statistic()`, `population_distribution()`, and `sampling_distribution()`. The code in these functions is similar to what we saw in the previous lesson, but let’s explore these together.\n",
    "\n",
    "    `choose_statistic()` allows us to choose a statistic we want to calculate for our sampling and population distributions. It contains two parameters:\n",
    "\n",
    "    - `x`: An array of numbers\n",
    "    - `sample_stat_text`: A string that tells the function which statistic to calculate on `x`. It takes on three values: “Mean”, “Minimum”, or “Variance”.\n",
    "    \n",
    "    `population_distribution()` allows us to plot the population distribution of a dataframe with one function call. It takes the following parameter:\n",
    "\n",
    "    - `population_data`: the dataframe being passed into the function\n",
    "    \n",
    "    `sampling_distribution()` allows us to plot a simulated sampling distribution of a statistic. The simulated sampling distribution is created by taking random samples of some size, calculating a particular statistic, and plotting a histogram of those sample statistics. It contains three parameters:\n",
    "\n",
    "    - `population_data`: the dataframe being sampled from\n",
    "    - `samp_size`: the size of each sample\n",
    "    - `stat`: the specific statistic being measured for each sample — either “Mean”, “Minimum”, or “Variance”\n",
    "    \n",
    "    Read through these functions in `helper_function.py` to familiarize yourself with them. Click the hint to see examples of `population_distribution()` and `sampling_distribution()` being used.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "Here is an example of how to use `population_distribution()`:\n",
    "\n",
    "```py\n",
    "# example function use case\n",
    "population_distribution(population)\n",
    "```\n",
    "Here is an example of how to use `sampling_distribution()`:\n",
    "\n",
    "```py\n",
    "# example function use case for sampling distribution of the mean\n",
    "sampling_distribution(population, \"Mean\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# task 4:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sampling Distribution Exploration\n",
    "5. Now that our data is loaded into **script.py** and we have gone over the functions in **helper_functions.py** let’s start our sampling distributions exploration. Make sure to write your code in **script.py**.\n",
    "\n",
    "    To start off, let’s use the `population_distribution()` function to graph distribution of `song_tempos`.\n",
    "\n",
    "    When you click run, you should see a graph with the following title:\n",
    "    ```py\n",
    "    Population Distribution\n",
    "    ```\n",
    "    How would you describe this distribution?\n",
    "\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To use the `population_distribution()` function, do the following:\n",
    "\n",
    "```py\n",
    "population_distribution(_____)\n",
    "```\n",
    "In the blank, you should put `song_tempos` because it is our population data.\n",
    "\n",
    "The population distribution is approximately normal with a little bit of right-skewness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 5: plot the population distribution with the mean labeled\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Now let’s plot the sampling distribution of the sample mean with sample sizes of 30 songs. To do this, use the `sampling_distribution()` helper function.\n",
    "\n",
    "    Once you hit run, you should see a graph with the following title:\n",
    "    ```py\n",
    "    Sampling Distribution of the Mean\n",
    "    Mean of the Sample Means: {Mean of the Sample Means} \n",
    "    Population Mean: {Population Mean}\n",
    "    ```\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To use the `sampling_distribution()` function, do the following:\n",
    "\n",
    "```py\n",
    "sampling_distribution(_____, _____, _____)\n",
    "```\n",
    "In the first blank, you should put `song_tempos` because it is our population data. In the second blank, you should put 30 since we want each sample to be of size `30`. In the last blank, you should put `\"Mean\"` since we want to analyze the sampling distribution of the sample mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 6: sampling distribution of the sample mean\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Compare your sampling distribution of the sample means to the population mean. Is the sample mean an unbiased or biased estimator of the population?\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "The mean is also an unbiased estimator as the mean of the sampling distribution of the mean is always approximately the same as the population mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 7:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8. Now let’s plot the sampling distribution of the sample minimum with sample sizes of 30 songs. To do this, use the `sampling_distribution()` helper function.\n",
    "\n",
    "    Once you hit run, you should see a graph with the following title:\n",
    "    ```py\n",
    "    Sampling Distribution of the Minimum\n",
    "    Mean of the Sample Minimums: {Mean of the Sample Minimums}\n",
    "    Population Mean: {Population Mean}\n",
    "    ```\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To use the `sampling_distribution()` function, do the following:\n",
    "\n",
    "```py\n",
    "sampling_distribution(_____, _____, _____)\n",
    "```\n",
    "In the first blank, you should put `song_tempos` because it is our population data. In the second blank, you should put `30` since we want each sample to be of size 30. In the last blank, you should put `\"Minimum\"` since we want to analyze the sampling distribution of the sample minimum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 8: sampling distribution of the sample minimum\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9. Compare your sampling distribution of the sample minimums to the population minimum. Is the sample minimum an unbiased or biased estimator of the population?\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "Notice that the mean of the sample minimums is consistently much higher than the population minimum. Since you are looking for high-tempo songs for the party, this is actually a good thing! You will want to avoid having a lot of low-tempo songs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 9:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10. Now let’s plot the sampling distribution of the sample variance with sample sizes of 30 songs. To do this, use the `sampling_distribution()` helper function.\n",
    "\n",
    "    Once you hit run, you should see a graph with the following title:\n",
    "    ```py\n",
    "    Sampling Distribution of the Variance\n",
    "    Mean of the Sample Variances: {Mean of the Sample Variances}\n",
    "    Population Variance: {Population Variance}\n",
    "    ```\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "To use the `sampling_distribution()` function, do the following:\n",
    "\n",
    "```py\n",
    "sampling_distribution(_____, _____, _____)\n",
    "```\n",
    "In the first blank, you should put `song_tempos` because it is our population data. In the second blank, you should put `30` since we want each sample to be of size 30. In the last blank, you should put `\"Variance\"` since we want to analyze the sampling distribution of the sample variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# task 10: sampling distribution of the sample variance\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11. Compare your sampling distribution of the sample variance to the population variance. Does the sample variance appear to be an unbiased or biased estimator of the population?\n",
    "\n",
    "    Click the hint for more information about sample variance.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "The mean of the sample variances is consistently slightly less than the population variance, meaning it is a biased estimator. However, it is super close. Let’s dig into this.\n",
    "\n",
    "We calculated the sample variance the same way we calculate population variance..\n",
    "\n",
    "However, the formulas for sample variance and population variance are actually distinct. As we have seen, population variance is calculated as:\n",
    "\n",
    "$$ population\\,variance = \\frac{\\sum(observation - \\\\mu)^2}{n} $$\n",
    "​ \n",
    "When we measure the sample variance using the same formula, it turns out that we tend to underestimate the population variance. Because of this, we divide by n-1 instead of n:\n",
    "\n",
    "$$ sample\\,variance = \\frac{\\sum(observation - sample\\,mean)^2}{n - 1} $$\n",
    "\n",
    "Using this formula, sample variance becomes an unbiased estimator of the population variance. Let’s apply this in the next task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 11:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "12. Go to line 17 in **helper_functions.py**. You should see the following line of code:\n",
    "    ```py\n",
    "    np.var(x)\n",
    "    ```\n",
    "    Change this to:\n",
    "    ```py\n",
    "    np.var(x, ddof=1)\n",
    "    ```\n",
    "    Adding this `ddof=1` parameter will divide our input by *n-1* instead of *n*, therefore applying the sample variance formula.\n",
    "\n",
    "    After changing this line of code, run **script.py**. Does the sample variance appear to be an unbiased or biased estimator of the population?\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "By changing the way we calculate sample variance, we have made it an unbiased parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 12:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating Probabilities\n",
    "13. We have a good sense of some sample statistics now that we’ve investigated sampling distributions. Let’s take our analysis further by calculating probabilities.\n",
    "\n",
    "    First, calculate the population mean and population standard deviation of `song_tempos`. Save these values in two separate variables called `population_mean` and `population_std`.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "Use the .mean() and .std() NumPy methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 13: calculate the population mean and standard deviation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "14. Use `population_mean` and `population_std` to calculate the standard error of the sampling distribution of the sample mean with a sample size of 30.\n",
    "\n",
    "    Save this value in a variable called `standard_error`.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "The formula for the standard error of the sample mean is:\n",
    "\n",
    "$$ standard~error = \\frac{standard~deviation}{square~root~of~the~sample~size}\n",
    "\n",
    "​\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 14: calculate the standard error\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "15. You are afraid that if the average tempo of the songs you randomly select is less than 140bpm that your party will not be enjoyable.\n",
    "\n",
    "    Using `population_mean` and `standard_error` in a CDF, calculate the probability that the sample mean of 30 selected songs is less than 140bpm.\n",
    "\n",
    "    Remember to print your result into the output terminal.\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "Use the `.cdf()` method from the SciPy library.\n",
    "\n",
    "As a reminder to use the `.cdf()` method, do the following:\n",
    "\n",
    "```py\n",
    "print(stats.norm.cdf(value_of_interest, mean, standard_error))\n",
    "```\n",
    "In this case, our `value_of_interest` is 140; our mean is `population_mean`; our `standard_error` is also called `standard_error`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 15: calculate the probability of observing an average tempo of 140bpm or lower from a sample of 30 songs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "16. You know the party will be truly epic if the randomly sampled songs have an average tempo of greater than 150bpm.\n",
    "\n",
    "    Using `population_mean` and `standard_error` in a CDF, calculate the probability that the sample mean of 30 selected songs is GREATER than 150bpm.\n",
    "\n",
    "    Remember to print your result into the output terminal.\n",
    "\n",
    "    Does this probability make you feel confident about the party?\n",
    "\n",
    "<details>\n",
    "<summary><b>Hint</b></summary>\n",
    "\n",
    "Use the `.cdf()` method from the SciPy library.\n",
    "\n",
    "As a reminder, to use the `.cdf()` method to calculate the probability of observing some value of interest or greater, do the following:\n",
    "\n",
    "```py\n",
    "print(1 - stats.norm.cdf(value_of_interest, mean, standard_error))\n",
    "```\n",
    "In this case, our `value_of_interest` is 150; our mean is `population_mean`; our `standard_error` is also called `standard_error`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task 16: calculate the probability of observing an average tempo of 150bpm or higher from a sample of 30 songs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extras\n",
    "17. Awesome job! You are ready to throw an awesome party! If you want to do some more exploration of sampling distributions, here are some more opportunities:\n",
    "\n",
    "    - Add another sample statistic to the `choose_statistic()` function in **helper_functions.py** — such as median, mode, or maximum.\n",
    "    - Explore a different column of data from the **spotify_data.csv** dataset.\n",
    "    - Use the sampling distribution of the sample minimum to estimate the probability of observing a specific sample minimum. For example, from the plot, what is the chance of getting a sample minimum that is less than 130bpm?\n",
    "    Happy coding!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EXTRA"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myenv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}