catalyst-cooperative · bendnorman · Dec 16, 2023 · Aug 30, 2023 · Sep 1, 2023 · Sep 1, 2023
diff --git a/README.rst b/README.rst
@@ -47,36 +47,81 @@ it's often difficult to work with. PUDL takes the original spreadsheets, CSV fil
 and databases and turns them into a unified resource. This allows users to spend more
 time on novel analysis and less time on data preparation.
 
-Who is PUDL for?
-----------------
-
 The project is focused on serving researchers, activists, journalists, policy makers,
-and small businesses that might not otherwise be able to afford access to this data from
-commercial sources and who may not have the time or expertise to do all the data
-processing themselves from scratch.
+and small businesses that might not otherwise be able to afford access to this data
+from commercial sources and who may not have the time or expertise to do all the
+data processing themselves from scratch.
 
 We want to make this data accessible and easy to work with for as wide an audience as
-possible: anyone from a grassroots youth climate organizers working with Google sheets
-to university researchers with access to scalable cloud computing resources and everyone
-in between!
+possible: anyone from a grassroots youth climate organizers working with Google
+sheets to university researchers with access to scalable cloud computing
+resources and everyone in between!
+
+PUDL is comprised of three core components:
+
+- **Raw Data Archives**
+
+  - PUDL `archives <https://github.com/catalyst-cooperative/pudl-archiver>`__
+    all the raw data inputs on `Zenodo <https://zenodo.org/communities/catalyst-cooperative/?page=1&size=20>`__
+    to ensure perminant, versioned access to the data. In the event that an agency
+    changes how they publish data or deletes old files, the ETL will still have access
+    to the original inputs. Each of the data inputs may have several different versions
+    archived, and all are assigned a unique DOI and made available through the REST API.
+    You can read more about the Raw Data Archives in the
+    `docs <https://catalystcoop-pudl.readthedocs.io/en/dev/intro.html#raw-data-archives>`__.
+- **ETL Pipeline**
+
+  - The ETL pipeline (this repo) ingests the raw archives, cleans them,
+    integrates them, and outputs them to a series of tables stored in SQLite Databases,
+    Parquet files, and pickle files (the Data Warehouse). Each release of the PUDL
+    Python package is embedded with a set of of DOIs to indicate which version of the
+    raw inputs it is meant to process. This process helps ensure that the ETL and it's
+    outputs are replicable. You can read more about the ETL in the
+    `docs <https://catalystcoop-pudl.readthedocs.io/en/dev/intro.html#the-etl-process>`__.
+- **Data Warehouse**
+
+  - The outputs from the ETL, sometimes called "PUDL outputs",
+    are stored in a data warehouse as a collection of SQLite and Parquet files so that
+    users can access the data without having to run any code. Learn more about how to
+    access the data `here <https://catalystcoop-pudl.readthedocs.io/en/dev/data_access.html>`__.
 
 What data is available?
 -----------------------
 
 PUDL currently integrates data from:
 
-* `EIA Form 860 <https://www.eia.gov/electricity/data/eia860/>`__: 2001 - 2022
-* `EIA Form 860m <https://www.eia.gov/electricity/data/eia860m/>`__: 2023-06
-* `EIA Form 861 <https://www.eia.gov/electricity/data/eia861/>`__: 2001 - 2022
-* `EIA Form 923 <https://www.eia.gov/electricity/data/eia923/>`__: 2001 - 2023-08
-* `EPA Continuous Emissions Monitoring System (CEMS) <https://campd.epa.gov/>`__: 1995 - 2022
-* `FERC Form 1 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annual>`__: 1994-2021
-* `FERC Form 714 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric/data>`__: 2006-2020
-* `US Census Demographic Profile 1 Geodatabase <https://www.census.gov/geographies/mapping-files/2010/geo/tiger-data.html>`__: 2010
+* **EIA Form 860**: 2001-2022
+  - `Source Docs <https://www.eia.gov/electricity/data/eia860/>`__
+  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/eia860.html>`__
+* **EIA Form 860m**: 2023-06
+  - `Source Docs <https://www.eia.gov/electricity/data/eia860m/>`__
+* **EIA Form 861**: 2001-2022
+  - `Source Docs <https://www.eia.gov/electricity/data/eia861/>`__
+  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/eia861.html>`__
+* **EIA Form 923**: 2001-2022
+  - `Source Docs <https://www.eia.gov/electricity/data/eia923/>`__
+  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/eia923.html>`__
+* **EPA Continuous Emissions Monitoring System (CEMS)**: 1995-2022
+  - `Source Docs <https://campd.epa.gov/>`__
+  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/epacems.html>`__
+* **FERC Form 1**: 1994-2021
+  - `Source Docs <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annual>`__
+  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/ferc1.html>`__
+* **FERC Form 714**: 2006-2020
+  - `Source Docs <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric/data>`__
+  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/ferc714.html>`__
+* **FERC Form 2**: 2021 (raw only)
+  - `Source Docs <https://www.ferc.gov/industries-data/natural-gas/industry-forms/form-2-2a-3-q-gas-historical-vfp-data>`__
+* **FERC Form 6**: 2021 (raw only)
+  - `Source Docs <https://www.ferc.gov/general-information-1/oil-industry-forms/form-6-6q-historical-vfp-data>`__
+* **FERC Form 60**: 2021 (raw only)
+  - `Source Docs <https://www.ferc.gov/form-60-annual-report-centralized-service-companies>`__
+* **US Census Demographic Profile 1 Geodatabase**: 2010
+  - `Source Docs <https://www.census.gov/geographies/mapping-files/2010/geo/tiger-data.html>`__
 
 Thanks to support from the `Alfred P. Sloan Foundation Energy & Environment
 Program <https://sloan.org/programs/research/energy-and-environment>`__, from
-2021 to 2024 we will be integrating the following data as well:
+2021 to 2024 we will be cleaning and integrating the following data as well:
 
 * `EIA Form 176 <https://www.eia.gov/dnav/ng/TblDefs/NG_DataSources.html#s176>`__
   (The Annual Report of Natural Gas Supply and Disposition)

diff --git a/devtools/debug-eia-etl.ipynb b/devtools/debug-eia-etl.ipynb
@@ -263,7 +263,7 @@
    "outputs": [],
    "source": [
     "%%time\n",
-    "asset_key = \"fuel_receipts_costs_eia923\"\n",
+    "asset_key = \"core_eia923__monthly_fuel_receipts_costs\"\n",
     "df = defs.load_asset_value(AssetKey(asset_key))\n",
     "\n",
     "df.head()"

diff --git a/devtools/debug-ferc1-etl.ipynb b/devtools/debug-ferc1-etl.ipynb
@@ -99,7 +99,7 @@
    },
    "outputs": [],
    "source": [
-    "ferc1_xbrl_raw_dfs[\"fuel_ferc1\"][\"duration\"].report_year"
+    "ferc1_xbrl_raw_dfs[\"core_ferc1__yearly_steam_plants_fuel_sched402\"][\"duration\"].report_year"
    ]
   },
   {
@@ -206,7 +206,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "table_name = \"other_regulatory_liabilities_ferc1\"\n",
+    "table_name = \"core_ferc1__yearly_other_regulatory_liabilities_sched278\"\n",
     "TRANSFORMER = transformers[table_name]  # add a table here"
    ]
   },
@@ -326,8 +326,8 @@
    "source": [
     "transformed_tables = {}\n",
     "for table_name, transformer in transformers.items():\n",
-    "    if table_name == \"plants_steam_ferc1\":\n",
-    "        # plants_steam_ferc1 is a special case. It depends on the transformed fuel_ferc1 table.\n",
+    "    if table_name == \"core_ferc1__yearly_steam_plants_sched402\":\n",
+    "        # core_ferc1__yearly_steam_plants_sched402 is a special case. It depends on the transformed core_ferc1__yearly_steam_plants_fuel_sched402 table.\n",
     "        continue\n",
     "    transformed_tables[transformer.table_id.value] = transformer.transform(\n",
     "        raw_dbf=ferc1_dbf_raw_dfs[transformer.table_id.value],\n",
@@ -345,13 +345,13 @@
    },
    "outputs": [],
    "source": [
-    "# Handle special case for \"plants_steam_ferc1\"\n",
-    "transformer = transformers[\"plants_steam_ferc1\"]\n",
+    "# Handle special case for \"core_ferc1__yearly_steam_plants_sched402\"\n",
+    "transformer = transformers[\"core_ferc1__yearly_steam_plants_sched402\"]\n",
     "transformed_tables[transformer.table_id.value] = transformer.transform(\n",
     "    raw_dbf=ferc1_dbf_raw_dfs[transformer.table_id.value],\n",
     "    raw_xbrl_instant=ferc1_xbrl_raw_dfs[transformer.table_id.value][\"instant\"],\n",
     "    raw_xbrl_duration=ferc1_xbrl_raw_dfs[transformer.table_id.value][\"duration\"],\n",
-    "    transformed_fuel=transformed_tables[\"fuel_ferc1\"],\n",
+    "    transformed_fuel=transformed_tables[\"core_ferc1__yearly_steam_plants_fuel_sched402\"],\n",
     ")"
    ]
   }
@@ -372,7 +372,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.5"
   }
  },
  "nbformat": 4,

diff --git a/devtools/ferc1-eia-glue/training_data/create_FERC1-EIA_manual_mapping_spreadsheets.ipynb b/devtools/ferc1-eia-glue/training_data/create_FERC1-EIA_manual_mapping_spreadsheets.ipynb
@@ -43,7 +43,7 @@
     "# Local libraries\n",
     "import pudl\n",
     "from pudl.workspace.setup import PudlPaths\n",
-    "from pudl.analysis.ferc1_eia_train import *"
+    "from pudl.analysis.eia_ferc1_train import *"
    ]
   },
   {

diff --git a/...rc1-eia-glue/training_data/validate_and_integrate_FERC1-EIA_manually_mapped_records.ipynb b/...rc1-eia-glue/training_data/validate_and_integrate_FERC1-EIA_manually_mapped_records.ipynb
@@ -44,7 +44,7 @@
     "# Local libraries\n",
     "import pudl\n",
     "from pudl.workspace.setup import PudlPaths\n",
-    "from pudl.analysis.ferc1_eia_train import *"
+    "from pudl.analysis.eia_ferc1_train import *"
    ]
   },
   {
@@ -188,7 +188,7 @@
    "outputs": [],
    "source": [
     "current_training_df = pd.read_csv(\n",
-    "    importlib.resources.files(\"pudl.package_data.glue\").joinpath(\"ferc1_eia_train.csv\")\n",
+    "    importlib.resources.files(\"pudl.package_data.glue\").joinpath(\"eia_ferc1_train.csv\")\n",
     ")\n",
     "path_to_overrides = \"./add_to_training/\"\n",
     "override_files = [\n",
@@ -326,10 +326,10 @@
     "# Get paths to CSVs.\n",
     "from importlib import resources\n",
     "one_to_many = path_to_one_to_many=resources.files(\"pudl.package_data.glue\").joinpath(\n",
-    "                \"ferc1_eia_one_to_many.csv\",\n",
+    "                \"eia_ferc1_one_to_many.csv\",\n",
     "            )\n",
     "nulls = path_to_one_to_many=resources.files(\"pudl.package_data.glue\").joinpath(\n",
-    "                \"ferc1_eia_null.csv\",\n",
+    "                \"eia_ferc1_null.csv\",\n",
     "            )"
    ]
   },

diff --git a/devtools/inspect-assets.ipynb b/devtools/inspect-assets.ipynb
@@ -50,10 +50,61 @@
     "\n",
     "from pudl.etl import defs\n",
     "\n",
-    "asset_key = \"raw_generator_existing_eia860\"\n",
+    "asset_key = \"exploded_balance_sheet_assets_ferc1\"\n",
     "df = defs.load_asset_value(AssetKey(asset_key))\n",
     "\n",
-    "df.head()"
+    "#df[df.row_type_xbrl == \"correction\"].xbrl_factoid.value_counts()\n",
+    "#df[(df.xbrl_factoid.isin([\"operation_expense\", \"maintenance_expense\"]))&(df.rel_diff.notnull())&(df.rel_diff!=0)].sort_values(['utility_id_ferc1', 'report_year', 'xbrl_factoid', 'rel_diff']).head(50)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b2d99594",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df[(df.xbrl_factoid==\"accumulated_depreciation\")&(df.plant_status==\"in_service\")&(df.plant_function==\"total\")]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "467111b1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df[df.xbrl_factoid.isin(factoids)&(df.utility_id_ferc1==9)&(df.report_year==1998)]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c6f7427a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "factoids = ['distribution_maintenance_expense_electric',\n",
+    " 'hydraulic_power_generation_maintenance_expense',\n",
+    " 'maintenance_of_general_plant',\n",
+    " 'nuclear_power_generation_maintenance_expense',\n",
+    " 'other_power_generation_maintenance_expense',\n",
+    " 'regional_market_maintenance_expense',\n",
+    " 'steam_power_generation_maintenance_expense',\n",
+    " 'transmission_maintenance_expense_electric']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "951b718d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "asset_key = \"calculation_components_xbrl_ferc1\"\n",
+    "calcs = defs.load_asset_value(AssetKey(asset_key))\n",
+    "\n",
+    "calcs[(calcs.xbrl_factoid_parent == \"accumulated_depreciation\")].head(50)"
    ]
   },
   {
@@ -77,10 +128,24 @@
     "\n",
     "from pudl.etl import defs\n",
     "\n",
-    "asset_key = \"fuel_receipts_costs_eia923\"\n",
+    "asset_key = \"emissions_unit_ids_epacems\"\n",
     "df = defs.load_asset_value(AssetKey(asset_key))\n",
     "\n",
-    "df.head()"
+    "df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9f0d118b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pudl.output.epacems import epacems\n",
+    "\n",
+    "test_epacems = epacems(states = [\"ID\"], years = [2022])\n",
+    "\n",
+    "test_epacems[test_epacems.operating_datetime_utc>=\"2022-01-04\"].head(40)"
    ]
   }
  ],
@@ -100,7 +165,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.11.5"
   }
  },
  "nbformat": 4,

diff --git a/devtools/python-output-table-conversion-debug.ipynb b/devtools/python-output-table-conversion-debug.ipynb
@@ -48,14 +48,14 @@
     "\n",
     "You can create an asset by creating a new function and adding the `@asset` decorator. For now, the only attribute you should add to the decorator is the `compute_type = \"Python\"`. All this does is add a cute tag to the asset in the dag to let people know how the asset is being processed.\n",
     "\n",
-    "Next you'll want to figure out what tables the output table depends on. Read through the old output function to see which normalized tables or output functions are being used as inputs to the joins and imputations. Once you have the input table names, add them to the asset function parameters. For example, the `utilities_eia860()` function merges `utilities_entity_eia`, `utilities_eia860`, and `utilities_eia` tables together so the asset would look like this:\n",
+    "Next you'll want to figure out what tables the output table depends on. Read through the old output function to see which normalized tables or output functions are being used as inputs to the joins and imputations. Once you have the input table names, add them to the asset function parameters. For example, the `utilities_eia860()` function merges `core_eia__entity_utilities`, `core_eia860__scd_utilities`, and `core_pudl__assn_eia_pudl_utilities` tables together so the asset would look like this:\n",
     "\n",
     "```python\n",
     "@asset(compute_kind=\"Python\")\n",
     "def denorm_utilities_eia860(\n",
-    "    utilities_entity_eia: pd.DataFrame,\n",
-    "    utilities_eia860: pd.DataFrame,\n",
-    "    utilities_eia: pd.DataFrame,\n",
+    "    core_eia__entity_utilities: pd.DataFrame,\n",
+    "    core_eia860__scd_utilities: pd.DataFrame,\n",
+    "    core_pudl__assn_eia_pudl_utilities: pd.DataFrame,\n",
     "):\n",
     "    ... # joining logic\n",
     "    return joined_df\n",
@@ -108,9 +108,9 @@
     "```python\n",
     "@asset(io_manager_key=\"pudl_sqlite_io_manager\", compute_kind=\"Python\")\n",
     "def denorm_utilities_eia860(\n",
-    "    utilities_entity_eia: pd.DataFrame,\n",
-    "    utilities_eia860: pd.DataFrame,\n",
-    "    utilities_eia: pd.DataFrame,\n",
+    "    core_eia__entity_utilities: pd.DataFrame,\n",
+    "    core_eia860__scd_utilities: pd.DataFrame,\n",
+    "    core_pudl__assn_eia_pudl_utilities: pd.DataFrame,\n",
     "):\n",
     "    ... # joining logic\n",
     "    return joined_df\n",

diff --git a/docs/data_access.rst b/docs/data_access.rst
@@ -2,12 +2,17 @@
 Data Access
 =======================================================================================
 
-We publish the :doc:`PUDL pipeline <intro>` outputs in several ways to serve
+We publish the PUDL pipeline outputs in several ways to serve
 different users and use cases. We're always trying to increase accessibility of the
 PUDL data, so if you have a suggestion please `open a GitHub issue
 <https://github.com/catalyst-cooperative/pudl/issues>`__. If you have a question you
 can `create a GitHub discussion <https://github.com/orgs/catalyst-cooperative/discussions/new?category=help-me>`__.
 
+PUDL's primary data output is the ``pudl.sqlite`` database. We recommend working with
+tables with the ``out_`` prefix, as these tables contain the most complete and easiest
+to work with data. For more information about the different types
+of tables, read through :ref:`PUDL's naming conventions <asset-naming>`.
+
 .. _access-modes:
 
 ---------------------------------------------------------------------------------------