Merge branch 'main' into lateral

* main: (22 commits)
  add s3 creds to monthly workflow for dra upload
  add missing variable to monthly download workflow
  fix #496 (#595)
  Elk crossing update 2024-11-05 (#594)
  Added migration folder v0.5.3 (#592)
  CARR-Update-2024-11-04 (#593)
  QUES-Update-2024-11-01 (#591)
  more description clarification
  clearer intro of feature type/connectivity params
  fix #588
  STUL_STUR_MIDR_TAKL_UTRE_LTRE_modelled_fix (#586)
  fix #584
  swap CH/CO descriptions, fix #582
  fix #583, include CH in MIDR/TAKL watershed groups as per CWF direction
  BULK-Updates-Habitat-2024-10-23 (#581)
  BULK-Updates-2024-10-23 (#580)
  Fixed spelling error and missing variable assignment in auto_rank.py (#579)
  Additions for Tuzist'ol T'ah WCRP (#578)
  write modelled crossings cache and fptwg output to object storage, re… (#574)
  HORS_2024-10-08 (#577)
  ...

.github/workflows/production-bcfishpass.yaml

@@ -8,7 +8,11 @@ env:
   DATABASE_URL: ${{ secrets.DATABASE_URL }}
   AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
   AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
-  AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
+  AWS_ENDPOINT_URL: https://nrs.objectstore.gov.bc.ca
+  AWS_S3_ENDPOINT: nrs.objectstore.gov.bc.ca
+  AWS_VIRTUAL_HOSTING: FALSE
+  AWS_EC2_METADATA_DISABLED: True
+  CPL_VSISTDIN_BUFFER_LIMIT: -1
 jobs:
   load-csv:
     runs-on: ubuntu-latest

.github/workflows/production-load-monthly.yaml

@@ -6,6 +6,14 @@ on:
   workflow_dispatch:
 env:
   DATABASE_URL: ${{ secrets.DATABASE_URL }}
+  DRAPWD: ${{ secrets.DRAPWD }}
+  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+  AWS_ENDPOINT_URL: https://nrs.objectstore.gov.bc.ca
+  AWS_S3_ENDPOINT: nrs.objectstore.gov.bc.ca
+  AWS_VIRTUAL_HOSTING: FALSE
+  AWS_EC2_METADATA_DISABLED: True
+  CPL_VSISTDIN_BUFFER_LIMIT: -1
 jobs:
   data-refresh:
     runs-on: ubuntu-latest
@@ -15,4 +23,6 @@ jobs:
       - name: Check out repository code
         uses: actions/checkout@v4
       - name: Run the jobs
-        run: jobs/load_monthly
+        run: |
+          jobs/replicate_dra
+          jobs/load_monthly

data/pscis_modelledcrossings_streams_xref.csv

@@ -2066,9 +2066,9 @@ stream_crossing_id,modelled_crossing_id,linear_feature_id,watershed_group_code,r
 123694,24703652,,BULK,,
 123695,1801535,,BULK,,
 123696,1801540,,BULK,,
-123697,,17039631,BULK,,
+123697,,,BULK,,
 123698,1801539,,BULK,,
-123699,,17040032,BULK,,
+123699,,,BULK,,
 123700,1800741,,BULK,,
 123701,1800334,,BULK,,
 123702,24700294,,BULK,,

data/user_habitat_classification.csv

@@ -5196,8 +5196,6 @@
 356345430,807.1,12637.43,TAKL,SK,spawning,t,SN,2023-08-17,Pacific Salmon Foundation,
 356345430,12586,12686,TAKL,SK,spawning,t,SN,2023-08-17,FISS spawning points,
 356345496,0,156.21,LNIC,ST,spawning,t,SN,2023-08-11,Pacific Salmon Foundation,
-356345496,2149,2249,LNIC,CH,spawning,t,SN,2023-08-17,FISS spawning points,
-356345496,2149,2249,LNIC,CO,spawning,t,SN,2023-08-17,FISS spawning points,
 356345543,109.96,552.14,LSAL,CH,spawning,t,SN,2023-08-11,Pacific Salmon Foundation,
 356345574,150.6,4938.43,UFRA,CH,spawning,t,SN,2023-08-11,Pacific Salmon Foundation,
 356345574,4888,4988,UFRA,CH,spawning,t,SN,2023-08-17,FISS spawning points,
@@ -15303,3 +15301,10 @@
 355995081,0,4932,LNIC,CH,spawning,t,LB,2024-07-30,Pacific Salmon Foundation,
 355995081,0,4932,LNIC,CO,spawning,t,LB,2024-07-30,Pacific Salmon Foundation,
 355995081,0,4932,LNIC,ST,spawning,t,LB,2024-07-30,Pacific Salmon Foundation,
+360871856,963,1786.330265,BULK,ch,spawning,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2022,Gramophone Creek
+360871856,963,2637,BULK,ch,rearing,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2023,Gramophone Creek
+360871856,963,2637,BULK,co,rearing,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2026,Gramophone Creek
+360871856,963,2637,BULK,co,spawning,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2024,Gramophone Creek
+360736668,0,99,BULK,co,spawning,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2025,Wiggs Creek
+360736668,0,99,BULK,co,rearing,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2027,Wiggs Creek
+360884108,1057,1397,BULK,co,rearing,f,LB,2024-10-23,Bulkley Watershed Fish Passage Restoration Planning 2028,Side channel off Gramophone Creek

data/user_pscis_barrier_status.csv

@@ -1350,4 +1350,4 @@ stream_crossing_id,user_barrier_status,watershed_group_code,reviewer_name,review
 197860,PASSABLE,ELKR,MC,2024-06-27,Ford crossing based on assessment
 62245,PASSABLE,ELKR,MC,2024-06-27,This crossing has been rehabilitated
 62293,POTENTIAL,ELKR,MC,2024-06-27,"Channel blew out and formed a new channel through road. Not clear if this ""ford"" is passable."
-124249,PASSABLE,HORS,MC,2024-09-26,Was backwatered due to beaver dam. This is a barrier.
+124249,BARRIER,HORS,MC,2024-09-26,Scores as passable because currently backwatered by impassable beaver dam. Undersized. This is a barrier.

data/wcrp_watersheds.csv

@@ -1,14 +1,20 @@
-watershed_group_code,ch,cm,co,pk,sk,st,wct,notes
-ATNA,t,,t,,t,,,
-BELA,t,,t,,t,,,
-BONP,t,,t,,t,,,
-BOWR,t,,t,,t,,,
-BULK,t,,t,,t,t,,
-CARR,t,,t,,t,,,
-ELKR,,,,,,,t,
-HORS,t,,t,,t,,,
-LNIC,t,,t,,,t,,
-NECL,t,,t,,t,,,
-QUES,t,,t,,t,,,
-SHUL,t,,t,,t,,,
-USHU,t,,t,,t,,,
+watershed_group_code,ch,cm,co,pk,sk,st,wct,notes,wcrp
+ATNA,t,,t,,t,,,,bela_atna_necl
+BELA,t,,t,,t,,,,bela_atna_necl
+BONP,t,,t,,t,,,,tho_shu
+BOWR,t,,t,,t,,,,bowr_ques_carr
+BULK,t,,t,,t,t,,,bulk
+CARR,t,,t,,t,,,,bowr_ques_carr
+ELKR,,,,,,,t,,elkr
+HORS,t,,t,,t,,,,hors
+LNIC,t,,t,,,t,,,lnic
+NECL,t,,t,,t,,,,bela_atna_necl
+QUES,t,,t,,t,,,,bowr_ques_carr
+SHUL,t,,t,,t,,,,tho_shu
+USHU,t,,t,,t,,,,tho_shu
+TAKL,t,,,,t,,,,tuzistol_tah
+MIDR,t,,,,t,,,,tuzistol_tah
+UTRE,t,,,,t,,,,tuzistol_tah
+LTRE,t,,,,t,,,,tuzistol_tah
+STUL,t,,,,t,,,,tuzistol_tah
+STUR,t,,,,t,,,,tuzistol_tah

data/wsg_species_presence.csv

@@ -133,7 +133,7 @@ MESC,,t,,,,,,,t,,,,
 MESI,t,,,,,t,t,,t,,,,Mackenzie basin
 MFRA,,t,,,,,,,t,,,,
 MFRT,,,,,,,t,,,,,,Mackenzie basin
-MIDR,t,,,,,t,,,t,t,,,
+MIDR,t,t,,,,t,,,t,t,,,
 MILL,,,,,,,t,,,,,,Mackenzie basin
 MMUS,t,,,,,t,t,,t,,,,Mackenzie basin
 MORI,,,t,t,t,t,,t,t,t,t,,
@@ -198,7 +198,7 @@ SWIR,t,t,,,,t,t,,,,,,
 TABR,t,t,,,,t,,,t,,,,
 TAHR,t,,,,t,t,,,t,t,t,,
 TAHS,,t,t,t,t,t,,t,t,t,t,,
-TAKL,t,,,,,t,,,t,t,,,
+TAKL,t,t,,,,t,,,t,t,,,
 TASR,t,,,,,t,,,t,t,,,
 TATR,,t,,,,t,t,,t,t,,,
 TAYR,t,,,,,t,,,t,,t,,

db/v0.5.3/migrate.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+set -euxo pipefail
+
+# add wcrp name column to wcrp table
+psql $DATABASE_URL -v ON_ERROR_STOP=1 -c "alter table bcfishpass.wcrp_watersheds add column wcrp varchar(32);"
+
+# add wcrp tracking views
+echo "On systems supporting CWF WCRP reporting, add join_tracking_table_crossings_view.sql"
+echo "psql $DATABASE_URL -v ON_ERROR_STOP=1 -f sql/join_tracking_table_crossings_view.sql"
+
+# note version
+psql $DATABASE_URL -v ON_ERROR_STOP=1 -c "update bcfishpass.db_version set tag = '${PWD##*/}'"

db/v0.5.3/sql/join_tracking_table_crossings_vw.sql

@@ -0,0 +1,24 @@
+-- This function joins the tracking table to bcfishpass.crossings_wcrp_vw on the barrier ID
+create or replace function wcrp_hors.join_tracking_table_crossings_wcrp_vw(p_wcrp text)
+	returns void
+as
+$$
+begin 
+	execute format('create or replace view wcrp_%I.combined_tracking_table_crossings_wcrp_vw_%I as
+				   select 
+				   	cv.*, 
+				   	tt.*
+				   from bcfishpass.crossings_wcrp_vw cv
+				   join wcrp_%I.combined_tracking_table_%I tt 
+				   	on tt.barrier_id = cv.aggregated_crossings_id', p_wcrp, p_wcrp, p_wcrp, p_wcrp);
+end
+$$
+language plpgsql;
+
+-- loop through each WCRP and create a view for each one
+with data (wcrp) as (
+	select wcrp
+	from bcfishpass.wcrp_watersheds
+)
+select wcrp_hors.join_tracking_table_crossings_wcrp_vw(wcrp)
+from data;

db/v0.5.4/migrate.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -euxo pipefail
+
+# update DRA table
+psql $DATABASE_URL -v ON_ERROR_STOP=1 -f sql/dra.sql
+
+# note version
+psql $DATABASE_URL -v ON_ERROR_STOP=1 -c "update bcfishpass.db_version set tag = '${PWD##*/}'"

db/v0.5.4/sql/dra.sql

@@ -0,0 +1,81 @@
+-- drop non public columns from dra transport line
+BEGIN;
+
+ALTER TABLE whse_basemapping.transport_line 
+  drop column create_integration_session_id,
+  drop column create_integration_date,
+  drop column modify_integration_session_id,
+  drop column modify_integration_date,
+  drop column create_partner_org_id,
+  drop column create_partner_org,
+  drop column modify_partner_org_id,
+  drop column modify_partner_org,
+  drop column custodian_partner_org_id,
+  drop column z_value_derived_ind,
+  drop column deactivation_date,
+  drop column transport_line_divided_code,
+  drop column travel_direction_code,
+  drop column speed_limit,
+  drop column left_number_of_lanes,
+  drop column right_number_of_lanes,
+  drop column under_construction_ind,
+  drop column virtual_ind,
+  drop column disaster_route_ind,
+  drop column truck_route_ind,
+  drop column left_locality_id,
+  drop column left_locality,
+  drop column right_locality_id,
+  drop column right_locality,
+  drop column left_regional_district_id,
+  drop column right_regional_district_id,
+  drop column structured_name_1_id,
+  drop column structured_name_2_id,
+  drop column structured_name_3_id,
+  drop column structured_name_4_id,
+  drop column structured_name_5_id,
+  drop column structured_name_6_id,
+  drop column structured_name_6,
+  drop column structured_name_7_id,
+  drop column structured_name_7,
+  drop column highway_route_2,
+  drop column highway_route_3,
+  drop column industry_name_1,
+  drop column industry_name_2,
+  drop column industry_name_3,
+  drop column single_house_number,
+  drop column left_house_num_scheme_code,
+  drop column from_left_house_number,
+  drop column to_left_house_number,
+  drop column right_house_num_scheme_code,
+  drop column from_right_house_number,
+  drop column to_right_house_number,
+  drop column lane_restriction_code,
+  drop column access_restriction_code,
+  drop column from_traffic_impactor_code,
+  drop column to_traffic_impactor_code,
+  drop column from_left_turn_time_code,
+  drop column from_centre_turn_time_code,
+  drop column from_right_turn_time_code,
+  drop column to_left_turn_time_code,
+  drop column to_centre_turn_time_code,
+  drop column to_right_turn_time_code,
+  drop column from_vehicle_max_weight_kg,
+  drop column to_vehicle_max_weight_kg,
+  drop column from_vehicle_max_width_metre,
+  drop column to_vehicle_max_width_metre,
+  drop column from_vehicle_max_height_metre,
+  drop column to_vehicle_max_height_metre,
+  drop column ministry_of_transport_id,
+  drop column ministry_of_transport_name,
+  drop column integration_notes,
+  drop column excluded_rules,
+  drop column demographic_ind,
+  drop column extended_data,
+  drop column ministry_of_transport_data,
+  drop column from_navigation_rules,
+  drop column along_navigation_rules,
+  drop column to_navigation_rules,
+  drop column from_transport_node_point_id,
+  drop column to_transport_node_point_id;
+
+COMMIT;

docs/02_model_access.md

@@ -26,9 +26,13 @@ Natural barriers identified by stakeholders include:
 (gradient_barriers)=
 ## 2. Generate gradient barriers
 
-FWA stream network lines hold standardized Z values; each vertex of a stream line holds an associated elevation value derived from the BC Digital Elevation Model. Absolute elevation accuracy is subject to error in the DEM, but all elevations have been processed to ensure *relative* elevation is clean - all streams flow downhill. With these clean Z values, we can confidently calculate a reasonable estimate of the gradient of a stream at any point.
+FWA stream network lines hold standardized Z values; each vertex of a stream line holds an associated elevation value derived from the BC Digital Elevation Model. Absolute elevation accuracy is subject to error in the DEM, but all elevations have been processed to ensure *relative* elevation is clean - all streams flow downhill (see the [Technical Specification](ftp://ftp.gdbc.gov.bc.ca/sections/outgoing/bmgs/FWA_Public//Documents/FWA-SegmentElevation.TechnicalSpecification.v0.7.pdf) for details). With these clean Z values, we can confidently calculate a reasonable estimate of the gradient of a stream at any point.
 
 To identify locations where a stream's slope exceeds a given threshold, the model starts at the mouth of a stream (identified by the `blue_line_key`) and iterates through each vertex of the stream flow line.  At each vertex, it calculates the slope of the stream from the vertex to 100m upstream. Wherever the measured slope exceeds the value of the given threshold(s), this location and slope is recorded as a potential 'gradient barrier'.
+100m was chosen as the interval because:
+
+- 100m is the traditional minimum length used for measuring slope and defining reaches in BC, as per the [Fish Stream Guidebook](https://www2.gov.bc.ca/assets/gov/environment/plants-animals-and-ecosystems/fish-data-information/fishstream.pdf)
+- smoothing the FWA stream profile over 100m removes small irregularities in the data (spikes)
 
 The gradient threshold used for a given model is species dependent. Thresholds applied to existing models are:
 
@@ -137,6 +141,12 @@ TRIM features were delineated through air photo interpretation and have varying
 
 Conversely, in the drier, interior portions of the province, TRIM may over-represent the number or magnitude of streams.  These may be ephemeral or intermittent streams which only have water in them at the wettest times of the year.  Field crews often report finding only a ‘dry draw’ at locations where a stream has been shown on the mapping.
 
+Also, while Z values of FWA geometries have been smoothed to ensure that all streams flow downhill, elevation errors are still present:
+- error present in the DEM will also be present in the FWA
+- resolution of elevation is limited by the resolution of the DEM
+- resolution of elevation is further limited by the smoothing/cleaning processing
+- despite the smoothing to FWA elevations, some small elevations spikes remain
+
 ### Known barriers and observations
 
 The barrier and fish observation inventories have similar limitations.  Fish observation data is based on the results of field sampling and fish collection permits.  The distribution and scale of this work has typically been driven by development requirements in the resource industries.  As such, the model is based on unevenly distributed data points.  Areas that have seen more development and resource extraction tend to have a higher density of sampling and observation points while other, less developed areas tend to have sparse or non-existent fish observation and barrier data.
@@ -182,7 +192,7 @@ Model inputs and outputs are refreshed on a scheduled basis, and processed via g
 
 The weekly model processing workflow generates a file based data package (as geopackage), posted to object storage for public distribution:
 
-- download link: [freshwater_fish_habitat_accessibility_MODEL.gpkg.zip](https://bcfishpass.s3.us-west-2.amazonaws.com/freshwater_fish_habitat_accessibility_MODEL.gpkg.zip)
+- download link: [freshwater_fish_habitat_accessibility_MODEL.gpkg.zip](https://nrs.objectstore.gov.bc.ca/bchamp/freshwater_fish_habitat_accessibility_MODEL.gpkg.zip)
 
 - general metadata: [BC Data Catalogue](https://catalogue.data.gov.bc.ca/dataset/freshwater-fish-habitat-accessibility-model-pacific-salmon-and-steelhead)