Back-end refactor and data validation master issue

[jkwening]

This issue refers to back-end tasks to facilitate maintainability of the codebase and validation of the data served up to the front-end. In addition to the primary goals of refactoring the codebase and verifying and validating our existing data, outstanding unimplemented features/tasks related to expanding our data source and server maintenance and automation are included.

Refactor

• Refactor '_populate_zone_facts' to be cleaner (Refactor _populate_zone_facts to be cleaner #563)
  • see test_load_data.py failing due to refactored _populate_zone_facts #594, Inaccurate residential_units counts for at least three census tracts  #596, Passes sums of residential_units (by zone) to other LoadData methods. #556, and # 493 for more context
• Data ingestion workflow refactor - nearly completed in PR 470 IngestionMediator class #593.
  • Verify that row-level skipping when error occurs in ingestion workflow Add row-level skipping when error occurs to all classes in the sources folder #524
  • Refactor HILogger to better fit in new ingestion design pattern
    • currently aggregate log is hard to review, maybe consider subset of logs as needed per process: download, cleaning, and database upload?
  • Persist missing address id to avoid additional mar api calls later in cleaning process (Persist missing address id in create_project_subsidy_csv() when returned from mar api call #592)
• review remaining codebase for opportunities to simplify, streamline, and reduce repetition
  • Is manifest still the best way to track data sources and update?
  • Can we automate meta.json update workflow?

Verify, validate, and clean up existing data

• Write description of how we dedupe our sources of projects to share w/ office of planning
• Update data documentation for all tables currently in the database, and meta.json entries
• Zone-based data should not have null values (Zone-based data should not have null values #456)
• Accurately capture the percent of unites in a zone that are subsidized (Calculate percent subsidized by zone #564)
  • investigate issue with inaccurate residential_units counts (Inaccurate residential_units counts for at least three census tracts  #596)
• Track all sources that a project was found in during ingestion (Populate 'in_prescat', 'in_dhcd', 'in_dchousing' fields #577)
  • see Adds active_subsidized_units & percent_units_subsidized to zone_facts. #574 for more context
• Implement end-to-end unit testing of our data ingestion workflow
  • identify missing and incomplete unit tests
  • review and improve existing unit tests
• Investigate how to better handle invalid address issues (DHCD addresses not found in MAR #603)

Additional data sources

• Add ANC's to weighting; add ANC to zone facts.
• Add loan information from DHCD to the subsidy records being added from Quickbase
• Populate total unit counts at the project level DCHousing is missing total unit counts #495
• Add percent ami - DHCD data has this at project level; other prescat sources we can use a mapping based on program type that was created by Karynna.

Server

• Create a 'sources' class that checks an S3 bucket for new files and downloads them during get_api_data().
  • we have updated Prescat data that can be used to test this (Preservation Catalog November 2017 update #613)
• Handle API connection errors - 104 error (Handle error 104 Connection reset #512), 503 error (add "503" error handling #479)
• Add nested json of proj_addre to filter and project endpoints Add nested json of proj_addre to the /api/filter endpoint and /api/project endpoint #510
• Add all project fields to the filter api endpoint (Add all project fields to the filter api endpoint #585)
  • based on comment in Add some new front end fields to data choices #586 might already be resolved just needs to be deployed? Also reference
• CORS / Timeout: CORS / Timeout error #557

#500

• Protect API against sql injection (Protect api against sql injection #474)
  • in progress see PR Protect API - addressing issue 474 #567