Process Form 497 Page 2 (Late Expenditures) #31
Conversation
This is a somewhat complicated implementation, but out of necessity: Form 497 rows do not contain the "Sup_Opp_Cd" column needed to bucket an expenditure into either supporting or opposing a ballot measure. So we must guess. Also, I suspect this will double-count in some cases like Just Cause -- they gave money to a committee that is basically themselves. I don't think we should count this money twice. I opened #29 to track this.
This changes nothing, yet. Form 497 doesn't have an Expn_Code column so I've bucketed them all in "Not Stated".
This changes nothing, yet, because no candidates have data here.
| end | ||
| elsif sup_opp_cd == 'O' | ||
| opposing_by_measure_name[row['Bal_Name']] ||= [] | ||
| opposing_by_measure_name[row['Bal_Name']] << row |
There was a problem hiding this comment.
This doesn't look right, aren't you appending the row twice?
| @@ -29,6 +39,37 @@ def fetch | |||
| end | |||
| end | |||
|
|
|||
| late_expenditures.each do |row| | |||
| sup_opp_cd = guess_whether_committee_supports_measure(row['Filer_ID'], row['Bal_Name']) | |||
There was a problem hiding this comment.
Does this ever come up empty? In that case, we just don't count the money?
There was a problem hiding this comment.
seems like if we can't guess then we drop it. Its not likely to happen too often.
On a related note, I think I saw the same ballot measure referred by two different names.
There was a problem hiding this comment.
Yep, we drop it. I created #35 as a issue to track our tracking of how many records / how much money we skip because we can't find a match. We can keep adding more to that issue as we introduce more fuzzy matching into our processing.
| # Stated". | ||
| late_expenditures = ActiveRecord::Base.connection.execute(<<-SQL) | ||
| SELECT DISTINCT ON ("Filer_ID") | ||
| "Filer_ID", '' AS "Expn_Code", SUM("Amount") AS "Total" |
There was a problem hiding this comment.
Should '' be 'Not stated'? Or maybe null so it's easier to handle on the frontend?
|
Not sure I'm reading the Ruby right, hopefully my comments are helpful :) |
* origin/master: Update build artifacts for 9e5b9b8 Update contributions_received stub to zero Limit Form 497 results to contributions only Add Form 497 to candidate total contributions Add Form 497 to candidate contributions by type
| @_guess_cache ||= | ||
| begin | ||
| guesses = ActiveRecord::Base.connection.execute(<<-SQL) | ||
| SELECT "Filer_ID", "Bal_Name", "Sup_Opp_Cd" |
There was a problem hiding this comment.
You could use SELECT DISTINCT rather than GROUP BY, its probably its processed about the same but I think it is clearer that is what you are doing.
There was a problem hiding this comment.
Yeah, I didn't use the Report_Num here because it seemed unlikely to me that the Sup_Opp_Cd would change between amendments.
| @@ -29,6 +39,37 @@ def fetch | |||
| end | |||
| end | |||
|
|
|||
| late_expenditures.each do |row| | |||
| sup_opp_cd = guess_whether_committee_supports_measure(row['Filer_ID'], row['Bal_Name']) | |||
There was a problem hiding this comment.
seems like if we can't guess then we drop it. Its not likely to happen too often.
On a related note, I think I saw the same ballot measure referred by two different names.
This adds Late Expenditures to:
The first one is somewhat difficult since Form 497 has no "Sup_Opp_Cd" field, so we must infer it based on the Schedule E (Payments Made). (We could instead use Schedule D, but it seems like the data is more complete for E).
Not much changed for candidates, since there is only one candidate entry on the 497 and it is a contribution, not an expenditure.
[Fixes #27]