TSE data utilities

Scraps all the from the election results from Tribunal Superior Eleitoral files at:

https://www.tse.jus.br/eleicoes/eleicoes-2022/interessados-na-divulgacao-de-resultados-2022

Please read their policy and docs before using the tool.

TSE Docs:

https://www.tse.jus.br/eleicoes/eleicoes-2022/arquivos/interessados

https://www.tse.jus.br/eleicoes/eleicoes-2022/documentacao-tecnica-do-software-da-urna-eletronica

English terms took (mostly) from:

https://international.tse.jus.br/en/

Prerequisites

Python 3.6+
Python Poetry: https://python-poetry.org/docs/#installation
- Or use brew install poetry
Install the dependencies poetry install
Activate shell/virtual environment poetry shell

Usage

Run scrapy crawl divulga to update all the files
- Scrapping is incremental and can be peformed continuously to get latest data if available,
- Files are downloaded to data/download/...
Run scrapy crawl urna to download all the original files transmitted from the voting machines (bulletins, logs, etc),
- Beware that it's above 472k electoral sections with 6 files and about 200kb per section, totalling 2.8 million files 90gb
Edit tse/setting.py to customize paths, network usage, narrow down filters, etc.