Overview

This project is centered around data mining and parse json. I am using data from propublica, NewsAPI, Opensecrets, Trade.gov, CKan. I make a request to each API create a model from the json response and then create a python object from the parsed Json Response. I will use pandas and numpy to perform calculations on the data and persent the data to the use.

Third Party Libraries

1. NewsAPI - get news articles
2. JsonTraverseParser - parse json easier
3. scrapy - creating spiders
4. pyfolio - stock risk analysis
5. requests - REST Calls
6. gensim - NLP
7. scapy - NLP
8. NLTK- NPL toolkit
9. newspaper - extract articles and information from website
10. Civic Information - political data
11. Political - political data
12. Youtube - videos

Documentation

1. NewsAPI - https://newsapi.org/docs
2. UnData - http://data.un.org/ws/
3. Opensecrets - https://www.opensecrets.org/api/admin/index.php?function=user_api_list
4. Scrapy - https://docs.scrapy.org/en/latest/
5. pyfolio - https://quantopian.github.io/pyfolio/
6. gensim - https://radimrehurek.com/gensim/tutorial.html
7. scapy - https://spacy.io/api/doc
8. newspaper - https://github.com/codelucas/newspaper
9. Civic Information - https://developers.google.com/civic-information/docs/v2/
10. Youtube - https://developers.google.com/youtube/v3/code_samples/python

Helpful Links

1. Open Data - https://project-open-data.cio.gov/

Pyenv

source ~/.bashrc set pycharm to use 3.5.2, checking setting in pycharm and make sure it is using pyenv version. check local version of python, which python.