Download adapters and components #52
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sses feat(ticker splitter): component that splits tickers in groups test(ticker splitter)
… a list of possible parameter values
fix(Parameter validator): items() instead of enumerate() fix(Parameter validator): not more checking parameter is defined by default fix(match param gen): now returns the generated method test(Parameter validator)
… by adapter's parameters style(adapters): formatted file
…ve for all sub-components feat(Tradier): started adapter translation
feat(TickerSplitter): parametrised output ticker groups name feat(Request): added debug mode fix(Request): fixed status code, fixed json response test(tradier_timeseries)
feat(TickerExtractor): added type and length checks test(TickerExtractor)
feat(Pram valid.): added new datetime validation test(param valid): added dt valid. tests
del(TradierTimeseries)
… reuse the same adapter for different downloads
|
Ayo bro let me review that for you |
|
Ayo thanks bro |
CremaLuca
commented
Sep 11, 2021
CremaLuca
commented
Sep 11, 2021
CremaLuca
commented
Sep 11, 2021
fix: missing exception string format parameter
|
Still hard to read and decipher, needs some work. |
refactor: applied autopeep8 formatting
refactor: Adapter init initialises components list if not defined
ricdezen
reviewed
Sep 14, 2021
There was a problem hiding this comment.
Seems sort of fine overall, I only have two doubts I pointed out. Also I believe docstrings could be a bit more generous with explanations/examples. I don't think I would have understood what some of the components do if I could not have looked directly into the code.
added 8 commits
September 14, 2021 12:38
feat: changed tradier metadata to adapters
fix: time_handler handles non-existing seconds feat: Renaming component
… first parameter anymore
fix: date validation properly uses required fix: datetimetoepoch component won't translate datetime if it's None
added 10 commits
September 16, 2021 10:16
test: tradier checking provider and ticker in atoms test: fixed some adapter component tests
fix: Tradier checking header presence before using it
…nt lists delete: unused components, for clarity rename: TickerChunkComp to ChunkerComp (more generic) move: Parameter validation methods factory to validators file
refactor: SubAdapter does not make a difference between prep and ret components
test: updated components tests
|
Something that will need some work is how to pass a limiter to the request component at initialization, as of now the limiter is instantiated when the file is parsed and cannot be changed. |
added 3 commits
October 5, 2021 18:25
cleanup: removed RealtimeDownloader and TradierRealtime
CremaLuca
commented
Oct 11, 2021
CremaLuca
commented
Oct 11, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Premise
The download phase was very much specific for each provider, leading to difficulty in spotting bugs, hard to test code, and multiple failure points.
Features
feat(Adapter): Class that holds components, performs the prepare, retrieve and atomize phases calling each one of its components per phase.
feat(AdapterComponent): Implements one or more of the three phases of the download, just like a middleware. They should be as generic as possible, making them harder to test but suitable for more providers.
Components
Parameter validation: Every downloader needs parameter validation because the provider only supports some values (eg. interval "1min" or "1m"), so it's a component that given the parameter name, and a validation method applies the method to check for validity of the parameter's value.
Request: Performs bare HTTP requests, taking values from parameters or constant ones given in the constructor. It can build a URL by using
BASE_URL + url parameters + '?' + query parameters(not done manually, but by the requests module). Also supports HTTP headers, request timeout time, automatic JSON of the response, request limiter."Chunker": Splits an array of elements in an array of chunks of a given maximum size. eg
max_size=3then[A, B, C, D, E] - > [[A, B, C], [D, E]]. Used when the provider supports multiple tickers but sets a maximum number of them per request.Notes
I also wanted to keep downloaders and the filtering/streaming structures separated, in order for the downloader part to be an MVP.
Picture of something weird