Elasticsearch storage backend #25
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # .github/workflows/ci.yml # .idea/inspectionProfiles/Project_Default.xml # .idea/misc.xml # README.md # archive_query_log/legacy/cli/make.py # archive_query_log/legacy/results/test/generate_tests.py # archive_query_log/legacy/results/test/test_ask_serp_parsing.py # archive_query_log/legacy/results/test/test_brave_serp_parsing.py # archive_query_log/legacy/results/test/test_chefkoch_serp_parsing.py # archive_query_log/legacy/results/test/test_duckduckgo_serp_parsing.py # archive_query_log/legacy/results/test/test_ecosia_serp_parsing.py # archive_query_log/legacy/results/test/test_qwant_serp_parsing.py # data/selected-services.yaml # pyproject.toml # slurm-cli # slurm-setup
janheinrichmerker
changed the title
Signed-off-by: Jan Heinrich Reimer <[email protected]>
Signed-off-by: Jan Heinrich Reimer <[email protected]>
Signed-off-by: Jan Heinrich Reimer <[email protected]>
Simplify Dockerfile
Signed-off-by: Jan Heinrich Reimer <[email protected]>
Upstream changes
Signed-off-by: Jan Heinrich Reimer <[email protected]>
- removed boxes and added url and text in DirectAnswer - changed xpath str to xpaths List[str]
Direct answers
Signed-off-by: Jan Heinrich Merker <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Restucture the crawing and parsing to store structured data in Elasticsearch indices instead of in the file system.
Also store WARCs in S3 instead of raw files.
The new storage backends should be flexible enough to allow for re-parsing parts of the dataset without having to delete anything.
The second key requirement is to be able to scale up massively by only interacting with standard ES/S3 APIs instead of having to mount a shared file system on all nodes.