Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-factor to simplify hats loading #461

Merged
merged 6 commits into from
Oct 29, 2024
Merged

Pre-factor to simplify hats loading #461

merged 6 commits into from
Oct 29, 2024

Conversation

delucchi-cmu
Copy link
Contributor

Complements (and requires) astronomy-commons/hats#394

Closes #460.

Puts all the file reading on loading a catalog into a single module. This reduces the complexity for reading the code, reduces code duplication, and also reduces the number of times we need to read the properties file, and perform file system stat calls to determine if files exist. This should speed up catalog loading on remote file systems where there is a latency associated with each file read.

Enough of the code has changed, that we no longer need the additional (and mostly unused) argument for catalog type, in order to appease mypy type checking.

Copy link

codecov bot commented Oct 25, 2024

Codecov Report

Attention: Patch coverage is 98.90110% with 1 line in your changes missing coverage. Please review.

Project coverage is 97.65%. Comparing base (f1a3713) to head (0ebb3e8).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/lsdb/loaders/hats/read_hats.py 98.83% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #461      +/-   ##
==========================================
- Coverage   97.71%   97.65%   -0.06%     
==========================================
  Files          44       39       -5     
  Lines        1619     1537      -82     
==========================================
- Hits         1582     1501      -81     
+ Misses         37       36       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Oct 25, 2024

Before [f1a3713] After [4aa3b83] Ratio Benchmark (Parameter)
failed 17.7±0.2ms n/a benchmarks.time_box_filter_on_partition
failed 14.4±0.08s n/a benchmarks.time_create_large_catalog
failed 2.16±0.01s n/a benchmarks.time_create_midsize_catalog
failed 105±0.8ms n/a benchmarks.time_kdtree_crossmatch
failed 32.4±0.7ms n/a benchmarks.time_polygon_search

Click here to view all benchmarks.

@hombit hombit linked an issue Oct 28, 2024 that may be closed by this pull request
3 tasks
@delucchi-cmu delucchi-cmu merged commit c9779aa into main Oct 29, 2024
10 of 11 checks passed
@delucchi-cmu delucchi-cmu deleted the maps branch October 29, 2024 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants