Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extension installation rework #4066

Merged
merged 19 commits into from
Aug 23, 2024
Merged

Extension installation rework #4066

merged 19 commits into from
Aug 23, 2024

Conversation

acquamarin
Copy link
Collaborator

@acquamarin acquamarin commented Aug 13, 2024 •
edited
Loading

This PR reworks the extension installation framework.
For extension which has dependencies (for example: duckdb), it should provide an additional loader and installer:
installer: installs the necessary dependency libraries.
loader: loads the necesssary dependency libraries.

@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: e90380ea89fd37a7fd29bf933d7e81ab1af51318
Branch commit hash: 1c080982aeca1cc19756a7242f94d2dd2ff9fa09

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 679.34 673.41 5.93 (0.88%)
aggregation q28 11645.21 11855.83 -210.62 (-1.78%)
filter q14 161.27 150.16 11.11 (7.40%)
filter q15 157.97 154.33 3.63 (2.36%)
filter q16 340.22 325.04 15.19 (4.67%)
filter q17 481.29 475.59 5.70 (1.20%)
filter q18 1970.39 1926.26 44.13 (2.29%)
fixed_size_expr_evaluator q07 575.08 564.10 10.97 (1.95%)
fixed_size_expr_evaluator q08 785.86 775.26 10.60 (1.37%)
fixed_size_expr_evaluator q09 784.90 773.71 11.19 (1.45%)
fixed_size_expr_evaluator q10 276.83 267.27 9.56 (3.58%)
fixed_size_expr_evaluator q11 273.10 262.27 10.83 (4.13%)
fixed_size_expr_evaluator q12 269.73 258.08 11.65 (4.51%)
fixed_size_expr_evaluator q13 1499.96 1489.55 10.40 (0.70%)
fixed_size_seq_scan q23 155.71 143.99 11.72 (8.14%)
join q31 12.27 11.80 0.47 (3.98%)
ldbc_snb_ic q35 809.92 766.22 43.70 (5.70%)
ldbc_snb_ic q36 43.99 46.78 -2.79 (-5.96%)
ldbc_snb_is q32 9.24 9.51 -0.28 (-2.89%)
ldbc_snb_is q33 17.82 18.00 -0.18 (-0.99%)
ldbc_snb_is q34 8.28 8.17 0.11 (1.39%)
multi-rel multi-rel-large-scan 2890.23 5573.63 -2683.39 (-48.14%)
multi-rel multi-rel-lookup 63.60 50.17 13.43 (26.77%)
multi-rel multi-rel-small-scan 59.16 74.62 -15.46 (-20.72%)
order_by q25 166.09 156.62 9.47 (6.05%)
order_by q26 479.18 478.48 0.70 (0.15%)
order_by q27 1450.07 1422.63 27.44 (1.93%)
scan_after_filter q01 209.57 200.39 9.18 (4.58%)
scan_after_filter q02 194.51 187.77 6.74 (3.59%)
shortest_path_ldbc100 q39 111.74 112.21 -0.47 (-0.42%)
var_size_expr_evaluator q03 2118.02 2078.03 40.00 (1.92%)
var_size_expr_evaluator q04 2265.86 2212.82 53.04 (2.40%)
var_size_expr_evaluator q05 2686.48 2731.49 -45.01 (-1.65%)
var_size_expr_evaluator q06 1410.11 1374.81 35.31 (2.57%)
var_size_seq_scan q19 1507.20 1467.09 40.11 (2.73%)
var_size_seq_scan q20 3164.72 3135.81 28.92 (0.92%)
var_size_seq_scan q21 2445.05 2402.99 42.06 (1.75%)
var_size_seq_scan q22 138.77 131.96 6.81 (5.16%)

@codecov Codecov
Copy link

codecov bot commented Aug 21, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 90.42553% with 9 lines in your changes missing coverage. Please review.

Project coverage is 84.14%. Comparing base (e76f943) to head (bb4fb2e).
Report is 1 commits behind head on master.

Files Patch % Lines
src/extension/extension.cpp 87.50% 7 Missing ⚠️
src/extension/extension_installer.cpp 83.33% 1 Missing ⚠️
src/processor/operator/simple/load_extension.cpp 92.85% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4066      +/-   ##
==========================================
+ Coverage   84.12%   84.14%   +0.01%     
==========================================
  Files        1314     1316       +2     
  Lines       51789    51860      +71     
  Branches     7187     7190       +3     
==========================================
+ Hits        43567    43636      +69     
- Misses       8067     8069       +2     
  Partials      155      155

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: e90380ea89fd37a7fd29bf933d7e81ab1af51318
Branch commit hash: 8ba8dff01dbb92722a14804f52da0712bd1a2f48

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 680.96 673.41 7.55 (1.12%)
aggregation q28 11548.19 11855.83 -307.64 (-2.59%)
filter q14 158.80 150.16 8.65 (5.76%)
filter q15 161.73 154.33 7.40 (4.80%)
filter q16 340.30 325.04 15.26 (4.70%)
filter q17 481.53 475.59 5.94 (1.25%)
filter q18 1990.01 1926.26 63.74 (3.31%)
fixed_size_expr_evaluator q07 574.49 564.10 10.38 (1.84%)
fixed_size_expr_evaluator q08 789.11 775.26 13.85 (1.79%)
fixed_size_expr_evaluator q09 783.81 773.71 10.09 (1.30%)
fixed_size_expr_evaluator q10 276.36 267.27 9.09 (3.40%)
fixed_size_expr_evaluator q11 272.55 262.27 10.28 (3.92%)
fixed_size_expr_evaluator q12 270.20 258.08 12.12 (4.70%)
fixed_size_expr_evaluator q13 1508.39 1489.55 18.84 (1.26%)
fixed_size_seq_scan q23 153.83 143.99 9.84 (6.83%)
join q31 12.24 11.80 0.44 (3.73%)
ldbc_snb_ic q35 752.32 766.22 -13.90 (-1.81%)
ldbc_snb_ic q36 39.37 46.78 -7.41 (-15.84%)
ldbc_snb_is q32 10.04 9.51 0.52 (5.48%)
ldbc_snb_is q33 16.07 18.00 -1.92 (-10.69%)
ldbc_snb_is q34 8.10 8.17 -0.07 (-0.82%)
multi-rel multi-rel-large-scan 2867.55 5573.63 -2706.08 (-48.55%)
multi-rel multi-rel-lookup 78.84 50.17 28.67 (57.14%)
multi-rel multi-rel-small-scan 54.33 74.62 -20.29 (-27.19%)
order_by q25 162.69 156.62 6.07 (3.87%)
order_by q26 480.04 478.48 1.56 (0.33%)
order_by q27 1450.57 1422.63 27.94 (1.96%)
scan_after_filter q01 208.11 200.39 7.72 (3.85%)
scan_after_filter q02 194.88 187.77 7.11 (3.79%)
shortest_path_ldbc100 q39 133.47 112.21 21.26 (18.94%)
var_size_expr_evaluator q03 2123.70 2078.03 45.68 (2.20%)
var_size_expr_evaluator q04 2280.42 2212.82 67.60 (3.05%)
var_size_expr_evaluator q05 2695.56 2731.49 -35.93 (-1.32%)
var_size_expr_evaluator q06 1413.85 1374.81 39.04 (2.84%)
var_size_seq_scan q19 1511.23 1467.09 44.14 (3.01%)
var_size_seq_scan q20 3175.60 3135.81 39.79 (1.27%)
var_size_seq_scan q21 2445.47 2402.99 42.47 (1.77%)
var_size_seq_scan q22 139.45 131.96 7.50 (5.68%)

@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: 7427dc8cb0383cef48762bc03f139fcae491fc85
Branch commit hash: cbfe6d6666fe7d8fb38f9dbb792a985a7a17824d

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 678.32 671.86 6.46 (0.96%)
aggregation q28 11662.69 11445.44 217.25 (1.90%)
filter q14 158.58 151.59 7.00 (4.62%)
filter q15 162.63 154.45 8.17 (5.29%)
filter q16 340.77 325.86 14.90 (4.57%)
filter q17 480.33 474.80 5.53 (1.16%)
filter q18 1969.40 1922.41 46.99 (2.44%)
fixed_size_expr_evaluator q07 577.77 563.90 13.86 (2.46%)
fixed_size_expr_evaluator q08 782.95 771.68 11.27 (1.46%)
fixed_size_expr_evaluator q09 786.20 772.71 13.49 (1.75%)
fixed_size_expr_evaluator q10 276.41 264.03 12.38 (4.69%)
fixed_size_expr_evaluator q11 271.44 259.23 12.21 (4.71%)
fixed_size_expr_evaluator q12 269.72 258.56 11.16 (4.32%)
fixed_size_expr_evaluator q13 1504.31 1482.59 21.72 (1.46%)
fixed_size_seq_scan q23 155.37 145.87 9.50 (6.51%)
join q31 11.74 11.88 -0.14 (-1.16%)
ldbc_snb_ic q35 1072.42 1534.01 -461.59 (-30.09%)
ldbc_snb_ic q36 43.48 45.72 -2.24 (-4.90%)
ldbc_snb_is q32 8.05 8.52 -0.47 (-5.53%)
ldbc_snb_is q33 14.75 17.60 -2.85 (-16.19%)
ldbc_snb_is q34 8.03 8.26 -0.23 (-2.72%)
multi-rel multi-rel-large-scan 2912.73 2780.40 132.34 (4.76%)
multi-rel multi-rel-lookup 72.44 57.59 14.85 (25.79%)
multi-rel multi-rel-small-scan 52.16 49.19 2.97 (6.04%)
order_by q25 166.11 157.96 8.15 (5.16%)
order_by q26 479.30 479.33 -0.03 (-0.01%)
order_by q27 1449.51 1429.79 19.72 (1.38%)
scan_after_filter q01 208.20 198.50 9.69 (4.88%)
scan_after_filter q02 193.71 188.16 5.54 (2.95%)
shortest_path_ldbc100 q39 117.48 115.41 2.07 (1.79%)
var_size_expr_evaluator q03 2115.48 2065.29 50.20 (2.43%)
var_size_expr_evaluator q04 2285.86 2224.53 61.33 (2.76%)
var_size_expr_evaluator q05 2687.63 2703.70 -16.08 (-0.59%)
var_size_expr_evaluator q06 1412.93 1373.94 38.99 (2.84%)
var_size_seq_scan q19 1501.04 1471.75 29.29 (1.99%)
var_size_seq_scan q20 3151.04 3135.13 15.91 (0.51%)
var_size_seq_scan q21 2449.88 2407.74 42.14 (1.75%)
var_size_seq_scan q22 138.69 131.35 7.34 (5.59%)

@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: 700b10f71c4e9ea19b4cd7e57ec864a38033225e
Branch commit hash: 2bdf3c76738e88022d1ec4dd28dca462ea9d8957

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 690.84 668.53 22.30 (3.34%)
aggregation q28 11405.38 11953.42 -548.05 (-4.58%)
filter q14 164.43 144.67 19.77 (13.66%)
filter q15 165.83 147.65 18.18 (12.31%)
filter q16 348.26 321.50 26.75 (8.32%)
filter q17 492.50 465.22 27.28 (5.86%)
filter q18 2114.67 1993.28 121.39 (6.09%)
fixed_size_expr_evaluator q07 575.15 555.62 19.53 (3.51%)
fixed_size_expr_evaluator q08 790.86 777.35 13.52 (1.74%)
fixed_size_expr_evaluator q09 787.55 778.60 8.95 (1.15%)
fixed_size_expr_evaluator q10 280.75 257.71 23.03 (8.94%)
fixed_size_expr_evaluator q11 275.97 251.64 24.33 (9.67%)
fixed_size_expr_evaluator q12 271.46 250.60 20.86 (8.32%)
fixed_size_expr_evaluator q13 1501.14 1484.57 16.57 (1.12%)
fixed_size_seq_scan q23 159.69 133.27 26.42 (19.83%)
join q31 12.63 11.39 1.24 (10.88%)
ldbc_snb_ic q35 911.52 771.29 140.24 (18.18%)
ldbc_snb_ic q36 50.95 48.19 2.77 (5.74%)
ldbc_snb_is q32 9.37 8.69 0.68 (7.80%)
ldbc_snb_is q33 18.61 15.30 3.31 (21.65%)
ldbc_snb_is q34 8.41 7.73 0.68 (8.84%)
multi-rel multi-rel-large-scan 2971.97 2881.53 90.44 (3.14%)
multi-rel multi-rel-lookup 72.14 66.20 5.94 (8.97%)
multi-rel multi-rel-small-scan 56.00 53.93 2.07 (3.84%)
order_by q25 168.06 148.91 19.16 (12.86%)
order_by q26 487.64 464.11 23.53 (5.07%)
order_by q27 1473.52 1422.50 51.02 (3.59%)
scan_after_filter q01 210.30 192.10 18.21 (9.48%)
scan_after_filter q02 195.72 182.94 12.77 (6.98%)
shortest_path_ldbc100 q39 164.07 97.14 66.92 (68.89%)
var_size_expr_evaluator q03 2153.28 2071.60 81.68 (3.94%)
var_size_expr_evaluator q04 2275.50 2263.46 12.04 (0.53%)
var_size_expr_evaluator q05 2725.64 2624.07 101.57 (3.87%)
var_size_expr_evaluator q06 1436.99 1337.83 99.16 (7.41%)
var_size_seq_scan q19 1529.05 1484.97 44.08 (2.97%)
var_size_seq_scan q20 3214.18 3194.70 19.48 (0.61%)
var_size_seq_scan q21 2453.41 2430.11 23.30 (0.96%)
var_size_seq_scan q22 139.59 133.10 6.49 (4.88%)

ray6080
ray6080 approved these changes Aug 22, 2024
View reviewed changes
Makefile
@@ -171,7 +171,6 @@ extension-test-build:
$(call run-cmake-release, \
-DBUILD_EXTENSIONS="httpfs;duckdb;json;postgres;sqlite" \
-DBUILD_EXTENSION_TESTS=TRUE \
-DENABLE_ADDRESS_SANITIZER=TRUE \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove asan from extension test?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duckdb doesn't compile with asan, if we compile with asan linking with duckdb will cause a linking error in windows.

test/c_api/database_test.cpp Show resolved Hide resolved
@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: e76f9436587ffe61b10ce147fcd7e892420b20ac
Branch commit hash: c0565e312dd0984adf1b6ea18efe47082e96d553

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 680.81 674.29 6.52 (0.97%)
aggregation q28 11141.29 11774.66 -633.37 (-5.38%)
filter q14 160.22 152.05 8.17 (5.38%)
filter q15 157.48 150.94 6.55 (4.34%)
filter q16 339.04 331.63 7.41 (2.23%)
filter q17 484.58 472.36 12.22 (2.59%)
filter q18 1974.50 1880.61 93.89 (4.99%)
fixed_size_expr_evaluator q07 583.51 562.66 20.85 (3.71%)
fixed_size_expr_evaluator q08 799.80 771.77 28.03 (3.63%)
fixed_size_expr_evaluator q09 795.02 771.53 23.49 (3.04%)
fixed_size_expr_evaluator q10 277.04 266.69 10.35 (3.88%)
fixed_size_expr_evaluator q11 272.22 260.66 11.56 (4.44%)
fixed_size_expr_evaluator q12 271.86 258.35 13.51 (5.23%)
fixed_size_expr_evaluator q13 1522.52 1486.41 36.12 (2.43%)
fixed_size_seq_scan q23 153.98 143.77 10.22 (7.11%)
join q31 13.37 13.26 0.12 (0.88%)
ldbc_snb_ic q35 759.53 783.25 -23.72 (-3.03%)
ldbc_snb_ic q36 48.71 46.06 2.65 (5.75%)
ldbc_snb_is q32 9.24 10.65 -1.41 (-13.22%)
ldbc_snb_is q33 15.99 19.22 -3.23 (-16.81%)
ldbc_snb_is q34 8.22 9.03 -0.81 (-9.00%)
multi-rel multi-rel-large-scan 2826.79 2777.32 49.47 (1.78%)
multi-rel multi-rel-lookup 40.63 48.41 -7.78 (-16.07%)
multi-rel multi-rel-small-scan 49.41 50.50 -1.09 (-2.15%)
order_by q25 161.10 158.44 2.66 (1.68%)
order_by q26 482.73 483.32 -0.59 (-0.12%)
order_by q27 1457.44 1420.47 36.98 (2.60%)
scan_after_filter q01 208.67 197.17 11.50 (5.83%)
scan_after_filter q02 195.56 187.71 7.85 (4.18%)
shortest_path_ldbc100 q39 95.18 93.02 2.17 (2.33%)
var_size_expr_evaluator q03 2098.73 2081.97 16.76 (0.80%)
var_size_expr_evaluator q04 2282.18 2199.58 82.60 (3.76%)
var_size_expr_evaluator q05 2665.87 2740.26 -74.38 (-2.71%)
var_size_expr_evaluator q06 1416.58 1386.71 29.88 (2.15%)
var_size_seq_scan q19 1518.94 1474.07 44.87 (3.04%)
var_size_seq_scan q20 3073.92 3198.15 -124.23 (-3.88%)
var_size_seq_scan q21 2420.05 2418.20 1.85 (0.08%)
var_size_seq_scan q22 139.62 131.66 7.97 (6.05%)

@acquamarin acquamarin merged commit ef4e729 into master Aug 23, 2024
31 checks passed
@acquamarin acquamarin deleted the extension-installation-fix branch August 23, 2024 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ray6080 ray6080 ray6080 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@acquamarin @ray6080 @mewim