Enforce levelization in libxrpl with CMake #5111

thejohnfreeman · 2024-08-28T04:39:22Z

Adds two CMake functions:

add_module(library subdirectory): Declares an OBJECT "library" (a CMake abstraction for a collection of object files) with sources from the given subdirectory of the given library, representing a module. Isolates the module's headers by creating a subdirectory in the build directory, e.g. .build/tmp123, that contains just a symlink, e.g. .build/tmp123/basics, to the module's header directory, e.g. include/xrpl/basics, in the source directory, and putting .build/tmp123 (but not include/xrpl) on the include path of the module sources. This prevents the module sources from including headers not explicitly linked to the module in CMake with target_link_libraries.
target_link_modules(library scope modules...): Links the library target to each of the module targets, and removes their sources from its source list (so they are not compiled and linked twice).

Uses these functions to separate and explicitly link modules in libxrpl:

Level 01: beast
Level 02: basics
Level 03: json, crypto
Level 04: protocol
Level 05: resource, server

Define each "module" of libxrpl as a separate OBJECT library in CMake. Link each one only to modules of a lower level. Then link all of them into libxrpl itself.

codecov · 2024-08-28T05:02:23Z

Codecov Report

Attention: Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (9d58f11) to head (ad16bbd).
Report is 3 commits behind head on develop.

Files with missing lines	Patch %	Lines
include/xrpl/basics/SHAMapHash.h	0.0%	2 Missing ⚠️
src/xrpld/app/tx/detail/Payment.cpp	88.9%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5111     +/-   ##
=========================================
- Coverage     77.9%   77.9%   -0.0%     
=========================================
  Files          782     781      -1     
  Lines        66616   66618      +2     
  Branches      8161    8127     -34     
=========================================
- Hits         51902   51896      -6     
- Misses       14714   14722      +8

Files with missing lines	Coverage Δ
include/xrpl/basics/Number.h	`100.0% <ø> (ø)`
include/xrpl/basics/base_uint.h	`96.8% <100.0%> (+0.1%)`	⬆️
include/xrpl/basics/partitioned_unordered_map.h	`99.2% <100.0%> (+<0.1%)`	⬆️
include/xrpl/protocol/AmountConversions.h	`87.7% <ø> (ø)`
include/xrpl/protocol/FeeUnits.h	`90.4% <ø> (ø)`
include/xrpl/protocol/Fees.h	`100.0% <ø> (ø)`
include/xrpl/protocol/IOUAmount.h	`100.0% <ø> (ø)`
include/xrpl/protocol/LedgerHeader.h	`100.0% <ø> (ø)`
include/xrpl/protocol/MPTAmount.h	`100.0% <100.0%> (ø)`
include/xrpl/protocol/PayChan.h	`100.0% <ø> (ø)`
... and 36 more

... and 7 files with indirect coverage changes

ximinez · 2024-08-29T19:35:56Z

Not a review yet, but this is such a cool idea.

cmake/isolate_headers.cmake

cmake/add_module.cmake

Bronek · 2024-09-20T20:31:27Z

cmake/RippledCore.cmake

+  PRIVATE
+    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src>


I think this is probably not needed ?

actually, upon some testing, this whole target_include_directories seem to be not needed.

It's not needed now that all of the subdirectories in libxrpl are modules. It was necessary as I migrated them one-by-one, and it could be necessary in the future if someone adds a non-module subdirectory. Could be nice if it "just works" without tampering with the CMake. What do you think?

Perhaps comment out and put a comment above, to explain why this is not necessary and in what condition it might become necessary again ?

Bronek · 2024-09-20T20:52:36Z

I like how you moved extract to appropriate locations, but it seems that extract is function-wise the same as hash.

Given we already specialize both boost::hash (e.g. in IPEndpoint.h) and std::hash (e.g. in Book.h) I think that cleaning this up belongs in a different PR (also converting extract to hash), so maybe just take a note for future refactoring.

Bronek · 2024-09-23T13:43:46Z

I do wonder whether Builds/levelization is becoming superfluous with this change. Why should we keep it, if the ordering of modules (and header isolation) enforce it for us ?

include/xrpl/protocol/XRPAmount.h

src/libxrpl/basics/partitioned_unordered_map.cpp

Bronek

Overall very nice change. Some comments in the middle, unsure if they require changes in this PR or not.

thejohnfreeman · 2024-09-23T20:56:55Z

Builds/levelization is still used to diagnose cycles in xrpld. I want to remove it after we've eliminated all of the cycles.

Bronek · 2024-09-24T11:03:05Z

Builds/levelization is still used to diagnose cycles in xrpld. I want to remove it after we've eliminated all of the cycles.

Makes sense. All the more reason to make an effort moving parts of xrpld into libxrpl, but that's separate discussion and more than one PR.

Bronek

Nice improvement, thanks !

ximinez · 2024-10-24T22:48:35Z

Bad news right out the gate on Windows builds:

$ cmake build/cmake/msvc19.ON
-- Using Conan toolchain: C:/Dev/rippled/review1/build/conan.msvc19/build/generators/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 20 with extensions OFF
-- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.19045.
[...]
-- Conan: Component target declared 'xxHash::xxhash'
-- Conan: Including build module from 'C:/.conan/4b4fe1f3/1/lib/cmake/protobuf/protobuf-generate.cmake'
-- Conan: Including build module from 'C:/.conan/4b4fe1f3/1/lib/cmake/protobuf/protobuf-module.cmake'
-- Conan: Including build module from 'C:/.conan/4b4fe1f3/1/lib/cmake/protobuf/protobuf-options.cmake'
CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.beast/xrpl/beast':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:69 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.basics/xrpl/basics':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:76 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.json/xrpl/json':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:80 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.crypto/xrpl/crypto':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:83 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.protocol/xrpl/protocol':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:87 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.resource/xrpl/resource':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:94 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.server/xrpl/server':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:97 (add_module)
  CMakeLists.txt:123 (include)


-- xrpl.libxrpl.basics with 14 sources took 14 sources from xrpl.libxrpl
-- xrpl.libxrpl.beast with 15 sources took 15 sources from xrpl.libxrpl
-- xrpl.libxrpl.crypto with 3 sources took 3 sources from xrpl.libxrpl
-- xrpl.libxrpl.json with 9 sources took 9 sources from xrpl.libxrpl
-- xrpl.libxrpl.protocol with 51 sources took 51 sources from xrpl.libxrpl
-- xrpl.libxrpl.resource with 4 sources took 4 sources from xrpl.libxrpl
-- xrpl.libxrpl.server with 2 sources took 2 sources from xrpl.libxrpl
-- Configuring incomplete, errors occurred!

thejohnfreeman · 2024-10-29T22:50:31Z

@ximinez ok, try again.

ximinez · 2024-10-30T15:59:59Z

It's building successfully now. Neat workaround. Super annoying that this is an issue, though.

I remember there were some file(CREATE_LINK calls elsewhere that were simply disabled under Windows. Would it be worth coming back to those to use this same technique?

Otherwise, I'll come back to the rest of this review as soon as I get some other priorities knocked out.

godexsoft

This looks good 👍 Just one question - not a blocker by any means.

godexsoft · 2024-11-11T23:16:41Z

cmake/add_module.cmake

+# add_module(parent a)
+# add_module(parent b)
+# target_link_libraries(project.libparent.b PUBLIC project.libparent.a)
+function(add_module parent name)


Wouldn't it be even nicer if you could optionally pass the libraries to link the target with right here too? Is it possible to add the PUBLIC lib1 lib2 through args somehow?

Enforce levelization with CMake

e364dbd

Define each "module" of libxrpl as a separate OBJECT library in CMake. Link each one only to modules of a lower level. Then link all of them into libxrpl itself.

thejohnfreeman requested review from Bronek and ximinez August 28, 2024 04:39

thejohnfreeman added 2 commits September 6, 2024 13:03

Merge branch 'develop'

3560a56

Merge branch 'develop'

1368e2b

Bronek reviewed Sep 20, 2024

View reviewed changes

cmake/isolate_headers.cmake Outdated Show resolved Hide resolved

Bronek reviewed Sep 20, 2024

View reviewed changes

cmake/add_module.cmake Show resolved Hide resolved

Bronek reviewed Sep 20, 2024

View reviewed changes

Rename module isolation directory

4e11039

Bronek reviewed Sep 23, 2024

View reviewed changes

include/xrpl/protocol/XRPAmount.h Show resolved Hide resolved

Bronek reviewed Sep 23, 2024

View reviewed changes

include/xrpl/protocol/XRPAmount.h Show resolved Hide resolved

Bronek reviewed Sep 23, 2024

View reviewed changes

src/libxrpl/basics/partitioned_unordered_map.cpp Outdated Show resolved Hide resolved

Bronek reviewed Sep 23, 2024

View reviewed changes

thejohnfreeman added 3 commits September 24, 2024 08:44

Disable private headers for libxrpl

e0f3c56

Merge branch 'develop'

cbdd49f

Disable non-module headers for libxrpl

947440a

Bronek self-requested a review September 24, 2024 16:23

Bronek approved these changes Sep 24, 2024

View reviewed changes

thejohnfreeman added 2 commits October 16, 2024 12:52

Merge branch 'develop'

529af81

Merge branch 'develop'

0929033

Fix symlinks on non-admin Windows

b0a2105

Merge branch 'develop'

9cd6f06

thejohnfreeman added 3 commits November 2, 2024 14:23

Fix symbolic links of install

8890adc

Merge branch 'develop'

0e11fa7

Merge branch 'develop'

1f953dc

godexsoft approved these changes Nov 11, 2024

View reviewed changes

thejohnfreeman mentioned this pull request Nov 12, 2024

[feature] Single Asset Vault thejohnfreeman/rippled#10

Draft

Move IOUAmount from basics module to protocol

ad16bbd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enforce levelization in libxrpl with CMake #5111

Enforce levelization in libxrpl with CMake #5111

thejohnfreeman commented Aug 28, 2024 •

edited

Loading

codecov bot commented Aug 28, 2024 •

edited

Loading

ximinez commented Aug 29, 2024

Bronek Sep 20, 2024

Bronek Sep 20, 2024

thejohnfreeman Sep 20, 2024

Bronek Sep 23, 2024 •

edited

Loading

Bronek commented Sep 20, 2024

Bronek commented Sep 23, 2024

Bronek left a comment

thejohnfreeman commented Sep 23, 2024

Bronek commented Sep 24, 2024

Bronek left a comment

ximinez commented Oct 24, 2024

thejohnfreeman commented Oct 29, 2024

ximinez commented Oct 30, 2024

godexsoft left a comment

godexsoft Nov 11, 2024

Enforce levelization in libxrpl with CMake #5111

Are you sure you want to change the base?

Enforce levelization in libxrpl with CMake #5111

Conversation

thejohnfreeman commented Aug 28, 2024 • edited Loading

codecov bot commented Aug 28, 2024 • edited Loading

Codecov Report

ximinez commented Aug 29, 2024

Bronek Sep 20, 2024

Choose a reason for hiding this comment

Bronek Sep 20, 2024

Choose a reason for hiding this comment

thejohnfreeman Sep 20, 2024

Choose a reason for hiding this comment

Bronek Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

Bronek commented Sep 20, 2024

Bronek commented Sep 23, 2024

Bronek left a comment

Choose a reason for hiding this comment

thejohnfreeman commented Sep 23, 2024

Bronek commented Sep 24, 2024

Bronek left a comment

Choose a reason for hiding this comment

ximinez commented Oct 24, 2024

thejohnfreeman commented Oct 29, 2024

ximinez commented Oct 30, 2024

godexsoft left a comment

Choose a reason for hiding this comment

godexsoft Nov 11, 2024

Choose a reason for hiding this comment

thejohnfreeman commented Aug 28, 2024 •

edited

Loading

codecov bot commented Aug 28, 2024 •

edited

Loading

Bronek Sep 23, 2024 •

edited

Loading