Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CMake build system for valkey #1196

Open
wants to merge 2 commits into
base: unstable
Choose a base branch
from

Conversation

eifrah-aws
Copy link

@eifrah-aws eifrah-aws commented Oct 20, 2024

Replacing this PR #1082

NOTE:

This PR was tested on:

  • Amazon Linux 2
  • Ubuntu 22.04
  • AlmaLinux 9
  • macOS Sonoma 14.7
  • FreeBSD 9 - note on FreeBSD, we require gmake for building valkey

With this PR, users are able to build valkey using CMake.

Example usage:

Build valkey-server in Release mode with TLS enabled and using jemalloc as the allocator:

mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DCMAKE_INSTALL_PREFIX=/tmp/valkey-install \
         -DWITH_MALLOC=jemalloc -DWITH_TLS=1
make -j$(nproc) install

# start valkey
/tmp/valkey-install/bin/valkey-server

Build valkey-unit-tests:

mkdir build-release-ut
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DWITH_MALLOC=jemalloc -DBUILD_TESTS=1
make -j$(nproc)

# Run the tests
./bin/valkey-unit-tests 

Current features supported by this PR:

  • Building against different allocators: (jemalloc, tcmalloc, tcmalloc_minimal and libc), e.g. to enable jemalloc pass -DWITH_MALLOC=jemalloc to cmake
  • OpenSSL builds (to enable TLS, pass -DWITH_TLS=1 to cmake)
  • Sanitizier: pass -DWITH_SANITIZER=<address|thread|undefined> to cmake
  • Install target + redis symbolic links
  • Build valkey-unit-tests executable
  • Standard CMake variables are supported. e.g. to install valkey under /home/you/root pass -DCMAKE_INSTALL_PREFIX=/home/you/root

Why using CMake? To list some of the advantages of using CMake:

  • Superior IDE integrations: cmake generates the file compile_commands.json which is required by clangd to get a compiler accuracy code completion (in other words: your VScode will thank you)
  • Out of the source build tree: with the current build system, object files are created all over the place polluting the build source tree, the best practice is to build the project on a separate folder
  • Multiple build types co-existing: with the current build system, it is often hard to have multiple build configurations. With cmake you can do it easily:
  • It is the de-facto standard for C/C++ project these days

More build examples:

ASAN build:

mkdir build-asan
cd $_
cmake .. -DWITH_SANITIZER=address -DWITH_MALLOC=libc
make -j$(nproc)

ASAN with jemalloc:

mkdir build-asan-jemalloc
cd $_
cmake .. -DWITH_SANITIZER=address -DWITH_MALLOC=jemalloc 
make -j$(nproc)

As seen by the previous examples, any combination is allowed and co-exist on the same source tree.

Valkey installation

With this new CMake, it is possible to install the binary by running make install or creating a package make package (currently supported on Debian like distros)

Example 1: build & install using make install:

mkdir build-release
cd $_
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/valkey-install -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) install
# valkey is now installed under $HOME/valkey-install

Example 2: create a .deb installer:

mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) package
# ... CPack deb generation output
sudo gdebi -n ./valkey_8.1.0_amd64.deb
# valkey is now installed under /opt/valkey

Example 3: create installer for non Debian systems (e.g. FreeBSD or macOS):

mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) package
mkdir -p /opt/valkey && ./valkey-8.1.0-Darwin.sh --prefix=/opt/valkey  --exclude-subdir
# valkey-server is now installed under /opt/valkey

Current CMake PR status:

  • Targets:
    • valkey-server
    • valkey-cli
    • valkey-benchmark
    • valkey-sentinel
    • RDMA module
    • TLS as module
    • valkey-unit-tests
    • valkey-check-aof
    • valkey-check-rdb
  • Features
    • SSL
    • Sanitizer
    • Different allocators (jemalloc, libc, tcmalloc and tcmalloc_minimal)
    • Support .deb packaging
    • Support for script based installer for non debian systems
  • Python code generation
    • generate release.h
    • generate commands.def
    • generate fmtargs.h
    • generate test_files.h

Signed-off-by: Eran Ifrah <[email protected]>
@eifrah-aws eifrah-aws marked this pull request as ready for review October 20, 2024 21:20
@eifrah-aws eifrah-aws mentioned this pull request Oct 20, 2024
20 tasks
@eifrah-aws
Copy link
Author

@PingXie see this PR instead.

Copy link

codecov bot commented Oct 20, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.69%. Comparing base (2743b7e) to head (f06b624).

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #1196      +/-   ##
============================================
- Coverage     70.69%   70.69%   -0.01%     
============================================
  Files           114      114              
  Lines         63076    63076              
============================================
- Hits          44594    44592       -2     
- Misses        18482    18484       +2     
Files with missing lines Coverage Δ
src/debug.c 53.05% <ø> (ø)
src/server.c 88.77% <ø> (+0.03%) ⬆️

... and 16 files with indirect coverage changes

Copy link
Contributor

@pizhenwei pizhenwei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use WITH_TLS instead of USE_TLS? Then we have a single TLS related building variable.

@PingXie PingXie added the run-extra-tests Run extra tests on this PR (Runs all tests from daily except valgrind and RESP) label Oct 21, 2024
cmake/Modules/ValkeySetup.cmake Outdated Show resolved Hide resolved
cmake/Modules/ValkeySetup.cmake Outdated Show resolved Hide resolved
Copy link
Member

@PingXie PingXie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @eifrah-aws

.cmake-format.yaml Show resolved Hide resolved
.cmake-format.yaml Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
cmake/Modules/Packaging.cmake Outdated Show resolved Hide resolved
CMakeLists.txt Show resolved Hide resolved
cmake/Modules/SourceFiles.cmake Outdated Show resolved Hide resolved
cmake/Modules/ValkeySetup.cmake Outdated Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
@pizhenwei
Copy link
Contributor

Is it possible to generate config.h by cmake?
I guess this may be the next step.

- Force function names to lowercase in YAML config file
- Parse version from version.h instead of hard-coding it
- Fixed typo
- Moved utility functions into their own module `Utils.cmake`
- macOS: do not hard code "/usr/bin/clang", instead use `find_program`
- Support for WITH_TLS=module|yes|off|1|0
- Support for WITH_RDMA=module|off|0
- Fixed (never worked for the original Makefile as well): build TLS as module on macOS

Signed-off-by: Eran Ifrah <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-extra-tests Run extra tests on this PR (Runs all tests from daily except valgrind and RESP)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants