diff --git a/.bazelignore b/.bazelignore
index 75ea7dff..ad88eee7 100644
--- a/.bazelignore
+++ b/.bazelignore
@@ -1,3 +1,4 @@
bazel-kv-server
tools/wasm_example/
google_internal/piper/
+node_modules
diff --git a/.bazelrc b/.bazelrc
index e3d6c35d..717b8ec1 100644
--- a/.bazelrc
+++ b/.bazelrc
@@ -1,9 +1,8 @@
build --announce_rc
build --verbose_failures
build --compilation_mode=opt
-build --output_filter='^//((?!(third_party):).)*$'`
+build --output_filter='^//((?!(third_party):).)*$'
build --color=yes
-build --@io_bazel_rules_docker//transitions:enable=false
build --workspace_status_command="bash tools/get_workspace_status"
build --copt=-Werror=thread-safety
build --config=clang
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 76b12c7c..29b3684e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,14 +2,97 @@
All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
-## 0.17.1 (2024-08-26)
+## 1.0.0 (2024-10-14)
+
+
+### ⚠ BREAKING CHANGES
+
+* GA release
+
+### Features
+
+* Add 64 bit int sets support to key value cache
+* Add CBOR conversion for v2 objects
+* Add CBOR support to multi-partition flows in V2
+* add CORS headers for envoy config
+* Add data loading support for uint64 sets
+* Add documentation for uint64 sets
+* Add internal lookup rpc for uint64 sets
+* Add parameter notifier to get parameter update notification
+* Add partition-level metadata to UDF execution metadata
+* Add runSetQueryUInt64 udf hook
+* Add support for reading and writing uint64 sets to csv files
+* Add uint64 bitset wrapper.
+* CBOR conversion for Compresion Group
+* CborDecodeToProto implementation
+* Convert http ContentType header to a custom header in GCP
+* Download pre-built aws-otel-collector.rpm
+* Encode cbor content as bytestring and add partitionOutputs to CBOR converter
+* Fix release script
+* Flag to control chaffing for sharding for nonprod
+* GA release
+* Implement CBOR for validator
+* Implement internal GetUInt64ValueSet functionality
+* Implement InternalRunSetQueryUInt64 rpc (local lookup)
+* Implement InternalRunSetQueryUInt64 rpc (sharded lookup)
+* multiple partition support
+* Pass partition level metadata to UDF
+* Process v2 padded requests
+* Put server logs in the response DebugInfo for consented requests
+* Refactor cache logic for bitsets into it's own class
+* Set up AWS terraform resources for logging verbosity parameter notification
+* Start parameter notifier to get logging verbosity updates
+* Support dataVersion field in PA partition output
+* Support set operations for 64 bit int sets
+* Update AWS sqs cleanup function to clean up sqs for parameter updates
+* Update common repo and set the verbosity level for PS_VLOG with new API
+* Update v2 contract
+* Update v2 headers
+* Upgrade common repo to 9c5c93e
+* Upgrade rules_oci to 2.0 and deprecate rules_docker
+* Use proper ohttp media types for encryption
+* When using the wrong inline set type in query, resolve the result
### Bug Fixes
+* Add missing include directive
+* Add missing internal testing parameters
+* Allow CORS OPTIONS for preflight
+* Correct fork logic
+* Correct output_filter typo
+* Destroy terraform before doing perfgate exporting
* Enable a second kv on aws deployment.
* fix AppMesh health check.
+* logMessage should us PS LOGS
+* Make AL2023 work.
+* Remove "k" from ReceivedLowLatencyNotificationsCount metric name
+* Remove version from header
+* Rename BUILD to BUILD.bazel
* Resolve proxy subnet resources collision issue.
+* Response partition id should come from the request
+* Temporary GCP V2 HTTP envoy fix
+* Update common repo to pick up the server crash fix
+* Update V2 handler and docs with proper ohttp response label.
+* Upgrade builders version to 0.69.0
+* Use specified release branch to cut release.
+* V2 should not return error status on UDF failure
+
+
+### Dependencies
+
+* **deps:** Upgrade build-system to 0.66.1
+* **deps:** Upgrade data-plane-shared-libraries to 144264c 2024-07-31
+
+
+### Documentation
+
+* Add aws update-function-code lambda update command to the AWS deployment doc
+* Add readme doc for diagnostic tool
+* Add screenshot for gcp server prod log location
+* Update docs to use docker compose instead of docker-compose
+* Update gcp deployment doc about console logging
+* Update playbook
## 0.17.0 (2024-07-08)
@@ -89,6 +172,11 @@ All notable changes to this project will be documented in this file. See [commit
* Use aws_platform bazel config
* Use local_{platform,instance} bazel configs
+### Image digests and PCR0s
+
+GCP: sha256:d09d5a6d340a8829df03213b71b74d4b431e4d5a138525c77269c347a367b004
+AWS: {"PCR0":"1e28ac4b72600ea40d61e1756e14f453a3d923a1bf94c360ae48d9777bff0714923d9322ed380823591859e357d2f825"}
+
## 0.16.0 (2024-04-05)
diff --git a/README.md b/README.md
index fcf6a733..e41c9fa8 100644
--- a/README.md
+++ b/README.md
@@ -11,47 +11,27 @@
---
-# ![Privacy Sandbox Logo](docs/assets/privacy_sandbox_logo.png) FLEDGE Key/Value service
+# ![Privacy Sandbox Logo](docs/assets/privacy_sandbox_logo.png) Protected Auction Key/Value service
-# Background
-
-FLEDGE API is a proposal to serve remarketing and other custom-audience ads without third-party
-cookies. FLEDGE executes the ad auction between the buyers (DSP) and the sellers (SSP) locally, and
-receives real-time signals from the FLEDGE K/V servers. To learn more about
+# State of the project
-- FLEDGE for the Web: [explainer](https://developer.chrome.com/en/docs/privacy-sandbox/fledge/)
- and the [developer guide](https://developer.chrome.com/blog/fledge-api/).
-- FLEDGE on Android:
- [design proposal](https://developer.android.com/design-for-safety/privacy-sandbox/fledge) and
- the
- [developer guide](https://developer.android.com/design-for-safety/privacy-sandbox/guides/fledge).
+The current codebase represents the implementation of the TEE-based Key/Value service by Privacy
+Sandbox.
-When the auction is executed, separate
-[FLEDGE K/V servers](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md)
-are queried for the buyers and sellers. When a buyer is making a bid, the DSP K/V server can be
-queried to receive real-time information to help determine the bid. To help the seller pick an
-auction winner, the SSP K/V server can be queried to receive any information about the creative to
-help score the ad.
+For
+[Protected Audience](https://developers.google.com/privacy-sandbox/private-advertising/protected-audience),
+the service can be used as a BYOS KV server. Soon it can be used to communicate with Chrome and the
+Bidding and Auction services using
+[V2 protocol](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md).
-# State of the project
+For
+[Protected App Signals](https://developers.google.com/privacy-sandbox/private-advertising/protected-audience/android/protected-app-signals),
+the service should be used as the ad retrieval server.
-The current codebase represents the initial implementation and setup of the Key/Value server. It can
-be integrated with Chrome and Android with the
+It can be integrated with Chrome and Android with the
[Privacy Sandbox unified origin trial](https://developer.chrome.com/blog/expanding-privacy-sandbox-testing/)
and
[Privacy Sandbox on Android Developer Preview](https://developer.android.com/design-for-safety/privacy-sandbox/program-overview).
-Our goal is to present the foundation of the project in a publicly visible way for early feedback.
-This feedback will help us shape the future versions.
-
-The implementation, and in particular the APIs, are in rapid development and may change as new
-versions are released. The query API conforms to the
-[API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md). At the
-moment, to load data, instead of calling the mutation API, you would place the data as files into a
-location that can be directly read by the server. See more details in the
-[data loading guide](/docs/data_loading/loading_data.md).
-
-Currently, this service can be deployed to 1 region of your choice. Multi-region configuration is up
-to the service owner to configure.
## Current features
@@ -120,6 +100,7 @@ products.
+
@@ -193,6 +174,7 @@ The implementation supports live traffic at scale
|
+
@@ -270,14 +252,7 @@ The implementation supports live traffic at scale
## Breaking changes
-While we make efforts to not introduce breaking changes, we expect that to happen occasionally.
-
-The release version follows the `[major change]-[minor change]-[patch]` scheme. All 0.x.x versions
-may contain breaking changes without notice. Refer to the [release changelog](/CHANGELOG.md) for the
-details of the breaking changes.
-
-At GA the version will become 1.0.0, we will establish additional channels for announcing breaking
-changes and major version will always be incremented for breaking changes.
+Backward-incompatible changes are expected to be rare and will result in a major version change.
# Key documents
@@ -304,8 +279,8 @@ changes and major version will always be incremented for breaking changes.
Contributions are welcome, and we will publish more detailed guidelines soon. In the meantime, if
you are interested,
-[open a new Issue](https://github.com/privacysandbox/fledge-key-value-service/issues) in the GitHub
-repository.
+[open a new Issue](https://github.com/privacysandbox/protected-auction-key-value-service/issues) in
+the GitHub repository.
# Feedback
diff --git a/WORKSPACE b/WORKSPACE
index 2813868a..2b824476 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -7,17 +7,19 @@ local_repository(
path = "testing/functionaltest-system",
)
-load("//builders/bazel:deps.bzl", "python_deps")
+load("//builders/bazel:deps.bzl", "python_deps", "python_register_toolchains")
-python_deps("//builders/bazel")
+python_deps()
+
+python_register_toolchains("//builders/bazel")
http_archive(
name = "google_privacysandbox_servers_common",
- # commit 34445c1 2024-07-01
- sha256 = "ce300bc178b1eedd88d7545b89d1d672b3b9bfb62c138ab3f4a845f159436285",
- strip_prefix = "data-plane-shared-libraries-37522d6ac55c8592060f636d68f50feddcb9598a",
+ # commit cc49da3 2024-10-09
+ sha256 = "7a0337420161304c7429c727b1f82394bc27e1e2586d2da30e6d6100ba92b437",
+ strip_prefix = "data-plane-shared-libraries-158593616a63df924af1cb689f3915b8d32e9db1",
urls = [
- "https://github.com/privacysandbox/data-plane-shared-libraries/archive/37522d6ac55c8592060f636d68f50feddcb9598a.zip",
+ "https://github.com/privacysandbox/data-plane-shared-libraries/archive/158593616a63df924af1cb689f3915b8d32e9db1.zip",
],
)
@@ -51,28 +53,10 @@ load(
cpp_repositories()
-http_archive(
- name = "io_bazel_rules_docker",
- sha256 = "b1e80761a8a8243d03ebca8845e9cc1ba6c82ce7c5179ce2b295cd36f7e394bf",
- urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.25.0/rules_docker-v0.25.0.tar.gz"],
-)
-
-load("@io_bazel_rules_docker//repositories:repositories.bzl", container_repositories = "repositories")
-
-container_repositories()
-
-load("@io_bazel_rules_docker//repositories:deps.bzl", io_bazel_rules_docker_deps = "deps")
-
-io_bazel_rules_docker_deps()
-
load("//third_party_deps:container_deps.bzl", "container_deps")
container_deps()
-load("@io_bazel_rules_docker//go:image.bzl", go_image_repos = "repositories")
-
-go_image_repos()
-
# googleapis
http_archive(
name = "com_google_googleapis", # master branch from 26.04.2022
@@ -88,6 +72,16 @@ http_archive(
urls = ["https://github.com/google/distributed_point_functions/archive/45da5f54836c38b73a1392e846c9db999c548711.tar.gz"],
)
+http_archive(
+ name = "libcbor",
+ build_file = "//third_party_deps:libcbor.BUILD",
+ patch_args = ["-p1"],
+ patches = ["//third_party_deps:libcbor.patch"],
+ sha256 = "9fec8ce3071d5c7da8cda397fab5f0a17a60ca6cbaba6503a09a47056a53a4d7",
+ strip_prefix = "libcbor-0.10.2/src",
+ urls = ["https://github.com/PJK/libcbor/archive/refs/tags/v0.10.2.zip"],
+)
+
# Dependencies for Flex/Bison build rules
http_archive(
name = "rules_m4",
@@ -132,6 +126,15 @@ latency_benchmark_install_deps()
word2vec_install_deps()
+http_archive(
+ name = "io_bazel_rules_go",
+ sha256 = "16e9fca53ed6bd4ff4ad76facc9b7b651a89db1689a2877d6fd7b82aa824e366",
+ urls = [
+ "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.34.0/rules_go-v0.34.0.zip",
+ "https://github.com/bazelbuild/rules_go/releases/download/v0.34.0/rules_go-v0.34.0.zip",
+ ],
+)
+
# Use nogo to run `go vet` with bazel
load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
diff --git a/builders/.pre-commit-config.yaml b/builders/.pre-commit-config.yaml
index e1ad672d..3f853aab 100644
--- a/builders/.pre-commit-config.yaml
+++ b/builders/.pre-commit-config.yaml
@@ -47,7 +47,7 @@ repos:
- id: shellcheck
- repo: https://github.com/pre-commit/mirrors-clang-format
- rev: v18.1.4
+ rev: v18.1.5
hooks:
- id: clang-format
types_or:
diff --git a/builders/.profiler.bazelrc b/builders/.profiler.bazelrc
new file mode 100644
index 00000000..42524d63
--- /dev/null
+++ b/builders/.profiler.bazelrc
@@ -0,0 +1,5 @@
+build:profiler --compilation_mode=opt
+build:profiler --dynamic_mode=off
+build:profiler --copt=-gmlt
+build:profiler --copt=-fno-omit-frame-pointer
+build:profiler --strip=never
diff --git a/builders/CHANGELOG.md b/builders/CHANGELOG.md
index 3ced1112..af695dc8 100644
--- a/builders/CHANGELOG.md
+++ b/builders/CHANGELOG.md
@@ -2,6 +2,112 @@
All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
+## 0.69.0 (2024-09-15)
+
+
+### Features
+
+* Pin build-debian ubuntu base image from 20.04 to focal-20240530
+
+## 0.68.1 (2024-08-21)
+
+### Bug Fixes
+
+* Fix load bazel_tools import for Python deps
+
+## 0.68.0 (2024-08-21)
+
+
+### Features
+
+* **deps:** Split python deps and registering toolchains
+* **deps:** Update rules_python to 0.35.0
+
+## 0.67.0 (2024-07-31)
+
+
+### Bug Fixes
+
+* Add EXTRA_CBUILD_ARGS to tools/bazel-* scripts
+
+
+### Dependencies
+
+* **deps:** Update buildozer to 6.1.1
+* **deps:** Upgrade amazonlinux2023 to 5.20240722.0
+
+## 0.66.1 (2024-06-24)
+
+
+### Bug Fixes
+
+* Add --compilation_mode=opt to build:profiler config
+
+## 0.66.0 (2024-06-20)
+
+
+### Features
+
+* Add cpu-profiler flags to cbuild
+* Add profiler config in .profiler.bazelrc
+
+## 0.65.1 (2024-06-04)
+
+
+### Bug Fixes
+
+* Support multiple etc files in a single image
+
+## 0.65.0 (2024-06-04)
+
+
+### Features
+
+* Add DOCKER_NETWORK env var for test-tools
+
+## 0.64.1 (2024-05-29)
+
+
+### Bug Fixes
+
+* Support container reuse when --cmd not specified
+* Use find to identify bazel symlinks
+
+## 0.64.0 (2024-05-27)
+
+
+### Features
+
+* Support cmd-profiler mode with/without --cmd
+
+
+### Bug Fixes
+
+* cbuild should find container with exact name match
+* Ensure normalize-bazel-symlinks is in the workspace dir
+
+
+### Dependencies
+
+* **deps:** Upgrade clang-format pre-commit hook
+
+## 0.63.0 (2024-05-26)
+
+
+### Features
+
+* Support cmd-profiler mode with/without --cmd
+
+
+### Bug Fixes
+
+* Ensure normalize-bazel-symlinks is in the workspace dir
+
+
+### Dependencies
+
+* **deps:** Upgrade clang-format pre-commit hook
+
## 0.62.0 (2024-05-10)
diff --git a/builders/bazel/deps.bzl b/builders/bazel/deps.bzl
index 0fe6743b..9e46f9ca 100644
--- a/builders/bazel/deps.bzl
+++ b/builders/bazel/deps.bzl
@@ -15,21 +15,24 @@
"""Load definitions for use in WORKSPACE files."""
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
-def python_deps(bazel_package):
- """Load rules_python and register container-based python toolchain
+def python_deps():
+ """Load rules_python. Use python_register_toolchains to also resgister container-based python toolchain."""
+ maybe(
+ http_archive,
+ name = "rules_python",
+ sha256 = "be04b635c7be4604be1ef20542e9870af3c49778ce841ee2d92fcb42f9d9516a",
+ strip_prefix = "rules_python-0.35.0",
+ url = "https://github.com/bazelbuild/rules_python/releases/download/0.35.0/rules_python-0.35.0.tar.gz",
+ )
+
+def python_register_toolchains(bazel_package):
+ """Register container-based python toolchain.
Note: the bazel_package arg will depend on the import/submodule location in your workspace
Args:
bazel_package: repo-relative bazel package to builders/bazel/BUILD eg. "//builders/bazel"
"""
- http_archive(
- name = "rules_python",
- sha256 = "0a8003b044294d7840ac7d9d73eef05d6ceb682d7516781a4ec62eeb34702578",
- strip_prefix = "rules_python-0.24.0",
- urls = [
- "https://github.com/bazelbuild/rules_python/releases/download/0.24.0/rules_python-0.24.0.tar.gz",
- ],
- )
native.register_toolchains("{}:py_toolchain".format(bazel_package))
diff --git a/builders/images/build-amazonlinux2023/Dockerfile b/builders/images/build-amazonlinux2023/Dockerfile
index 268bbcba..c7d3509b 100644
--- a/builders/images/build-amazonlinux2023/Dockerfile
+++ b/builders/images/build-amazonlinux2023/Dockerfile
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM amazonlinux:2023.4.20240416.0
+FROM amazonlinux:2023.5.20240722.0
COPY /install_apps install_golang_apps install_go.sh generate_system_bazelrc .bazelversion /scripts/
COPY get_workspace_mount /usr/local/bin
diff --git a/builders/images/build-amazonlinux2023/install_apps b/builders/images/build-amazonlinux2023/install_apps
index 442021c4..16378fa5 100755
--- a/builders/images/build-amazonlinux2023/install_apps
+++ b/builders/images/build-amazonlinux2023/install_apps
@@ -47,8 +47,8 @@ function install_python() {
function install_nitro() {
dnf install -y \
- "aws-nitro-enclaves-cli-1.2.*" \
- "aws-nitro-enclaves-cli-devel-1.2.*"
+ "aws-nitro-enclaves-cli-1.3.*" \
+ "aws-nitro-enclaves-cli-devel-1.3.*"
}
function install_gcc() {
diff --git a/builders/images/build-debian/Dockerfile b/builders/images/build-debian/Dockerfile
index 735d9b66..4b615819 100644
--- a/builders/images/build-debian/Dockerfile
+++ b/builders/images/build-debian/Dockerfile
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-ARG BASE_IMAGE=ubuntu:20.04
+ARG BASE_IMAGE=ubuntu:focal-20240530
# ignore this hadolint error as BASE_IMAGE contains an image tag
# hadolint ignore=DL3006
diff --git a/builders/images/presubmit/.bazelversion b/builders/images/presubmit/.bazelversion
new file mode 120000
index 00000000..8f79a5af
--- /dev/null
+++ b/builders/images/presubmit/.bazelversion
@@ -0,0 +1 @@
+../../etc/.bazelversion
\ No newline at end of file
diff --git a/builders/images/presubmit/Dockerfile b/builders/images/presubmit/Dockerfile
index 53ee3778..827e2584 100644
--- a/builders/images/presubmit/Dockerfile
+++ b/builders/images/presubmit/Dockerfile
@@ -14,7 +14,7 @@
FROM ubuntu:24.04
-COPY install_apps install_go.sh .pre-commit-config.yaml /scripts/
+COPY install_apps install_go.sh install_golang_apps .bazelversion .pre-commit-config.yaml /scripts/
COPY gitconfig /etc
ARG PRE_COMMIT_VENV_DIR=/usr/pre-commit-venv
@@ -28,6 +28,7 @@ ENV BUILD_ARCH="${TARGETARCH}" \
RUN \
chmod 644 /etc/gitconfig && \
/usr/bin/env -v PRE_COMMIT_VENV_DIR=${PRE_COMMIT_VENV_DIR} /scripts/install_apps && \
+ /scripts/install_golang_apps && \
rm -rf /scripts
ENV PATH="${PATH}:/usr/local/go/bin"
diff --git a/builders/images/presubmit/install_golang_apps b/builders/images/presubmit/install_golang_apps
new file mode 120000
index 00000000..acc9d5a3
--- /dev/null
+++ b/builders/images/presubmit/install_golang_apps
@@ -0,0 +1 @@
+../install_golang_apps
\ No newline at end of file
diff --git a/builders/images/release/install_release_apps b/builders/images/release/install_release_apps
index e6c12253..dc99134f 100755
--- a/builders/images/release/install_release_apps
+++ b/builders/images/release/install_release_apps
@@ -5,4 +5,4 @@ npm install --global commit-and-tag-version@10.1.0
# Install the GitHub CLI tool (https://cli.github.com/)
apk add github-cli
-GOBIN=/usr/local/go/bin go install github.com/bazelbuild/buildtools/buildozer@6.0.1
+GOBIN=/usr/local/go/bin go install github.com/bazelbuild/buildtools/buildozer@6.1.1
diff --git a/builders/tests/data/hashes/build-amazonlinux2023 b/builders/tests/data/hashes/build-amazonlinux2023
index 5fe991bf..7d1e615d 100644
--- a/builders/tests/data/hashes/build-amazonlinux2023
+++ b/builders/tests/data/hashes/build-amazonlinux2023
@@ -1 +1 @@
-8d01333fe93d2ac2102dd8360a58717724b7b594d51fe4e412ec20aae181efce
+59a82d2db8173784b0b49959c9f82ead6c2e6da78a6be21cdc78520aa43741e3
diff --git a/builders/tests/data/hashes/build-debian b/builders/tests/data/hashes/build-debian
index 57095aed..b0a607dd 100644
--- a/builders/tests/data/hashes/build-debian
+++ b/builders/tests/data/hashes/build-debian
@@ -1 +1 @@
-c194dafd287978093f8fe6e16e981fb22028e37345e20a4d7ca84caa43f0d4c0
+83fab12505490f9ed41e5d8747f3c8844f6aee8740ad05c47eca61fd7b42a8d1
diff --git a/builders/tests/data/hashes/presubmit b/builders/tests/data/hashes/presubmit
index b02b21b0..a3e4b6ff 100644
--- a/builders/tests/data/hashes/presubmit
+++ b/builders/tests/data/hashes/presubmit
@@ -1 +1 @@
-afaf1932764d07d480c4e833e6b08877f069abae87401bdac4782277c535a298
+560a5a1726e7b6fd2a507f72f2563eb3938d153a7d4b4aada575a7fe772873b0
diff --git a/builders/tests/data/hashes/release b/builders/tests/data/hashes/release
index e5218ba2..affaf218 100644
--- a/builders/tests/data/hashes/release
+++ b/builders/tests/data/hashes/release
@@ -1 +1 @@
-d60fb40a53b1704f7ac353d0d036f49eac10bfd08ccb19f9b436acf8bdf2cb79
+398787e442bb10bcf7383bc3beec85ab27fb0145f80a23d8ee0eeb4992e5cb81
diff --git a/builders/tools/bazel-debian b/builders/tools/bazel-debian
index 843b3b6e..c32cf7b9 100755
--- a/builders/tools/bazel-debian
+++ b/builders/tools/bazel-debian
@@ -19,6 +19,7 @@
# BAZEL_STARTUP_ARGS Additional startup arguments to pass to bazel invocations
# BAZEL_EXTRA_ARGS Additional command arguments to pass to bazel invocations
# EXTRA_DOCKER_RUN_ARGS Additional arguments to pass to docker run invocations
+# EXTRA_CBUILD_ARGS Additional arguments to pass to tools/cbuild
set -o pipefail
set -o errexit
@@ -63,7 +64,8 @@ declare -a APP_ARGS
declare -r -a ARGLIST=("$@")
partition_array ARGLIST BAZEL_ARGS APP_ARGS
-"${CBUILD}" --seccomp-unconfined --image "${IMAGE}" --cmd "
+# shellcheck disable=SC2086
+"${CBUILD}" ${EXTRA_CBUILD_ARGS} --seccomp-unconfined --image "${IMAGE}" --cmd "
printf 'bazel output_base: [%s]\n' \"\$(bazel info output_base 2>/dev/null)\"
bazel ${BAZEL_STARTUP_ARGS} ${BAZEL_ARGS[*]@Q} ${BAZEL_EXTRA_ARGS} ${APP_ARGS[*]@Q}
"
diff --git a/builders/tools/cbuild b/builders/tools/cbuild
index 17da1336..31ee0fcb 100755
--- a/builders/tools/cbuild
+++ b/builders/tools/cbuild
@@ -36,7 +36,6 @@ function usage() {
usage:
$0
--cmd bash command string to execute within the docker container
- --cmd-profiler enable profiler for the command
--image Image name for the build runtime. Valid names:
USAGE
@@ -57,6 +56,11 @@ USAGE
--seccomp-unconfined Run docker container without a seccomp profile
--verbose Enable verbose output
+ Profiler flags:
+ --cmd-profiler enable profiler for the command
+ --cpu-profiler-signal unix signal to use to trigger profiler output. Default: ${CPU_PROFILER_SIGNAL}
+ --cpu-profiler-filename path for the cpu profiler output. Default: ${CPU_PROFILER_FILENAME}
+
Environment variables (all optional):
WORKSPACE Full path to the workspace (repo root)
WORKSPACE_MOUNT Full path to the workspace on the host filesystem
@@ -79,6 +83,8 @@ DOCKER_NETWORK="${DOCKER_NETWORK:-bridge}"
declare -i DOCKER_SECCOMP_UNCONFINED=0
declare -i KEEP_CONTAINER_RUNNING=0
declare LONG_RUNNING_CONTAINER_TIMEOUT=8h
+declare CPU_PROFILER_FILENAME=benchmark.prof
+declare -i CPU_PROFILER_SIGNAL=12
while [[ $# -gt 0 ]]; do
case "$1" in
@@ -90,6 +96,14 @@ while [[ $# -gt 0 ]]; do
WITH_CMD_PROFILER=1
shift
;;
+ --cpu-profiler-filename)
+ CPU_PROFILER_FILENAME="$2"
+ shift 2 || usage
+ ;;
+ --cpu-profiler-signal)
+ CPU_PROFILER_SIGNAL=$2
+ shift 2 || usage
+ ;;
--env)
ENV_VARS+=("$2")
shift 2 || usage
@@ -199,7 +213,8 @@ if [[ ${WITH_CMD_PROFILER} -eq 1 ]]; then
fi
DOCKER_RUN_ARGS+=(
"--env=CMD_PROFILER=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libprofiler.so"
- "--env=CPUPROFILE=benchmark.prof"
+ "--env=CPUPROFILE=${CPU_PROFILER_FILENAME}"
+ "--env=CPUPROFILESIGNAL=${CPU_PROFILER_SIGNAL}"
)
fi
@@ -252,8 +267,8 @@ function running_container_for() {
declare -r name="$1"
declare -a docker_args=(
container ls
- --filter "name=${name}"
- --format "{{print .Names}}"
+ "--filter=name=^${name}$"
+ "--format={{print .Names}}"
)
local -r exited="$(docker "${docker_args[@]}" --all --filter "status=exited")"
if [[ -n ${exited} ]]; then
@@ -269,13 +284,7 @@ function long_running_container() {
local -r docker_running_container="$(running_container_for "${container_name}")"
if [[ -z ${docker_running_container} ]]; then
printf "starting a new container [%s]\n" "${container_name}" &>/dev/stderr
- if [[ -z ${CMD} ]]; then
- # shellcheck disable=SC2068
- docker run \
- ${DOCKER_RUN_ARGS[@]} \
- "${DOCKER_EXEC_RUN_ARGS[@]}" \
- "${IMAGE_TAGGED}"
- else
+ if [[ -n ${CMD} ]]; then
# shellcheck disable=SC2068
docker run \
${DOCKER_RUN_ARGS[@]} \
@@ -293,11 +302,26 @@ timeout ${LONG_RUNNING_CONTAINER_TIMEOUT} tail --pid=\${pid} -f /dev/null
}
if [[ ${KEEP_CONTAINER_RUNNING} -eq 1 ]]; then
- DOCKER_RUNNING_CONTAINER="$(long_running_container "${DOCKER_CONTAINER_NAME}")"
- docker exec \
- "${DOCKER_EXEC_RUN_ARGS[@]}" \
- "${DOCKER_RUNNING_CONTAINER}" \
- /bin/bash -c "${CMD}"
+ if [[ -z ${CMD} ]]; then
+ # shellcheck disable=SC2068
+ docker run \
+ ${DOCKER_RUN_ARGS[@]} \
+ "${DOCKER_EXEC_RUN_ARGS[@]}" \
+ "${IMAGE_TAGGED}"
+ else
+ DOCKER_RUNNING_CONTAINER="$(long_running_container "${DOCKER_CONTAINER_NAME}")"
+ if [[ ${WITH_CMD_PROFILER} -eq 1 ]]; then
+ docker exec \
+ "${DOCKER_EXEC_RUN_ARGS[@]}" \
+ "${DOCKER_RUNNING_CONTAINER}" \
+ /bin/bash -c "'${TOOLS_RELDIR}'/normalize-bazel-symlinks; env \${CMD_PROFILER} ${CMD:-/bin/sh}"
+ else
+ docker exec \
+ "${DOCKER_EXEC_RUN_ARGS[@]}" \
+ "${DOCKER_RUNNING_CONTAINER}" \
+ /bin/bash -c "${CMD:-/bin/sh}"
+ fi
+ fi
else
if [[ -z ${CMD} ]]; then
# shellcheck disable=SC2068
@@ -319,6 +343,6 @@ else
${DOCKER_RUN_ARGS[@]} \
"${DOCKER_EXEC_RUN_ARGS[@]}" \
"${IMAGE_TAGGED}" \
- --login -c "$CMD"
+ --login -c "${CMD}"
fi
fi
diff --git a/builders/tools/get-builder-image-tagged b/builders/tools/get-builder-image-tagged
index 35371aea..bc14958a 100755
--- a/builders/tools/get-builder-image-tagged
+++ b/builders/tools/get-builder-image-tagged
@@ -199,7 +199,7 @@ function _tar_for_dir() {
# shellcheck disable=SC2012
ls -A -1 "${FILEPATH}" "${ETC_DIR}" | sort | uniq -d
ls -A -1 "${WORKSPACE}"
- } | sort | uniq -d)"
+ } | sort | uniq -d | tr '\n' ' ')"
# create a deterministic tarball of the collected files
docker run \
--rm \
diff --git a/builders/tools/normalize-bazel-symlinks b/builders/tools/normalize-bazel-symlinks
index 8506ac96..05e3e7ef 100755
--- a/builders/tools/normalize-bazel-symlinks
+++ b/builders/tools/normalize-bazel-symlinks
@@ -43,14 +43,11 @@ if [[ -f /.dockerenv ]]; then
_normalize_fn=normalize_symlink_docker
fi
-declare -a -r LINK_DIRS=(
- bazel-bin
- bazel-out
- bazel-testlogs
- bazel-workspace
-)
-for link in "${LINK_DIRS[@]}"; do
- if [[ -L ${link} ]]; then
- ${_normalize_fn} "${link}"
- fi
+source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"/builder.sh
+cd "${WORKSPACE}" || true
+
+declare -a links
+mapfile -t links < <(find . -maxdepth 1 -type l -name "bazel-*" -exec basename {} \;)
+for link in "${links[@]}"; do
+ ${_normalize_fn} "${link}"
done
diff --git a/builders/tools/test-tool b/builders/tools/test-tool
index 6a275370..908249db 100755
--- a/builders/tools/test-tool
+++ b/builders/tools/test-tool
@@ -13,7 +13,9 @@
# limitations under the License.
# environment variables (all optional):
-# WORKSPACE repo root directory, must be an absolute path
+# WORKSPACE repo root directory, must be an absolute path
+# DOCKER_NETWORK docker run --network arg, defaults to "host", set to
+# blank to avoid setting --network
set -o errexit
@@ -57,6 +59,9 @@ readonly REL_PWD
WORKSPACE_MOUNT="$(builder::get_docker_workspace_mount)"
readonly WORKSPACE_MOUNT
+# respect an empty DOCKER_NETWORK value
+DOCKER_NETWORK=${DOCKER_NETWORK-host}
+
declare -a DOCKER_RUN_ARGS=(
"--rm"
"--interactive"
@@ -64,6 +69,11 @@ declare -a DOCKER_RUN_ARGS=(
"--volume=${WORKSPACE_MOUNT}:/src/workspace"
"--workdir=/src/workspace/${REL_PWD}"
)
+if [[ -n ${DOCKER_NETWORK} ]]; then
+ DOCKER_RUN_ARGS+=(
+ "--network=${DOCKER_NETWORK}"
+ )
+fi
if [[ -n ${EXTRA_DOCKER_RUN_ARGS} ]]; then
# shellcheck disable=SC2207
DOCKER_RUN_ARGS+=(
diff --git a/builders/version.txt b/builders/version.txt
index 7e9253a3..acdcb836 100644
--- a/builders/version.txt
+++ b/builders/version.txt
@@ -1 +1 @@
-0.62.0
\ No newline at end of file
+0.69.0
\ No newline at end of file
diff --git a/components/aws/BUILD.bazel b/components/aws/BUILD.bazel
index f35d8299..6bf9b9db 100644
--- a/components/aws/BUILD.bazel
+++ b/components/aws/BUILD.bazel
@@ -12,12 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+load("@rules_oci//oci:defs.bzl", "oci_image", "oci_load", "oci_push")
load(
- "@io_bazel_rules_docker//container:container.bzl",
- "container_image",
- "container_layer",
- "container_push",
+ "@rules_pkg//pkg:mappings.bzl",
+ "pkg_attributes",
+ "pkg_files",
)
+load("@rules_pkg//pkg:tar.bzl", "pkg_tar")
load("@rules_python//python:defs.bzl", "py_library", "py_test")
py_library(
@@ -35,35 +36,58 @@ py_test(
],
)
-container_layer(
- name = "lambda_binary_layer",
- directory = "/var/task", # ${LAMBDA_TASK_ROOT}
- files = [
+pkg_files(
+ name = "lambda_binaries",
+ srcs = [
"sqs_cleanup.py",
"sqs_cleanup_manager.py",
],
+ attributes = pkg_attributes(mode = "0555"),
+ prefix = "/var/task", # ${LAMBDA_TASK_ROOT}
)
-container_image(
- name = "sqs_lambda",
+pkg_tar(
+ name = "lambda_binary_tar",
+ srcs = [":lambda_binaries"],
+)
+
+oci_image(
+ name = "sqs_lambda_image",
base = select({
- "@platforms//cpu:arm64": "@aws-lambda-python-arm64//image",
- "@platforms//cpu:x86_64": "@aws-lambda-python-amd64//image",
+ "@platforms//cpu:arm64": "@aws-lambda-python-arm64",
+ "@platforms//cpu:x86_64": "@aws-lambda-python-amd64",
}),
cmd = ["sqs_cleanup.handler"],
- layers = [
- ":lambda_binary_layer",
+ tars = [
+ ":lambda_binary_tar",
],
+)
+
+oci_load(
+ name = "sqs_lambda",
+ image = ":sqs_lambda_image",
+ repo_tags = ["bazel/components/aws:sqs_lambda"],
+)
+
+filegroup(
+ name = "sqs_lambda_tarball_file",
+ srcs = [":sqs_lambda"],
+ output_group = "tarball",
+)
+
+genrule(
+ name = "sqs_lambda_tarball",
+ srcs = [":sqs_lambda_tarball_file"],
+ outs = ["sqs_lambda.tar"],
+ cmd = "cp $< $@",
visibility = ["//production/packaging:__subpackages__"],
)
-container_push(
+oci_push(
name = "sqs_lambda_push_aws_ecr",
- format = "Docker",
- image = ":sqs_lambda",
- registry = "$${AWS_ECR}",
- repository = "sqs_lambda",
- tag = "latest",
+ image = ":sqs_lambda_image",
+ remote_tags = ["latest"],
+ repository = "$${AWS_ECR}/sqs_lambda",
)
exports_files(
diff --git a/components/aws/sqs_cleanup.py b/components/aws/sqs_cleanup.py
index 4717cbff..713a6c87 100644
--- a/components/aws/sqs_cleanup.py
+++ b/components/aws/sqs_cleanup.py
@@ -44,9 +44,27 @@ def handler(event, context):
deleted_realtime_queues, deleted_realtime_subscriptions = find_and_cleanup(
realtime_sns_topic, realtime_queue_prefix, timeout_secs
)
+ logging_verbosity_updates_sns_topic = event.get(
+ "logging_verbosity_updates_sns_topic"
+ )
+ parameter_queue_prefix = event.get("parameter_queue_prefix")
+ if logging_verbosity_updates_sns_topic is None:
+ raise Exception("no logging verbosity updates topic")
+ if parameter_queue_prefix is None:
+ raise Exception("no parameter queue prefix")
+
+ (
+ deleted_logging_verbosity_parameter_queues,
+ deleted_logging_verbosity_parameter_subscriptions,
+ ) = find_and_cleanup(
+ logging_verbosity_updates_sns_topic, parameter_queue_prefix, timeout_secs
+ )
+
return {
"deleted_queues": deleted_queues,
"deleted_subscriptions": deleted_subscriptions,
"deleted_realtime_queues": deleted_realtime_queues,
"deleted_realtime_subscriptions": deleted_realtime_subscriptions,
+ "deleted_logging_verbosity_parameter_queues": deleted_logging_verbosity_parameter_queues,
+ "deleted_logging_verbosity_parameter_subscriptions": deleted_logging_verbosity_parameter_subscriptions,
}
diff --git a/components/cloud_config/instance_client_gcp.cc b/components/cloud_config/instance_client_gcp.cc
index 5ad5f919..b6d58ffc 100644
--- a/components/cloud_config/instance_client_gcp.cc
+++ b/components/cloud_config/instance_client_gcp.cc
@@ -250,9 +250,7 @@ class GcpInstanceClient : public InstanceClient {
const ExecutionResult& result,
const GetInstanceDetailsByResourceNameResponse& response) {
if (result.Successful()) {
- // TODO(b/342614468): Temporarily turn off this vlog until
- // verbosity setting API in the common repo is fixed
- // PS_VLOG(2, log_context_) << response.DebugString();
+ PS_VLOG(2, log_context_) << response.DebugString();
instance_id_ =
std::string{response.instance_details().instance_id()};
environment_ =
diff --git a/components/cloud_config/parameter_client_local.cc b/components/cloud_config/parameter_client_local.cc
index c051a649..377bb63c 100644
--- a/components/cloud_config/parameter_client_local.cc
+++ b/components/cloud_config/parameter_client_local.cc
@@ -55,6 +55,9 @@ ABSL_FLAG(std::string, data_loading_file_format,
"possible values.");
ABSL_FLAG(std::int32_t, logging_verbosity_level, 0,
"Loggging verbosity level.");
+ABSL_FLAG(std::int32_t, logging_verbosity_backup_poll_frequency_secs, 300,
+ "Loggging verbosity level back up poll frequency in seconds.");
+
ABSL_FLAG(absl::Duration, udf_timeout, absl::Seconds(5),
"Timeout for one UDF invocation");
ABSL_FLAG(absl::Duration, udf_update_timeout, absl::Seconds(30),
@@ -132,6 +135,9 @@ class LocalParameterClient : public ParameterClient {
absl::GetFlag(FLAGS_udf_num_workers)});
int32_t_flag_values_.insert({"kv-server-local-logging-verbosity-level",
absl::GetFlag(FLAGS_logging_verbosity_level)});
+ int32_t_flag_values_.insert(
+ {"kv-server-local-logging-verbosity-backup-poll-frequency-secs",
+ absl::GetFlag(FLAGS_logging_verbosity_backup_poll_frequency_secs)});
int32_t_flag_values_.insert(
{"kv-server-local-udf-timeout-millis",
absl::ToInt64Milliseconds(absl::GetFlag(FLAGS_udf_timeout))});
diff --git a/components/cloud_config/parameter_update/BUILD.bazel b/components/cloud_config/parameter_update/BUILD.bazel
new file mode 100644
index 00000000..747283e8
--- /dev/null
+++ b/components/cloud_config/parameter_update/BUILD.bazel
@@ -0,0 +1,44 @@
+load("@rules_cc//cc:defs.bzl", "cc_library", "cc_test")
+
+package(default_visibility = [
+ "//components:__subpackages__",
+ "//tools:__subpackages__",
+])
+
+cc_library(
+ name = "parameter_notifier",
+ srcs = select({
+ "//:aws_platform": ["parameter_notifier_aws.cc"],
+ "//:gcp_platform": ["parameter_notifier_gcp.cc"],
+ "//:local_platform": ["parameter_notifier_local.cc"],
+ }) + ["parameter_notifier.cc"],
+ hdrs = [
+ "parameter_notifier.h",
+ ],
+ deps = [
+ "//components/data/common:change_notifier",
+ "//components/data/common:thread_manager",
+ "//components/util:sleepfor",
+ "@com_google_absl//absl/log",
+ "@com_google_absl//absl/status",
+ "@com_google_absl//absl/status:statusor",
+ "@com_google_absl//absl/synchronization",
+ "@google_privacysandbox_servers_common//src/logger:request_context_logger",
+ ],
+)
+
+cc_test(
+ name = "parameter_notifier_test",
+ size = "small",
+ srcs = select({
+ "//:aws_platform": ["parameter_notifier_test_aws.cc"],
+ "//:gcp_platform": ["parameter_notifier_test_gcp.cc"],
+ "//:local_platform": ["parameter_notifier_test_local.cc"],
+ }),
+ deps = [
+ ":parameter_notifier",
+ "//components/data/common:mocks",
+ "//components/util:sleepfor_mock",
+ "@com_google_googletest//:gtest_main",
+ ],
+)
diff --git a/components/cloud_config/parameter_update/parameter_notifier.cc b/components/cloud_config/parameter_update/parameter_notifier.cc
new file mode 100644
index 00000000..984a3320
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier.cc
@@ -0,0 +1,57 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+
+#include
+#include
+
+namespace kv_server {
+
+using privacy_sandbox::server_common::ExpiringFlag;
+
+absl::Status ParameterNotifier::Stop() {
+ absl::Status status = sleep_for_->Stop();
+ status.Update(thread_manager_->Stop());
+ return status;
+}
+
+bool ParameterNotifier::IsRunning() const {
+ return thread_manager_->IsRunning();
+}
+
+absl::StatusOr ParameterNotifier::ShouldGetParameter(
+ ExpiringFlag& expiring_flag) {
+ if (!expiring_flag.Get()) {
+ PS_VLOG(5, log_context_)
+ << "Backup poll on parameter update " << parameter_name_;
+ return true;
+ }
+ absl::StatusOr notification =
+ WaitForNotification(expiring_flag.GetTimeRemaining(),
+ [this]() { return thread_manager_->ShouldStop(); });
+
+ if (absl::IsDeadlineExceeded(notification.status())) {
+ // Deadline exceeded while waiting, trigger backup poll
+ PS_VLOG(5, log_context_)
+ << "Backup poll on parameter update " << parameter_name_;
+ return true;
+ }
+ if (!notification.ok()) {
+ return notification.status();
+ }
+ return true;
+}
+
+} // namespace kv_server
diff --git a/components/cloud_config/parameter_update/parameter_notifier.h b/components/cloud_config/parameter_update/parameter_notifier.h
new file mode 100644
index 00000000..e0d3e556
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier.h
@@ -0,0 +1,162 @@
+/*
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef COMPONENTS_CLOUD_CONFIG_PARAMETER_UPDATE_PARAMETER_NOTIFIER_H_
+#define COMPONENTS_CLOUD_CONFIG_PARAMETER_UPDATE_PARAMETER_NOTIFIER_H_
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "absl/status/status.h"
+#include "components/data/common/change_notifier.h"
+#include "components/data/common/notifier_metadata.h"
+#include "components/data/common/thread_manager.h"
+#include "components/util/sleepfor.h"
+#include "src/logger/request_context_logger.h"
+
+namespace kv_server {
+
+// The ParameterNotifier watches the cloud pubsub notification on the
+// changes of the value for a given parameter. When a notification is received
+// or the poll period deadline is reached, this class executes the get parameter
+// value callback to retrieve the updated parameter value, and executes the
+// apply parameter value callback to use the retrieved parameter to do some
+// operation (e.g. update logging verbosity with the updated verbosity level
+// parameter value)
+class ParameterNotifier {
+ public:
+ explicit ParameterNotifier(
+ std::unique_ptr notifier, std::string parameter_name,
+ const absl::Duration poll_frequency, std::unique_ptr sleep_for,
+ privacy_sandbox::server_common::SteadyClock& clock,
+ privacy_sandbox::server_common::log::PSLogContext& log_context)
+ : notifier_(std::move(notifier)),
+ parameter_name_(std::move(parameter_name)),
+ thread_manager_(
+ ThreadManager::Create("Parameter Notifier " + parameter_name_)),
+ poll_frequency_(poll_frequency),
+ sleep_for_(std::move(sleep_for)),
+ clock_(clock),
+ log_context_(log_context) {}
+ virtual ~ParameterNotifier() = default;
+ // Starts watching the parameter updates. The ParamType is the data type of
+ // the value for the given parameter name. The data type can be int, bool or
+ // string etc, depending on how the callbacks are defined.
+ template
+ absl::Status Start(
+ std::function(std::string_view param_name)>
+ get_param_callback,
+ std::function apply_param_callback);
+
+ // Blocks until `IsRunning` is False.
+ virtual absl::Status Stop();
+
+ // Returns False before calling `Start` or after `Stop` is
+ // successful.
+ virtual bool IsRunning() const;
+
+ static absl::StatusOr> Create(
+ NotifierMetadata notifier_metadata, std::string parameter_name,
+ const absl::Duration poll_frequency,
+ privacy_sandbox::server_common::log::PSLogContext& log_context =
+ const_cast(
+ privacy_sandbox::server_common::log::kNoOpContext));
+
+ private:
+ template
+ // Starts thread for watching the parameter updates
+ void Watch(
+ std::function(std::string_view param_name)>
+ get_param_callback,
+ std::function apply_param_callback);
+ // Gets notification from pubsub, returns error status or the notification
+ // message
+ absl::StatusOr WaitForNotification(
+ absl::Duration wait_duration,
+ const std::function& should_stop_callback);
+ absl::StatusOr ShouldGetParameter(
+ privacy_sandbox::server_common::ExpiringFlag& expiring_flag);
+ std::unique_ptr notifier_;
+ const std::string parameter_name_;
+ std::unique_ptr thread_manager_;
+ const absl::Duration poll_frequency_;
+ std::unique_ptr sleep_for_;
+ privacy_sandbox::server_common::SteadyClock& clock_;
+ privacy_sandbox::server_common::log::PSLogContext& log_context_;
+};
+
+template
+absl::Status ParameterNotifier::Start(
+ std::function(std::string_view)>
+ get_param_callback,
+ std::function apply_param_callback) {
+ return thread_manager_->Start(
+ [this, get_param_callback = std::move(get_param_callback),
+ apply_param_callback = std::move(apply_param_callback)]() {
+ Watch(std::move(get_param_callback), std::move(apply_param_callback));
+ });
+}
+
+template
+void ParameterNotifier::Watch(
+ std::function(std::string_view)>
+ get_param_callback,
+ std::function apply_param_callback) {
+ PS_LOG(INFO, log_context_)
+ << "Started to watch " << parameter_name_ << " parameter update";
+ privacy_sandbox::server_common::ExpiringFlag expiring_flag(clock_);
+ uint32_t sequential_failures = 0;
+ while (!thread_manager_->ShouldStop()) {
+ const absl::StatusOr should_get_parameter =
+ ShouldGetParameter(expiring_flag);
+ if (!should_get_parameter.ok()) {
+ ++sequential_failures;
+ const absl::Duration backoff_time =
+ std::min(expiring_flag.GetTimeRemaining(),
+ ExponentialBackoffForRetry(sequential_failures));
+ PS_LOG(ERROR, log_context_)
+ << "Failed to get parameter update notifications: " << parameter_name_
+ << ", " << should_get_parameter.status() << ". Waiting for "
+ << backoff_time;
+ if (!sleep_for_->Duration(backoff_time)) {
+ PS_LOG(ERROR, log_context_)
+ << "Failed to sleep for " << backoff_time << ". SleepFor invalid.";
+ }
+ continue;
+ }
+ sequential_failures = 0;
+ if (!*should_get_parameter) {
+ continue;
+ }
+ expiring_flag.Set(poll_frequency_);
+ auto param_result = get_param_callback(parameter_name_);
+ if (param_result.ok()) {
+ apply_param_callback(std::move(*param_result));
+ PS_VLOG(5, log_context_) << "Applied the callback on the parameter";
+ } else {
+ PS_LOG(ERROR, log_context_) << "Failed to get parameter value for "
+ << parameter_name_ << param_result.status();
+ }
+ }
+}
+
+} // namespace kv_server
+
+#endif // COMPONENTS_CLOUD_CONFIG_PARAMETER_UPDATE_PARAMETER_NOTIFIER_H_
diff --git a/components/cloud_config/parameter_update/parameter_notifier_aws.cc b/components/cloud_config/parameter_update/parameter_notifier_aws.cc
new file mode 100644
index 00000000..06b814d5
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier_aws.cc
@@ -0,0 +1,52 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+
+namespace kv_server {
+
+using privacy_sandbox::server_common::SteadyClock;
+
+absl::StatusOr ParameterNotifier::WaitForNotification(
+ absl::Duration wait_duration,
+ const std::function& should_stop_callback) {
+ absl::StatusOr> changes =
+ notifier_->GetNotifications(wait_duration, should_stop_callback);
+ if (!changes.ok()) {
+ return changes.status();
+ }
+ if ((*changes).empty()) {
+ return absl::DataLossError("Empty message in the notification");
+ }
+ // return the last element
+ PS_VLOG(5, log_context_) << "Received notification for parameter update";
+ return std::string((*changes).back());
+}
+
+absl::StatusOr> ParameterNotifier::Create(
+ NotifierMetadata notifier_metadata, std::string parameter_name,
+ const absl::Duration poll_frequency,
+ privacy_sandbox::server_common::log::PSLogContext& log_context) {
+ auto cloud_notifier_metadata =
+ std::get(notifier_metadata);
+ cloud_notifier_metadata.queue_prefix = "ParameterNotifier_";
+ PS_ASSIGN_OR_RETURN(
+ auto notifier,
+ ChangeNotifier::Create(std::move(cloud_notifier_metadata), log_context));
+ return std::make_unique(
+ std::move(notifier), std::move(parameter_name), poll_frequency,
+ std::make_unique(), SteadyClock::RealClock(), log_context);
+}
+
+} // namespace kv_server
diff --git a/components/cloud_config/parameter_update/parameter_notifier_gcp.cc b/components/cloud_config/parameter_update/parameter_notifier_gcp.cc
new file mode 100644
index 00000000..62597722
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier_gcp.cc
@@ -0,0 +1,45 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+
+namespace kv_server {
+
+using privacy_sandbox::server_common::SteadyClock;
+
+absl::StatusOr ParameterNotifier::WaitForNotification(
+ absl::Duration wait_duration,
+ const std::function& should_stop_callback) {
+ // TODO(b/356110894): Use change_notifier_gcp to get notifications from gcp
+ // pubsub once it is ready
+ sleep_for_->Duration(wait_duration);
+ return absl::DeadlineExceededError(
+ "Trigger backup poll before GCP change notifier is "
+ "implemented.");
+}
+
+absl::StatusOr> ParameterNotifier::Create(
+ NotifierMetadata notifier_metadata, std::string parameter_name,
+ const absl::Duration poll_frequency,
+ privacy_sandbox::server_common::log::PSLogContext& log_context) {
+ PS_ASSIGN_OR_RETURN(
+ auto notifier,
+ ChangeNotifier::Create(std::get(notifier_metadata),
+ log_context));
+ return std::make_unique(
+ std::move(notifier), std::move(parameter_name), poll_frequency,
+ std::make_unique(), SteadyClock::RealClock(), log_context);
+}
+
+} // namespace kv_server
diff --git a/components/cloud_config/parameter_update/parameter_notifier_local.cc b/components/cloud_config/parameter_update/parameter_notifier_local.cc
new file mode 100644
index 00000000..59d71712
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier_local.cc
@@ -0,0 +1,38 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+
+namespace kv_server {
+
+using privacy_sandbox::server_common::SteadyClock;
+
+absl::StatusOr ParameterNotifier::WaitForNotification(
+ absl::Duration wait_duration,
+ const std::function& should_stop_callback) {
+ sleep_for_->Duration(wait_duration);
+ return absl::DeadlineExceededError(
+ "Parameter pubsub notification"
+ "does not support local platform");
+}
+
+absl::StatusOr> ParameterNotifier::Create(
+ NotifierMetadata notifier_metadata, std::string parameter_name,
+ const absl::Duration poll_frequency,
+ privacy_sandbox::server_common::log::PSLogContext& log_context) {
+ return std::make_unique(
+ nullptr, std::move(parameter_name), poll_frequency,
+ std::make_unique(), SteadyClock::RealClock(), log_context);
+}
+} // namespace kv_server
diff --git a/components/cloud_config/parameter_update/parameter_notifier_test_aws.cc b/components/cloud_config/parameter_update/parameter_notifier_test_aws.cc
new file mode 100644
index 00000000..f99df5cc
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier_test_aws.cc
@@ -0,0 +1,182 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include
+#include
+#include
+
+#include "absl/synchronization/notification.h"
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+#include "components/data/common/mocks.h"
+#include "components/util/sleepfor_mock.h"
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+namespace kv_server {
+namespace {
+
+using privacy_sandbox::server_common::SimulatedSteadyClock;
+using testing::_;
+using testing::AllOf;
+using testing::Field;
+using testing::Return;
+
+class ParameterNotifierAWSTest : public ::testing::Test {
+ protected:
+ void SetUp() override {
+ std::unique_ptr mock_change_notifier =
+ std::make_unique();
+ change_notifier_ = mock_change_notifier.get();
+ std::unique_ptr mock_sleep_for =
+ std::make_unique();
+ sleep_for_ = mock_sleep_for.get();
+ notifier_ = std::make_unique(
+ std::move(mock_change_notifier), parameter_name_, poll_frequency_,
+ std::move(mock_sleep_for), sim_clock_,
+ const_cast(
+ privacy_sandbox::server_common::log::kNoOpContext));
+ }
+ std::unique_ptr notifier_;
+ MockChangeNotifier* change_notifier_;
+ MockSleepFor* sleep_for_;
+ SimulatedSteadyClock sim_clock_;
+ absl::Duration poll_frequency_ = absl::Minutes(5);
+ std::string parameter_name_ = "test_parameter";
+};
+
+TEST_F(ParameterNotifierAWSTest, NotRunning) {
+ ASSERT_FALSE(notifier_->IsRunning());
+}
+
+TEST_F(ParameterNotifierAWSTest, StartsAndStops) {
+ absl::Status status = notifier_->Start(
+ [](std::string_view param_name) { return "test_value"; },
+ [](std::string param_value) {});
+ ASSERT_TRUE(status.ok());
+ EXPECT_TRUE(notifier_->IsRunning());
+ status = notifier_->Stop();
+ ASSERT_TRUE(status.ok());
+ EXPECT_FALSE(notifier_->IsRunning());
+}
+
+TEST_F(ParameterNotifierAWSTest, NotifiesWithParameterUpdateIncludeFailures) {
+ std::string param_update_triggerred_by_notification = "value_n";
+ std::string param_update_triggerred_by_backpoll = "value_b";
+ EXPECT_CALL(*change_notifier_, GetNotifications(_, _))
+ .WillOnce(Return(absl::InvalidArgumentError("stuff")))
+ .WillOnce(Return(absl::InvalidArgumentError("stuff")))
+ .WillOnce(Return(std::vector({"pubsub_update"})))
+ .WillOnce(Return(absl::DeadlineExceededError("no message")))
+ .WillRepeatedly(Return(std::vector()));
+ absl::Notification finished;
+ testing::MockFunction(
+ std::string_view param_name)>
+ get_parameter_callback;
+ EXPECT_CALL(get_parameter_callback, Call)
+ .Times(3)
+ // called from initial poll
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll;
+ })
+ // called when notification is received
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_notification;
+ })
+ // called when there is no message during the notification wait period
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll;
+ });
+ testing::MockFunction apply_parameter_callback;
+ EXPECT_CALL(apply_parameter_callback, Call)
+ .Times(3)
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll);
+ })
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_notification);
+ })
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll);
+ finished.Notify();
+ });
+ EXPECT_CALL(*sleep_for_, Duration(_)).WillRepeatedly(Return(true));
+ absl::Status status =
+ notifier_->Start(get_parameter_callback.AsStdFunction(),
+ apply_parameter_callback.AsStdFunction());
+ ASSERT_TRUE(status.ok());
+ EXPECT_TRUE(notifier_->IsRunning());
+ finished.WaitForNotification();
+ status = notifier_->Stop();
+ ASSERT_TRUE(status.ok());
+ EXPECT_FALSE(notifier_->IsRunning());
+}
+
+TEST_F(ParameterNotifierAWSTest, BackupPollOnly) {
+ std::string param_update_triggerred_by_backpoll_1 = "value_1";
+ std::string param_update_triggerred_by_backpoll_2 = "value_2";
+ std::string param_update_triggerred_by_backpoll_3 = "value_3";
+ absl::Notification finished;
+ testing::MockFunction(
+ std::string_view param_name)>
+ get_parameter_callback;
+ EXPECT_CALL(get_parameter_callback, Call)
+ .Times(3)
+ // called from initial poll
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll_1;
+ })
+ // called due to expiring flag
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll_2;
+ })
+ // called due to timeout
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll_3;
+ });
+ testing::MockFunction apply_parameter_callback;
+ EXPECT_CALL(apply_parameter_callback, Call)
+ .Times(3)
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll_1);
+ })
+ .WillOnce([&](std::string param_value) {
+ sim_clock_.AdvanceTime(poll_frequency_ + absl::Seconds(1));
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll_2);
+ })
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll_3);
+ finished.Notify();
+ });
+ EXPECT_CALL(*change_notifier_, GetNotifications(_, _))
+ .WillOnce(Return(absl::DeadlineExceededError("time out")))
+ .WillRepeatedly(Return(std::vector()));
+ absl::Status status =
+ notifier_->Start(get_parameter_callback.AsStdFunction(),
+ apply_parameter_callback.AsStdFunction());
+ ASSERT_TRUE(status.ok());
+ EXPECT_TRUE(notifier_->IsRunning());
+ finished.WaitForNotification();
+ status = notifier_->Stop();
+ ASSERT_TRUE(status.ok());
+ EXPECT_FALSE(notifier_->IsRunning());
+}
+} // namespace
+
+} // namespace kv_server
diff --git a/components/cloud_config/parameter_update/parameter_notifier_test_gcp.cc b/components/cloud_config/parameter_update/parameter_notifier_test_gcp.cc
new file mode 100644
index 00000000..8f63e345
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier_test_gcp.cc
@@ -0,0 +1,127 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include
+#include
+#include
+
+#include "absl/synchronization/notification.h"
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+#include "components/data/common/mocks.h"
+#include "components/util/sleepfor_mock.h"
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+namespace kv_server {
+namespace {
+
+using privacy_sandbox::server_common::SimulatedSteadyClock;
+using testing::_;
+using testing::AllOf;
+using testing::Field;
+using testing::Return;
+
+// TODO(b/356110894) Remove or combine this test with AWS test once
+// change_notifier_gcp is ready to use
+class ParameterNotifierGCPTest : public ::testing::Test {
+ protected:
+ void SetUp() override {
+ std::unique_ptr mock_change_notifier =
+ std::make_unique();
+ change_notifier_ = mock_change_notifier.get();
+ std::unique_ptr mock_sleep_for =
+ std::make_unique();
+ sleep_for_ = mock_sleep_for.get();
+ notifier_ = std::make_unique(
+ std::move(mock_change_notifier), parameter_name_, poll_frequency_,
+ std::move(mock_sleep_for), sim_clock_,
+ const_cast(
+ privacy_sandbox::server_common::log::kNoOpContext));
+ }
+ std::unique_ptr notifier_;
+ MockChangeNotifier* change_notifier_;
+ MockSleepFor* sleep_for_;
+ SimulatedSteadyClock sim_clock_;
+ absl::Duration poll_frequency_ = absl::Minutes(5);
+ std::string parameter_name_ = "test_parameter";
+};
+
+TEST_F(ParameterNotifierGCPTest, NotRunning) {
+ ASSERT_FALSE(notifier_->IsRunning());
+}
+
+TEST_F(ParameterNotifierGCPTest, StartsAndStops) {
+ absl::Status status = notifier_->Start(
+ [](std::string_view param_name) { return "test_value"; },
+ [](std::string param_value) {});
+ ASSERT_TRUE(status.ok());
+ EXPECT_TRUE(notifier_->IsRunning());
+ status = notifier_->Stop();
+ ASSERT_TRUE(status.ok());
+ EXPECT_FALSE(notifier_->IsRunning());
+}
+
+TEST_F(ParameterNotifierGCPTest, BackupPollOnly) {
+ std::string param_update_triggerred_by_backpoll_1 = "value_1";
+ std::string param_update_triggerred_by_backpoll_2 = "value_2";
+ std::string param_update_triggerred_by_backpoll_3 = "value_3";
+ absl::Notification finished;
+ testing::MockFunction(
+ std::string_view param_name)>
+ get_parameter_callback;
+ EXPECT_CALL(get_parameter_callback, Call)
+ // called from initial poll
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll_1;
+ })
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll_2;
+ })
+ .WillOnce([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return param_update_triggerred_by_backpoll_3;
+ })
+ .WillRepeatedly([&](std::string_view param_name) {
+ EXPECT_EQ(param_name, parameter_name_);
+ return "";
+ });
+ testing::MockFunction apply_parameter_callback;
+ EXPECT_CALL(apply_parameter_callback, Call)
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll_1);
+ })
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll_2);
+ })
+ .WillOnce([&](std::string param_value) {
+ EXPECT_EQ(param_value, param_update_triggerred_by_backpoll_3);
+ finished.Notify();
+ })
+ .WillRepeatedly(
+ [&](std::string param_value) { EXPECT_EQ(param_value, ""); });
+ EXPECT_CALL(*sleep_for_, Duration(_)).WillRepeatedly(Return(true));
+ absl::Status status =
+ notifier_->Start(get_parameter_callback.AsStdFunction(),
+ apply_parameter_callback.AsStdFunction());
+ ASSERT_TRUE(status.ok());
+ EXPECT_TRUE(notifier_->IsRunning());
+ finished.WaitForNotification();
+ status = notifier_->Stop();
+ ASSERT_TRUE(status.ok());
+ EXPECT_FALSE(notifier_->IsRunning());
+}
+} // namespace
+} // namespace kv_server
diff --git a/components/cloud_config/parameter_update/parameter_notifier_test_local.cc b/components/cloud_config/parameter_update/parameter_notifier_test_local.cc
new file mode 100644
index 00000000..6a1ed7a7
--- /dev/null
+++ b/components/cloud_config/parameter_update/parameter_notifier_test_local.cc
@@ -0,0 +1,69 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include
+#include
+#include
+
+#include "absl/synchronization/notification.h"
+#include "components/cloud_config/parameter_update/parameter_notifier.h"
+#include "components/data/common/mocks.h"
+#include "components/util/sleepfor_mock.h"
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+namespace kv_server {
+namespace {
+
+using privacy_sandbox::server_common::SimulatedSteadyClock;
+using testing::_;
+using testing::AllOf;
+using testing::Field;
+using testing::Return;
+
+// The runtime parameter notification does not support local platform,
+// we only need to test start and stop here.
+
+class ParameterNotifierLocalTest : public ::testing::Test {
+ protected:
+ void SetUp() override {
+ notifier_ = std::make_unique(
+ nullptr, std::move(parameter_name_), poll_frequency_,
+ std::make_unique(), sim_clock_,
+ const_cast(
+ privacy_sandbox::server_common::log::kNoOpContext));
+ }
+ std::unique_ptr notifier_;
+ MockSleepFor* sleep_for_;
+ SimulatedSteadyClock sim_clock_;
+ absl::Duration poll_frequency_ = absl::Minutes(5);
+ std::string parameter_name_ = "test_parameter";
+};
+
+TEST_F(ParameterNotifierLocalTest, NotRunningSmokeTest) {
+ ASSERT_FALSE(notifier_->IsRunning());
+}
+
+TEST_F(ParameterNotifierLocalTest, StartsAndStopsSmokeTest) {
+ absl::Status status = notifier_->Start(
+ [](std::string_view param_name) { return "test_value"; },
+ [](std::string param_value) {});
+ ASSERT_TRUE(status.ok());
+ EXPECT_TRUE(notifier_->IsRunning());
+ status = notifier_->Stop();
+ ASSERT_TRUE(status.ok());
+ EXPECT_FALSE(notifier_->IsRunning());
+}
+} // namespace
+} // namespace kv_server
diff --git a/components/data/blob_storage/blob_storage_change_notifier_s3_test.cc b/components/data/blob_storage/blob_storage_change_notifier_s3_test.cc
index 69bc4e8f..dd9863bd 100644
--- a/components/data/blob_storage/blob_storage_change_notifier_s3_test.cc
+++ b/components/data/blob_storage/blob_storage_change_notifier_s3_test.cc
@@ -55,14 +55,7 @@ class MockSqsClient : public ::Aws::SQS::SQSClient {
// https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html
class BlobStorageChangeNotifierS3Test : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
void CreateRequiredSqsCallExpectations() {
static const std::string mock_sqs_url("mock sqs url");
EXPECT_CALL(mock_message_service_, IsSetupComplete)
diff --git a/components/data/blob_storage/blob_storage_client_gcp_test.cc b/components/data/blob_storage/blob_storage_client_gcp_test.cc
index ac4b4900..6f6256ff 100644
--- a/components/data/blob_storage/blob_storage_client_gcp_test.cc
+++ b/components/data/blob_storage/blob_storage_client_gcp_test.cc
@@ -51,14 +51,7 @@ class GcpBlobStorageClientTest : public ::testing::Test {
protected:
PlatformInitializer initializer_;
privacy_sandbox::server_common::log::NoOpContext no_op_context_;
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
};
TEST_F(GcpBlobStorageClientTest, DeleteBlobSucceeds) {
diff --git a/components/data/blob_storage/blob_storage_client_s3_test.cc b/components/data/blob_storage/blob_storage_client_s3_test.cc
index 938810b8..17b215f1 100644
--- a/components/data/blob_storage/blob_storage_client_s3_test.cc
+++ b/components/data/blob_storage/blob_storage_client_s3_test.cc
@@ -55,14 +55,7 @@ class MockS3Client : public ::Aws::S3::S3Client {
class BlobStorageClientS3Test : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
privacy_sandbox::server_common::log::NoOpContext no_op_context_;
private:
diff --git a/components/data/blob_storage/seeking_input_streambuf_test.cc b/components/data/blob_storage/seeking_input_streambuf_test.cc
index 845f6995..4ec47c36 100644
--- a/components/data/blob_storage/seeking_input_streambuf_test.cc
+++ b/components/data/blob_storage/seeking_input_streambuf_test.cc
@@ -66,14 +66,7 @@ class StringBlobInputStreambuf : public SeekingInputStreambuf {
class SeekingInputStreambufTest
: public testing::TestWithParam {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
StringBlobInputStreambuf CreateStringBlobStreambuf(std::string_view blob) {
TelemetryProvider::Init("test", "test");
return StringBlobInputStreambuf(blob, GetParam());
diff --git a/components/data/common/change_notifier_aws_test.cc b/components/data/common/change_notifier_aws_test.cc
index 8850a1ef..cb2d6542 100644
--- a/components/data/common/change_notifier_aws_test.cc
+++ b/components/data/common/change_notifier_aws_test.cc
@@ -55,14 +55,7 @@ class MockSqsClient : public ::Aws::SQS::SQSClient {
class ChangeNotifierAwsTest : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
private:
PlatformInitializer initializer_;
diff --git a/components/data/converters/BUILD.bazel b/components/data/converters/BUILD.bazel
new file mode 100644
index 00000000..ed98b203
--- /dev/null
+++ b/components/data/converters/BUILD.bazel
@@ -0,0 +1,80 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load("@rules_cc//cc:defs.bzl", "cc_library", "cc_test")
+
+cc_library(
+ name = "scoped_cbor",
+ hdrs = [
+ "scoped_cbor.h",
+ ],
+ deps = [
+ "@libcbor//:cbor",
+ ],
+)
+
+cc_library(
+ name = "cbor_converter_utils",
+ srcs = [
+ "cbor_converter_utils.cc",
+ ],
+ hdrs = [
+ "cbor_converter_utils.h",
+ ],
+ deps = [
+ "@com_google_absl//absl/status:statusor",
+ "@com_google_absl//absl/strings",
+ "@libcbor//:cbor",
+ ],
+)
+
+cc_library(
+ name = "cbor_converter",
+ srcs = [
+ "cbor_converter.cc",
+ ],
+ hdrs = [
+ "cbor_converter.h",
+ ],
+ visibility = [
+ "//components/data_server:__subpackages__",
+ "//components/tools:__subpackages__",
+ "//infrastructure:__subpackages__",
+ ],
+ deps = [
+ ":cbor_converter_utils",
+ ":scoped_cbor",
+ "//public/applications/pa:api_overlay_cc_proto",
+ "//public/applications/pa:response_utils",
+ "//public/query/v2:get_values_v2_cc_proto",
+ "@com_google_absl//absl/status:statusor",
+ "@com_google_protobuf//:protobuf",
+ "@google_privacysandbox_servers_common//src/util/status_macro:status_macros",
+ "@libcbor//:cbor",
+ "@nlohmann_json//:lib",
+ ],
+)
+
+cc_test(
+ name = "cbor_converter_test",
+ size = "small",
+ srcs = ["cbor_converter_test.cc"],
+ deps = [
+ ":cbor_converter",
+ "//public/test_util:proto_matcher",
+ "@com_google_googletest//:gtest",
+ "@com_google_googletest//:gtest_main",
+ "@nlohmann_json//:lib",
+ ],
+)
diff --git a/components/data/converters/cbor_converter.cc b/components/data/converters/cbor_converter.cc
new file mode 100644
index 00000000..29370ae5
--- /dev/null
+++ b/components/data/converters/cbor_converter.cc
@@ -0,0 +1,314 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/data/converters/cbor_converter.h"
+
+#include
+#include
+#include
+
+#include "absl/log/log.h"
+#include "components/data/converters/cbor_converter_utils.h"
+#include "components/data/converters/scoped_cbor.h"
+#include "google/protobuf/message.h"
+#include "google/protobuf/util/json_util.h"
+#include "nlohmann/json.hpp"
+#include "public/applications/pa/response_utils.h"
+#include "public/query/v2/get_values_v2.pb.h"
+#include "src/util/status_macro/status_macros.h"
+
+#include "cbor.h"
+
+namespace kv_server {
+
+namespace {
+inline constexpr char kCompressionGroups[] = "compressionGroups";
+inline constexpr char kCompressionGroupId[] = "compressionGroupId";
+inline constexpr char kTtlMs[] = "ttlMs";
+inline constexpr char kContent[] = "content";
+
+inline constexpr char kPartitionOutputs[] = "partitionOutputs";
+inline constexpr char kPartitionId[] = "id";
+inline constexpr char kKeyGroupOutputs[] = "keyGroupOutputs";
+inline constexpr char kTags[] = "tags";
+inline constexpr char kKeyValues[] = "keyValues";
+inline constexpr char kValue[] = "value";
+
+absl::StatusOr EncodeCompressionGroup(
+ v2::CompressionGroup& compression_group) {
+ const int compressionGroupKeysNumber = 3;
+ auto* cbor_internal = cbor_new_definite_map(compressionGroupKeysNumber);
+ if (compression_group.has_ttl_ms()) {
+ PS_RETURN_IF_ERROR(
+ CborSerializeUInt(kTtlMs, compression_group.ttl_ms(), *cbor_internal));
+ }
+ PS_RETURN_IF_ERROR(CborSerializeByteString(
+ kContent, std::move(compression_group.content()), *cbor_internal));
+ PS_RETURN_IF_ERROR(CborSerializeUInt(kCompressionGroupId,
+ compression_group.compression_group_id(),
+ *cbor_internal));
+
+ return cbor_internal;
+}
+
+absl::StatusOr EncodeCompressionGroups(
+ google::protobuf::RepeatedPtrField&
+ compression_groups) {
+ cbor_item_t* serialized_compression_groups =
+ cbor_new_definite_array(compression_groups.size());
+ for (auto& compression_group : compression_groups) {
+ PS_ASSIGN_OR_RETURN(auto* serialized_compression_group,
+ EncodeCompressionGroup(compression_group));
+ if (!cbor_array_push(serialized_compression_groups,
+ cbor_move(serialized_compression_group))) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ",
+ kCompressionGroups, " to CBOR. ",
+ compression_group));
+ }
+ }
+
+ return serialized_compression_groups;
+}
+
+absl::StatusOr EncodeKeyGroupOutput(
+ application_pa::KeyGroupOutput& key_group_output) {
+ const int keyGroupOutputKeysNumber = 2;
+ auto* cbor_internal = cbor_new_definite_map(keyGroupOutputKeysNumber);
+ // tags
+ cbor_item_t* serialized_tags =
+ cbor_new_definite_array(key_group_output.tags().size());
+
+ for (auto& tag : key_group_output.tags()) {
+ if (!cbor_array_push(serialized_tags, cbor_move(cbor_build_stringn(
+ tag.data(), tag.size())))) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ", kTags,
+ " to CBOR. ", key_group_output));
+ }
+ }
+ struct cbor_pair serialized_serialized_tags_pair = {
+ .key = cbor_move(cbor_build_stringn(kTags, sizeof(kTags) - 1)),
+ .value = serialized_tags,
+ };
+ if (!cbor_map_add(cbor_internal, serialized_serialized_tags_pair)) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ", kTags,
+ " to CBOR. ", key_group_output));
+ }
+ // key_values
+ cbor_item_t* serialized_key_values =
+ cbor_new_definite_map(key_group_output.key_values().size());
+ std::vector> kv_vector;
+ for (auto&& [key, value] : *(key_group_output.mutable_key_values())) {
+ std::string value_str = std::move(value.mutable_value()->string_value());
+ auto* cbor_internal_value = cbor_new_definite_map(1);
+ struct cbor_pair serialized_value_pair = {
+ .key = cbor_move(cbor_build_stringn(kValue, sizeof(kValue) - 1)),
+ .value =
+ cbor_move(cbor_build_stringn(value_str.c_str(), value_str.size())),
+ };
+
+ if (!cbor_map_add(cbor_internal_value, serialized_value_pair)) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ", kValue,
+ " to CBOR. ", key_group_output));
+ }
+ struct cbor_pair serialized_key_value_pair = {
+ .key = cbor_move(cbor_build_stringn(key.c_str(), key.size())),
+ .value = cbor_internal_value,
+ };
+ kv_vector.emplace_back(key, serialized_key_value_pair);
+ }
+ // Following the chromium implementation, we only need to check that
+ // the length and lexicographic order of the plaintext string
+ // https://chromium.googlesource.com/chromium/src/components/cbor/+/10d0a11b998d2cca774189ba26159ad4e1eacb7f/values.h#59
+ // https://chromium.googlesource.com/chromium/src/components/cbor/+/10d0a11b998d2cca774189ba26159ad4e1eacb7f/values.cc#109
+ std::sort(kv_vector.begin(), kv_vector.end(), [](auto& left, auto& right) {
+ const auto left_size = left.first.size();
+ const auto& left_str = left.first;
+ const auto right_size = right.first.size();
+ const auto& right_str = right.first;
+ return std::tie(left_size, left_str) < std::tie(right_size, right_str);
+ });
+ for (auto&& [key, serialized_key_value_pair] : kv_vector) {
+ if (!cbor_map_add(serialized_key_values, serialized_key_value_pair)) {
+ return absl::InternalError(absl::StrCat(
+ "Failed to serialize ", kKeyValues, " to CBOR. ", key_group_output));
+ }
+ }
+ struct cbor_pair serialized_key_values_pair = {
+ .key = cbor_move(cbor_build_stringn(kKeyValues, sizeof(kKeyValues) - 1)),
+ .value = serialized_key_values,
+ };
+ if (!cbor_map_add(cbor_internal, serialized_key_values_pair)) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ", kKeyValues,
+ " to CBOR. ", key_group_output));
+ }
+ return cbor_internal;
+}
+
+absl::StatusOr EncodePartitionOutput(
+ application_pa::PartitionOutput& partition_output) {
+ const int partitionKeysNumber = 2;
+ auto* cbor_internal = cbor_new_definite_map(partitionKeysNumber);
+ PS_RETURN_IF_ERROR(
+ CborSerializeUInt(kPartitionId, partition_output.id(), *cbor_internal));
+ cbor_item_t* serialized_key_group_outputs =
+ cbor_new_definite_array(partition_output.key_group_outputs().size());
+ for (auto& key_group_output :
+ *(partition_output.mutable_key_group_outputs())) {
+ PS_ASSIGN_OR_RETURN(auto* serialized_key_group_output,
+ EncodeKeyGroupOutput(key_group_output));
+ if (!cbor_array_push(serialized_key_group_outputs,
+ cbor_move(serialized_key_group_output))) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ",
+ kPartitionOutputs, " to CBOR",
+ partition_output));
+ }
+ }
+ struct cbor_pair serialized_key_group_outputs_pair = {
+ .key = cbor_move(
+ cbor_build_stringn(kKeyGroupOutputs, sizeof(kKeyGroupOutputs) - 1)),
+ .value = serialized_key_group_outputs,
+ };
+ if (!cbor_map_add(cbor_internal, serialized_key_group_outputs_pair)) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ",
+ kKeyGroupOutputs, " to CBOR. ",
+ partition_output));
+ }
+ return cbor_internal;
+}
+
+absl::Status EncodePartitionOutputs(
+ google::protobuf::RepeatedPtrField&
+ partition_outputs,
+ cbor_item_t* serialized_partition_outputs) {
+ for (auto& partition_output : partition_outputs) {
+ PS_ASSIGN_OR_RETURN(auto* serialized_partition_output,
+ EncodePartitionOutput(partition_output));
+ if (!cbor_array_push(serialized_partition_outputs,
+ cbor_move(serialized_partition_output))) {
+ return absl::InternalError(absl::StrCat("Failed to serialize ",
+ kPartitionOutputs, " to CBOR. ",
+ partition_output));
+ }
+ }
+ return absl::OkStatus();
+}
+
+} // namespace
+absl::StatusOr V2GetValuesResponseCborEncode(
+ v2::GetValuesResponse& response) {
+ if (response.has_single_partition()) {
+ return absl::InvalidArgumentError(
+ "single_partition is not supported for cbor content type");
+ }
+ const int getValuesResponseKeysNumber = 1;
+ ScopedCbor root(cbor_new_definite_map(getValuesResponseKeysNumber));
+ PS_ASSIGN_OR_RETURN(
+ auto* compression_groups,
+ EncodeCompressionGroups(*(response.mutable_compression_groups())));
+ struct cbor_pair serialized_compression_groups = {
+ .key = cbor_move(cbor_build_stringn(kCompressionGroups,
+ sizeof(kCompressionGroups) - 1)),
+ .value = compression_groups,
+ };
+ auto* cbor_internal = root.get();
+ if (!cbor_map_add(cbor_internal, serialized_compression_groups)) {
+ return absl::InternalError(absl::StrCat(
+ "Failed to serialize ", kCompressionGroups, " to CBOR. ", response));
+ }
+ return GetCborSerializedResult(*cbor_internal);
+}
+
+absl::StatusOr V2GetValuesRequestJsonStringCborEncode(
+ std::string_view serialized_json) {
+ nlohmann::json json_req = nlohmann::json::parse(serialized_json, nullptr,
+ /*allow_exceptions=*/false,
+ /*ignore_comments=*/true);
+ if (json_req.is_discarded()) {
+ return absl::InternalError(absl::StrCat(
+ "Unable to parse json req from string: ", serialized_json));
+ }
+ std::vector cbor_vec = nlohmann::json::to_cbor(json_req);
+ return std::string(cbor_vec.begin(), cbor_vec.end());
+}
+
+absl::StatusOr V2GetValuesRequestProtoToCborEncode(
+ const v2::GetValuesRequest& proto_req) {
+ std::string json_req_string;
+ if (const auto json_status = google::protobuf::json::MessageToJsonString(
+ proto_req, &json_req_string);
+ !json_status.ok()) {
+ return absl::InternalError(absl::StrCat(
+ "Unable to convert proto request to json string: ", proto_req));
+ }
+ return V2GetValuesRequestJsonStringCborEncode(json_req_string);
+}
+
+absl::StatusOr PartitionOutputsCborEncode(
+ google::protobuf::RepeatedPtrField&
+ partition_outputs) {
+ ScopedCbor root(cbor_new_definite_array(partition_outputs.size()));
+ auto* cbor_internal = root.get();
+ PS_RETURN_IF_ERROR(EncodePartitionOutputs(partition_outputs, cbor_internal));
+ return GetCborSerializedResult(*cbor_internal);
+}
+
+absl::StatusOr GetPartitionOutputsInJson(
+ const nlohmann::json& content_json) {
+ std::vector content_cbor = nlohmann::json::to_cbor(content_json);
+ std::string content_cbor_string =
+ std::string(content_cbor.begin(), content_cbor.end());
+ struct cbor_load_result result;
+ cbor_item_t* cbor_bytestring = cbor_load(
+ reinterpret_cast(content_cbor_string.data()),
+ content_cbor_string.size(), &result);
+ auto partition_output_cbor = cbor_bytestring_handle(cbor_bytestring);
+ auto cbor_bytestring_len = cbor_bytestring_length(cbor_bytestring);
+ return nlohmann::json::from_cbor(std::vector(
+ partition_output_cbor, partition_output_cbor + cbor_bytestring_len));
+}
+
+absl::StatusOr V2CompressionGroupCborEncode(
+ application_pa::V2CompressionGroup& comp_group) {
+ const int getCompressionGroupKeysNumber = 1;
+ ScopedCbor root(cbor_new_definite_map(getCompressionGroupKeysNumber));
+ cbor_item_t* partition_outputs =
+ cbor_new_definite_array(comp_group.partition_outputs().size());
+ PS_RETURN_IF_ERROR(EncodePartitionOutputs(
+ *(comp_group.mutable_partition_outputs()), partition_outputs));
+ struct cbor_pair serialized_partition_outputs = {
+ .key = cbor_move(
+ cbor_build_stringn(kPartitionOutputs, sizeof(kPartitionOutputs) - 1)),
+ .value = partition_outputs,
+ };
+ auto* cbor_internal = root.get();
+ if (!cbor_map_add(cbor_internal, serialized_partition_outputs)) {
+ return absl::InternalError(absl::StrCat(
+ "Failed to serialize ", kPartitionOutputs, " to CBOR. ", comp_group));
+ }
+ return GetCborSerializedResult(*cbor_internal);
+}
+
+absl::Status CborDecodeToNonBytesProto(std::string_view cbor_raw,
+ google::protobuf::Message& message) {
+ // TODO(b/353537363): Skip intermediate JSON conversion step
+ nlohmann::json json_from_cbor = nlohmann::json::from_cbor(
+ cbor_raw, /*strict=*/true, /*allow_exceptions=*/false);
+ if (json_from_cbor.is_discarded()) {
+ return absl::InternalError("Failed to convert raw CBOR buffer to JSON");
+ }
+ return google::protobuf::util::JsonStringToMessage(json_from_cbor.dump(),
+ &message);
+}
+
+} // namespace kv_server
diff --git a/components/data/converters/cbor_converter.h b/components/data/converters/cbor_converter.h
new file mode 100644
index 00000000..b545c39b
--- /dev/null
+++ b/components/data/converters/cbor_converter.h
@@ -0,0 +1,53 @@
+/*
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef COMPONENTS_DATA_CONVERTER_H
+#define COMPONENTS_DATA_CONVERTER_H
+
+#include
+
+#include "absl/status/statusor.h"
+#include "nlohmann/json.hpp"
+#include "public/applications/pa/api_overlay.pb.h"
+#include "public/query/v2/get_values_v2.pb.h"
+
+namespace kv_server {
+
+absl::StatusOr V2GetValuesResponseCborEncode(
+ v2::GetValuesResponse& response);
+
+absl::StatusOr V2CompressionGroupCborEncode(
+ application_pa::V2CompressionGroup& comp_group);
+
+absl::StatusOr V2GetValuesRequestJsonStringCborEncode(
+ std::string_view serialized_json);
+
+absl::StatusOr V2GetValuesRequestProtoToCborEncode(
+ const v2::GetValuesRequest& proto_req);
+
+absl::StatusOr PartitionOutputsCborEncode(
+ google::protobuf::RepeatedPtrField&
+ partition_outputs);
+
+absl::StatusOr GetPartitionOutputsInJson(
+ const nlohmann::json& content_json);
+
+// Converts a CBOR serialized string to a proto that does not contain a `bytes`
+// field. Will return error if the proto contains `bytes`.
+absl::Status CborDecodeToNonBytesProto(std::string_view cbor_raw,
+ google::protobuf::Message& message);
+} // namespace kv_server
+#endif // COMPONENTS_DATA_CONVERTER_H
diff --git a/components/data/converters/cbor_converter_test.cc b/components/data/converters/cbor_converter_test.cc
new file mode 100644
index 00000000..3a34100a
--- /dev/null
+++ b/components/data/converters/cbor_converter_test.cc
@@ -0,0 +1,549 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/data/converters/cbor_converter.h"
+
+#include
+
+#include "gmock/gmock.h"
+#include "google/protobuf/text_format.h"
+#include "gtest/gtest.h"
+#include "nlohmann/json.hpp"
+#include "public/query/v2/get_values_v2.pb.h"
+#include "public/test_util/proto_matcher.h"
+
+namespace kv_server {
+namespace {
+
+using json = nlohmann::json;
+using ordered_json = nlohmann::ordered_json;
+using google::protobuf::TextFormat;
+
+TEST(CborConverterTest, V2GetValuesResponseCborEncodeSuccess) {
+ // "abc" -> [97,98,99] as byte array
+ ordered_json json_etalon = nlohmann::ordered_json::parse(R"({
+ "compressionGroups": [
+ {
+ "ttlMs": 2,
+ "content": {"bytes":[97,98,99],"subtype":null},
+ "compressionGroupId": 1
+ }
+ ]
+ })");
+ v2::GetValuesResponse response;
+ TextFormat::ParseFromString(
+ R"pb(
+ compression_groups { compression_group_id: 1 ttl_ms: 2 content: "abc" }
+ )pb",
+ &response);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ V2GetValuesResponseCborEncode(response);
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok())
+ << cbor_encoded_proto_maybe.status();
+ EXPECT_EQ(
+ json_etalon.dump(),
+ nlohmann::ordered_json::from_cbor(*cbor_encoded_proto_maybe).dump());
+}
+
+TEST(CborConverterTest, V2GetValuesResponseCborEncode_SinglePartition_Failure) {
+ v2::GetValuesResponse response;
+ TextFormat::ParseFromString(
+ R"pb(
+ single_partition {}
+ )pb",
+ &response);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ V2GetValuesResponseCborEncode(response);
+ ASSERT_FALSE(cbor_encoded_proto_maybe.ok())
+ << cbor_encoded_proto_maybe.status();
+}
+
+TEST(CborConverterTest, V2GetValuesResponseCborEncodeArrayMsSuccess) {
+ ordered_json json_etalon = nlohmann::ordered_json::parse(R"({
+ "compressionGroups": [
+ {
+ "content": {"bytes":[97,98,99], "subtype":null },
+ "compressionGroupId": 1
+
+ },
+ {
+ "ttlMs": 2,
+ "content": {"bytes":[97,98,99,100], "subtype":null },
+ "compressionGroupId": 2
+ }
+ ]
+ })");
+
+ v2::GetValuesResponse response;
+ TextFormat::ParseFromString(
+ R"pb(
+ compression_groups { compression_group_id: 1 content: "abc" }
+ compression_groups { compression_group_id: 2 ttl_ms: 2 content: "abcd" }
+ )pb",
+ &response);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ V2GetValuesResponseCborEncode(response);
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok())
+ << cbor_encoded_proto_maybe.status();
+ EXPECT_EQ(json_etalon.dump(),
+ ordered_json::from_cbor(*cbor_encoded_proto_maybe).dump());
+}
+
+TEST(CborConverterTest, V2CompressionGroupCborEncodeSuccess) {
+ ordered_json json_etalon = ordered_json::parse(R"(
+ {
+ "partitionOutputs": [
+ {
+ "id": 0,
+ "keyGroupOutputs": [
+ {
+ "tags": [
+ "custom",
+ "keys"
+ ],
+ "keyValues": {
+ "hello": {
+ "value": "world"
+ }
+ }
+ },
+ {
+ "tags": [
+ "structured",
+ "groupNames"
+ ],
+ "keyValues": {
+ "hello": {
+ "value": "world"
+ }
+ }
+ }
+ ]
+ },
+ {
+ "id": 1,
+ "keyGroupOutputs": [
+ {
+ "tags": [
+ "custom",
+ "keys"
+ ],
+ "keyValues": {
+ "hello2": {
+ "value": "world2"
+ }
+ }
+ }
+ ]
+ }
+ ]
+ }
+)");
+
+ application_pa::V2CompressionGroup compression_group;
+ TextFormat::ParseFromString(
+ R"pb(partition_outputs {
+ id: 0
+ key_group_outputs {
+ tags: "custom"
+ tags: "keys"
+ key_values {
+ key: "hello"
+ value { value { string_value: "world" } }
+ }
+ }
+ key_group_outputs {
+ tags: "structured"
+ tags: "groupNames"
+ key_values {
+ key: "hello"
+ value { value { string_value: "world" } }
+ }
+ }
+ }
+ partition_outputs {
+ id: 1
+ key_group_outputs {
+ tags: "custom"
+ tags: "keys"
+ key_values {
+ key: "hello2"
+ value { value { string_value: "world2" } }
+ }
+ }
+ })pb",
+ &compression_group);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ V2CompressionGroupCborEncode(compression_group);
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok());
+ EXPECT_EQ(json_etalon.dump(),
+ ordered_json::from_cbor(*cbor_encoded_proto_maybe).dump());
+}
+
+TEST(CborConverterTest,
+ V2CompressionGroupEmptyKeyGroupOutputsCborEncodeSuccess) {
+ json json_etalon = R"(
+ {
+ "partitionOutputs": [
+ {
+ "id": 0,
+ "keyGroupOutputs": []
+ }
+ ]
+ }
+)"_json;
+
+ application_pa::V2CompressionGroup compression_group;
+ TextFormat::ParseFromString(R"pb(partition_outputs { id: 0 })pb",
+ &compression_group);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ V2CompressionGroupCborEncode(compression_group);
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok());
+ ASSERT_TRUE(json_etalon == json::from_cbor(*cbor_encoded_proto_maybe));
+}
+
+TEST(CborConverterTest, V2GetValuesRequestJsonStringCborEncodeSuccess) {
+ std::string json_request = R"({
+ "partitions": [
+ {
+ "id": 0,
+ "compressionGroupId": 0,
+ "arguments": [
+ {
+ "tags": [
+ "structured",
+ "groupNames"
+ ],
+ "data": [
+ "hello"
+ ]
+ }
+ ]
+ }
+ ]
+ })";
+ absl::StatusOr cbor_encoded_request_maybe =
+ V2GetValuesRequestJsonStringCborEncode(json_request);
+ ASSERT_TRUE(cbor_encoded_request_maybe.ok())
+ << cbor_encoded_request_maybe.status();
+ EXPECT_EQ(nlohmann::json::parse(json_request),
+ nlohmann::json::from_cbor(*cbor_encoded_request_maybe));
+}
+
+TEST(CborConverterTest,
+ V2GetValuesRequestJsonStringCborEncodeInvalidJsonFails) {
+ std::string json_request = R"({
+ "partitions": [
+ {
+ "id": 0,
+ "compressionGroupId": 0,
+ "arguments": [
+ {
+ "tags": [
+ "structured",
+ "groupNames"
+ ],
+ "data": [
+ "hello"
+ ]
+ }
+ ]
+ }
+ ],
+ })";
+ absl::StatusOr cbor_encoded_request_maybe =
+ V2GetValuesRequestJsonStringCborEncode(json_request);
+ ASSERT_FALSE(cbor_encoded_request_maybe.ok())
+ << cbor_encoded_request_maybe.status();
+}
+
+TEST(CborConverterTest, V2GetValuesRequestProtoToCborEncodeSuccess) {
+ v2::GetValuesRequest request;
+ TextFormat::ParseFromString(
+ R"pb(partitions {
+ id: 0
+ compression_group_id: 1
+ arguments { data { string_value: "hi" } }
+
+ })pb",
+ &request);
+ absl::StatusOr cbor_encoded_request_maybe =
+ V2GetValuesRequestProtoToCborEncode(request);
+ ASSERT_TRUE(cbor_encoded_request_maybe.ok())
+ << cbor_encoded_request_maybe.status();
+}
+
+TEST(CborConverterTest, CborDecodeToNonBytesProtoSuccess) {
+ v2::GetValuesRequest expected;
+ TextFormat::ParseFromString(R"pb(
+ client_version: "version1"
+ metadata {
+ fields {
+ key: "foo"
+ value { string_value: "bar1" }
+ }
+ }
+ partitions {
+ id: 1
+ compression_group_id: 1
+ metadata {
+ fields {
+ key: "partition_metadata"
+ value { string_value: "bar2" }
+ }
+ }
+ arguments {
+ tags {
+ values { string_value: "tag1" }
+ values { string_value: "tag2" }
+ }
+
+ data { string_value: "bar4" }
+ }
+ }
+ )pb",
+ &expected);
+ nlohmann::json json_message = R"(
+ {
+ "clientVersion": "version1",
+ "metadata": {
+ "foo": "bar1"
+ },
+ "partitions": [
+ {
+ "id": 1,
+ "compressionGroupId": 1,
+ "metadata": {
+ "partition_metadata": "bar2"
+ },
+ "arguments": {
+ "tags": [
+ "tag1",
+ "tag2"
+ ],
+ "data": "bar4"
+ }
+ }
+ ]
+}
+)"_json;
+ ::kv_server::v2::GetValuesRequest actual;
+ std::vector v = json::to_cbor(json_message);
+ std::string cbor_raw(v.begin(), v.end());
+ const auto status = CborDecodeToNonBytesProto(cbor_raw, actual);
+ ASSERT_TRUE(status.ok());
+ EXPECT_THAT(actual, EqualsProto(expected));
+}
+
+TEST(CborConverterTest, CborDecodeToNonBytesProtoGetValuesResponseFailure) {
+ json json_etalon = R"({
+ "compressionGroups": [
+ {
+ "compressionGroupId": 1,
+ "content": {"bytes":[97,98,99],"subtype":null},
+ "ttlMs": 2
+ }
+ ]
+ })"_json;
+ v2::GetValuesResponse actual;
+ std::vector v = json::to_cbor(json_etalon);
+ std::string cbor_raw(v.begin(), v.end());
+ const auto status = CborDecodeToNonBytesProto(cbor_raw, actual);
+ ASSERT_FALSE(status.ok()) << status;
+}
+
+TEST(CborConverterTest, CborDecodeToNonBytesProtoFailure) {
+ nlohmann::json json_message = R"(
+ {
+ "clientVersion": "version1",
+ "metadata": {
+ "foo": "bar1"
+ },
+ "partitions": [
+ {
+ "id": 1,
+ "compressionGroupId": 1,
+ "metadata": {
+ "partition_metadata": "bar2"
+ },
+ "arguments": {
+ "tags": [
+ "tag1",
+ "tag2"
+ ],
+ "data": "bar4"
+ }
+ }
+ ]
+}
+)"_json;
+ ::kv_server::v2::GetValuesRequest actual;
+ std::vector v = json::to_cbor(json_message);
+ std::string cbor_raw(v.begin(), --v.end());
+ const auto status = CborDecodeToNonBytesProto(cbor_raw, actual);
+ ASSERT_FALSE(status.ok());
+}
+
+TEST(CborConverterTest, PartitionOutputsCborEncodeSuccess) {
+ ordered_json json_etalon = ordered_json::parse(R"(
+ [
+ {
+ "id": 0,
+ "keyGroupOutputs": [
+ {
+ "tags": [
+ "custom",
+ "keys"
+ ],
+ "keyValues": {
+ "hello": {
+ "value": "world"
+ }
+ }
+ }
+ ]
+ },
+ {
+ "id": 1,
+ "keyGroupOutputs": [
+ {
+ "tags": [
+ "custom",
+ "keys"
+ ],
+ "keyValues": {
+ "hello2": {
+ "value": "world2"
+ }
+ }
+ }
+ ]
+ }
+ ]
+)");
+
+ application_pa::V2CompressionGroup compression_group;
+ TextFormat::ParseFromString(
+ R"pb(partition_outputs {
+ id: 0
+ key_group_outputs {
+ tags: "custom"
+ tags: "keys"
+ key_values {
+ key: "hello"
+ value { value { string_value: "world" } }
+ }
+ }
+ }
+ partition_outputs {
+ id: 1
+ key_group_outputs {
+ tags: "custom"
+ tags: "keys"
+ key_values {
+ key: "hello2"
+ value { value { string_value: "world2" } }
+ }
+ }
+ })pb",
+ &compression_group);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ PartitionOutputsCborEncode(
+ *compression_group.mutable_partition_outputs());
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok())
+ << cbor_encoded_proto_maybe.status();
+ EXPECT_EQ(json_etalon, ordered_json::from_cbor(*cbor_encoded_proto_maybe));
+}
+
+TEST(CborConverterTest, PartitionOutputsCborEncodeEmptyKeyGroupOutputsSuccess) {
+ ordered_json json_etalon = nlohmann::ordered_json::parse(R"(
+ [
+ {
+ "id": 0,
+ "keyGroupOutputs": []
+ }
+ ]
+)");
+
+ application_pa::V2CompressionGroup compression_group;
+ TextFormat::ParseFromString(R"pb(partition_outputs { id: 0 })pb",
+ &compression_group);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ PartitionOutputsCborEncode(
+ *compression_group.mutable_partition_outputs());
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok());
+ ASSERT_TRUE(json_etalon ==
+ ordered_json::from_cbor(*cbor_encoded_proto_maybe));
+}
+
+TEST(CborConverterTest, PartitionOutputsCborEncodeKeyValueMapOrderSuccess) {
+ ordered_json json_etalon = ordered_json::parse(R"(
+ [
+ {
+ "id": 0,
+ "keyGroupOutputs": [
+ {
+ "tags": [
+ "custom",
+ "keys"
+ ],
+ "keyValues": {
+ "a": {
+ "value": "first"
+ },
+ "b": {
+ "value": "second"
+ },
+ "ab": {
+ "value": "third"
+ }
+ }
+ }
+ ]
+ }
+ ]
+)");
+
+ application_pa::V2CompressionGroup compression_group;
+ TextFormat::ParseFromString(
+ R"pb(partition_outputs {
+ id: 0
+ key_group_outputs {
+ tags: "custom"
+ tags: "keys"
+ key_values {
+ key: "b"
+ value { value { string_value: "second" } }
+ }
+ key_values {
+ key: "a"
+ value { value { string_value: "first" } }
+ }
+ key_values {
+ key: "ab"
+ value { value { string_value: "third" } }
+ }
+ }
+ })pb",
+ &compression_group);
+ absl::StatusOr cbor_encoded_proto_maybe =
+ PartitionOutputsCborEncode(
+ *compression_group.mutable_partition_outputs());
+ ASSERT_TRUE(cbor_encoded_proto_maybe.ok())
+ << cbor_encoded_proto_maybe.status();
+ EXPECT_EQ(json_etalon, ordered_json::from_cbor(*cbor_encoded_proto_maybe));
+}
+
+} // namespace
+} // namespace kv_server
diff --git a/components/data/converters/cbor_converter_utils.cc b/components/data/converters/cbor_converter_utils.cc
new file mode 100644
index 00000000..e941182f
--- /dev/null
+++ b/components/data/converters/cbor_converter_utils.cc
@@ -0,0 +1,86 @@
+// Copyright 2024 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "components/data/converters/cbor_converter_utils.h"
+
+#include "absl/strings/str_cat.h"
+
+// TODO(b/355941387): move common methods to the common repo
+namespace kv_server {
+cbor_item_t* cbor_build_uint(uint32_t input) {
+ if (input <= 255) {
+ return cbor_build_uint8(input);
+ } else if (input <= 65535) {
+ return cbor_build_uint16(input);
+ }
+ return cbor_build_uint32(input);
+}
+
+absl::Status CborSerializeUInt(absl::string_view key, uint32_t value,
+ cbor_item_t& root) {
+ struct cbor_pair kv = {
+ .key = cbor_move(cbor_build_stringn(key.data(), key.size())),
+ .value = cbor_build_uint(value)};
+ if (!cbor_map_add(&root, kv)) {
+ return absl::InternalError(
+ absl::StrCat("Failed to serialize ", key, " to CBOR"));
+ }
+ return absl::OkStatus();
+}
+
+absl::Status CborSerializeString(absl::string_view key, absl::string_view value,
+ cbor_item_t& root) {
+ struct cbor_pair kv = {
+ .key = cbor_move(cbor_build_stringn(key.data(), key.size())),
+ .value = cbor_move(cbor_build_stringn(value.data(), value.size()))};
+ if (!cbor_map_add(&root, kv)) {
+ return absl::InternalError(
+ absl::StrCat("Failed to serialize ", key, " to CBOR"));
+ }
+
+ return absl::OkStatus();
+}
+
+absl::Status CborSerializeByteString(absl::string_view key,
+ absl::string_view value,
+ cbor_item_t& root) {
+ struct cbor_pair kv = {
+ .key = cbor_move(cbor_build_stringn(key.data(), key.size())),
+ .value = cbor_move(cbor_build_bytestring(
+ reinterpret_cast(value.data()), value.size()))};
+ if (!cbor_map_add(&root, kv)) {
+ return absl::InternalError(
+ absl::StrCat("Failed to serialize ", key, " to CBOR"));
+ }
+
+ return absl::OkStatus();
+}
+
+absl::StatusOr GetCborSerializedResult(
+ cbor_item_t& cbor_data_root) {
+ const size_t cbor_serialized_data_size =
+ cbor_serialized_size(&cbor_data_root);
+ if (!cbor_serialized_data_size) {
+ return absl::InternalError("Failed to serialize to CBOR (too large!)");
+ }
+ std::string byte_string;
+ byte_string.resize(cbor_serialized_data_size);
+ if (cbor_serialize(&cbor_data_root,
+ reinterpret_cast(byte_string.data()),
+ cbor_serialized_data_size) == 0) {
+ return absl::InternalError("Failed to serialize to CBOR");
+ }
+ return byte_string;
+}
+} // namespace kv_server
diff --git a/components/data/converters/cbor_converter_utils.h b/components/data/converters/cbor_converter_utils.h
new file mode 100644
index 00000000..978fe9be
--- /dev/null
+++ b/components/data/converters/cbor_converter_utils.h
@@ -0,0 +1,43 @@
+/*
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef COMPONENTS_DATA_CONVERTER_UTILS_H
+#define COMPONENTS_DATA_CONVERTER_UTILS_H
+
+#include
+
+#include "absl/status/statusor.h"
+
+#include "cbor.h"
+
+namespace kv_server {
+cbor_item_t* cbor_build_uint(uint32_t input);
+
+absl::Status CborSerializeUInt(absl::string_view key, uint32_t value,
+ cbor_item_t& root);
+
+absl::Status CborSerializeString(absl::string_view key, absl::string_view value,
+ cbor_item_t& root);
+
+absl::Status CborSerializeByteString(absl::string_view key,
+ absl::string_view value,
+ cbor_item_t& root);
+
+absl::StatusOr GetCborSerializedResult(
+ cbor_item_t& cbor_data_root);
+
+} // namespace kv_server
+#endif // COMPONENTS_DATA_CONVERTER_UTILS_H
diff --git a/components/data/converters/scoped_cbor.h b/components/data/converters/scoped_cbor.h
new file mode 100644
index 00000000..2d76508f
--- /dev/null
+++ b/components/data/converters/scoped_cbor.h
@@ -0,0 +1,64 @@
+/*
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef SERVICES_COMMON_UTIL_SCOPED_CBOR_H_
+#define SERVICES_COMMON_UTIL_SCOPED_CBOR_H_
+
+#include "cbor.h"
+
+// TODO(b/355941387): move to the common repo
+namespace kv_server {
+
+// Wrapper class for managing the ref-counted CBOR objects.
+class ScopedCbor {
+ public:
+ ScopedCbor() {}
+
+ // Expects a pointer to an object that is ref-incremented at the
+ // time of creation (e.g. an object created with cbor_build_string).
+ explicit ScopedCbor(cbor_item_t* val) { ptr_ = val; }
+ virtual ~ScopedCbor() {
+ if (ptr_) {
+ cbor_decref(&ptr_);
+ }
+ }
+ ScopedCbor& operator=(cbor_item_t* val) {
+ if (ptr_) {
+ cbor_decref(&ptr_);
+ }
+ ptr_ = val;
+ return *this;
+ }
+
+ cbor_item_t* operator->() const { return ptr_; }
+ cbor_item_t* operator*() const { return ptr_; }
+ cbor_item_t* get() const { return ptr_; }
+ cbor_item_t* release() {
+ cbor_item_t* tmp = ptr_;
+ ptr_ = nullptr;
+ return tmp;
+ }
+
+ explicit operator bool() const { return ptr_ != nullptr; }
+ bool operator!() const { return ptr_ == nullptr; }
+
+ private:
+ cbor_item_t* ptr_ = nullptr;
+};
+
+} // namespace kv_server
+
+#endif // SERVICES_COMMON_UTIL_SCOPED_CBOR_H_
diff --git a/components/data/realtime/delta_file_record_change_notifier_aws_test.cc b/components/data/realtime/delta_file_record_change_notifier_aws_test.cc
index e2f91460..4bd036eb 100644
--- a/components/data/realtime/delta_file_record_change_notifier_aws_test.cc
+++ b/components/data/realtime/delta_file_record_change_notifier_aws_test.cc
@@ -56,12 +56,7 @@ constexpr std::string_view kX64EncodedMessage =
class DeltaFileRecordChangeNotifierAwsTest : public ::testing::Test {
protected:
void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
+ kv_server::InitMetricsContextMap();
mock_change_notifier_ = std::make_unique();
}
std::unique_ptr mock_change_notifier_;
diff --git a/components/data/realtime/realtime_notifier_aws_test.cc b/components/data/realtime/realtime_notifier_aws_test.cc
index 238acdc1..f09bc0d5 100644
--- a/components/data/realtime/realtime_notifier_aws_test.cc
+++ b/components/data/realtime/realtime_notifier_aws_test.cc
@@ -36,14 +36,7 @@ using privacy_sandbox::server_common::GetTracer;
class RealtimeNotifierAwsTest : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
std::unique_ptr change_notifier_ =
std::make_unique();
std::unique_ptr mock_sleep_for_ =
diff --git a/components/data/realtime/realtime_notifier_gcp_test.cc b/components/data/realtime/realtime_notifier_gcp_test.cc
index fa6e5603..76e5ae33 100644
--- a/components/data/realtime/realtime_notifier_gcp_test.cc
+++ b/components/data/realtime/realtime_notifier_gcp_test.cc
@@ -46,14 +46,7 @@ using testing::Return;
class RealtimeNotifierGcpTest : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
std::unique_ptr mock_sleep_for_ =
std::make_unique();
std::shared_ptr mock_ =
diff --git a/components/data/realtime/realtime_thread_pool_manager_aws_test.cc b/components/data/realtime/realtime_thread_pool_manager_aws_test.cc
index 3ee12dc2..38eb0b34 100644
--- a/components/data/realtime/realtime_thread_pool_manager_aws_test.cc
+++ b/components/data/realtime/realtime_thread_pool_manager_aws_test.cc
@@ -34,14 +34,7 @@ using testing::Return;
class RealtimeThreadPoolNotifierAwsTest : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
int32_t thread_number_ = 4;
};
diff --git a/components/data/realtime/realtime_thread_pool_manager_gcp_test.cc b/components/data/realtime/realtime_thread_pool_manager_gcp_test.cc
index b56e4282..bb48b085 100644
--- a/components/data/realtime/realtime_thread_pool_manager_gcp_test.cc
+++ b/components/data/realtime/realtime_thread_pool_manager_gcp_test.cc
@@ -39,14 +39,7 @@ using testing::Return;
class RealtimeThreadPoolNotifierGcpTest : public ::testing::Test {
protected:
- void SetUp() override {
- privacy_sandbox::server_common::telemetry::TelemetryConfig config_proto;
- config_proto.set_mode(
- privacy_sandbox::server_common::telemetry::TelemetryConfig::PROD);
- kv_server::KVServerContextMap(
- privacy_sandbox::server_common::telemetry::BuildDependentConfig(
- config_proto));
- }
+ void SetUp() override { kv_server::InitMetricsContextMap(); }
int32_t thread_number_ = 4;
std::unique_ptr mock_sleep_for_ =
std::make_unique();
diff --git a/components/data_server/cache/BUILD.bazel b/components/data_server/cache/BUILD.bazel
index 7def83d5..c6812308 100644
--- a/components/data_server/cache/BUILD.bazel
+++ b/components/data_server/cache/BUILD.bazel
@@ -20,9 +20,9 @@ package(default_visibility = [
])
cc_library(
- name = "uint32_value_set",
- srcs = ["uint32_value_set.cc"],
- hdrs = ["uint32_value_set.h"],
+ name = "uint_value_set",
+ srcs = ["uint_value_set.cc"],
+ hdrs = ["uint_value_set.h"],
deps = [
"@com_google_absl//absl/container:btree",
"@com_google_absl//absl/container:flat_hash_map",
@@ -32,13 +32,13 @@ cc_library(
)
cc_test(
- name = "uint32_value_set_test",
+ name = "uint_value_set_test",
size = "small",
srcs = [
- "uint32_value_set_test.cc",
+ "uint_value_set_test.cc",
],
deps = [
- ":uint32_value_set",
+ ":uint_value_set",
"@com_google_googletest//:gtest",
"@com_google_googletest//:gtest_main",
"@roaring_bitmap//:c_roaring",
@@ -54,7 +54,7 @@ cc_library(
"get_key_value_set_result.h",
],
deps = [
- ":uint32_value_set",
+ ":uint_value_set",
"//components/container:thread_safe_hash_map",
"@com_google_absl//absl/container:flat_hash_map",
"@com_google_absl//absl/container:flat_hash_set",
@@ -74,6 +74,38 @@ cc_library(
],
)
+cc_library(
+ name = "uint_value_set_cache",
+ hdrs = [
+ "uint_value_set_cache.h",
+ ],
+ deps = [
+ ":get_key_value_set_result_impl",
+ ":uint_value_set",
+ "//components/container:thread_safe_hash_map",
+ "//components/util:request_context",
+ "//public:base_types_cc_proto",
+ "@com_google_absl//absl/base",
+ "@com_google_absl//absl/container:btree",
+ "@com_google_absl//absl/log",
+ "@google_privacysandbox_servers_common//src/telemetry:telemetry_provider",
+ ],
+)
+
+cc_test(
+ name = "uint_value_set_cache_test",
+ size = "small",
+ srcs = [
+ "uint_value_set_cache_test.cc",
+ ],
+ deps = [
+ ":uint_value_set_cache",
+ "@com_google_googletest//:gtest",
+ "@com_google_googletest//:gtest_main",
+ "@roaring_bitmap//:c_roaring",
+ ],
+)
+
cc_library(
name = "key_value_cache",
srcs = [
@@ -85,7 +117,8 @@ cc_library(
deps = [
":cache",
":get_key_value_set_result_impl",
- ":uint32_value_set",
+ ":uint_value_set",
+ ":uint_value_set_cache",
"//components/container:thread_safe_hash_map",
"//public:base_types_cc_proto",
"@com_google_absl//absl/base",
@@ -120,7 +153,7 @@ cc_library(
hdrs = ["mocks.h"],
deps = [
":cache",
- ":uint32_value_set",
+ ":uint_value_set",
"//components/container:thread_safe_hash_map",
"@com_google_googletest//:gtest",
],
diff --git a/components/data_server/cache/cache.h b/components/data_server/cache/cache.h
index 06081599..31f510e5 100644
--- a/components/data_server/cache/cache.h
+++ b/components/data_server/cache/cache.h
@@ -49,6 +49,10 @@ class Cache {
const RequestContext& request_context,
const absl::flat_hash_set& key_set) const = 0;
+ virtual std::unique_ptr GetUInt64ValueSet(
+ const RequestContext& request_context,
+ const absl::flat_hash_set& key_set) const = 0;
+
// Inserts or updates the key with the new value for a given prefix
virtual void UpdateKeyValue(
privacy_sandbox::server_common::log::PSLogContext& log_context,
@@ -69,6 +73,11 @@ class Cache {
std::string_view key, absl::Span value_set,
int64_t logical_commit_time, std::string_view prefix = "") = 0;
+ virtual void UpdateKeyValueSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix = "") = 0;
+
// Deletes a particular (key, value) pair for a given prefix.
virtual void DeleteKey(
privacy_sandbox::server_common::log::PSLogContext& log_context,
@@ -91,6 +100,11 @@ class Cache {
std::string_view key, absl::Span value_set,
int64_t logical_commit_time, std::string_view prefix = "") = 0;
+ virtual void DeleteValuesInSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix = "") = 0;
+
// Removes the values that were deleted before the specified
// logical_commit_time for a given prefix.
virtual void RemoveDeletedKeys(
diff --git a/components/data_server/cache/get_key_value_set_result.h b/components/data_server/cache/get_key_value_set_result.h
index f19aed41..b98246fe 100644
--- a/components/data_server/cache/get_key_value_set_result.h
+++ b/components/data_server/cache/get_key_value_set_result.h
@@ -22,7 +22,7 @@
#include "absl/container/flat_hash_set.h"
#include "components/container/thread_safe_hash_map.h"
-#include "components/data_server/cache/uint32_value_set.h"
+#include "components/data_server/cache/uint_value_set.h"
namespace kv_server {
// Class that holds the data retrieved from cache lookup and read locks for
@@ -36,6 +36,8 @@ class GetKeyValueSetResult {
std::string_view key) const = 0;
virtual const UInt32ValueSet* GetUInt32ValueSet(
std::string_view key) const = 0;
+ virtual const UInt64ValueSet* GetUInt64ValueSet(
+ std::string_view key) const = 0;
private:
// Adds key, value_set to the result data map, mantains the lock on `key`
@@ -43,14 +45,20 @@ class GetKeyValueSetResult {
virtual void AddKeyValueSet(
std::string_view key, absl::flat_hash_set value_set,
std::unique_ptr key_lock) = 0;
- virtual void AddUInt32ValueSet(
+ virtual void AddUIntValueSet(
std::string_view key,
ThreadSafeHashMap::ConstLockedNodePtr
value_set_node) = 0;
+ virtual void AddUIntValueSet(
+ std::string_view key,
+ ThreadSafeHashMap::ConstLockedNodePtr
+ value_set_node) = 0;
static std::unique_ptr Create();
friend class KeyValueCache;
+ template
+ friend class UIntValueSetCache;
};
} // namespace kv_server
diff --git a/components/data_server/cache/get_key_value_set_result_impl.cc b/components/data_server/cache/get_key_value_set_result_impl.cc
index 8e3b44ce..ac5385d3 100644
--- a/components/data_server/cache/get_key_value_set_result_impl.cc
+++ b/components/data_server/cache/get_key_value_set_result_impl.cc
@@ -27,6 +27,8 @@ namespace {
using UInt32ValueSetNodePtr =
ThreadSafeHashMap::ConstLockedNodePtr;
+using UInt64ValueSetNodePtr =
+ ThreadSafeHashMap::ConstLockedNodePtr;
// Class that holds the data retrieved from cache lookup and read locks for
// the lookup keys
@@ -51,8 +53,16 @@ class GetKeyValueSetResultImpl : public GetKeyValueSetResult {
}
const UInt32ValueSet* GetUInt32ValueSet(std::string_view key) const override {
- if (auto iter = uin32t_sets_map_.find(key);
- iter != uin32t_sets_map_.end() && iter->second.is_present()) {
+ if (auto iter = uint32_sets_map_.find(key);
+ iter != uint32_sets_map_.end() && iter->second.is_present()) {
+ return iter->second.value();
+ }
+ return nullptr;
+ }
+
+ const UInt64ValueSet* GetUInt64ValueSet(std::string_view key) const override {
+ if (auto iter = uint64_sets_map_.find(key);
+ iter != uint64_sets_map_.end() && iter->second.is_present()) {
return iter->second.value();
}
return nullptr;
@@ -68,15 +78,21 @@ class GetKeyValueSetResultImpl : public GetKeyValueSetResult {
data_map_.emplace(key, std::move(value_set));
}
- void AddUInt32ValueSet(std::string_view key,
- UInt32ValueSetNodePtr value_set_ptr) override {
- uin32t_sets_map_.emplace(key, std::move(value_set_ptr));
+ void AddUIntValueSet(std::string_view key,
+ UInt32ValueSetNodePtr value_set_ptr) override {
+ uint32_sets_map_.emplace(key, std::move(value_set_ptr));
+ }
+
+ void AddUIntValueSet(std::string_view key,
+ UInt64ValueSetNodePtr value_set_ptr) override {
+ uint64_sets_map_.emplace(key, std::move(value_set_ptr));
}
std::vector> read_locks_;
absl::flat_hash_map>
data_map_;
- absl::flat_hash_map uin32t_sets_map_;
+ absl::flat_hash_map uint32_sets_map_;
+ absl::flat_hash_map uint64_sets_map_;
};
} // namespace
diff --git a/components/data_server/cache/key_value_cache.cc b/components/data_server/cache/key_value_cache.cc
index fb2cf22a..5072c2f5 100644
--- a/components/data_server/cache/key_value_cache.cc
+++ b/components/data_server/cache/key_value_cache.cc
@@ -95,9 +95,31 @@ std::unique_ptr KeyValueCache::GetUInt32ValueSet(
ScopeLatencyMetricsRecorder
latency_recorder(request_context.GetInternalLookupMetricsContext());
- auto result = GetKeyValueSetResult::Create();
+ auto result = uint32_sets_cache_.GetValueSet(request_context, key_set);
+ for (const auto& key : key_set) {
+ if (result->GetUInt32ValueSet(key) != nullptr) {
+ LogCacheAccessMetrics(request_context, kKeyValueSetCacheHit);
+ } else {
+ LogCacheAccessMetrics(request_context, kKeyValueSetCacheMiss);
+ }
+ }
+ return result;
+}
+
+// Looks up and returns int64 value set result for the given key set.
+std::unique_ptr KeyValueCache::GetUInt64ValueSet(
+ const RequestContext& request_context,
+ const absl::flat_hash_set& key_set) const {
+ ScopeLatencyMetricsRecorder
+ latency_recorder(request_context.GetInternalLookupMetricsContext());
+ auto result = uint64_sets_cache_.GetValueSet(request_context, key_set);
for (const auto& key : key_set) {
- result->AddUInt32ValueSet(key, uint32_sets_map_.CGet(key));
+ if (result->GetUInt32ValueSet(key) != nullptr) {
+ LogCacheAccessMetrics(request_context, kKeyValueSetCacheHit);
+ } else {
+ LogCacheAccessMetrics(request_context, kKeyValueSetCacheMiss);
+ }
}
return result;
}
@@ -231,18 +253,23 @@ void KeyValueCache::UpdateKeyValueSet(
ScopeLatencyMetricsRecorder
latency_recorder(KVServerContextMap()->SafeMetric());
- if (auto prefix_max_time_node =
- uint32_sets_max_cleanup_commit_time_map_.CGet(prefix);
- prefix_max_time_node.is_present() &&
- logical_commit_time <= *prefix_max_time_node.value()) {
- return; // Skip old updates.
- }
- auto cached_set_node = uint32_sets_map_.Get(key);
- if (!cached_set_node.is_present()) {
- auto result = uint32_sets_map_.PutIfAbsent(key, UInt32ValueSet());
- cached_set_node = std::move(result.first);
- }
- cached_set_node.value()->Add(value_set, logical_commit_time);
+ PS_VLOG(9, log_context) << "Received update for [" << key << "] at "
+ << logical_commit_time;
+ uint32_sets_cache_.UpdateSetValues(log_context, key, value_set,
+ logical_commit_time, prefix);
+}
+
+void KeyValueCache::UpdateKeyValueSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix) {
+ ScopeLatencyMetricsRecorder
+ latency_recorder(KVServerContextMap()->SafeMetric());
+ PS_VLOG(9, log_context) << "Received update for [" << key << "] at "
+ << logical_commit_time;
+ uint64_sets_cache_.UpdateSetValues(log_context, key, value_set,
+ logical_commit_time, prefix);
}
void KeyValueCache::DeleteKey(
@@ -346,32 +373,23 @@ void KeyValueCache::DeleteValuesInSet(
ScopeLatencyMetricsRecorder
latency_recorder(KVServerContextMap()->SafeMetric());
- if (auto prefix_max_time_node =
- uint32_sets_max_cleanup_commit_time_map_.CGet(prefix);
- prefix_max_time_node.is_present() &&
- logical_commit_time <= *prefix_max_time_node.value()) {
- return; // Skip old deletes.
- }
- {
- auto cached_set_node = uint32_sets_map_.Get(key);
- if (!cached_set_node.is_present()) {
- auto result = uint32_sets_map_.PutIfAbsent(key, UInt32ValueSet());
- cached_set_node = std::move(result.first);
- }
- cached_set_node.value()->Remove(value_set, logical_commit_time);
- }
- {
- // Mark set as having deleted elements.
- auto prefix_deleted_sets_node = deleted_uint32_sets_map_.Get(prefix);
- if (!prefix_deleted_sets_node.is_present()) {
- auto result = deleted_uint32_sets_map_.PutIfAbsent(
- prefix, absl::btree_map>());
- prefix_deleted_sets_node = std::move(result.first);
- }
- auto* commit_time_sets =
- &(*prefix_deleted_sets_node.value())[logical_commit_time];
- commit_time_sets->insert(std::string(key));
- }
+ PS_VLOG(9, log_context) << "Received delete for [" << key << "] at "
+ << logical_commit_time;
+ uint32_sets_cache_.DeleteSetValues(log_context, key, value_set,
+ logical_commit_time, prefix);
+}
+
+void KeyValueCache::DeleteValuesInSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix) {
+ ScopeLatencyMetricsRecorder
+ latency_recorder(KVServerContextMap()->SafeMetric());
+ PS_VLOG(9, log_context) << "Received delete for [" << key << "] at "
+ << logical_commit_time;
+ uint64_sets_cache_.DeleteSetValues(log_context, key, value_set,
+ logical_commit_time, prefix);
}
void KeyValueCache::RemoveDeletedKeys(
@@ -382,7 +400,7 @@ void KeyValueCache::RemoveDeletedKeys(
latency_recorder(KVServerContextMap()->SafeMetric());
CleanUpKeyValueMap(log_context, logical_commit_time, prefix);
CleanUpKeyValueSetMap(log_context, logical_commit_time, prefix);
- CleanUpUInt32SetMap(log_context, logical_commit_time, prefix);
+ CleanUpUIntSetMaps(log_context, logical_commit_time, prefix);
}
void KeyValueCache::CleanUpKeyValueMap(
@@ -473,47 +491,17 @@ void KeyValueCache::CleanUpKeyValueSetMap(
}
}
-void KeyValueCache::CleanUpUInt32SetMap(
+void KeyValueCache::CleanUpUIntSetMaps(
privacy_sandbox::server_common::log::PSLogContext& log_context,
int64_t logical_commit_time, std::string_view prefix) {
ScopeLatencyMetricsRecorder
+ kCleanUpUIntSetMapLatency>
latency_recorder(KVServerContextMap()->SafeMetric());
- {
- if (auto max_cleanup_time_node =
- uint32_sets_max_cleanup_commit_time_map_.PutIfAbsent(
- prefix, logical_commit_time);
- *max_cleanup_time_node.first.value() < logical_commit_time) {
- *max_cleanup_time_node.first.value() = logical_commit_time;
- }
- }
- absl::flat_hash_set cleanup_sets;
- {
- auto prefix_deleted_sets_node = deleted_uint32_sets_map_.Get(prefix);
- if (!prefix_deleted_sets_node.is_present()) {
- return; // nothing to cleanup for this prefix.
- }
- absl::flat_hash_set cleanup_commit_times;
- for (const auto& [commit_time, deleted_sets] :
- *prefix_deleted_sets_node.value()) {
- if (commit_time > logical_commit_time) {
- break;
- }
- cleanup_commit_times.insert(commit_time);
- cleanup_sets.insert(deleted_sets.begin(), deleted_sets.end());
- }
- for (auto commit_time : cleanup_commit_times) {
- prefix_deleted_sets_node.value()->erase(
- prefix_deleted_sets_node.value()->find(commit_time));
- }
- }
- {
- for (const auto& set : cleanup_sets) {
- if (auto set_node = uint32_sets_map_.Get(set); set_node.is_present()) {
- set_node.value()->Cleanup(logical_commit_time);
- }
- }
- }
+ PS_VLOG(9, log_context)
+ << "Cleaning up uint set maps with a new cutoff timestamp: "
+ << logical_commit_time;
+ uint32_sets_cache_.CleanUpValueSets(log_context, logical_commit_time);
+ uint64_sets_cache_.CleanUpValueSets(log_context, logical_commit_time);
}
void KeyValueCache::LogCacheAccessMetrics(
diff --git a/components/data_server/cache/key_value_cache.h b/components/data_server/cache/key_value_cache.h
index 61e08c91..9e05638c 100644
--- a/components/data_server/cache/key_value_cache.h
+++ b/components/data_server/cache/key_value_cache.h
@@ -26,10 +26,10 @@
#include "absl/container/btree_map.h"
#include "absl/container/flat_hash_map.h"
#include "absl/container/flat_hash_set.h"
-#include "components/container/thread_safe_hash_map.h"
#include "components/data_server/cache/cache.h"
#include "components/data_server/cache/get_key_value_set_result.h"
-#include "components/data_server/cache/uint32_value_set.h"
+#include "components/data_server/cache/uint_value_set.h"
+#include "components/data_server/cache/uint_value_set_cache.h"
namespace kv_server {
// In-memory datastore.
@@ -51,6 +51,10 @@ class KeyValueCache : public Cache {
const RequestContext& request_context,
const absl::flat_hash_set& key_set) const override;
+ std::unique_ptr GetUInt64ValueSet(
+ const RequestContext& request_context,
+ const absl::flat_hash_set& key_set) const override;
+
// Inserts or updates the key with the new value for a given prefix
void UpdateKeyValue(
privacy_sandbox::server_common::log::PSLogContext& log_context,
@@ -71,6 +75,11 @@ class KeyValueCache : public Cache {
std::string_view key, absl::Span value_set,
int64_t logical_commit_time, std::string_view prefix = "") override;
+ void UpdateKeyValueSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix = "") override;
+
// Deletes a particular (key, value) pair for a given prefix.
void DeleteKey(privacy_sandbox::server_common::log::PSLogContext& log_context,
std::string_view key, int64_t logical_commit_time,
@@ -92,6 +101,11 @@ class KeyValueCache : public Cache {
std::string_view key, absl::Span value_set,
int64_t logical_commit_time, std::string_view prefix = "") override;
+ void DeleteValuesInSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix = "") override;
+
// Removes the values that were deleted before the specified
// logical_commit_time for a given prefix.
// TODO: b/267182790 -- Cache cleanup should be done periodically from a
@@ -127,6 +141,25 @@ class KeyValueCache : public Cache {
: last_logical_commit_time(logical_commit_time), is_deleted(deleted) {}
};
+ // Removes deleted keys from key-value map for a given prefix
+ void CleanUpKeyValueMap(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ int64_t logical_commit_time, std::string_view prefix);
+
+ // Removes deleted key-values from key-value_set map for a given prefix
+ void CleanUpKeyValueSetMap(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ int64_t logical_commit_time, std::string_view prefix);
+
+ void CleanUpUIntSetMaps(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ int64_t logical_commit_time, std::string_view prefix);
+
+ // Logs cache access metrics for cache hit or miss counts. The cache access
+ // event name is defined in server_definition.h file
+ void LogCacheAccessMetrics(const RequestContext& request_context,
+ std::string_view cache_access_event) const;
+
// mutex for key value map;
mutable absl::Mutex mutex_;
// mutex for key value set map;
@@ -178,32 +211,8 @@ class KeyValueCache : public Cache {
absl::flat_hash_map>>>
deleted_set_nodes_map_ ABSL_GUARDED_BY(set_map_mutex_);
- // Maps set key to its int32_t value set.
- ThreadSafeHashMap uint32_sets_map_;
- ThreadSafeHashMap
- uint32_sets_max_cleanup_commit_time_map_;
- ThreadSafeHashMap>>
- deleted_uint32_sets_map_;
-
- // Removes deleted keys from key-value map for a given prefix
- void CleanUpKeyValueMap(
- privacy_sandbox::server_common::log::PSLogContext& log_context,
- int64_t logical_commit_time, std::string_view prefix);
-
- // Removes deleted key-values from key-value_set map for a given prefix
- void CleanUpKeyValueSetMap(
- privacy_sandbox::server_common::log::PSLogContext& log_context,
- int64_t logical_commit_time, std::string_view prefix);
-
- void CleanUpUInt32SetMap(
- privacy_sandbox::server_common::log::PSLogContext& log_context,
- int64_t logical_commit_time, std::string_view prefix);
-
- // Logs cache access metrics for cache hit or miss counts. The cache access
- // event name is defined in server_definition.h file
- void LogCacheAccessMetrics(const RequestContext& request_context,
- std::string_view cache_access_event) const;
+ UIntValueSetCache uint32_sets_cache_;
+ UIntValueSetCache uint64_sets_cache_;
friend class KeyValueCacheTestPeer;
};
diff --git a/components/data_server/cache/key_value_cache_test.cc b/components/data_server/cache/key_value_cache_test.cc
index be8620f2..83a264a9 100644
--- a/components/data_server/cache/key_value_cache_test.cc
+++ b/components/data_server/cache/key_value_cache_test.cc
@@ -51,16 +51,6 @@ class KeyValueCacheTestPeer {
return c.map_;
}
- static const auto& ReadUint32Nodes(KeyValueCache& cache,
- std::string_view prefix = "") {
- return cache.uint32_sets_map_;
- }
-
- static const auto& ReadDeletedUint32Nodes(KeyValueCache& cache,
- std::string_view prefix = "") {
- return cache.deleted_uint32_sets_map_;
- }
-
static int GetDeletedSetNodesMapSize(const KeyValueCache& c,
std::string prefix = "") {
absl::MutexLock lock(&c.set_map_mutex_);
@@ -1423,30 +1413,6 @@ TEST_F(CacheTest, VerifyCleaningUpUInt32Sets) {
EXPECT_THAT(set->GetRemovedValues(),
UnorderedElementsAreArray(delete_values));
}
- const auto& deleted_nodes =
- KeyValueCacheTestPeer::ReadDeletedUint32Nodes(*cache);
- {
- auto prefix_nodes = deleted_nodes.CGet("");
- ASSERT_TRUE(prefix_nodes.is_present());
- auto iter = prefix_nodes.value()->find(2);
- ASSERT_NE(iter, prefix_nodes.value()->end());
- EXPECT_THAT(iter->first, 2);
- EXPECT_TRUE(iter->second.contains("set1"));
- }
- {
- cache->RemoveDeletedKeys(safe_path_log_context_, 3);
- auto prefix_nodes = deleted_nodes.CGet("");
- ASSERT_TRUE(prefix_nodes.is_present());
- auto iter = prefix_nodes.value()->find(2);
- ASSERT_EQ(iter, prefix_nodes.value()->end());
- }
- {
- auto result = cache->GetUInt32ValueSet(request_context, keys);
- auto* set = result->GetUInt32ValueSet("set1");
- ASSERT_TRUE(set != nullptr);
- EXPECT_THAT(set->GetValues(), UnorderedElementsAre(3, 4, 5));
- EXPECT_TRUE(set->GetRemovedValues().empty());
- }
}
} // namespace
diff --git a/components/data_server/cache/mocks.h b/components/data_server/cache/mocks.h
index 5661b6a7..17d892d5 100644
--- a/components/data_server/cache/mocks.h
+++ b/components/data_server/cache/mocks.h
@@ -20,7 +20,7 @@
#include "components/container/thread_safe_hash_map.h"
#include "components/data_server/cache/cache.h"
-#include "components/data_server/cache/uint32_value_set.h"
+#include "components/data_server/cache/uint_value_set.h"
#include "gmock/gmock.h"
namespace kv_server {
@@ -44,6 +44,10 @@ class MockCache : public Cache {
(const RequestContext&,
const absl::flat_hash_set&),
(const, override));
+ MOCK_METHOD((std::unique_ptr), GetUInt64ValueSet,
+ (const RequestContext&,
+ const absl::flat_hash_set&),
+ (const, override));
MOCK_METHOD(void, UpdateKeyValue,
(privacy_sandbox::server_common::log::PSLogContext&,
std::string_view, std::string_view, int64_t, std::string_view),
@@ -58,6 +62,11 @@ class MockCache : public Cache {
std::string_view, absl::Span, int64_t,
std::string_view),
(override));
+ MOCK_METHOD(void, UpdateKeyValueSet,
+ (privacy_sandbox::server_common::log::PSLogContext&,
+ std::string_view, absl::Span, int64_t,
+ std::string_view),
+ (override));
MOCK_METHOD(void, DeleteValuesInSet,
(privacy_sandbox::server_common::log::PSLogContext&,
std::string_view, absl::Span, int64_t,
@@ -68,6 +77,11 @@ class MockCache : public Cache {
std::string_view, absl::Span, int64_t,
std::string_view),
(override));
+ MOCK_METHOD(void, DeleteValuesInSet,
+ (privacy_sandbox::server_common::log::PSLogContext&,
+ std::string_view, absl::Span, int64_t,
+ std::string_view),
+ (override));
MOCK_METHOD(void, DeleteKey,
(privacy_sandbox::server_common::log::PSLogContext&,
std::string_view, int64_t, std::string_view),
@@ -87,12 +101,19 @@ class MockGetKeyValueSetResult : public GetKeyValueSetResult {
std::unique_ptr),
(override));
MOCK_METHOD((const UInt32ValueSet*), GetUInt32ValueSet, (std::string_view),
- (const override));
+ (const, override));
MOCK_METHOD(
- void, AddUInt32ValueSet,
+ void, AddUIntValueSet,
(std::string_view,
(ThreadSafeHashMap::ConstLockedNodePtr)),
(override));
+ MOCK_METHOD((const UInt64ValueSet*), GetUInt64ValueSet, (std::string_view),
+ (const, override));
+ MOCK_METHOD(
+ void, AddUIntValueSet,
+ (std::string_view,
+ (ThreadSafeHashMap::ConstLockedNodePtr)),
+ (override));
};
} // namespace kv_server
diff --git a/components/data_server/cache/noop_key_value_cache.h b/components/data_server/cache/noop_key_value_cache.h
index aa79f7e1..ad26e313 100644
--- a/components/data_server/cache/noop_key_value_cache.h
+++ b/components/data_server/cache/noop_key_value_cache.h
@@ -39,6 +39,11 @@ class NoOpKeyValueCache : public Cache {
const absl::flat_hash_set& key_set) const override {
return std::make_unique();
}
+ std::unique_ptr GetUInt64ValueSet(
+ const RequestContext& request_context,
+ const absl::flat_hash_set& key_set) const override {
+ return std::make_unique();
+ }
void UpdateKeyValue(
privacy_sandbox::server_common::log::PSLogContext& log_context,
std::string_view key, std::string_view value, int64_t logical_commit_time,
@@ -51,6 +56,10 @@ class NoOpKeyValueCache : public Cache {
privacy_sandbox::server_common::log::PSLogContext& log_context,
std::string_view key, absl::Span value_set,
int64_t logical_commit_time, std::string_view prefix = "") override {}
+ void UpdateKeyValueSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix = "") override {}
void DeleteKey(privacy_sandbox::server_common::log::PSLogContext& log_context,
std::string_view key, int64_t logical_commit_time,
std::string_view prefix) override {}
@@ -62,6 +71,10 @@ class NoOpKeyValueCache : public Cache {
privacy_sandbox::server_common::log::PSLogContext& log_context,
std::string_view key, absl::Span value_set,
int64_t logical_commit_time, std::string_view prefix = "") override {}
+ void DeleteValuesInSet(
+ privacy_sandbox::server_common::log::PSLogContext& log_context,
+ std::string_view key, absl::Span value_set,
+ int64_t logical_commit_time, std::string_view prefix = "") override {}
void RemoveDeletedKeys(
privacy_sandbox::server_common::log::PSLogContext& log_context,
int64_t logical_commit_time, std::string_view prefix) override {}
@@ -79,13 +92,21 @@ class NoOpKeyValueCache : public Cache {
std::string_view key) const override {
return nullptr;
}
+ const UInt64ValueSet* GetUInt64ValueSet(
+ std::string_view key) const override {
+ return nullptr;
+ }
void AddKeyValueSet(
std::string_view key, absl::flat_hash_set value_set,
std::unique_ptr key_lock) override {}
- void AddUInt32ValueSet(
+ void AddUIntValueSet(
std::string_view key,
ThreadSafeHashMap::ConstLockedNodePtr
value_set_node) override {}
+ void AddUIntValueSet(
+ std::string_view key,
+ ThreadSafeHashMap::ConstLockedNodePtr
+ value_set_node) override {}
};
};
diff --git a/components/data_server/cache/uint32_value_set.cc b/components/data_server/cache/uint32_value_set.cc
deleted file mode 100644
index 9f6e02a4..00000000
--- a/components/data_server/cache/uint32_value_set.cc
+++ /dev/null
@@ -1,100 +0,0 @@
-/*
- * Copyright 2024 Google LLC
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include "components/data_server/cache/uint32_value_set.h"
-
-#include
-
-namespace kv_server {
-
-absl::flat_hash_set UInt32ValueSet::GetValues() const {
- absl::flat_hash_set values;
- values.reserve(values_bitset_.cardinality());
- for (const auto& [value, metadata] : values_metadata_) {
- if (!metadata.is_deleted) {
- values.insert(value);
- }
- }
- return values;
-}
-
-const roaring::Roaring& UInt32ValueSet::GetValuesBitSet() const {
- return values_bitset_;
-}
-
-absl::flat_hash_set UInt32ValueSet::GetRemovedValues() const {
- absl::flat_hash_set removed_values;
- for (const auto& [_, values] : deleted_values_) {
- for (auto value : values) {
- removed_values.insert(value);
- }
- }
- return removed_values;
-}
-
-void UInt32ValueSet::AddOrRemove(absl::Span values,
- int64_t logical_commit_time, bool is_deleted) {
- for (auto value : values) {
- auto* metadata = &values_metadata_[value];
- if (metadata->logical_commit_time >= logical_commit_time) {
- continue;
- }
- metadata->logical_commit_time = logical_commit_time;
- metadata->is_deleted = is_deleted;
- if (is_deleted) {
- values_bitset_.remove(value);
- deleted_values_[logical_commit_time].insert(value);
- } else {
- values_bitset_.add(value);
- deleted_values_[logical_commit_time].erase(value);
- }
- }
- values_bitset_.runOptimize();
-}
-
-void UInt32ValueSet::Add(absl::Span values,
- int64_t logical_commit_time) {
- AddOrRemove(values, logical_commit_time, /*is_deleted=*/false);
-}
-
-void UInt32ValueSet::Remove(absl::Span values,
- int64_t logical_commit_time) {
- AddOrRemove(values, logical_commit_time, /*is_deleted=*/true);
-}
-
-void UInt32ValueSet::Cleanup(int64_t cutoff_logical_commit_time) {
- for (const auto& [logical_commit_time, values] : deleted_values_) {
- if (logical_commit_time > cutoff_logical_commit_time) {
- break;
- }
- for (auto value : values) {
- values_metadata_.erase(value);
- }
- }
- deleted_values_.erase(
- deleted_values_.begin(),
- deleted_values_.upper_bound(cutoff_logical_commit_time));
-}
-
-absl::flat_hash_set BitSetToUint32Set(
- const roaring::Roaring& bitset) {
- auto num_values = bitset.cardinality();
- auto data = std::make_unique(num_values);
- bitset.toUint32Array(data.get());
- return absl::flat_hash_set(data.get(), data.get() + num_values);
-}
-
-} // namespace kv_server
diff --git a/components/data_server/cache/uint32_value_set.h b/components/data_server/cache/uint32_value_set.h
deleted file mode 100644
index 49b1844a..00000000
--- a/components/data_server/cache/uint32_value_set.h
+++ /dev/null
@@ -1,75 +0,0 @@
-/*
- * Copyright 2024 Google LLC
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef COMPONENTS_DATA_SERVER_CACHE_UINT32_VALUE_SET_H_
-#define COMPONENTS_DATA_SERVER_CACHE_UINT32_VALUE_SET_H_
-
-#include "absl/container/btree_map.h"
-#include "absl/container/flat_hash_map.h"
-#include "absl/container/flat_hash_set.h"
-
-#include "roaring.hh"
-
-namespace kv_server {
-
-// Stores a set of `uint32_t` values associated with a `logical_commit_time`.
-// The `logical_commit_time` is used to support out of order set mutations,
-// i.e., calling `Remove({1, 2, 3}, 5)` and then `Add({1, 2, 3}, 3)` will result
-// in an empty set.
-//
-// The values in the set are also projected to `roaring::Roaring` bitset which
-// can be used for efficient set operations such as union, intersection, .e.t.c.
-class UInt32ValueSet {
- public:
- // Returns values not marked as removed from the set.
- absl::flat_hash_set GetValues() const;
- // Returns values not marked as removed from the set as a bitset.
- const roaring::Roaring& GetValuesBitSet() const;
- // Returns values marked as removed from the set.
- absl::flat_hash_set GetRemovedValues() const;
-
- // Adds values associated with `logical_commit_time` to the set. If a value
- // with the same or greater `logical_commit_time` already exists in the set,
- // then this is a noop.
- void Add(absl::Span values, int64_t logical_commit_time);
- // Marks values associated with `logical_commit_time` as removed from the set.
- // If a value with the same or greater `logical_commit_time` already exists in
- // the set, then this is a noop.
- void Remove(absl::Span values, int64_t logical_commit_time);
- // Cleans up space occupied by values (including value metadata) matching the
- // condition `logical_commit_time` <= `cutoff_logical_commit_time` and are
- // marked as removed.
- void Cleanup(int64_t cutoff_logical_commit_time);
-
- private:
- struct ValueMetadata {
- int64_t logical_commit_time;
- bool is_deleted;
- };
-
- void AddOrRemove(absl::Span values, int64_t logical_commit_time,
- bool is_deleted);
-
- roaring::Roaring values_bitset_;
- absl::flat_hash_map values_metadata_;
- absl::btree_map> deleted_values_;
-};
-
-absl::flat_hash_set BitSetToUint32Set(const roaring::Roaring& bitset);
-
-} // namespace kv_server
-
-#endif // COMPONENTS_DATA_SERVER_CACHE_UINT32_VALUE_SET_H_
diff --git a/components/data_server/cache/uint_value_set.cc b/components/data_server/cache/uint_value_set.cc
new file mode 100644
index 00000000..dd5b5f6d
--- /dev/null
+++ b/components/data_server/cache/uint_value_set.cc
@@ -0,0 +1,31 @@
+/*
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "components/data_server/cache/uint_value_set.h"
+
+namespace kv_server {
+
+absl::flat_hash_set BitSetToUint32Set(
+ const roaring::Roaring& bitset) {
+ return BitSetToUintSet(bitset);
+}
+
+absl::flat_hash_set BitSetToUint64Set(
+ const roaring::Roaring64Map& bitset) {
+ return BitSetToUintSet(bitset);
+}
+
+} // namespace kv_server
diff --git a/components/data_server/cache/uint_value_set.h b/components/data_server/cache/uint_value_set.h
new file mode 100644
index 00000000..b1f13d28
--- /dev/null
+++ b/components/data_server/cache/uint_value_set.h
@@ -0,0 +1,181 @@
+/*
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef COMPONENTS_DATA_SERVER_CACHE_UINT_VALUE_SET_H_
+#define COMPONENTS_DATA_SERVER_CACHE_UINT_VALUE_SET_H_
+
+#include
+
+#include "absl/container/btree_map.h"
+#include "absl/container/flat_hash_map.h"
+#include "absl/container/flat_hash_set.h"
+
+#include "roaring.hh"
+#include "roaring64map.hh"
+
+namespace kv_server {
+
+// Stores a set of unsigned int values associated with a `logical_commit_time`.
+// The `logical_commit_time` is used to support out of order set mutations,
+// i.e., calling `Remove({1, 2, 3}, 5)` and then `Add({1, 2, 3}, 3)` will result
+// in an empty set.
+//
+// The values in the set are also projected to `roaring::Roaring` bitset which
+// can be used for efficient set operations such as union, intersection, .e.t.c.
+template
+class UIntValueSet {
+ public:
+ using value_type = ValueType;
+ using bitset_type = BitsetType;
+
+ // Returns values not marked as removed from the set.
+ absl::flat_hash_set GetValues() const;
+ // Returns values not marked as removed from the set as a bitset.
+ const BitsetType& GetValuesBitSet() const;
+ // Returns values marked as removed from the set.
+ absl::flat_hash_set