Release 0.7.0

### Features * Add --platform flag to build_and_test_all_in_docker * Add a concurrent record reader for data files. * Add a helper service for protocol testing * Add a stub class for reading local blob files * add automated PCR0 updates to kokoro continuous * Add base streambuf with seeking support for reading blobs. * Add delta writer and custom audience data parser * add github personal access token validation for release scripts * Add instance id to metrics. * Add seeking to S3 blob reader. * Add support for Zipkin exports for local builds. * Add terraform logic to create SNS for real time updates * Check timestamps for cache update * Implement BinaryHTTP version of V2 API * Implement delta file record change notifier to retrieve high priority updates * Implement test OHTTP V2 query handling * Integrating high priority updates in data server * Memory cleanup for delete timestamps in the cache * Record metrics for all RetryUntilOk events. Export them to stdout or * Upgrade black to 23.1.0 * Upgrade to build-system 0.13.0 * Upgrade to build-system 0.14.0 * Upgrade to build-system 0.16.0 * Use concurrent reader for reading snapshot and delta files. ### Bug Fixes * Add docker compose config for testing locally. * add empty bug id to automated PCR0 CL * Add unit test for delta file backup poll. Fix bug where we can't * Don't ListBlobs to poll Delta files on notifications that don't * Don't use default number of cores for small test files. * fetch git remote for automated pcr0 updates * Fix typos and remove unreachable branches. * flaky delta_file_notifier test. * Listing non-delta files from bucket shouldn't cause state change. * Only read the most recent snapshot file. * path for local envoy * Prefer github release artifacts over archive artifacts * remove duplicate open telemetry entry. * remove spaces from automated PCR0 CL commit footer * Switch jaeger over to using OTLP directly. Jaeger otel component is * Upgrade to rules_buf 0.1.1 * Uprev Otel to pull in semantic resource convensions. Use them * Use shared libraries for proxy ### Build System * Hide build stdout/stderr for third_party ### Documentation * Add docs for data loading library. Change-Id: I37444494693c1ba3f3edfa2760ae48bf415e88de Bug: b/269120926 GitOrigin-RevId: 789c517d33605180cacfe578e8747e7129a0f092
privacysandbox · Feb 16, 2023 · 053c245 · 053c245
1 parent cebdb46
commit 053c245
Show file tree

Hide file tree

Showing 240 changed files with 7,330 additions and 3,876 deletions.
diff --git a/.bazelrc b/.bazelrc
@@ -3,6 +3,7 @@ build --client_env=CC=clang
 build --cxxopt=-std=c++17
 build --copt=-Werror=return-type
 build -c opt
+build --output_filter='^//((?!(third_party):).)*$'`
 build --color=yes
 build --@io_bazel_rules_docker//transitions:enable=false
 build --workspace_status_command="bash tools/get_workspace_status"

diff --git a/.gitignore b/.gitignore
@@ -14,6 +14,7 @@
 *.eif
 /dist/
 __pycache__/
+/test/run_local/dist
 
 # Hedron's Compile Commands Extractor for Bazel
 # https://github.com/hedronvision/bazel-compile-commands-extractor

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -137,7 +137,7 @@ repos:
     - --quiet
 
 - repo: https://github.com/psf/black
-  rev: 22.12.0
+  rev: 23.1.0
   hooks:
   - id: black
     name: black python formatter
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,68 @@
 
 All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
 
+## [0.7.0](https://github.com/privacysandbox/fledge-key-value-service/compare/release-0.6.0...release-0.7.0) (2023-02-16)
+
+
+### Features
+
+* Add --platform flag to build_and_test_all_in_docker
+* Add a concurrent record reader for data files.
+* Add a helper service for protocol testing
+* Add a stub class for reading local blob files
+* add automated PCR0 updates to kokoro continuous
+* Add base streambuf with seeking support for reading blobs.
+* Add delta writer and custom audience data parser
+* add github personal access token validation for release scripts
+* Add instance id to metrics.
+* Add seeking to S3 blob reader.
+* Add support for Zipkin exports for local builds.
+* Add terraform logic to create SNS for real time updates
+* Check timestamps for cache update
+* Implement BinaryHTTP version of V2 API
+* Implement delta file record change notifier to retrieve high priority updates
+* Implement test OHTTP V2 query handling
+* Integrating high priority updates in data server
+* Memory cleanup for delete timestamps in the cache
+* Record metrics for all RetryUntilOk events.  Export them to stdout or
+* Upgrade black to 23.1.0
+* Upgrade to build-system 0.13.0
+* Upgrade to build-system 0.14.0
+* Upgrade to build-system 0.16.0
+* Use concurrent reader for reading snapshot and delta files.
+
+
+### Bug Fixes
+
+* Add docker compose config for testing locally.
+* add empty bug id to automated PCR0 CL
+* Add unit test for delta file backup poll.  Fix bug where we can't
+* Don't ListBlobs to poll Delta files on notifications that don't
+* Don't use default number of cores for small test files.
+* fetch git remote for automated pcr0 updates
+* Fix typos and remove unreachable branches.
+* flaky delta_file_notifier test.
+* Listing non-delta files from bucket shouldn't cause state change.
+* Only read the most recent snapshot file.
+* path for local envoy
+* Prefer github release artifacts over archive artifacts
+* remove duplicate open telemetry entry.
+* remove spaces from automated PCR0 CL commit footer
+* Switch jaeger over to using OTLP directly.  Jaeger otel component is
+* Upgrade to rules_buf 0.1.1
+* Uprev Otel to pull in semantic resource convensions.  Use them
+* Use shared libraries for proxy
+
+
+### Build System
+
+* Hide build stdout/stderr for third_party
+
+
+### Documentation
+
+* Add docs for data loading library.
+
 ## [0.6.0](https://github.com/privacysandbox/fledge-key-value-service/compare/release-0.5.0...release-0.6.0) (2023-01-10)
 
 

diff --git a/WORKSPACE b/WORKSPACE
@@ -1,5 +1,12 @@
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
+http_archive(
+    name = "boringssl",
+    sha256 = "0cd64ecff9e5f757988b84b7685e968775de08ea9157656d0b9fee0fa62d67ec",
+    strip_prefix = "boringssl-c2837229f381f5fcd8894f0cca792a94b557ac52",
+    urls = ["https://github.com/google/boringssl/archive/c2837229f381f5fcd8894f0cca792a94b557ac52.tar.gz"],
+)
+
 http_archive(
     name = "bazel_skylib",
     sha256 = "74d544d96f4a5bb630d465ca8bbcfe231e3594e5aae57e1edbf17a6eb3ca2506",
@@ -15,9 +22,9 @@ bazel_skylib_workspace()
 
 http_archive(
     name = "com_google_protobuf",
-    sha256 = "8b28fdd45bab62d15db232ec404248901842e5340299a57765e48abe8a80d930",  # Last updated 2022-05-18
-    strip_prefix = "protobuf-3.20.1",
-    urls = ["https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.20.1.tar.gz"],
+    sha256 = "e51cc8fc496f893e2a48beb417730ab6cbcb251142ad8b2cd1951faa5c76fe3d",  # Last updated 2022-09-29
+    strip_prefix = "protobuf-3.20.3",
+    urls = ["https://github.com/protocolbuffers/protobuf/releases/download/v3.20.3/protobuf-cpp-3.20.3.tar.gz"],
 )
 
 http_archive(
@@ -92,9 +99,9 @@ http_archive(
 ### rules_buf (https://docs.buf.build/build-systems/bazel)
 http_archive(
     name = "rules_buf",
-    sha256 = "3fe244c9efa42a41edd83f63dee1b5570a1951a654030658b86bfaea6a268164",
-    strip_prefix = "rules_buf-0.1.0",
-    urls = ["https://github.com/bufbuild/rules_buf/archive/refs/tags/v0.1.0.zip"],
+    sha256 = "523a4e06f0746661e092d083757263a249fedca535bd6dd819a8c50de074731a",
+    strip_prefix = "rules_buf-0.1.1",
+    urls = ["https://github.com/bufbuild/rules_buf/archive/refs/tags/v0.1.1.zip"],
 )
 
 load("@rules_buf//buf:repositories.bzl", "rules_buf_dependencies", "rules_buf_toolchains")
@@ -128,6 +135,10 @@ load("//third_party:cpp_repositories.bzl", "cpp_repositories")
 
 cpp_repositories()
 
+load("//third_party:scp_deps.bzl", "scp_deps")
+
+scp_deps()
+
 load("//third_party:quiche.bzl", "quiche_dependencies")
 
 quiche_dependencies()

diff --git a/builders/CHANGELOG.md b/builders/CHANGELOG.md
@@ -2,6 +2,86 @@
 
 All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
 
+## [0.16.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.15.1...release-0.16.0) (2023-02-05)
+
+
+### Features
+
+* Run test tools in docker interactive mode to admit std streams
+
+### [0.15.1](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.15.0...release-0.15.1) (2023-02-04)
+
+
+### Bug Fixes
+
+* Return value from get_docker_workspace_mount()
+
+## [0.15.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.14.0...release-0.15.0) (2023-02-03)
+
+
+### Features
+
+* Use WORKSPACE_MOUNT if set
+
+
+### Bug Fixes
+
+* Pin commit-and-tag-version to v10.1.0
+
+## [0.14.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.13.1...release-0.14.0) (2023-01-27)
+
+
+### Features
+
+* Improve verbose output for get-builder-image-tagged
+
+### [0.13.1](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.13.0...release-0.13.1) (2023-01-26)
+
+
+### Bug Fixes
+
+* Upgrade software-properties-common
+
+## [0.13.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.12.0...release-0.13.0) (2023-01-23)
+
+
+### Features
+
+* Add ab tool
+* Add cassowary http load testing tool
+* Add h2load tool
+* Add slowhttptest tool
+* Adjust get-builder-image-tagged verbose output
+* Upgrade to packer v1.8.5
+
+
+### Bug Fixes
+
+* Re-pin dependencies in test-tools image
+* Relax version pins to semver
+* Upgrade amazonlinux2 base image
+* Upgrade git on amazonlinux2
+
+## [0.12.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.11.0...release-0.12.0) (2023-01-10)
+
+
+### Features
+
+* Modify ghz wrapper for generic use. Add curl
+* Use test-tools image for grpcurl
+
+## [0.11.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.10.0...release-0.11.0) (2023-01-09)
+
+
+### Features
+
+* Add chrpath
+
+
+### Bug Fixes
+
+* Clean up tmpdir via RETURN trap
+
 ## [0.10.0](https://team/kiwi-air-force-eng-team/build-system/compare/release-0.9.0...release-0.10.0) (2023-01-06)
 
 

diff --git a/builders/images/build-amazonlinux2/Dockerfile b/builders/images/build-amazonlinux2/Dockerfile
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM amazonlinux:2.0.20220912.1
+FROM amazonlinux:2.0.20221210.0
 
 COPY /install_apps install_golang_apps install_go.sh generate_system_bazelrc .bazelversion /scripts/
 COPY get_workspace_mount /usr/local/bin

diff --git a/builders/images/build-amazonlinux2/install_apps b/builders/images/build-amazonlinux2/install_apps
@@ -64,7 +64,7 @@ function install_gcc() {
 function install_misc() {
   yum install -y \
     gettext-0.19.8.1-3.amzn2 \
-    git-2.37.1-1.amzn2.0.1 \
+    git-2.38.1-1.amzn2.0.1 \
     tar-1.26-35.amzn2 \
     unzip-6.0-43.amzn2 \
     zip-3.0-11.amzn2.0.2
@@ -73,7 +73,7 @@ function install_misc() {
 function install_packer() {
   yum install -y yum-utils-1.1.31-46.amzn2.0.1.noarch
   yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
-  yum -y install packer-1.8.4-1
+  yum -y install packer-1.8.5-1
   update-alternatives --install /usr/local/bin/packer packer /usr/bin/packer 100
 
   /usr/local/bin/packer version

diff --git a/builders/images/build-debian/install_apps b/builders/images/build-debian/install_apps
@@ -50,22 +50,23 @@ function install_misc() {
   DEBIAN_FRONTEND=noninteractive apt-get --quiet install -y --no-install-recommends \
     apt-transport-https="2.0.9" \
     ca-certificates="20211016ubuntu0.20.04.1" \
-    libcurl4="7.68.0-1ubuntu2" \
-    curl="7.68.0-1ubuntu2" \
-    gettext="0.19.8.1-10build1" \
-    git="1:2.25.1-1ubuntu3.6" \
-    gnupg="2.2.19-3ubuntu2.2" \
-    locales="2.31-0ubuntu9.9" \
-    lsb-release="11.1.0ubuntu2" \
-    openjdk-11-jdk="11.0.17+8-1ubuntu2~20.04" \
-    python3.9-venv="3.9.5-3ubuntu0~20.04.1" \
-    rename="1.10-1" \
-    software-properties-common="0.99.9.8" \
-    unzip="6.0-25ubuntu1.1" \
-    wget="1.20.3-1ubuntu2" \
-    xz-utils="5.2.4-1ubuntu1.1" \
-    zip="3.0-11build1" \
-    zlib1g-dev="1:1.2.11.dfsg-2ubuntu1.5"
+    chrpath="0.16-*" \
+    libcurl4="7.68.0-*" \
+    curl="7.68.0-*" \
+    gettext="0.19.8.1-*" \
+    git="1:2.25.1-*" \
+    gnupg="2.2.19-*" \
+    locales="2.31-*" \
+    lsb-release="11.1.0*" \
+    openjdk-11-jdk="11.0.17*" \
+    python3.9-venv="3.9.5-*" \
+    rename="1.10-*" \
+    software-properties-common="0.99.9.10" \
+    unzip="6.0-*" \
+    wget="1.20.3-*" \
+    xz-utils="5.2.4-*" \
+    zip="3.0-*" \
+    zlib1g-dev="1:1.2.11.dfsg-*"
   mkdir -p /opt/bin
   update-alternatives --install /opt/bin/python3 python3 /usr/bin/python3.9 100
   update-alternatives --install /opt/bin/python python /usr/bin/python3.9 100

diff --git a/builders/images/presubmit/install_apps b/builders/images/presubmit/install_apps
@@ -63,7 +63,7 @@ function install_packages() {
     openjdk-11-jre="11.0.17+8-1ubuntu2~20.04" \
     python3.9-venv="3.9.5-3ubuntu0~20.04.1" \
     shellcheck="0.7.0-2build2" \
-    software-properties-common="0.99.9.8" \
+    software-properties-common="0.99.9.10" \
     wget="1.20.3-1ubuntu2"
   update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 100
 }

diff --git a/builders/images/release/install_release_apps b/builders/images/release/install_release_apps
@@ -1,6 +1,6 @@
 #!/bin/sh
 
-npm install --global commit-and-tag-version
+npm install --global commit-and-tag-version@10.1.0
 
 # Install the GitHub CLI tool (https://cli.github.com/)
 apk add github-cli

diff --git a/builders/images/test-tools/Dockerfile b/builders/images/test-tools/Dockerfile
@@ -12,9 +12,17 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+FROM alpine:3.16 as builder
+
+# hadolint ignore=DL3018
+RUN apk add --no-cache build-base git openssl-dev autoconf automake
+WORKDIR /build
+ADD https://github.com/shekyan/slowhttptest/archive/refs/tags/v1.9.0.tar.gz /build
+RUN tar xz --strip-components 1 -f v1.9.0.tar.gz && ls -l && ./configure && make
+
 FROM golang:1.19.4-alpine3.17 AS golang-1.19
 FROM fullstorydev/grpcurl:v1.8.7 AS grpcurl
-FROM alpine:3.17.0
+FROM alpine:3.17.1
 
 COPY --from=golang-1.19 /usr/local/go/ /usr/local/go/
 COPY --from=grpcurl /bin/grpcurl /usr/bin
@@ -27,3 +35,4 @@ ENV BUILD_ARCH="${TARGETARCH}" \
 COPY install_apps /scripts/
 
 RUN /scripts/install_apps
+COPY --from=builder /build/src/slowhttptest /usr/bin/
diff --git a/builders/images/test-tools/install_apps b/builders/images/test-tools/install_apps
@@ -4,15 +4,36 @@ set -o errexit
 
 function install_packages() {
   apk --no-cache add \
-    bash=5.2.15-r0 \
-    curl=7.87.0-r0 \
-    jq=1.6-r2
+    bash~=5.2.15 \
+    curl~=7.87.0 \
+    jq~=1.6 \
+    libstdc++
 }
 
 function install_ghz() {
   go install github.com/bojand/ghz/cmd/[email protected]
   ghz --help
 }
 
+function install_nghttp2() {
+  apk --no-cache add \
+    nghttp2~=1.51.0
+  h2load --version
+}
+
+function install_apache2_utils() {
+  apk --no-cache add \
+    apache2-utils~=2.4.55
+  ab -V
+}
+
+function install_cassowary() {
+  go install github.com/rogerwelin/cassowary/cmd/[email protected]
+  cassowary --help
+}
+
 install_packages
 install_ghz
+install_nghttp2
+install_apache2_utils
+install_cassowary
diff --git a/builders/images/utils/Dockerfile b/builders/images/utils/Dockerfile
@@ -15,5 +15,5 @@
 FROM alpine:3.16
 
 RUN apk --no-cache add \
-  unzip=6.0-r9 \
-  zip=3.0-r9
+  unzip~=6.0 \
+  zip~=3.0
diff --git a/builders/tests/data/hashes/build-amazonlinux2 b/builders/tests/data/hashes/build-amazonlinux2
@@ -1 +1 @@
-privacysandbox/builders/build-amazonlinux2:amd64-05c70ad9416a903975525483c5635833e57473a79ff1269ab8d17ee7889ca896
+6ce9eb8473f75cc825b4f12b46f559f53e24dfb641d04740ac909e1d6a0fd42d
diff --git a/builders/tests/data/hashes/build-debian b/builders/tests/data/hashes/build-debian
@@ -1 +1 @@
-privacysandbox/builders/build-debian:amd64-2a4167faeb729ba7684989d5158bac8806321e5c4f6e66c8d5c7b1c5addce7ba
+105b8dce756e8781d2a141cdcaf163093c05507977a7887f74ddfc75fca9b117
diff --git a/builders/tests/data/hashes/presubmit b/builders/tests/data/hashes/presubmit
@@ -1 +1 @@
-privacysandbox/builders/presubmit:amd64-e037ede2e2132b0f1534e612efe69c50b852556fbbdf4460a3989d11cbcd0faa
+97b58bb9dd2de43c41064ecbfdd43dcccb5106e988e55783ee54036955dc225e
diff --git a/builders/tests/data/hashes/release b/builders/tests/data/hashes/release
@@ -1 +1 @@
-privacysandbox/builders/release:amd64-1788ad91d6ac6760e5a6141fe9335bb1020ef45a42bc43bffdb05ec09c1788dc
+fadb118a254936caf3623deaeb4d0cbd44b5ba992644cce7cb1b994724725d5f