Release v2.0.2 (#2233)

* FEATURE: add cli argument to modify controller workqueue ratelimiter (#2186) * add cli argument to modify controller workqueue ratelimiter Signed-off-by: ImpSy <[email protected]> * add cli argument to modify controller workqueue ratelimiter support to helm chart Signed-off-by: ImpSy <[email protected]> --------- Signed-off-by: ImpSy <[email protected]> (cherry picked from commit d37a0e9) Signed-off-by: Yi Chen <[email protected]> * Fix ingress capability discovery (#2201) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 56b4974) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/aws/aws-sdk-go-v2 from 1.30.5 to 1.31.0 (#2207) Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.30.5 to 1.31.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@v1.30.5...v1.31.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit faa0822) Signed-off-by: Yi Chen <[email protected]> * Bump golang.org/x/net from 0.28.0 to 0.29.0 (#2205) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.28.0 to 0.29.0. - [Commits](golang/net@v0.28.0...v0.29.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 6106178) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/docker/docker from 27.0.3+incompatible to 27.1.1+incompatible (#2125) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 27.0.3+incompatible to 27.1.1+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v27.0.3...v27.1.1) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 316536f) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.58.3 to 1.63.3 (#2206) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.58.3 to 1.63.3. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.58.3...service/s3/v1.63.3) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 1972fb7) Signed-off-by: Yi Chen <[email protected]> * Update integration test workflow and add golangci lint check (#2197) * Update integration test workflow Signed-off-by: Yi Chen <[email protected]> * Update golangci lint config Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 143b16e) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/aws/aws-sdk-go-v2 from 1.31.0 to 1.32.0 (#2229) Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.31.0 to 1.32.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@v1.31.0...v1.32.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit a4dcfcb) Signed-off-by: Yi Chen <[email protected]> * Bump cloud.google.com/go/storage from 1.43.0 to 1.44.0 (#2228) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.43.0 to 1.44.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@pubsub/v1.43.0...spanner/v1.44.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 2542009) Signed-off-by: Yi Chen <[email protected]> * Bump manusa/actions-setup-minikube from 2.11.0 to 2.12.0 (#2226) Bumps [manusa/actions-setup-minikube](https://github.com/manusa/actions-setup-minikube) from 2.11.0 to 2.12.0. - [Release notes](https://github.com/manusa/actions-setup-minikube/releases) - [Commits](manusa/actions-setup-minikube@v2.11.0...v2.12.0) --- updated-dependencies: - dependency-name: manusa/actions-setup-minikube dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 4358fd4) Signed-off-by: Yi Chen <[email protected]> * Bump golang.org/x/time from 0.6.0 to 0.7.0 (#2227) Bumps [golang.org/x/time](https://github.com/golang/time) from 0.6.0 to 0.7.0. - [Commits](golang/time@v0.6.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/time dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 29ba4e7) Signed-off-by: Yi Chen <[email protected]> * fix: imagePullPolicy was ignored (#2222) Signed-off-by: xuqingtan <[email protected]> (cherry picked from commit 7fb14e6) Signed-off-by: Yi Chen <[email protected]> * fix: spark-submission failed due to lack of permission by user `spark` (#2223) error: Exception in thread "main" java.io.FileNotFoundException: /home/spark/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-511288aa-ce7c-4a38-9c8e-4869b71c68fa-1.0.xml (No such file or directory) Signed-off-by: xuqingtan <[email protected]> (cherry picked from commit d07821b) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.33 to 1.27.42 (#2231) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.33 to 1.27.42. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.33...config/v1.27.42) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 9be8dce) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/prometheus/client_golang from 1.19.1 to 1.20.4 (#2204) Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.19.1 to 1.20.4. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.19.1...v1.20.4) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit fe833fa) Signed-off-by: Yi Chen <[email protected]> * Remove `cap_net_bind_service` from image (#2216) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit ac761ef) Signed-off-by: Yi Chen <[email protected]> * fix: webhook panics due to logging (#2232) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 247e834) Signed-off-by: Yi Chen <[email protected]> * Add check for generating manifests and code (#2234) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit c75d99f) Signed-off-by: Yi Chen <[email protected]> * Spark Operator Official Release v2.0.2 Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: ImpSy <[email protected]> Signed-off-by: Yi Chen <[email protected]> Signed-off-by: Jacob Salway <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: xuqingtan <[email protected]> Co-authored-by: Sébastien Maintrot <[email protected]> Co-authored-by: Jacob Salway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nick Tan <[email protected]>
kubeflow · Oct 10, 2024 · ef9a2a1 · ef9a2a1
1 parent 94cdc52
commit ef9a2a1
Show file tree

Hide file tree

Showing 36 changed files with 531 additions and 303 deletions.
diff --git a/.github/workflows/integration.yaml b/.github/workflows/integration.yaml
@@ -16,39 +16,80 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  build-api-docs:
+  code-check:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout source code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
 
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
           go-version-file: go.mod
 
-      - name: The API documentation hasn't changed
+      - name: Run go mod tidy
         run: |
-          make build-api-docs
-          if ! git diff --quiet -- docs/api-docs.md; then
-            echo "Need to re-run 'make build-api-docs' and commit the changes"
-            git diff -- docs/api-docs.md;
+          go mod tidy
+          if ! git diff --quiet; then
+            echo "Please run 'go mod tidy' and commit the changes."
+            git diff
             false
           fi
 
-  build-sparkctl:
+      - name: Generate code
+        run: |
+          make generate
+          if ! git diff --quiet; then
+            echo "Need to re-run 'make generate' and commit the changes."
+            git diff
+            false
+          fi
+
+      - name: Run go fmt check
+        run: |
+          make go-fmt
+          if ! git diff --quiet; then
+            echo "Need to re-run 'make go-fmt' and commit the changes."
+            git diff
+            false
+          fi
+
+      - name: Run go vet check
+        run: |
+          make go-vet
+          if ! git diff --quiet; then
+            echo "Need to re-run 'make go-vet' and commit the changes."
+            git diff
+            false
+          fi
+
+      - name: Run golangci-lint
+        run: |
+          make go-lint
+
+  build-api-docs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout source code
         uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
 
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
           go-version-file: go.mod
 
-      - name: build sparkctl
-        run: make build-sparkctl
+      - name: Build API docs
+        run: |
+          make build-api-docs
+          if ! git diff --quiet; then
+            echo "Need to re-run 'make build-api-docs' and commit the changes."
+            git diff
+            false
+          fi
 
   build-spark-operator:
     runs-on: ubuntu-latest
@@ -63,18 +104,28 @@ jobs:
         with:
           go-version-file: go.mod
 
-      - name: Run go fmt check
-        run: make go-fmt
-
-      - name: Run go vet check
-        run: make go-vet
-
-      - name: Run unit tests
+      - name: Run go unit tests
         run: make unit-test
 
       - name: Build Spark operator
         run: make build-operator
 
+  build-sparkctl:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+
+      - name: Build sparkctl
+        run: make build-sparkctl
+
   build-helm-chart:
     runs-on: ubuntu-latest
     steps:
@@ -103,6 +154,18 @@ jobs:
       - name: Set up chart-testing
         uses: helm/[email protected]
 
+      - name: Generate manifests
+        run: |
+          make manifests
+          if ! git diff --quiet; then
+            echo "Need to re-run 'make manifests' and commit the changes."
+            git diff
+            false
+          fi
+
+      - name: Detect CRDs drift between chart and manifest
+        run: make detect-crds-drift
+
       - name: Run chart-testing (list-changed)
         id: list-changed
         env:
@@ -119,22 +182,18 @@ jobs:
           BRANCH: ${{ steps.get_branch.outputs.BRANCH }}
         run: ct lint --check-version-increment=false --target-branch $BRANCH
 
-      - name: Detect CRDs drift between chart and manifest
-        if: steps.list-changed.outputs.changed == 'true'
-        run: make detect-crds-drift
-
       - name: Produce the helm documentation
         if: steps.list-changed.outputs.changed == 'true'
         run: |
           make helm-docs
           if ! git diff --quiet -- charts/spark-operator-chart/README.md; then
-            echo "Need to re-run 'make helm-docs' and commit the changes"
+            echo "Need to re-run 'make helm-docs' and commit the changes."
             false
           fi
 
       - name: setup minikube
         if: steps.list-changed.outputs.changed == 'true'
-        uses: manusa/actions-setup-minikube@v2.11.0
+        uses: manusa/actions-setup-minikube@v2.12.0
         with:
           minikube version: v1.33.0
           kubernetes version: v1.30.0

diff --git a/.golangci.yaml b/.golangci.yaml
@@ -1,39 +1,66 @@
 run:
-  deadline: 5m
+  # Timeout for analysis, e.g. 30s, 5m.
+  # Default: 1m
+  timeout: 1m
 
 linters:
+  # Enable specific linters.
+  # https://golangci-lint.run/usage/linters/#enabled-by-default
   enable:
-  - revive
-  - gci
-  - depguard
-  - godot
-  - testifylint
-  - unconvert
+    # Detects places where loop variables are copied.
+    - copyloopvar
+    # Checks for duplicate words in the source code.
+    - dupword
+    # Tool for detection of FIXME, TODO and other comment keywords.
+    # - godox
+    # Check import statements are formatted according to the 'goimport' command.
+    - goimports
+    # Enforces consistent import aliases.
+    - importas
+    # Find code that shadows one of Go's predeclared identifiers.
+    - predeclared
+    # Check that struct tags are well aligned.
+    - tagalign
+    # Remove unnecessary type conversions.
+    - unconvert
+    # Checks Go code for unused constants, variables, functions and types.
+    - unused
 
 issues:
-  exclude-rules:
-  # Disable errcheck linter for test files.
-  - path: _test.go
-    linters:
-    - errcheck
+  # Which dirs to exclude: issues from them won't be reported.
+  # Can use regexp here: `generated.*`, regexp is applied on full path,
+  # including the path prefix if one is set.
+  # Default dirs are skipped independently of this option's value (see exclude-dirs-use-default).
+  # "/" will be replaced by current OS file path separator to properly work on Windows.
+  # Default: []
+  exclude-dirs:
+    - sparkctl
+  # Maximum issues count per one linter.
+  # Set to 0 to disable.
+  # Default: 50
+  max-issues-per-linter: 50
+  # Maximum count of issues with the same text.
+  # Set to 0 to disable.
+  # Default: 3
+  max-same-issues: 3
 
 linters-settings:
-  gci:
-    sections:
-      - standard
-      - default
-      - prefix(github.com/kubeflow/spark-operator)
-  depguard:
-    Main:
-      files:
-        - $all
-        - "!$test"
-      listMode: Lax
-      deny:
-        reflect: Please don't use reflect package
-    Test:
-      files:
-        - $test
-      listMode: Lax
-      deny:
-        reflect: Please don't use reflect package
+  importas:
+    # List of aliases
+    alias:
+    - pkg: k8s.io/api/admissionregistration/v1
+      alias: admissionregistrationv1
+    - pkg: k8s.io/api/apps/v1
+      alias: appsv1
+    - pkg: k8s.io/api/batch/v1
+      alias: batchv1
+    - pkg: k8s.io/api/core/v1
+      alias: corev1
+    - pkg: k8s.io/api/extensions/v1beta1
+      alias: extensionsv1beta1
+    - pkg: k8s.io/api/networking/v1
+      alias: networkingv1
+    - pkg: k8s.io/apimachinery/pkg/apis/meta/v1
+      alias: metav1
+    - pkg: sigs.k8s.io/controller-runtime
+      alias: ctrl
diff --git a/Dockerfile b/Dockerfile
@@ -20,10 +20,6 @@ FROM golang:1.23.1 AS builder
 
 WORKDIR /workspace
 
-RUN apt-get update \
-    && apt-get install -y libcap2-bin \
-    && rm -rf /var/lib/apt/lists/*
-
 RUN --mount=type=cache,target=/go/pkg/mod/ \
     --mount=type=bind,source=go.mod,target=go.mod \
     --mount=type=bind,source=go.sum,target=go.sum \
@@ -36,7 +32,6 @@ ARG TARGETARCH
 RUN --mount=type=cache,target=/go/pkg/mod/ \
     --mount=type=cache,target="/root/.cache/go-build" \
     CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on make build-operator
-RUN setcap 'cap_net_bind_service=+ep' /workspace/bin/spark-operator
 
 FROM ${SPARK_IMAGE}
 
@@ -46,9 +41,9 @@ RUN apt-get update \
     && apt-get install -y tini \
     && rm -rf /var/lib/apt/lists/*
 
-RUN mkdir -p /etc/k8s-webhook-server/serving-certs && \
+RUN mkdir -p /etc/k8s-webhook-server/serving-certs /home/spark && \
     chmod -R g+rw /etc/k8s-webhook-server/serving-certs && \
-    chown -R spark /etc/k8s-webhook-server/serving-certs
+    chown -R spark /etc/k8s-webhook-server/serving-certs /home/spark
 
 USER spark
 

diff --git a/Makefile b/Makefile
@@ -55,7 +55,7 @@ KIND_VERSION ?= v0.23.0
 ENVTEST_VERSION ?= release-0.18
 # ENVTEST_K8S_VERSION refers to the version of kubebuilder assets to be downloaded by envtest binary.
 ENVTEST_K8S_VERSION ?= 1.29.3
-GOLANGCI_LINT_VERSION ?= v1.57.2
+GOLANGCI_LINT_VERSION ?= v1.61.0
 GEN_CRD_API_REFERENCE_DOCS_VERSION ?= v0.3.0
 HELM_VERSION ?= v3.15.3
 HELM_UNITTEST_VERSION ?= 0.5.1
@@ -136,13 +136,13 @@ go-vet: ## Run go vet against code.
 	@echo "Running go vet..."
 	go vet ./...
 
-.PHONY: lint
-lint: golangci-lint ## Run golangci-lint linter.
+.PHONY: go-lint
+go-lint: golangci-lint ## Run golangci-lint linter.
 	@echo "Running golangci-lint run..."
 	$(GOLANGCI_LINT) run
 
-.PHONY: lint-fix
-lint-fix: golangci-lint ## Run golangci-lint linter and perform fixes.
+.PHONY: go-lint-fix
+go-lint-fix: golangci-lint ## Run golangci-lint linter and perform fixes.
 	@echo "Running golangci-lint run --fix..."
 	$(GOLANGCI_LINT) run --fix
 

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-v2.0.1
+v2.0.2
diff --git a/api/v1beta1/sparkapplication_types.go b/api/v1beta1/sparkapplication_types.go
@@ -70,8 +70,8 @@ type SparkApplicationSpec struct {
 	// spark-submit.
 	// Optional.
 	SparkConf map[string]string `json:"sparkConf,omitempty"`
-	// HadoopConf carries user-specified Hadoop configuration properties as they would use the  the "--conf" option
-	// in spark-submit.  The SparkApplication controller automatically adds prefix "spark.hadoop." to Hadoop
+	// HadoopConf carries user-specified Hadoop configuration properties as they would use the "--conf" option
+	// in spark-submit. The SparkApplication controller automatically adds prefix "spark.hadoop." to Hadoop
 	// configuration properties.
 	// Optional.
 	HadoopConf map[string]string `json:"hadoopConf,omitempty"`

diff --git a/api/v1beta2/defaults.go b/api/v1beta2/defaults.go
@@ -54,7 +54,6 @@ func SetSparkApplicationDefaults(app *SparkApplication) {
 }
 
 func setDriverSpecDefaults(spec *DriverSpec, sparkConf map[string]string) {
-
 	if _, exists := sparkConf["spark.driver.cores"]; !exists && spec.Cores == nil {
 		spec.Cores = new(int32)
 		*spec.Cores = 1

diff --git a/api/v1beta2/defaults_test.go b/api/v1beta2/defaults_test.go
@@ -121,7 +121,6 @@ func TestSetSparkApplicationDefaultsOnFailureRestartPolicyShouldSetDefaultValueF
 }
 
 func TestSetSparkApplicationDefaultsDriverSpecDefaults(t *testing.T) {
-
 	//Case1: Driver config not set.
 	app := &SparkApplication{
 		Spec: SparkApplicationSpec{},

diff --git a/api/v1beta2/doc.go b/api/v1beta2/doc.go
@@ -15,7 +15,6 @@ limitations under the License.
 */
 
 // +k8s:deepcopy-gen=package,register
-// go:generate controller-gen crd:trivialVersions=true paths=. output:dir=.
 
 // Package v1beta2 is the v1beta2 version of the API.
 // +groupName=sparkoperator.k8s.io

diff --git a/api/v1beta2/sparkapplication_types.go b/api/v1beta2/sparkapplication_types.go
@@ -71,8 +71,8 @@ type SparkApplicationSpec struct {
 	// spark-submit.
 	// +optional
 	SparkConf map[string]string `json:"sparkConf,omitempty"`
-	// HadoopConf carries user-specified Hadoop configuration properties as they would use the  the "--conf" option
-	// in spark-submit.  The SparkApplication controller automatically adds prefix "spark.hadoop." to Hadoop
+	// HadoopConf carries user-specified Hadoop configuration properties as they would use the "--conf" option
+	// in spark-submit. The SparkApplication controller automatically adds prefix "spark.hadoop." to Hadoop
 	// configuration properties.
 	// +optional
 	HadoopConf map[string]string `json:"hadoopConf,omitempty"`

diff --git a/charts/spark-operator-chart/Chart.yaml b/charts/spark-operator-chart/Chart.yaml
@@ -20,9 +20,9 @@ name: spark-operator
 
 description: A Helm chart for Spark on Kubernetes operator.
 
-version: 2.0.1
+version: 2.0.2
 
-appVersion: 2.0.1
+appVersion: 2.0.2
 
 keywords:
 - apache spark