Skip to content

Commit eb5b90d

Browse files
committed
[SPARK-56142] Add worker StatefulSet replica validation to SparkCluster e2e test
### What changes were proposed in this pull request? This PR adds a worker StatefulSet replica validation to the SparkCluster e2e test. A new assertion file (`spark-cluster-worker-statefulset.yaml`) verifies that the worker StatefulSet's `spec.replicas` and `status.readyReplicas` match the `initWorkers` configuration. ### Why are the changes needed? The existing SparkCluster e2e test only validates the state transition history (`Submitted` → `RunningHealthy`). It does not verify that the worker StatefulSet is actually created with the correct number of replicas. This gap means the test could pass even if the worker scaling is broken. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added validation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.6) Closes #577 from dongjoon-hyun/SPARK-56142. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 56e75a8 commit eb5b90d

2 files changed

Lines changed: 32 additions & 0 deletions

File tree

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
apiVersion: apps/v1
19+
kind: StatefulSet
20+
metadata:
21+
name: spark-cluster-succeeded-test-worker
22+
namespace: ($SPARK_APP_NAMESPACE)
23+
spec:
24+
replicas: 1
25+
status:
26+
readyReplicas: 1

tests/e2e/state-transition/chainsaw-test.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,12 @@ spec:
7070
value: default
7171
timeout: 60s
7272
file: "../assertions/spark-cluster/spark-cluster-state-transition.yaml"
73+
- assert:
74+
bindings:
75+
- name: SPARK_APP_NAMESPACE
76+
value: default
77+
timeout: 60s
78+
file: "../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
7379
catch:
7480
- describe:
7581
apiVersion: spark.apache.org/v1

0 commit comments

Comments
 (0)