Skip to content

Commit 56e75a8

Browse files
dongjoon-hyundongjoon-hyun
committed
[SPARK-56139] Add E2E test for ApplicationTimeoutConfig with driverStartTimeoutMillis
### What changes were proposed in this pull request? This PR adds an E2E test for `ApplicationTimeoutConfig` with `driverStartTimeoutMillis`. ### Why are the changes needed? There is no existing E2E test coverage for `ApplicationTimeoutConfig`. This test ensures the `driverStartTimeoutMillis` configuration works correctly end-to-end in a real Kubernetes environment. The test uses a non-existent container image (`apache/spark:non-existent-image`) to trigger a driver start timeout and validates the state transition: `Submitted` → `DriverRequested` → `DriverStartTimedOut` → `ResourceReleased`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added E2E test, `tests/e2e/driver-start-timeout/`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-4-6) Closes #575 from dongjoon-hyun/SPARK-56139. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 7018cf0 commit 56e75a8

4 files changed

Lines changed: 124 additions & 0 deletions

File tree

‎.github/workflows/build_and_test.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ jobs:
9595
- resource-retain-duration
9696
- resource-selector
9797
- watched-namespaces
98+
- driver-start-timeout
9899
exclude:
99100
- mode: dynamic
100101
test-group: spark-versions
@@ -106,6 +107,8 @@ jobs:
106107
test-group: resource-retain-duration
107108
- mode: dynamic
108109
test-group: resource-selector
110+
- mode: dynamic
111+
test-group: driver-start-timeout
109112
- mode: static
110113
test-group: watched-namespaces
111114
- mode: static
@@ -120,6 +123,8 @@ jobs:
120123
test-group: resource-retain-duration
121124
- mode: selector
122125
test-group: watched-namespaces
126+
- mode: selector
127+
test-group: driver-start-timeout
123128
steps:
124129
- name: Checkout repository
125130
uses: actions/checkout@v6

‎tests/e2e/assertions/spark-application/spark-driver-start-timeout-state-transition.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
apiVersion: spark.apache.org/v1
19+
kind: SparkApplication
20+
metadata:
21+
name: spark-example-driver-start-timeout
22+
namespace: ($SPARK_APP_NAMESPACE)
23+
status:
24+
stateTransitionHistory:
25+
(*.currentStateSummary):
26+
- "Submitted"
27+
- "DriverRequested"
28+
- "DriverStartTimedOut"
29+
- "ResourceReleased"

‎tests/e2e/driver-start-timeout/chainsaw-test.yaml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
apiVersion: chainsaw.kyverno.io/v1alpha1
19+
kind: Test
20+
metadata:
21+
name: spark-operator-driver-start-timeout
22+
spec:
23+
scenarios:
24+
- bindings:
25+
- name: TEST_NAME
26+
value: driver-start-timeout
27+
- name: APPLICATION_FILE_NAME
28+
value: spark-example-driver-start-timeout.yaml
29+
- name: SPARK_APPLICATION_NAME
30+
value: spark-example-driver-start-timeout
31+
steps:
32+
- try:
33+
- script:
34+
env:
35+
- name: FILE_NAME
36+
value: ($APPLICATION_FILE_NAME)
37+
content: kubectl apply -f $FILE_NAME
38+
- assert:
39+
bindings:
40+
- name: SPARK_APP_NAMESPACE
41+
value: default
42+
timeout: 120s
43+
file: "../assertions/spark-application/spark-driver-start-timeout-state-transition.yaml"
44+
catch:
45+
- describe:
46+
apiVersion: spark.apache.org/v1
47+
kind: SparkApplication
48+
namespace: default
49+
finally:
50+
- script:
51+
env:
52+
- name: SPARK_APPLICATION_NAME
53+
value: ($SPARK_APPLICATION_NAME)
54+
timeout: 120s
55+
content: |
56+
kubectl delete sparkapplication $SPARK_APPLICATION_NAME --ignore-not-found=true

‎tests/e2e/driver-start-timeout/spark-example-driver-start-timeout.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
apiVersion: spark.apache.org/v1
19+
kind: SparkApplication
20+
metadata:
21+
name: spark-example-driver-start-timeout
22+
namespace: default
23+
spec:
24+
mainClass: "org.apache.spark.examples.SparkPi"
25+
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
26+
applicationTolerations:
27+
applicationTimeoutConfig:
28+
driverStartTimeoutMillis: 30000
29+
sparkConf:
30+
spark.executor.instances: "1"
31+
spark.kubernetes.container.image: "apache/spark:non-existent-image"
32+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
33+
runtimeVersions:
34+
sparkVersion: "4.1.1"

0 commit comments

Comments
 (0)