Skip to content

[FEA] Add selective test execution for PR builds #931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bala-bhargav wants to merge 16 commits into
base: main
Choose a base branch
from
Open

[FEA] Add selective test execution for PR builds #931

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b93bd54
feat(ci): add selective test execution for PR builds (#861)
bala-bhargav Mar 4, 2026
09489bf
refactor(ci): use changed-files action and extract shared component l…
bala-bhargav Mar 17, 2026
60af7c6
Merge branch 'main' into feat/ci-selective-test-execution-861
rgsl888prabhu Mar 17, 2026
1e33a30
fix(ci): address review comments - MIP→LP cascade and narrow include …
bala-bhargav Mar 22, 2026
675eac2
Merge branch 'release/26.04' into feat/ci-selective-test-execution-861
rgsl888prabhu Apr 1, 2026
a39469c
fix tests
rgsl888prabhu Apr 1, 2026
616ab1b
add changes
rgsl888prabhu Apr 1, 2026
88eb4cc
Merge branch 'release/26.04' into feat/ci-selective-test-execution-861
rgsl888prabhu Apr 8, 2026
620ec72
Address review changes
rgsl888prabhu Apr 8, 2026
11d3454
Merge branch 'feat/ci-selective-test-execution-861' of github.com:bal…
rgsl888prabhu Apr 8, 2026
808ff45
update
rgsl888prabhu Apr 8, 2026
b59a15b
fix style
rgsl888prabhu Apr 8, 2026
47bc15e
move files
rgsl888prabhu Apr 8, 2026
c0eeb44
updates
rgsl888prabhu Apr 8, 2026
cf79254
Merge branch 'main' into feat/ci-selective-test-execution-861
rgsl888prabhu Apr 9, 2026
52a11a4
Merge branch 'main' into feat/ci-selective-test-execution-861
rgsl888prabhu Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 76 additions & 3 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,67 @@ jobs:
- '!.cursor-plugin/**'
- '!.claude-plugin/**'
- '!gemini-extension.json'
test_routing:
Comment thread
rgsl888prabhu marked this conversation as resolved.
Outdated
- 'cpp/src/routing/**'
- 'cpp/src/distance/**'
- 'cpp/tests/routing/**'
- 'cpp/tests/distance_engine/**'
- 'cpp/tests/examples/routing/**'
- 'python/cuopt/cuopt/routing/**'
- 'python/cuopt/cuopt/tests/routing/**'
- 'regression/routing*'
- 'cpp/include/**'
Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- 'cpp/include/**'
- 'cpp/include/cuopt/error.hpp'
- 'cpp/include/cuopt/routing/**'
- 'cpp/inlcude/cuopt/utilities/**'

- 'cpp/cmake/**'
- 'cpp/CMakeLists.txt'
- 'cpp/src/utilities/**'
- 'cpp/tests/CMakeLists.txt'
- 'cpp/tests/utilities/**'
- 'python/cuopt/cuopt/*.py'
- 'python/cuopt/cuopt/distance_engine/**'
- 'python/cuopt/cuopt/utils/**'
- 'python/libcuopt/**'
- 'conda/**'
- 'dependencies.yaml'
test_lp:
- 'cpp/src/dual_simplex/**'
- 'cpp/src/barrier/**'
- 'cpp/src/pdlp/**'
- 'cpp/src/math_optimization/**'
- 'cpp/tests/linear_programming/**'
- 'cpp/tests/dual_simplex/**'
- 'cpp/tests/qp/**'
- 'python/cuopt/cuopt/tests/linear_programming/**'
- 'python/cuopt/cuopt/tests/quadratic_programming/**'
- 'regression/lp*'
- 'cpp/include/**'
Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- 'cpp/include/**'
- 'cpp/include/cuopt/error.hpp'
- 'cpp/include/cuopt/linear_programming/**'
- 'cpp/inlcude/cuopt/utilities/**'

- 'cpp/cmake/**'
- 'cpp/CMakeLists.txt'
- 'cpp/src/utilities/**'
- 'cpp/tests/CMakeLists.txt'
- 'cpp/tests/utilities/**'
- 'python/cuopt/cuopt/*.py'
- 'python/cuopt/cuopt/distance_engine/**'
- 'python/cuopt/cuopt/utils/**'
- 'python/libcuopt/**'
- 'conda/**'
- 'dependencies.yaml'
test_mip:
# Note: MIP has no separate Python tests (tested through LP tests).
# If MIP-specific Python tests are added, include their paths here.
- 'cpp/src/branch_and_bound/**'
Comment thread
coderabbitai[bot] marked this conversation as resolved.
- 'cpp/src/cuts/**'
- 'cpp/src/mip_heuristics/**'
- 'cpp/tests/mip/**'
- 'regression/mip*'
- 'cpp/include/**'
Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- 'cpp/include/**'
- 'cpp/include/cuopt/error.hpp'
- 'cpp/include/cuopt/linear_programming/**'
- 'cpp/inlcude/cuopt/utilities/**'

- 'cpp/cmake/**'
- 'cpp/CMakeLists.txt'
- 'cpp/src/utilities/**'
- 'cpp/tests/CMakeLists.txt'
- 'cpp/tests/utilities/**'
- 'python/libcuopt/**'
- 'conda/**'
- 'dependencies.yaml'
checks:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
Expand All @@ -296,7 +357,11 @@ jobs:
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp
with:
build_type: pull-request
script: ci/test_cpp.sh
script: |
export CUOPT_ROUTING_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_routing }}"
export CUOPT_LP_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_lp }}"
export CUOPT_MIP_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_mip }}"
ci/test_cpp.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.conda_test_filter }}
secrets:
script-env-secret-1-key: CUOPT_DATASET_S3_URI
Expand All @@ -320,7 +385,11 @@ jobs:
with:
run_codecov: false
build_type: pull-request
script: ci/test_python.sh
script: |
export CUOPT_ROUTING_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_routing }}"
export CUOPT_LP_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_lp }}"
export CUOPT_MIP_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_mip }}"
ci/test_python.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.conda_test_filter }}
secrets:
script-env-secret-1-key: CUOPT_DATASET_S3_URI
Expand Down Expand Up @@ -381,7 +450,11 @@ jobs:
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_wheels
with:
build_type: pull-request
script: ci/test_wheel_cuopt.sh
script: |
export CUOPT_ROUTING_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_routing }}"
export CUOPT_LP_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_lp }}"
export CUOPT_MIP_CHANGED="${{ fromJSON(needs.changed-files.outputs.changed_file_groups).test_mip }}"
ci/test_wheel_cuopt.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.wheel_lean_filter }}
secrets:
script-env-secret-1-key: CUOPT_DATASET_S3_URI
Expand Down
24 changes: 24 additions & 0 deletions ci/derive_test_components.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Derives CUOPT_TEST_COMPONENTS from changed-files env vars
# (CUOPT_ROUTING_CHANGED, CUOPT_LP_CHANGED, CUOPT_MIP_CHANGED).
#
# When none of these env vars are set (e.g. nightly / non-PR builds),
# defaults to "all" so every test runs.
#
# Usage: source ./ci/derive_test_components.sh

if [[ -z "${CUOPT_ROUTING_CHANGED:-}" && -z "${CUOPT_LP_CHANGED:-}" && -z "${CUOPT_MIP_CHANGED:-}" ]]; then
export CUOPT_TEST_COMPONENTS="all"
else
components=""
[[ "${CUOPT_ROUTING_CHANGED:-true}" == "true" ]] && components="${components:+${components},}routing"
[[ "${CUOPT_LP_CHANGED:-true}" == "true" ]] && components="${components:+${components},}lp"
[[ "${CUOPT_MIP_CHANGED:-true}" == "true" ]] && components="${components:+${components},}mip"
# Fallback to "all" if all components are false (defensive — the job-level
# 'if' gate in pr.yaml would normally skip the job entirely in this case).
export CUOPT_TEST_COMPONENTS="${components:-all}"
fi
echo "CUOPT_TEST_COMPONENTS=${CUOPT_TEST_COMPONENTS}"
46 changes: 44 additions & 2 deletions ci/run_ctests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,44 @@

set -euo pipefail

# Determine which test binary belongs to which component
should_run_test() {
local test_name="$1"
local components="${CUOPT_TEST_COMPONENTS:-all}"

# If all components should run, return true
if [[ "${components}" == "all" ]]; then
return 0
fi

# Map test binary names to components
case "${test_name}" in
ROUTING_*|WAYPOINT_*|VEHICLE_*|OBJECTIVE_*|RETAIL_*)
Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a suggestion, please explore if you can come across better approach.

I am trying to see how to make it easier to manage, if we can have these tests installed per module, like routing, lp, milp, it would be easier to just add those directory instead of adding each new test to this list.

We may have to update cpp/test/CmakeLists.txt

    install(
        TARGETS ${CMAKE_TEST_NAME}
        COMPONENT testing
        DESTINATION bin/gtests/libcuopt
        EXCLUDE_FROM_ALL
    )

to keep the lower folders intact so we can use them

Image

Copy link
Copy Markdown
Author

@bala-bhargav bala-bhargav Mar 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rgsl888prabhu Great suggestion! I think we can update ConfigureTest() to install tests into per-module subdirectories (like bin/gtests/libcuopt/routing/, lp/,

mip/
) so

run_ctests.sh
can just iterate the right folder instead of maintaining the case statement.

Since it touches the CMake build system and all the test CMakeLists, I'd prefer to keep it as a separate follow-up PR to avoid mixing it with the CI changes here. I'll open one once this is merged. Does that sound good?

Copy link
Copy Markdown
Contributor

@mlubin mlubin Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered cmake test labels? I'm pretty concerned about the maintainability of having hardcoded test names in this file

Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, harcoded test cases might create missing tests. I was thinking to tackle this in a follow-up PR, but since we have crossed 26.04 release, may be we can push updates as well in the same PR so there are no breakages in between.

Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mlubin Added label based gtests

Copy link
Copy Markdown
Collaborator

@rgsl888prabhu rgsl888prabhu Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mlubin May I get your review on this ?

[[ "${components}" == *"routing"* ]] && return 0
;;
LP_*|PDLP_*|C_API_*|DUAL_SIMPLEX_*|QP_*)
[[ "${components}" == *"lp"* ]] && return 0
;;
MIP_*|PROBLEM_*|ELIM_*|STANDARDIZATION_*|MULTI_PROBE_*|INCUMBENT_*|DOC_EXAMPLE_*|CUTS_*|EMPTY_*|DETERMINISM_*)
Comment thread
rgsl888prabhu marked this conversation as resolved.
Outdated
[[ "${components}" == *"mip"* ]] && return 0
;;
# UNIT_TEST and PRESOLVE_TEST can belong to LP or MIP
UNIT_TEST|PRESOLVE_TEST)
[[ "${components}" == *"mip"* || "${components}" == *"lp"* ]] && return 0
;;
# CLI_TEST is a general utility test, run if any component is selected
CLI_TEST)
return 0
;;
# Unknown tests: run them to be safe
*)
return 0
;;
esac

return 1
}

# Support customizing the gtests' install location
# First, try the installed location (CI/conda environments)
installed_test_location="${INSTALL_PREFIX:-${CONDA_PREFIX:-/usr}}/bin/gtests/libcuopt/"
Expand All @@ -23,8 +61,12 @@ fi

for gt in "${GTEST_DIR}"/*_TEST; do
test_name=$(basename "${gt}")
echo "Running gtest ${test_name}"
"${gt}" "$@"
if should_run_test "${test_name}"; then
echo "Running gtest ${test_name}"
"${gt}" "$@"
else
echo "Skipping gtest ${test_name} (not in CUOPT_TEST_COMPONENTS=${CUOPT_TEST_COMPONENTS:-all})"
fi
done

# Run C_API_TEST with CPU memory for local solves (excluding time limit tests)
Expand Down
28 changes: 26 additions & 2 deletions ci/run_cuopt_pytests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,30 @@ set -euo pipefail
# It is essential to cd into python/cuopt/cuopt as `pytest-xdist` + `coverage` seem to work only at this directory level.

# Support invoking run_cuopt_pytests.sh outside the script directory
cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../python/cuopt/cuopt/
cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../python/cuopt/cuopt/"

pytest -s --cache-clear "$@" tests
# Build the list of test directories based on CUOPT_TEST_COMPONENTS
COMPONENTS="${CUOPT_TEST_COMPONENTS:-all}"
TEST_DIRS=""

if [[ "${COMPONENTS}" == "all" ]]; then
TEST_DIRS="tests"
else
if [[ "${COMPONENTS}" == *"routing"* ]]; then
TEST_DIRS="${TEST_DIRS} tests/routing"
fi
if [[ "${COMPONENTS}" == *"lp"* ]]; then
TEST_DIRS="${TEST_DIRS} tests/linear_programming tests/quadratic_programming"
fi
# MIP does not have separate Python tests (tested through LP tests)

# If no Python test dirs matched, skip
if [[ -z "${TEST_DIRS}" ]]; then
echo "No Python test directories match CUOPT_TEST_COMPONENTS=${COMPONENTS}, skipping."
exit 0
fi
fi

echo "Running pytest on: ${TEST_DIRS} (CUOPT_TEST_COMPONENTS=${COMPONENTS})"
# shellcheck disable=SC2086
pytest -s --cache-clear "$@" ${TEST_DIRS}
31 changes: 24 additions & 7 deletions ci/test_cpp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,32 @@ rapids-print-env
rapids-logger "Check GPU usage"
nvidia-smi

rapids-logger "Download datasets"
./datasets/linear_programming/download_pdlp_test_dataset.sh
./datasets/mip/download_miplib_test_dataset.sh
# Derive CUOPT_TEST_COMPONENTS from changed-files env vars
source ./ci/derive_test_components.sh

rapids-logger "Download datasets"
RAPIDS_DATASET_ROOT_DIR="$(realpath datasets)"
export RAPIDS_DATASET_ROOT_DIR
pushd "${RAPIDS_DATASET_ROOT_DIR}"
./get_test_data.sh
popd

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"lp"* ]]; then
./datasets/linear_programming/download_pdlp_test_dataset.sh
else
rapids-logger "Skipping LP dataset download (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"mip"* ]]; then
./datasets/mip/download_miplib_test_dataset.sh
else
rapids-logger "Skipping MIP dataset download (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"routing"* ]]; then
pushd "${RAPIDS_DATASET_ROOT_DIR}"
./get_test_data.sh
popd
else
rapids-logger "Skipping routing dataset downloads (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

EXITCODE=0
trap "EXITCODE=1" ERR
Expand All @@ -51,7 +68,7 @@ set +e
# Run gtests from libcuopt-tests package
export GTEST_OUTPUT=xml:${RAPIDS_TESTS_DIR}/

rapids-logger "Run gtests"
rapids-logger "Run gtests (components: ${CUOPT_TEST_COMPONENTS})"
timeout 40m ./ci/run_ctests.sh

rapids-logger "Test script exiting with value: $EXITCODE"
Expand Down
30 changes: 24 additions & 6 deletions ci/test_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,32 @@ mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}"

rapids-print-env

# Derive CUOPT_TEST_COMPONENTS from changed-files env vars
source ./ci/derive_test_components.sh

rapids-logger "Download datasets"
RAPIDS_DATASET_ROOT_DIR="$(realpath datasets)"
export RAPIDS_DATASET_ROOT_DIR
./datasets/linear_programming/download_pdlp_test_dataset.sh
./datasets/mip/download_miplib_test_dataset.sh
pushd "${RAPIDS_DATASET_ROOT_DIR}"
./get_test_data.sh
popd

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"lp"* ]]; then
./datasets/linear_programming/download_pdlp_test_dataset.sh
else
rapids-logger "Skipping LP dataset download (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"mip"* ]]; then
./datasets/mip/download_miplib_test_dataset.sh
else
rapids-logger "Skipping MIP dataset download (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"routing"* ]]; then
pushd "${RAPIDS_DATASET_ROOT_DIR}"
./get_test_data.sh
popd
else
rapids-logger "Skipping routing dataset downloads (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

rapids-logger "Check GPU usage"
nvidia-smi
Expand All @@ -57,7 +75,7 @@ export OMP_NUM_THREADS=1
rapids-logger "Test cuopt_cli"
timeout 10m bash ./python/libcuopt/libcuopt/tests/test_cli.sh

rapids-logger "pytest cuopt"
rapids-logger "pytest cuopt (components: ${CUOPT_TEST_COMPONENTS})"
timeout 30m ./ci/run_cuopt_pytests.sh \
--junitxml="${RAPIDS_TESTS_DIR}/junit-cuopt.xml" \
--cov-config=.coveragerc \
Expand Down
30 changes: 24 additions & 6 deletions ci/test_wheel_cuopt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,33 @@ elif command -v dnf &> /dev/null; then
dnf -y install file unzip
fi

./datasets/linear_programming/download_pdlp_test_dataset.sh
./datasets/mip/download_miplib_test_dataset.sh
cd ./datasets
./get_test_data.sh --solomon
./get_test_data.sh --tsp
cd -
# Derive CUOPT_TEST_COMPONENTS from changed-files env vars
source ./ci/derive_test_components.sh

RAPIDS_DATASET_ROOT_DIR="$(realpath datasets)"
export RAPIDS_DATASET_ROOT_DIR

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"lp"* ]]; then
./datasets/linear_programming/download_pdlp_test_dataset.sh
else
echo "Skipping LP dataset download (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"mip"* ]]; then
./datasets/mip/download_miplib_test_dataset.sh
else
echo "Skipping MIP dataset download (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

if [[ "${CUOPT_TEST_COMPONENTS}" == "all" || "${CUOPT_TEST_COMPONENTS}" == *"routing"* ]]; then
cd ./datasets
./get_test_data.sh --solomon
./get_test_data.sh --tsp
cd -
else
echo "Skipping routing dataset downloads (not needed for components: ${CUOPT_TEST_COMPONENTS})"
fi

# Run CLI tests
timeout 10m bash ./python/libcuopt/libcuopt/tests/test_cli.sh

Expand All @@ -61,6 +78,7 @@ timeout 10m bash ./python/libcuopt/libcuopt/tests/test_cli.sh
# Due to race condition in certain cases UCX might not be able to cleanup properly, so we set the number of threads to 1
export OMP_NUM_THREADS=1

echo "Running cuopt pytests (components: ${CUOPT_TEST_COMPONENTS})"
timeout 30m ./ci/run_cuopt_pytests.sh --verbose --capture=no

# run thirdparty integration tests for only nightly builds
Expand Down