[ci_gen_kustomize_values] Co-locate provisionserver with metal3 to prevent DHCP failures by mnietoji · Pull Request #3738 · openstack-k8s-operators/ci-framework

mnietoji · 2026-03-04T11:27:30Z

…ures

When metal3-dnsmasq pod restarts during a node's DHCP lease renewal on the provisioning network (172.23.0.0/24), NetworkManager fails to renew and sets ipv4.method=disabled. NMState operator then preserves this disabled state, causing permanent loss of provisioning network connectivity on that node.

The issue occurs when OpenStackProvisionServer and metal3 pods run on different nodes. If metal3 restarts while a node is attempting DHCP renewal, the temporary unavailability of metal3-dnsmasq causes the renewal to fail.

Solution:
Automatically detect the node running metal3 pod (via k8s-app=metal3 label) and configure provisionServerNodeSelector in baremetalSetTemplate to schedule OpenStackProvisionServer on the same node. This ensures provisioning network connectivity is maintained because metal3-static-ip-manager maintains a static IP (172.23.0.3) on the metal3 node regardless of dnsmasq restarts.

openshift-ci · 2026-03-04T11:27:36Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tosky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2026-03-04T11:38:10Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/ci-framework for 3738,d660efa12350eb88ab3c89b1d91a04abcbc82293

softwarefactory-project-zuul · 2026-03-04T11:43:08Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/ci-framework for 3738,369ae185bc2b7d5a266e63c93224f86f1d2723cd

softwarefactory-project-zuul · 2026-03-04T11:49:32Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/ci-framework for 3738,3fa51c9d28a6f3c53f0c99dbbdef1baf476724d5

softwarefactory-project-zuul · 2026-03-04T12:02:11Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3217375613864e0d83f7f88f394dcfaa

❌ openstack-k8s-operators-content-provider FAILURE in 7m 16s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 25s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 21s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 49s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 44s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 54s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 24s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 14s

softwarefactory-project-zuul · 2026-03-10T17:18:55Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0b04bcb1f4f54d518d017da862888f74

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 03m 30s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 08s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 25m 42s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 49m 34s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 28s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 35s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 49s
❌ cifmw-pod-pre-commit TIMED_OUT in 31m 04s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 29s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 33s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 07s

github-actions · 2026-04-09T04:15:49Z

This PR is stale because it has been for over 15 days with no activity.
Remove stale label or comment or this will be closed in 7 days.

michburk

Has this been tested? If so, could you link a jira ticket with any relevant downstream links hidden behind comments or descriptions that are marked as 'Red Hat Employee'?

Thanks!

michburk · 2026-04-09T19:48:42Z

+{%   for key, value in _original_baremetal_template.items()                            %}
+    {{ key }}: {{ value }}
+{%   endfor                                                                            %}


Would it be possible to use something like | to_nice_yaml for this instead of manually deconstructing and reconstructing the yaml key: values?

yes, improved the code

michburk · 2026-04-09T19:55:26Z

      ansible.builtin.include_role:
        name: run_hook

+    - name: Detect metal3 pod node for baremetal nodeset provisioning


Would it make more sense to put these tasks somewhere other than execute_step.yml? Would these tasks be better-suited to living in some dedicated hook, rather than being part of this generic execute_step.yml file?

yes, you are right. moved the code to a different place

…event DHCP failures When metal3-dnsmasq pod restarts during a node's DHCP lease renewal on the provisioning network (172.23.0.0/24), NetworkManager fails to renew and sets ipv4.method=disabled. NMState operator then preserves this disabled state, causing permanent loss of provisioning network connectivity on that node. The issue occurs when OpenStackProvisionServer and metal3 pods run on different nodes. If metal3 restarts while a node is attempting DHCP renewal, the temporary unavailability of metal3-dnsmasq causes the renewal to fail. Solution: Automatically detect the node running metal3 pod (via k8s-app=metal3 label) and configure provisionServerNodeSelector in baremetalSetTemplate to schedule OpenStackProvisionServer on the same node. This ensures provisioning network connectivity is maintained because metal3-static-ip-manager maintains a static IP (172.23.0.3) on the metal3 node regardless of dnsmasq restarts. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Miguel Angel Nieto Jimenez <mnietoji@redhat.com>

mnietoji added the do-not-merge/work-in-progress label Mar 4, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from f5c2a2c to d660efa Compare March 4, 2026 11:37

openshift-merge-robot added the needs-rebase label Mar 4, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from d660efa to 369ae18 Compare March 4, 2026 11:42

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 369ae18 to 3fa51c9 Compare March 4, 2026 11:49

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 3fa51c9 to d0cf92f Compare March 4, 2026 11:52

openshift-merge-robot removed the needs-rebase label Mar 4, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from d0cf92f to 6b9c8b0 Compare March 4, 2026 14:44

github-actions bot added the Ready For Review label Mar 4, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 6b9c8b0 to 1339a1d Compare March 4, 2026 17:48

github-actions bot added Ready For Review and removed Ready For Review labels Mar 4, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 1339a1d to cf58db9 Compare March 5, 2026 10:55

github-actions bot removed Ready For Review labels Mar 5, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch 3 times, most recently from e29b915 to 08a2b2b Compare March 10, 2026 15:14

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 08a2b2b to a7b18cb Compare March 12, 2026 21:29

github-actions bot added the Ready For Review label Mar 12, 2026

mnietoji removed the do-not-merge/work-in-progress label Mar 18, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from a7b18cb to 4ec3152 Compare March 18, 2026 15:15

github-actions bot removed Ready For Review labels Mar 18, 2026

github-actions bot added Stale Ready For Review and removed Ready For Review labels Apr 9, 2026

michburk reviewed Apr 9, 2026

View reviewed changes

github-actions bot added Ready For Review and removed Stale Ready For Review labels Apr 10, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 3c8b076 to dc09396 Compare April 13, 2026 09:17

github-actions bot added Ready For Review and removed Ready For Review labels Apr 13, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from dc09396 to 309d835 Compare April 14, 2026 15:09

github-actions bot removed the Ready For Review label Apr 14, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 309d835 to 073a7c2 Compare April 14, 2026 15:11

github-actions bot added the Ready For Review label Apr 14, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 073a7c2 to 6ed9e2a Compare April 14, 2026 21:16

github-actions bot removed the Ready For Review label Apr 14, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch 7 times, most recently from 7720fe4 to 2ffc089 Compare April 14, 2026 21:42

mnietoji changed the title ~~[multiple] Co-locate provisionserver with metal3 to prevent DHCP fail…~~ [ci_gen_kustomize_values] Co-locate provisionserver with metal3 to prevent DHCP failures Apr 14, 2026

github-actions bot added the Ready For Review label Apr 15, 2026

mnietoji force-pushed the dhcp_provisioning_with_fix branch from 2ffc089 to 2fc519d Compare April 15, 2026 10:46

github-actions bot added Ready For Review and removed Ready For Review labels Apr 15, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci_gen_kustomize_values] Co-locate provisionserver with metal3 to prevent DHCP failures#3738

[ci_gen_kustomize_values] Co-locate provisionserver with metal3 to prevent DHCP failures#3738
mnietoji wants to merge 1 commit intoopenstack-k8s-operators:mainfrom
mnietoji:dhcp_provisioning_with_fix

mnietoji commented Mar 4, 2026

Uh oh!

openshift-ci bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 10, 2026

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

michburk left a comment

Uh oh!

michburk Apr 9, 2026

Uh oh!

mnietoji Apr 15, 2026

Uh oh!

michburk Apr 9, 2026

Uh oh!

mnietoji Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mnietoji commented Mar 4, 2026

Uh oh!

openshift-ci bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 4, 2026

Uh oh!

softwarefactory-project-zuul bot commented Mar 10, 2026

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

michburk left a comment

Choose a reason for hiding this comment

Uh oh!

michburk Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

mnietoji Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

michburk Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

mnietoji Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants