fix source freshness upload for dbt fusion

[michael-myaskovsky]

Fix process_freshness_result crash on dbt-fusion (#2245)

dbt source freshness on dbt-fusion crashes the on_run_end hook — fusion returns None for node on some results, and the macro accesses result_dict.node.unique_id unguarded. Fix: skip None-node results in process_freshness_result, filter them in upload_source_freshness, and guard timing in flatten_source_freshness. Stops the crash; such results are skipped, not uploaded.

[github-actions]

👋 @michael-myaskovsky
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 18b5ff92-852f-41ba-81fe-7be84ab55df9

📥 Commits

Reviewing files that changed from the base of the PR and between 1c3a526 and d0ab63b.

📒 Files selected for processing (1)

• macros/edr/dbt_artifacts/upload_source_freshness.sql

---

📝 Walkthrough

Walkthrough

Source freshness result processing macros are refactored to normalize output, filter unidentifiable records where node metadata is missing, and use defensive iteration to prevent crashes when optional fields are absent. Conditionally appended results skip none values returned for dbt-fusion records lacking node context.

Changes

Source freshness robustness

Layer / File(s)	Summary
Result normalization, filtering, and defensive extraction macros/edr/dbt_artifacts/upload_source_freshness.sql	process_freshness_result() early-returns none when result_dict.node is none, preventing unidentifiable records. upload_source_freshness() conditionally appends only non-none results. flatten_source_freshness() uses safe iteration over timing entries via .get("timing") with fallback to empty list.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 Three macros align, defensive and kind,
Filtering nulls, no crashes to find,
Safe timing loops guard each freshness row,
Result by result, the pipeline will flow!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix source freshness upload for dbt fusion' directly describes the main change: addressing a bug in source freshness upload functionality specifically for dbt-fusion integration.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-source-freshness-upload-for-dbt-fusion

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

integration_tests/tests/test_dbt_artifacts/test_artifacts.py (1)

204-257: 💤 Low value

Consider asserting the errored source's result is also uploaded with error status.

The test validates that healthy sources aren't blocked by errored ones, which covers the main fix. For more complete coverage, consider also verifying that the errored source's result row exists with status = 'runtime error' — this would confirm the error handling path end-to-end.

That said, the current test adequately covers the critical regression scenario.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@integration_tests/tests/test_dbt_artifacts/test_artifacts.py` around lines
204 - 257, Add an assertion that the errored source's freshness result row is
present and marked as an error: after the existing healthy-source read_table
call, call dbt_project.read_table on "dbt_source_freshness_results" with where
matching unique_id = f"source.elementary_tests.test_source.{errored_name}" (use
the errored_name variable) and assert the returned row's status column equals
"runtime error" (use raise_if_empty=True to ensure presence and then check the
status field). This verifies the errored path is uploaded with status 'runtime
error' in addition to the healthy source check.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@integration_tests/tests/test_dbt_artifacts/test_artifacts.py`:
- Around line 204-257: Add an assertion that the errored source's freshness
result row is present and marked as an error: after the existing healthy-source
read_table call, call dbt_project.read_table on "dbt_source_freshness_results"
with where matching unique_id =
f"source.elementary_tests.test_source.{errored_name}" (use the errored_name
variable) and assert the returned row's status column equals "runtime error"
(use raise_if_empty=True to ensure presence and then check the status field).
This verifies the errored path is uploaded with status 'runtime error' in
addition to the healthy source check.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a1d39111-1c3c-4185-9084-abda79ebd378

📥 Commits

Reviewing files that changed from the base of the PR and between 02e6e0f and 1c3a526.

📒 Files selected for processing (2)

• integration_tests/tests/test_dbt_artifacts/test_artifacts.py
• macros/edr/dbt_artifacts/upload_source_freshness.sql