Skip to content

Use X-Databricks-Workspace-Id for workspace routing#1436

Merged
Divyansh-db merged 13 commits into
mainfrom
new-workspace-parameter
May 28, 2026
Merged

Use X-Databricks-Workspace-Id for workspace routing#1436
Divyansh-db merged 13 commits into
mainfrom
new-workspace-parameter

Conversation

@Divyansh-db
Copy link
Copy Markdown
Contributor

@Divyansh-db Divyansh-db commented May 20, 2026

Summary

Workspace-scoped API requests now identify the target workspace via the X-Databricks-Workspace-Id header instead of X-Databricks-Org-Id. The value source is unchanged: it still comes from Config.workspace_id (or the DATABRICKS_WORKSPACE_ID environment variable). Response-side reads of /api/2.0/preview/scim/v2/Me continue to read the workspace ID from the legacy X-Databricks-Org-Id response header — this PR is outbound-only.

Why

On unified Databricks hosts that serve multiple workspaces, the SDK sends a routing header on every workspace-scoped HTTP request so the gateway can dispatch it to the correct workspace. Until now that header was X-Databricks-Org-Id.

The Databricks platform is consolidating workspace addressing onto a single header, X-Databricks-Workspace-Id, which accepts a broader range of workspace identifier formats and replaces the older single-purpose channels going forward. The previous header continues to be accepted by the platform, so older SDK versions keep working, but new SDK versions should use the new name.

This PR is the Python SDK's part of that migration.

What changed

Interface changes

None. Config.workspace_id keeps the same field name, type (str), and environment variable (DATABRICKS_WORKSPACE_ID). The accompanying comment is widened to note that the value may now be either a classic numeric workspace ID or another workspace identifier format that the server understands.

Behavioral changes

  • Workspace-scoped requests no longer send X-Databricks-Org-Id. They now send X-Databricks-Workspace-Id (with the same value, gated on Config.workspace_id being non-empty). Account-scoped requests are unaffected.
  • Response-side reads are unchanged. The server still emits the workspace ID in the legacy X-Databricks-Org-Id response header on /api/2.0/preview/scim/v2/Me, so WorkspaceClient.get_workspace_id(), the storage-proxy workspace probe in FilesExt._resolve_workspace_id(), and the SQL HTTP path resolver in Config._workspace_id_from_jdbc_path() all continue to read the old header name.

Internal changes

  • Regenerated databricks/sdk/service/*.py (including the newly-added bundle.py), databricks/sdk/__init__.py, and the generated test fixtures under tests/databricks/sdk/service/ from the corresponding generator template change. Pure mechanical literal swap on workspace-scoped request emission.
  • Updated the hand-written sites that emit the same header on workspace-scoped calls but are not covered by the generator: databricks/sdk/mixins/workspace.py (upload/download), databricks/sdk/mixins/sharing.py (list shares).
  • Widened the docstring on Config.workspace_id to note the new accepted formats.

Test changes

  • Renamed two request-side unit tests in tests/test_config.py so the names reflect the new request-side header (test_*_org_id_header_*test_*_workspace_id_header_*).
  • Upgraded the third renamed test into a direct unit test on WorkspaceClient.get_workspace_id(): test_get_workspace_id_reads_org_id_response_header_when_config_missing_workspace_id now mocks /api/2.0/preview/scim/v2/Me returning an X-Databricks-Org-Id response header and asserts both (a) the SDK does not send the new request header when Config.workspace_id is empty, and (b) the SDK correctly parses the legacy response header.
  • Added a wire-level integration test test_spog_unified_host_sends_workspace_id_header in tests/integration/test_unified_profile.py that mounts a transport-level capture adapter on the SDK's requests.Session and probes /api/2.0/preview/scim/v2/Me against a real unified host, asserting (1) the request carries X-Databricks-Workspace-Id, (2) the request does not carry the legacy X-Databricks-Org-Id, and (3) the response still echoes X-Databricks-Org-Id. Mirrors the Go SDK's equivalent acceptance test.

How is this tested?

  • make test passes locally (2073 passed, 3 skipped).
  • Unit coverage:
    • The two renamed request-side tests in tests/test_config.py cover request-header emission on workspace-scoped vs account-scoped calls on unified hosts.
    • test_get_workspace_id_reads_org_id_response_header_when_config_missing_workspace_id is a direct unit test on WorkspaceClient.get_workspace_id() covering both directions (no new request header + legacy response header parsed correctly) on the SCIM /Me probe.
    • The storage-proxy / SCIM-/Me response-parsing paths in mixins/files.py are exercised by tests/test_files.py (mock fixtures continue to emit X-Databricks-Org-Id, matching the unchanged response-side behavior).
  • Integration coverage:
    • tests/integration/test_unified_profile.py::test_spog_unified_host_sends_workspace_id_header verifies wire-level header behavior against a real unified host (runs in the OAuth M2M-equipped matrices).
  • Spot-checked the regenerated diff: every line change in databricks/sdk/service/*.py is exclusively the header literal swap; no collateral changes.

Mechanical literal swap matching the updated pysdkv0/pysdkv1 generator
templates that emit the workspace routing header on workspace-scoped
operations. 35 generated files, 969 occurrences (symmetric +/-):

- 30 generated service modules under databricks/sdk/service/
- databricks/sdk/__init__.py (WorkspaceClient.get_workspace_id())
- 4 generated test fixtures under tests/databricks/sdk/service/

No behavioral change beyond the header name; emission remains gated on
the existing `cfg.workspace_id` check.

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Generator can't touch these (no DO NOT EDIT marker), or need
post-generation reformatting because the new literal is longer:

- config.py: SQL HTTP path resolver header read + widen workspace_id
  field comment (now accepts non-classic workspace identifier formats)
- mixins/workspace.py: upload/download header injection (2 sites)
- mixins/sharing.py: shares-list header injection
- mixins/files.py: storage-proxy workspace probe (2 sites)
- tests/test_config.py: renamed three header tests
  (test_workspace_org_id_header_* -> test_workspace_id_header_*,
   test_no_org_id_header_* -> test_no_workspace_id_header_*)
  and swapped asserted literals/docstrings
- tests/test_files.py: updated mock response header fixtures
- tests/integration/test_unified_profile.py: comment refresh
- NEXT_CHANGELOG.md: internal-changes entry
- __init__.py: black-wrap of the get_workspace_id() probe line that
  exceeded 120 chars after the longer literal substitution

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Mechanical literal swap on services that picked up the old header
literal after the most recent SDK regen (#1438):
- new file: databricks/sdk/service/bundle.py (14 sites)
- new methods in databricks/sdk/service/dashboards.py (1 site)
- new methods in databricks/sdk/service/postgres.py (1 site)

Symmetric 16/16 diff; no logic change.

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
@Divyansh-db Divyansh-db force-pushed the new-workspace-parameter branch from b2d0839 to 370975a Compare May 22, 2026 14:17
@Divyansh-db Divyansh-db temporarily deployed to test-trigger-is May 22, 2026 14:17 — with GitHub Actions Inactive
@Divyansh-db Divyansh-db temporarily deployed to test-trigger-is May 22, 2026 14:20 — with GitHub Actions Inactive
@Divyansh-db Divyansh-db requested a review from hectorcast-db May 22, 2026 15:51
The server still emits the workspace ID in the legacy
`X-Databricks-Org-Id` response header on `/api/2.0/preview/scim/v2/Me`,
so reverts the response-side reads back to the old name. The
request-side migration to `X-Databricks-Workspace-Id` is unchanged.

8 sites reverted on the same API (GET /api/2.0/preview/scim/v2/Me):
- __init__.py: WorkspaceClient.get_workspace_id() docstring + the two
  response reads (response_headers arg + dict-key read)
- mixins/files.py: FilesExt._resolve_workspace_id() storage-proxy probe
- config.py: Config._workspace_id_from_jdbc_path() SQL HTTP path resolver
- tests/test_files.py: two mock-response fixtures simulating the server

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Signed-off-by: Divyansh Vijayvergia <171924202+Divyansh-db@users.noreply.github.com>
Pure mechanical reformatting from running `make fmt` (ruff format)
against files that still carried the prior black wrapping convention.
Two patterns swapped:
- `assert (cond), msg` with parens around the condition collapses to
  `assert cond, (msg)` with parens around the message
- short `self._api.do(...)` calls that black had wrapped now fit on
  one line under ruff's 120-char limit

7 files touched, no behavioral changes.

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
The prior reformat commit (db2a26f) was made with ruff 0.15.x, which
applies a different `assert cond, (msg)` parenthesization than the
pinned ruff 0.5.6 (which keeps `assert (cond), msg`). This commit
reverts those five test files back to the layout that the pinned
formatter produces, so `make lint` stays green.

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Rewrites the previous `test_no_workspace_id_header_on_regular_workspace`
test (which probed an unrelated `/api/2.0/test` path via a raw ApiClient
purely to assert header absence) into a direct test of
`WorkspaceClient.get_workspace_id()` that exercises both sides of the
header migration on the SCIM /Me probe:

- Request side: when `Config.workspace_id` is empty, the SDK must NOT
  send the new `X-Databricks-Workspace-Id` header.
- Response side: the SDK must read the workspace ID from the legacy
  `X-Databricks-Org-Id` response header, since the server-side migration
  hasn't happened yet.

Mirrors the Go SDK's `TestCurrentWorkspaceIDOmitsWorkspaceIdHeaderWhenConfigMissingWorkspaceID`
in `workspace_functions_test.go`.

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Mirrors the Go SDK's TestAccUnifiedHostSendsWorkspaceIdHeader in
internal/unified_host_test.go. Probes /api/2.0/preview/scim/v2/Me
against a real unified (SPOG) host with a transport-level capture
adapter mounted on the SDK's session and asserts three things at
the wire:

1. SDK sends X-Databricks-Workspace-Id on workspace-scoped requests
2. SDK does not send the legacy X-Databricks-Org-Id request header
3. Server still echoes X-Databricks-Org-Id on the response (server
   side has not migrated, which is why get_workspace_id() still
   reads the legacy name)

Requires UNIFIED_HOST, THIS_WORKSPACE_ID, TEST_ACCOUNT_ID,
TEST_DATABRICKS_CLIENT_ID, TEST_DATABRICKS_CLIENT_SECRET in the
ucws isolated env (consistent with the other test_spog_workspace_*
tests in this file).

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Signed-off-by: Divyansh Vijayvergia <171924202+Divyansh-db@users.noreply.github.com>
Divyansh-db and others added 2 commits May 28, 2026 08:00
`_CaptureHeadersAdapter` was copying request/response headers into a
plain `dict()`, which preserves the original wire casing as keys. The
Azure unified host (HTTP/2) emits headers in lowercase, so the assertion
`"X-Databricks-Org-Id" in transport.resp_headers` failed even though the
server was sending the header — its key was `x-databricks-org-id`.

Switch to `requests.structures.CaseInsensitiveDict` for both bags so
assertions match what the production SDK code does (e.g.
`_base_client.py` reads `response.headers.get(...)` on a
`CaseInsensitiveDict`, which is why the SDK itself was never affected).

Signed-off-by: Divyansh Vijayvergia <divyansh.vijayvergia@databricks.com>
Signed-off-by: Divyansh Vijayvergia <171924202+Divyansh-db@users.noreply.github.com>
@Divyansh-db Divyansh-db temporarily deployed to test-trigger-is May 28, 2026 08:10 — with GitHub Actions Inactive
@github-actions
Copy link
Copy Markdown

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-py

Inputs:

  • PR number: 1436
  • Commit SHA: 15a5e5e00eb5b14f4414ee9bd7416bc1c3577820

Checks will be approved automatically on success.

@Divyansh-db Divyansh-db temporarily deployed to test-trigger-is May 28, 2026 08:11 — with GitHub Actions Inactive
Copy link
Copy Markdown
Contributor

@hectorcast-db hectorcast-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Divyansh-db Divyansh-db added this pull request to the merge queue May 28, 2026
Merged via the queue into main with commit 64861e9 May 28, 2026
14 checks passed
@Divyansh-db Divyansh-db deleted the new-workspace-parameter branch May 28, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants