Skip to content

fix(storage): Invocation ID is not retained on multipart upload retries#8190

Draft
thiyaguk09 wants to merge 21 commits into
googleapis:storage-node-18from
thiyaguk09:fix/multipart-invocation-id
Draft

fix(storage): Invocation ID is not retained on multipart upload retries#8190
thiyaguk09 wants to merge 21 commits into
googleapis:storage-node-18from
thiyaguk09:fix/multipart-invocation-id

Conversation

@thiyaguk09
Copy link
Copy Markdown
Contributor

Hoists the generation of persistentInvocationId to the beginning of the
upload process in Bucket.upload and File.save. This ensures that retried multipart upload attempts reuse the same invocation ID in the x-goog-api-client header, rather than generating a new one for each attempt.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@thiyaguk09 thiyaguk09 requested a review from a team as a code owner May 6, 2026 12:56
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label May 6, 2026
@thiyaguk09 thiyaguk09 requested review from a team as code owners May 6, 2026 12:58
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements persistent invocationId tracking across the upload lifecycle and refactors the StorageTransport.makeRequest method to improve project ID resolution and header management. Feedback focuses on critical issues in the makeRequest refactor, including a bug where the request body is incorrectly assigned to an undefined property, thread-safety risks from unconditionally clearing shared interceptors, and an inconsistent response handler that mutates data and breaks return type expectations.

Comment thread handwritten/storage/src/storage-transport.ts Outdated
Comment thread handwritten/storage/src/storage-transport.ts Outdated
Comment on lines +193 to +202
const responseHandler = (resp: GaxiosResponse<T>) => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const data = resp.data as any;
if (data !== null && typeof data === 'object') {
data.headers = resp.headers;
data.status = resp.status;
return data;
}
return resp;
};
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The responseHandler introduces an inconsistency in the return type of makeRequest. If the response data is an object, it returns the data mutated with headers and status. If the response data is not an object (e.g., a string or null), it returns the entire GaxiosResponse object. This violates the expected return type T and will likely cause runtime errors in callers expecting raw data. Additionally, mutating the data object directly is risky as it might be a frozen object or a class instance that doesn't support these extra properties. It is recommended to return resp.data consistently as before.

      const responseHandler = (resp: GaxiosResponse<T>) => resp.data;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required for retry operations

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please elaborate as stated in the above comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To preserve backward compatibility for legacy SDK methods that expect resource metadata objects to carry .headers and .status properties without introducing runtime errors or mutating binary data, we added a strict validation check to ensure these properties are strictly injected into plain JSON response objects and never onto Buffer or Stream payloads.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with using isPlanObject to avoid mutating Buffer or Stream payloads.

However the issue mentioned above refers to the fallback "return resp;" statement. If the payload is a Buffer or Stream and the check fails, returning the entire GaxiosResponse wrapper instead of the raw data might break backward compatibility for binary downloads, since it previously resolved with just the data.

What do you think about returning data instead in that fallback case to keep the return types consistent?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch. Simply returning data broke resumable uploads, which rely on resp.headers.location when the API returns an empty string. I've updated the logic to handle all three scenarios correctly:

  • Plain objects: Mutate and return data.
  • Buffers/Streams (non-plain objects): Return raw data to protect binary downloads.
  • Primitives (empty strings/null): Return the full resp wrapper so session URIs can still be read from the headers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any presubmits or checks to prevent such kind of breakages ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, our conformance test suite acts as the safety net for this (referenced in draft PR #7917). In fact, it was the conformance tests that caught this exact breakage. When I tested changing the fallback to just return data, the bucketUploadResumable test scenarios immediately failed because they couldn't extract the session URI from the headers. These test suites run during presubmit to prevent regressions like this from merging.

@thiyaguk09 thiyaguk09 force-pushed the storage-node-18 branch 2 times, most recently from da2ee2d to b4ee48b Compare May 7, 2026 07:55
@thiyaguk09 thiyaguk09 requested review from a team as code owners May 7, 2026 07:55
Hoists the generation of `persistentInvocationId` to the beginning of
the
upload process in `Bucket.upload` and `File.save`. This ensures that
retried multipart upload attempts reuse the same invocation ID in the
`x-goog-api-client` header, rather than generating a new one for
each attempt.
@thiyaguk09 thiyaguk09 force-pushed the fix/multipart-invocation-id branch from 5b69a10 to e232f4e Compare May 7, 2026 10:00
@thiyaguk09 thiyaguk09 removed request for a team May 7, 2026 10:20
…sport (googleapis#8283)

- Remove Service.ts and common.ts files from handwritten/storage

- Migrate remaining functionality to StorageTransport

- chore(ci): upgrade conformance tests to Node 18
…sport (googleapis#8283)

- Remove Service.ts and common.ts files from handwritten/storage

- Migrate remaining functionality to StorageTransport

- chore(ci): upgrade conformance tests to Node 18
…sport (googleapis#8283)

- Remove Service.ts and common.ts files from handwritten/storage

- Migrate remaining functionality to StorageTransport

- chore(ci): upgrade conformance tests to Node 18
…sport (googleapis#8283)

- Remove Service.ts and common.ts files from handwritten/storage

- Migrate remaining functionality to StorageTransport

- chore(ci): upgrade conformance tests to Node 18
@thiyaguk09 thiyaguk09 requested a review from Dhriti07 May 18, 2026 14:05
Comment thread handwritten/storage/src/storage-transport.ts Outdated
Comment thread handwritten/storage/src/storage-transport.ts Outdated
…sport (googleapis#8283)

- Remove Service.ts and common.ts files from handwritten/storage

- Migrate remaining functionality to StorageTransport

- chore(ci): upgrade conformance tests to Node 18
@thiyaguk09 thiyaguk09 requested a review from Dhriti07 May 19, 2026 10:06
@thiyaguk09 thiyaguk09 force-pushed the fix/multipart-invocation-id branch from eff2c10 to 9822554 Compare May 21, 2026 04:07
@thiyaguk09 thiyaguk09 marked this pull request as draft May 21, 2026 08:17
@thiyaguk09
Copy link
Copy Markdown
Contributor Author

Moved to draft; its required PR is #8235 to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants