fix(storage): Invocation ID is not retained on multipart upload retries#8190
fix(storage): Invocation ID is not retained on multipart upload retries#8190thiyaguk09 wants to merge 21 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements persistent invocationId tracking across the upload lifecycle and refactors the StorageTransport.makeRequest method to improve project ID resolution and header management. Feedback focuses on critical issues in the makeRequest refactor, including a bug where the request body is incorrectly assigned to an undefined property, thread-safety risks from unconditionally clearing shared interceptors, and an inconsistent response handler that mutates data and breaks return type expectations.
| const responseHandler = (resp: GaxiosResponse<T>) => { | ||
| // eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
| const data = resp.data as any; | ||
| if (data !== null && typeof data === 'object') { | ||
| data.headers = resp.headers; | ||
| data.status = resp.status; | ||
| return data; | ||
| } | ||
| return resp; | ||
| }; |
There was a problem hiding this comment.
The responseHandler introduces an inconsistency in the return type of makeRequest. If the response data is an object, it returns the data mutated with headers and status. If the response data is not an object (e.g., a string or null), it returns the entire GaxiosResponse object. This violates the expected return type T and will likely cause runtime errors in callers expecting raw data. Additionally, mutating the data object directly is risky as it might be a frozen object or a class instance that doesn't support these extra properties. It is recommended to return resp.data consistently as before.
const responseHandler = (resp: GaxiosResponse<T>) => resp.data;There was a problem hiding this comment.
required for retry operations
There was a problem hiding this comment.
Please elaborate as stated in the above comment.
There was a problem hiding this comment.
To preserve backward compatibility for legacy SDK methods that expect resource metadata objects to carry .headers and .status properties without introducing runtime errors or mutating binary data, we added a strict validation check to ensure these properties are strictly injected into plain JSON response objects and never onto Buffer or Stream payloads.
There was a problem hiding this comment.
I agree with using isPlanObject to avoid mutating Buffer or Stream payloads.
However the issue mentioned above refers to the fallback "return resp;" statement. If the payload is a Buffer or Stream and the check fails, returning the entire GaxiosResponse wrapper instead of the raw data might break backward compatibility for binary downloads, since it previously resolved with just the data.
What do you think about returning data instead in that fallback case to keep the return types consistent?
There was a problem hiding this comment.
Great catch. Simply returning data broke resumable uploads, which rely on resp.headers.location when the API returns an empty string. I've updated the logic to handle all three scenarios correctly:
- Plain objects: Mutate and return
data. - Buffers/Streams (non-plain objects): Return raw
datato protect binary downloads. - Primitives (empty strings/null): Return the full
respwrapper so session URIs can still be read from the headers.
There was a problem hiding this comment.
Do we have any presubmits or checks to prevent such kind of breakages ?
There was a problem hiding this comment.
Yes, our conformance test suite acts as the safety net for this (referenced in draft PR #7917). In fact, it was the conformance tests that caught this exact breakage. When I tested changing the fallback to just return data, the bucketUploadResumable test scenarios immediately failed because they couldn't extract the session URI from the headers. These test suites run during presubmit to prevent regressions like this from merging.
da2ee2d to
b4ee48b
Compare
b4ee48b to
b2e12e7
Compare
Hoists the generation of `persistentInvocationId` to the beginning of the upload process in `Bucket.upload` and `File.save`. This ensures that retried multipart upload attempts reuse the same invocation ID in the `x-goog-api-client` header, rather than generating a new one for each attempt.
5b69a10 to
e232f4e
Compare
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
011f5b5 to
1ae557f
Compare
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
1ae557f to
cc411a0
Compare
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
cc411a0 to
9010041
Compare
…ipart-invocation-id
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
9010041 to
0e8f067
Compare
…ipart-invocation-id
…ion flag to storage transport
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
0e8f067 to
72c17d7
Compare
…ipart-invocation-id
…ransport and skip interceptor test
eff2c10 to
9822554
Compare
72c17d7 to
eacb087
Compare
…ipart-invocation-id
|
Moved to draft; its required PR is #8235 to merge. |
Hoists the generation of
persistentInvocationIdto the beginning of theupload process in
Bucket.uploadandFile.save. This ensures that retried multipart upload attempts reuse the same invocation ID in thex-goog-api-clientheader, rather than generating a new one for each attempt.Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> 🦕