ML pipelines: RunInference - OSS Image Object detection, OSS Image Captioning, OSS Image Classification#37186
ML pipelines: RunInference - OSS Image Object detection, OSS Image Captioning, OSS Image Classification#37186Amar3tto wants to merge 30 commits into
Conversation
Summary of ChangesHello @Amar3tto, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances Apache Beam's machine learning capabilities by integrating a new PyTorch-based image object detection pipeline. The pipeline leverages the RunInference transform for efficient batched GPU inference with open-source TorchVision models, processing images from cloud storage and outputting structured detection results to BigQuery. This addition is complemented by a new performance benchmark and corresponding documentation, ensuring that the pipeline's efficiency and resource usage can be consistently monitored and evaluated. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
Amar3tto
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #37186 +/- ##
=============================================
- Coverage 55.28% 36.33% -18.96%
Complexity 1676 1676
=============================================
Files 1067 1069 +2
Lines 167148 167178 +30
Branches 1208 1208
=============================================
- Hits 92415 60737 -31678
- Misses 72551 104259 +31708
Partials 2182 2182
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Amar3tto
changed the title
|
Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
@Abacn Could you please help with review? |
| # limitations under the License. | ||
|
|
||
| name: Inference Python Benchmarks Dataflow | ||
| name: Inference Python Benchmarks Dataflow (1 part) |
There was a problem hiding this comment.
Is there a reason to split these into different workflows? If it is just about minimizing the time it takes to run, could we do one workflow with 2 jobs?
There was a problem hiding this comment.
Or if there is a reason, could you add a comment explaining it? (maybe what I'm suggesting would exhaust resources and we need different cron schedules?)
There was a problem hiding this comment.
Yes, initially my idea was to have two parts with different cron schedules, because for example there is a quota for GPU machines. Also it is hard to read one large .yml file, easy to make a mistake.
There was a problem hiding this comment.
for example there is a quota for GPU machines
Could we solve this problem by just having things run sequentially? (e.g. job2 depends on job1)
Actually just running the steps sequentially would work just as well
Also it is hard to read one large .yml file, easy to make a mistake.
I'm not sure splitting it really helps; it now means that if something is going wrong in benchmarks (the primary reason you'd need to read these files), there are 2 places to look.
There was a problem hiding this comment.
I think we should consolidate back to a single file
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces three new ML inference pipelines for image classification, object detection, and image captioning using PyTorch, along with their corresponding benchmarks and documentation. The pipelines are well-structured and showcase advanced Beam features like RunInference with custom model handlers and stateful DoFns. My review focuses on improving scalability, robustness, and maintainability. I've identified a few key areas for improvement, including a scalability bottleneck in the data loading pipelines, several instances of broad exception handling that could mask errors, some potentially buggy logic, and a few copy-paste errors in the new documentation pages. Overall, this is a valuable contribution, and the suggested changes aim to make these examples more robust and easier to understand.
|
Reminder, please take a look at this pr: @claudevdm @liferoad @shunping |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @jrmccluskey for label python. Available commands:
|
|
Reminder, please take a look at this pr: @jrmccluskey @damccorm |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @shunping for label python. Available commands:
|
|
waiting on author |
Amar3tto
force-pushed
the
oss-image-detection
branch
from
0148ee2 to
7dfdc4a
Compare
| @@ -0,0 +1,41 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
For large PRs like this, please avoid rebasing in between reviews. It breaks GitHub's "Changes since your last review" feature, and forces me to mostly start reviewing from scratch.
| # limitations under the License. | ||
|
|
||
| name: Inference Python Benchmarks Dataflow | ||
| name: Inference Python Benchmarks Dataflow (1 part) |
There was a problem hiding this comment.
for example there is a quota for GPU machines
Could we solve this problem by just having things run sequentially? (e.g. job2 depends on job1)
Actually just running the steps sequentially would work just as well
Also it is hard to read one large .yml file, easy to make a mistake.
I'm not sure splitting it really helps; it now means that if something is going wrong in benchmarks (the primary reason you'd need to read these files), there are 2 places to look.
| # limitations under the License. | ||
|
|
||
| name: Inference Python Benchmarks Dataflow | ||
| name: Inference Python Benchmarks Dataflow (1 part) |
There was a problem hiding this comment.
I think we should consolidate back to a single file
| return int(time.time() * 1000) | ||
|
|
||
|
|
||
| def load_image_from_uri(uri: str) -> bytes: |
There was a problem hiding this comment.
I think we can just inline this, I think separating it out makes things less readable (should be same number of lines, and its unclear what kind of exception we're protecting against)
|
|
||
|
|
||
| def sha1_hex(s: str) -> str: | ||
| import hashlib |
There was a problem hiding this comment.
Can this be a top level import? Then we also don't need a separate function since its a one-liner
| images = [] | ||
| uris = [] | ||
| bytes_list = [] | ||
| for x in batch: | ||
| b = x["image_bytes"] | ||
| bytes_list.append(b) | ||
| uris.append(x.get("uri", "")) | ||
| try: | ||
| images.append(decode_pil(b)) | ||
| except Exception as e: | ||
| # fallback: a blank image (so pipeline keeps going) | ||
| logging.warning("Failed to decode image %s: %s", uris[-1], e) | ||
| images.append(PILImage.new("RGB", (224, 224), color=(0, 0, 0))) |
There was a problem hiding this comment.
Generating this image seems more natural as a preprocessing map function (not part of the ModelHandler)
There was a problem hiding this comment.
Then we wouldn't need to repeat it in both model handlers as well
| return int(time.time() * 1000) | ||
|
|
||
|
|
||
| def load_image_from_uri(uri: str) -> bytes: |
There was a problem hiding this comment.
Same as above - does this need to be a separate function?
| """Produce (image_id, uri) where image_id is stable for dedup and keys.""" | ||
| def process(self, element: str): | ||
| uri = element | ||
| image_id = sha1_hex(uri) |
There was a problem hiding this comment.
Same as above, can we just use the uri as a key?
| yield image_id, uri | ||
|
|
||
|
|
||
| class DedupDoFn(beam.DoFn): |
There was a problem hiding this comment.
Why do we need to dedupe here? Also, can we just use https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.deduplicate.html
| tried = [64, 32, 16] if desired_batch is None else [desired_batch] | ||
|
|
||
| # Device | ||
| device = 'GPU' if known_args.device.upper() == 'GPU' else 'CPU' |
There was a problem hiding this comment.
Isn't the point of this pipeline to use right fitting with resource hints? https://docs.cloud.google.com/dataflow/docs/guides/right-fitting
2 participants
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.