Add progress.md documenting Java extractor implementation

Comprehensive documentation of the metadata extraction replacement project:
- Problem statement and engineering feedback
- Solution implementation details
- Testing results (1008 options vs 461)
- Architecture decisions and rationale
- New workflow description
- Lessons learned and next steps

Provides complete context for future reference.

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

Author: llewellyn-sl

docs/progress.md

@@ -0,0 +1,392 @@
+# CLI Documentation Automation - Implementation Progress
+
+## Overview
+
+This document tracks the implementation of deterministic CLI metadata extraction for the tower-cli project, replacing brittle Python regex parsing with Java reflection-based extraction.
+
+## Timeline
+
+**Start Date:** 2026-01-22
+**Branch:** `ll-cli-docs-automation-infrastructure`
+**PR:** [#574](https://github.com/seqeralabs/tower-cli/pull/574)
+
+## Problem Statement
+
+### Original Approach Issues
+
+The initial Python-based metadata extractor (`docs/scripts/extract-cli-metadata.py`) had several limitations:
+
+1. **Brittle regex parsing** - String concatenation, multiline annotations, and formatting variations could break extraction
+2. **Incomplete constant resolution** - Constants defined in other files/packages might not resolve
+3. **Mixin limitations** - Platform/provider mixin options weren't automatically captured
+4. **Non-deterministic** - Same input could produce different outputs based on parsing edge cases
+5. **Limited coverage** - Only captured 461 options out of 1000+
+
+### Engineering Feedback
+
+Received feedback from CLI engineer to:
+
+1. **Replace text-based analysis** with deterministic approach
+2. **Use Java reflection** (or Claude) to extract metadata deterministically
+3. **Integrate into PR workflow** as final step before merging (not on release)
+
+## Solution Implemented
+
+### 1. Java Reflection-Based Metadata Extractor
+
+**File:** `src/main/java/io/seqera/tower/cli/utils/metadata/CliMetadataExtractor.java` (551 lines)
+
+**Approach:**
+- Uses picocli's `CommandSpec` API for reflection-based extraction
+- Instantiates the Tower CLI application at runtime
+- Walks the complete command tree recursively
+- Automatically resolves all `@Mixin` annotations
+- Extracts full type information (String, boolean, enums, etc.)
+- Outputs structured JSON to `docs/cli-metadata.json`
+
+**Key Methods:**
+- `extractMetadataAsJson()` - Main entry point, returns processed JSON
+- `buildCommandHierarchy()` - Recursively walks command tree
+- `extractCommandMetadata()` - Extracts data for single command
+- `extractOptionMetadata()` - Extracts option details with types
+- `extractParameterMetadata()` - Extracts positional parameters
+
+**Benefits:**
+- ✅ Deterministic - same input always produces same output
+- ✅ Complete - captures all options including mixins
+- ✅ Type-safe - compile-time checks for API changes
+- ✅ Maintainable - standard Java code in same repo
+- ✅ No external dependencies - uses picocli API already in classpath
+
+### 2. Gradle Task Integration
+
+**File:** `build.gradle`
+
+**Added task:**
+```gradle
+task extractCliMetadata(type: JavaExec) {
+    group = 'documentation'
+    description = 'Extract CLI metadata using Java reflection (deterministic, includes resolved mixins)'
+    classpath = sourceSets.main.runtimeClasspath
+    mainClass = 'io.seqera.tower.cli.utils.metadata.CliMetadataExtractor'
+    args = [file('docs/cli-metadata.json').absolutePath]
+    dependsOn classes
+}
+```
+
+**Usage:**
+```bash
+./gradlew extractCliMetadata
+```
+
+**Output:**
+- `docs/cli-metadata.json` - Processed metadata for docs consumption (992KB)
+- `docs/command-spec.json` - Raw picocli CommandSpec data for debugging (1.2MB, gitignored)
+
+### 3. PR Template with Checklist
+
+**File:** `.github/pull_request_template.md`
+
+**Key section:**
+```markdown
+## Pre-Merge Checklist
+
+- [ ] CLI metadata updated (if CLI commands/options were added/modified)
+  - [ ] Ran `./gradlew extractCliMetadata` to regenerate `docs/cli-metadata.json`
+  - [ ] Committed the updated metadata file
+```
+
+**Purpose:**
+- Enforces metadata updates as final step before merge
+- Makes developers responsible for keeping metadata current
+- Eliminates need for automated workflow on every PR
+
+### 4. Updated GitHub Actions Workflow
+
+**File:** `.github/workflows/trigger-docs-release-update.yml`
+
+**Changes:**
+- Clarified that metadata should already be current from PR checklist
+- Enhanced error messaging if metadata is missing
+- Workflow now just notifies docs repo (metadata already updated)
+- Simplified from "extract and notify" to "verify and notify"
+
+**Trigger:** Still runs on `release.published` event
+
+### 5. Documentation Updates
+
+**docs/README.md** - Complete rewrite
+- Removed all Python script references
+- Documented Java reflection approach
+- Updated workflow description
+- Added troubleshooting section
+- Metrics comparison table
+
+**.claude/skills/enrich-cli-help/SKILL.md**
+- Updated metadata extraction commands
+- Changed `python docs/scripts/extract-cli-metadata.py` → `./gradlew extractCliMetadata`
+- Removed absolute path references
+- Updated working directory to project root
+
+**.claude/README.md**
+- Updated repository structure diagram
+- Replaced Python references throughout
+- Clarified automatic mixin resolution
+- Added metadata update to contributor guide
+
+### 6. Cleanup
+
+**Removed:**
+- `docs/scripts/extract-cli-metadata.py` (replaced by Java)
+- `docs/scripts/extract-cli-examples.py` (out of scope)
+- `docs/cli-examples.json` (out of scope)
+- Empty `docs/scripts/` directory
+
+**Updated .gitignore:**
+- Added `docs/command-spec.json` (raw debug output)
+
+## Testing Results
+
+### Extraction Test (2026-01-22)
+
+```bash
+./gradlew extractCliMetadata
+```
+
+**Output:**
+```
+CLI metadata written to: /Users/llewelyn-van-der-berg/Documents/GitHub/tower-cli/docs/cli-metadata.json
+Total commands: 164
+Total options: 1008
+Total parameters: 12
+Raw command spec written to: /Users/llewelyn-van-der-berg/Documents/GitHub/tower-cli/docs/command-spec.json
+
+BUILD SUCCESSFUL
+```
+
+**Verification:**
+- ✅ File generated: `docs/cli-metadata.json` (992KB)
+- ✅ Valid JSON structure
+- ✅ Complete command hierarchy
+- ✅ All options with resolved mixins
+- ✅ Full type information included
+
+## Metrics Comparison
+
+### Python Regex vs Java Reflection
+
+| Metric | Python (Regex) | Java (Reflection) | Improvement |
+|--------|----------------|-------------------|-------------|
+| **Commands** | 163 | 164 | +1 (+0.6%) |
+| **Options** | 461 | 1008 | **+547 (+118%)** |
+| **Parameters** | 9 | 12 | +3 (+33%) |
+| **Mixin resolution** | Partial (manual) | Automatic | ✅ Complete |
+| **Type information** | Limited | Complete | ✅ Full Java types |
+| **Deterministic** | No (brittle regex) | Yes | ✅ Guaranteed |
+| **Dependencies** | Python 3 | Java (already required) | ✅ No new deps |
+| **Maintenance** | External script | In-repo, type-safe | ✅ Better DX |
+
+### Key Insight
+
+The **118% increase in options** (461 → 1008) reveals that the Python approach was missing more than half of the CLI options! This was primarily due to:
+- Platform/provider mixin classes not being resolved
+- Complex annotation patterns that regex couldn't handle
+- Inheritance patterns that weren't tracked
+
+## New Workflow
+
+### For Developers Making CLI Changes
+
+1. **Develop** - Add/modify CLI commands and options in Java source
+2. **Improve descriptions** - Apply quality standards to `@Option` descriptions
+3. **Extract metadata** - Run `./gradlew extractCliMetadata` before merge
+4. **Commit both** - Java changes + updated `docs/cli-metadata.json`
+5. **PR checklist** - Verify metadata update checkbox is checked
+6. **Merge** - Metadata is now current in master branch
+
+### On Release
+
+1. **Create release** - Tag version and publish
+2. **Workflow triggers** - GitHub Actions runs automatically
+3. **Verify metadata exists** - Checks `docs/cli-metadata.json` in release tag
+4. **Notify docs repo** - Sends repository dispatch event
+5. **Docs repo** - Fetches metadata, generates docs, creates PR
+
+**Result:** Zero manual documentation steps after release
+
+## Architecture Decisions
+
+### Why Java Reflection Over Claude?
+
+**Considered options:**
+1. Java reflection (chosen)
+2. Claude-based extraction
+3. Hybrid (Java + Claude)
+
+**Decision rationale:**
+- Java reflection is **deterministic** (same input = same output)
+- Claude-based would require API calls and careful prompt engineering
+- Claude results may vary between runs
+- Java reflection uses official picocli API (guaranteed to be correct)
+- No external API dependency or rate limits
+- Type-safe and compile-time checked
+- Faster execution (no API latency)
+
+**Trade-offs accepted:**
+- Requires compilation (but build already required)
+- Requires GitHub credentials for tower-java-sdk (already needed)
+- Slightly more complex than script (but more robust)
+
+### Why PR Checklist Over Automated Workflow?
+
+**Considered options:**
+1. PR checklist (chosen)
+2. Automated workflow on every PR
+3. Pre-commit git hook
+
+**Decision rationale:**
+- **PR checklist** gives developers control and visibility
+- Automated workflow would create bot commits on every PR (noise)
+- Many PRs don't change CLI structure (unnecessary workflow runs)
+- Pre-commit hooks require local setup (not enforced)
+- Checklist is simple, explicit, and educational
+
+**Trade-offs accepted:**
+- Relies on developer compliance (but checklist makes it obvious)
+- Not automatically enforced (but PR reviewers can check)
+- Could be forgotten (but template makes it prominent)
+
+## Implementation Commits
+
+1. `6895d301` - Replace Python metadata extractor with Java reflection approach
+   - Added CliMetadataExtractor.java
+   - Added extractCliMetadata Gradle task
+   - Removed Python scripts
+   - Created PR template
+   - Updated workflow and documentation
+
+2. `506c07d5` - Update enrich-cli-help skill to reference Java extractor
+   - Changed metadata extraction commands
+   - Removed absolute paths
+   - Updated working directory references
+
+3. `4538b39a` - Update .claude/README.md to reference Java extractor
+   - Updated repository structure
+   - Replaced Python references
+   - Updated workflow pipeline
+
+4. `bc3ece0f` - Test Java metadata extractor and update cli-metadata.json
+   - Successfully tested extraction
+   - Generated fresh metadata (1008 options)
+   - Added command-spec.json to gitignore
+
+## Current State
+
+### Completed ✅
+
+- ✅ Java reflection-based metadata extractor implemented
+- ✅ Gradle task integrated and tested
+- ✅ PR template with checklist created
+- ✅ GitHub Actions workflow updated
+- ✅ All documentation updated (docs/README.md, .claude/)
+- ✅ Python scripts removed
+- ✅ Fresh metadata generated (1008 options)
+- ✅ PR created: #574
+
+### Branch Status
+
+**Branch:** `ll-cli-docs-automation-infrastructure`
+**Status:** Ready for review
+**PR:** https://github.com/seqeralabs/tower-cli/pull/574
+
+### Files Changed Summary
+
+**Added:**
+- `src/main/java/io/seqera/tower/cli/utils/metadata/CliMetadataExtractor.java` (551 lines)
+- `.github/pull_request_template.md` (new)
+
+**Modified:**
+- `build.gradle` (added extractCliMetadata task)
+- `.github/workflows/trigger-docs-release-update.yml` (clarified comments)
+- `docs/README.md` (complete rewrite)
+- `.claude/skills/enrich-cli-help/SKILL.md` (updated references)
+- `.claude/README.md` (updated references)
+- `.gitignore` (added command-spec.json)
+- `docs/cli-metadata.json` (regenerated with 1008 options)
+
+**Removed:**
+- `docs/scripts/extract-cli-metadata.py`
+- `docs/scripts/extract-cli-examples.py`
+- `docs/cli-examples.json`
+
+## Next Steps
+
+### Immediate (Post-Merge)
+
+1. **Merge PR #574** - Get Java extractor into master
+2. **Update other branch** - `ll-metadata-extractor-and-docs-automation` (help text PR)
+   - Rebase on master to get Java extractor
+   - Regenerate metadata with `./gradlew extractCliMetadata`
+   - Update that PR to include fresh metadata
+
+### Future Enhancements
+
+**Short term:**
+- Add metadata validation tests (ensure all commands have descriptions)
+- Add GitHub Action to verify metadata is current in PRs (non-blocking check)
+- Document common patterns for new command implementations
+
+**Long term:**
+- Changelog generation (compare metadata between versions)
+- Deprecation tracking (flag removed commands/options)
+- Multi-format output (man pages, shell completions)
+- Metadata coverage metrics dashboard
+
+## Lessons Learned
+
+### What Worked Well
+
+1. **Java reflection approach** - Deterministic, complete, maintainable
+2. **PR checklist integration** - Simple, explicit, developer-friendly
+3. **Porting from experimental branch** - Already proven in prototype
+4. **Comprehensive documentation** - Clear process for contributors
+
+### Challenges Encountered
+
+1. **Build requires GitHub credentials** - Need tower-java-sdk access
+   - Solution: Documented in troubleshooting section
+2. **Multiple options discovered** - Initially considered Claude-based approach
+   - Solution: Engineering feedback helped prioritize Java reflection
+3. **Outdated skill references** - Claude skills had Python script paths
+   - Solution: Systematically updated all references
+
+### Process Improvements
+
+1. **Always test before committing** - Verified extractor works before PR
+2. **Update all documentation** - Ensured no stale references remain
+3. **Clear commit messages** - Each commit explains what and why
+4. **Comparison metrics** - Showed concrete improvement (118% more options)
+
+## References
+
+### Implementation Files
+- Java extractor: `src/main/java/io/seqera/tower/cli/utils/metadata/CliMetadataExtractor.java`
+- Gradle task: `build.gradle` (extractCliMetadata)
+- PR template: `.github/pull_request_template.md`
+- Workflow: `.github/workflows/trigger-docs-release-update.yml`
+
+### Documentation
+- Process guide: `docs/README.md`
+- Claude skill: `.claude/skills/enrich-cli-help/SKILL.md`
+- Contributor guide: `.claude/README.md`
+
+### External
+- PR #574: https://github.com/seqeralabs/tower-cli/pull/574
+- Picocli docs: https://picocli.info/
+- Original conversation: Downloaded text file with experimental branch discussion
+
+---
+
+**Last Updated:** 2026-01-22
+**Status:** Implementation complete, PR ready for review
+**Contributors:** Llewelyn van der Berg, Claude Sonnet 4.5