diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md deleted file mode 100644 index 641b31a..0000000 --- a/.github/copilot-instructions.md +++ /dev/null @@ -1,103 +0,0 @@ -# LLVM Prebuilt - AI Coding Agent Instructions - -## Project Overview - -This repository builds custom cross-platform LLVM/Clang distributions and Halide packages for **x86_64** and **aarch64** architectures across **Windows**, **macOS**, **Ubuntu 22.04**, and **Ubuntu 24.04**. The build system uses CMake toolchain files and applies version-specific patches to produce minimal, installable distributions. - -## Core Architecture - -### Build Pipeline Structure -- **Stage 1 (Host Tools)**: Native build of `llvm-tblgen`, `clang-tblgen`, `llvm-config`, and clang-tools-extra generators -- **Stage 2 (Cross-Compilation)**: Target build using host tools and architecture-specific toolchain files -- **Distribution**: Uses `install-distribution` target with custom component lists from [cmake/llvm-distribution.cmake](../cmake/llvm-distribution.cmake) - -### Critical Build Workflow -```bash -# 1. Build host tools (always native architecture) -cmake -G Ninja -S llvm-project/llvm -B llvm-host \ - -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" \ - -DCMAKE_BUILD_TYPE=Release -cmake --build llvm-host --target llvm-tblgen clang-tblgen llvm-config - -# 2. Configure target build with toolchain -cmake -G Ninja -S llvm-project/llvm -B llvm-build \ - -DCMAKE_TOOLCHAIN_FILE=cmake/-.cmake \ - -C cmake/llvm-distribution.cmake \ - -DLLVM_NATIVE_TOOL_DIR=llvm-host/bin - -# 3. Build and install distribution -cmake --build llvm-build -cmake --build llvm-build --target install-distribution -``` - -## Toolchain File System - -Toolchain files in [cmake/](../cmake/) follow a strict **include hierarchy**: -- Platform-specific configs (e.g., `x86_64-ubuntu-22.04.cmake`) include base configs (e.g., `x86_64-linux.cmake`) -- Base configs set `CMAKE_SYSTEM_NAME`, `CMAKE_SYSTEM_PROCESSOR`, cross-compilation flags, and default target triples -- **Always test both Ubuntu versions separately** - they require different library packages (see workflow deps) - -### Key Variables -- `LLVM_NATIVE_TOOL_DIR`: Points to host-built tablegen tools (required for cross-compilation) -- `LLVM_DISTRIBUTION_COMPONENTS`: Defines what gets installed (includes dev headers, libraries, lld components, tools) -- `PACKAGE_VENDOR`: Set to `"awakecoding"` for branding - -## Patch Management - -Patches in [patches/](../patches/) are **version-specific** (18.x vs 20.x) and must be applied before configuration: -- **lld-install-targets**: Adds missing `install-lld-headers` and `install-lld-libraries` targets -- **llvm-name-prefix**: Renames `llc`/`lli`/`opt` to avoid conflicts with system tools -- **force-disable-clang-ast-introspection**: Disables feature causing cross-compilation issues - -**Pattern**: When updating LLVM versions, verify all three patch categories apply cleanly. Test on Linux first (fastest builds). - -## Dependency Handling - -### Linux (Ubuntu) -Cross-compilation requires **arm64 library installation**. Use the [Get-UbuntuPackage.ps1](../scripts/Get-UbuntuPackage.ps1) script to automatically download packages: -```powershell -# Automatically resolves latest package version from packages.ubuntu.com -./scripts/Get-UbuntuPackage.ps1 -PackageName libxml2-dev -Release 22.04 -Architecture arm64 -./scripts/Get-UbuntuPackage.ps1 -PackageName zlib1g-dev -Release 24.04 -Architecture arm64 -``` - -The script: -1. Maps release numbers to codenames (22.04 → jammy, 24.04 → noble) -2. Scrapes packages.ubuntu.com to find the latest .deb download URL -3. Downloads and extracts the package (handles .zst, .xz, .gz compression) -4. Returns the extraction directory path - -**Why this matters**: Hardcoding package filenames (e.g., `libxml2-dev_2.9.13+dfsg-1build1_arm64.deb`) breaks when Ubuntu updates packages. This script eliminates that brittleness. - -### Windows -Uses vcpkg for static dependencies: `zlib:-windows-static-release` and `libxml2:-windows-static-release` - -### macOS -System libraries work directly; specify `CMAKE_OSX_ARCHITECTURES` for universal builds - -## Testing Changes - -**Local testing pattern** (fastest verification): -1. Test on Ubuntu x86_64 first (native build, no cross-compilation complexity) -2. Verify patches apply: `git -C llvm-project apply ../patches/llvm--*.patch` -3. Check distribution components: `cmake --build llvm-build --target help | grep install-` -4. Validate package contents: `tar -tvf clang+llvm-*.tar.xz | grep -E '(bin|lib|include)'` - -## Halide Integration - -The [halide-prebuilt.yml](../.github/workflows/halide-prebuilt.yml) workflow: -- **Depends on llvm-prebuilt artifacts** (downloads from previous workflow run via `llvm_run_id` input) -- Uses downloaded LLVM as `LLVM_DIR` for Halide CMake configuration -- Applies parallel Halide patches (disable autoschedulers, add host tools dir option) - -## Common Pitfalls - -1. **Cross-compilation failures**: Always ensure `LLVM_NATIVE_TOOL_DIR` points to host-built tools -2. **Missing install targets**: Verify lld patches applied if `install-lld-*` targets fail -3. **Library linking errors on Linux aarch64**: Double-check aarch64 library extraction and sudo copy steps -4. **Windows static linking**: Must use vcpkg's `-static-release` variants, not shared libs -5. **macOS deployment target**: Set `MACOSX_DEPLOYMENT_TARGET=10.13` for compatibility - -## File Naming Conventions - -Artifacts follow pattern: `clang+llvm---.tar.xz` (e.g., `clang+llvm-20.1.8-aarch64-ubuntu-22.04.tar.xz`) diff --git a/.github/prompts/plan-migrateLlvm16To20.prompt.md b/.github/prompts/plan-migrateLlvm16To20.prompt.md deleted file mode 100644 index 0cd1dac..0000000 --- a/.github/prompts/plan-migrateLlvm16To20.prompt.md +++ /dev/null @@ -1,84 +0,0 @@ -# Migration Plan: LLVM 16.0.6 → 20.1.0 - -## Overview -Drop LLVM 16.x support and add LLVM 20.1.0 support, maintaining 18.1.8 as the older stable version. - -## 1. Workflow Updates - -### llvm-prebuilt.yml -- [ ] Line 12: Change `version: [ 16.0.6, 18.1.8 ]` → `version: [ 18.1.8, 20.1.0 ]` -- [ ] Lines 15-16: Remove 16.0.6 branch mapping -- [ ] Add: `- version: 20.1.0` / `branch: release/20.x` -- [ ] Lines 104-110: Update patch conditional logic - - Remove: `if ('${{matrix.version}}' -eq '16.0.6')` block - - Add: `elseif ('${{matrix.version}}' -eq '20.1.0')` block for 20.x patches - -### halide-prebuilt.yml -- [ ] Line 18: Change `version: [ 16.0.0, 18.0.0 ]` → `version: [ 18.0.0, 20.0.0 ]` -- [ ] Line 67: Change `$LlvmVersion = "16.0.6"` → `$LlvmVersion = "18.1.8"` or handle dynamically -- [ ] Line 166: Update patch conditional logic - - Remove: `if ('${{matrix.version}}' -eq '16.0.0')` block - - Add: `elseif ('${{matrix.version}}' -eq '20.0.0')` block for 20.x patches - -## 2. Create LLVM 20 Patches - -Need to test if 18.x patches apply cleanly to 20.x, or create new versions: - -### LLVM Patches (copy from 18.x as baseline) -- [ ] `llvm-20-add-lld-install-targets.patch` - - Adds missing `install-lld-headers` and `install-lld-libraries` targets - - Test on LLVM 20.1.0 release branch - -- [ ] `llvm-20-add-llvm-name-prefix-to-llc-lli-opt-tools.patch` - - Renames `llc`/`lli`/`opt` to avoid conflicts with system tools - - Verify tool names haven't changed in 20.x - -- [ ] `llvm-20-force-disable-clang-ast-introspection.patch` - - Disables feature causing cross-compilation issues - - Check if issue still exists in 20.x - -### Halide Patches -- [ ] `halide-20-add-halide-host-tools-dir-cmake-option.patch` -- [ ] `halide-20-disable-autoschedulers.patch` -- [ ] `halide-20-disable-clang-verbose-build-warnings.patch` - -**Note:** Halide version 20.0.0 may not exist yet. Verify Halide release schedule matches LLVM. - -## 3. Delete LLVM 16 Patches - -- [ ] `patches/llvm-16-add-lld-install-targets.patch` -- [ ] `patches/llvm-16-add-llvm-name-prefix-to-llc-lli-opt-tools.patch` -- [ ] `patches/llvm-16-force-disable-clang-ast-introspection.patch` -- [ ] `patches/halide-16-add-halide-host-tools-dir-cmake-option.patch` -- [ ] `patches/halide-16-disable-autoschedulers.patch` -- [ ] `patches/halide-16-disable-clang-verbose-build-warnings.patch` - -## 4. Documentation Updates - -- [ ] `.github/copilot-instructions.md` line 103: Update example from `clang+llvm-16.0.6-...` to `clang+llvm-20.1.0-...` - -## 5. Testing Strategy - -1. **Verify LLVM 20.1.0 branch exists**: Check https://github.com/llvm/llvm-project/tree/release/20.x -2. **Test patch application**: Clone LLVM 20.1.0 and apply 18.x patches first -3. **Build verification**: Test native x86_64 Ubuntu build first (fastest validation) -4. **Cross-compilation test**: Verify aarch64 builds work with new patches -5. **Halide compatibility**: Confirm Halide 20.0.0 release exists or adjust version - -## 6. Potential Issues - -- **LLVM 20.x branch availability**: May need to use `llvmorg-20.1.0` tag instead of release branch -- **Halide 20.0.0**: Halide release cadence may not align with LLVM - may need to use 18.0.0 with LLVM 20.1.0 -- **API changes**: LLVM 20.x may have CMake or API changes requiring patch adjustments -- **Distribution component names**: Verify `LLVM_DISTRIBUTION_COMPONENTS` names unchanged - -## Implementation Order - -1. Create LLVM 20 patches (test on local LLVM 20.1.0 checkout) -2. Update llvm-prebuilt.yml workflow -3. Verify LLVM builds work in CI -4. Create/verify Halide 20 patches (if Halide 20.0.0 exists) -5. Update halide-prebuilt.yml workflow -6. Delete 16.x patches -7. Update documentation -8. Commit and test full CI pipeline diff --git a/.github/workflows/cctools-prebuilt.yml b/.github/workflows/cctools-prebuilt.yml index 145f72d..f93a330 100644 --- a/.github/workflows/cctools-prebuilt.yml +++ b/.github/workflows/cctools-prebuilt.yml @@ -77,7 +77,7 @@ jobs: tar -cJf cctools-${{matrix.arch}}-${{matrix.os}}.tar.xz cctools-${{matrix.arch}}-${{matrix.os}} - name: Upload cctools package - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: cctools-${{matrix.arch}}-${{matrix.os}} path: cctools-${{matrix.arch}}-${{matrix.os}}.tar.xz diff --git a/.github/workflows/github-release.yml b/.github/workflows/github-release.yml index 15362a4..4438b20 100644 --- a/.github/workflows/github-release.yml +++ b/.github/workflows/github-release.yml @@ -5,7 +5,7 @@ on: version: description: 'release version' required: true - default: '2025.1.0' + default: '2026.1.0' llvm_run_id: description: 'llvm workflow run id' default: "latest" diff --git a/.github/workflows/halide-prebuilt.yml b/.github/workflows/halide-prebuilt.yml index 1ef8611..333b437 100644 --- a/.github/workflows/halide-prebuilt.yml +++ b/.github/workflows/halide-prebuilt.yml @@ -36,9 +36,9 @@ jobs: steps: - name: Set up Python - uses: actions/setup-python@v4 + uses: actions/setup-python@v6 with: - python-version: '3.9' + python-version: '3.12' - name: Configure Windows runner if: runner.os == 'Windows' @@ -68,9 +68,9 @@ jobs: shell: pwsh run: | $LlvmVersion = "18.1.8" - $LlvmRunId = '${{ github.event.inputs.run_id }}' + $LlvmRunId = '${{ github.event.inputs.llvm_run_id }}' if ($LlvmRunId -eq 'latest') { - $LlvmRunId = $(gh run list -w 'LLVM prebuilt' --json 'status,databaseId,conclusion') | + $LlvmRunId = $(gh run list -R $Env:GITHUB_REPOSITORY -w 'LLVM prebuilt' --json 'status,databaseId,conclusion') | ConvertFrom-Json | Where-Object { ($_.status -eq 'completed') -and ($_.conclusion -eq 'success') } | Select-Object -First 1 -ExpandProperty databaseId } @@ -152,12 +152,12 @@ jobs: } - name: Clone project - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: path: llvm-prebuilt - name: Clone Halide ${{matrix.version}} - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: repository: halide/Halide ref: v${{matrix.version}} @@ -264,7 +264,7 @@ jobs: } - name: Upload Halide package - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: halide-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}} path: halide-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}}.tar.xz diff --git a/.github/workflows/llvm-prebuilt.yml b/.github/workflows/llvm-prebuilt.yml index fb93f66..e25c06e 100644 --- a/.github/workflows/llvm-prebuilt.yml +++ b/.github/workflows/llvm-prebuilt.yml @@ -2,7 +2,7 @@ name: LLVM prebuilt on: workflow_dispatch jobs: build: - name: LLVM ${{matrix.version}} [${{matrix.arch}}-${{matrix.os}}] ${{matrix.distro}} + name: LLVM ${{matrix.version}} [${{matrix.arch}}-${{matrix.os}}] runs-on: ${{matrix.runner}} strategy: fail-fast: false @@ -41,7 +41,7 @@ jobs: osx_arch: arm64 steps: - name: Clone project - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: path: llvm-prebuilt @@ -49,10 +49,10 @@ jobs: if: runner.os == 'Windows' run: | choco install --no-progress ninja 7zip - git clone https://github.com/microsoft/vcpkg.git + git clone --depth 1 --branch 2025.10.17 https://github.com/microsoft/vcpkg.git cd vcpkg .\bootstrap-vcpkg.bat - $VcpkgRootDir = "$PWD/vcpkg" -Replace '\\','/' + $VcpkgRootDir = "$PWD" -Replace '\\','/' $VcpkgArch = @{'x86_64'='x64';'aarch64'='arm64'}['${{matrix.arch}}'] .\vcpkg install "zlib:${VcpkgArch}-windows-static-release" .\vcpkg install "libxml2:${VcpkgArch}-windows-static-release" @@ -92,7 +92,7 @@ jobs: sudo cp -R "$ZlibPath/usr/lib/aarch64-linux-gnu/" /usr/lib/aarch64-linux-gnu/ - name: Clone LLVM ${{matrix.version}} - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: repository: llvm/llvm-project ref: ${{matrix.branch}} @@ -225,7 +225,7 @@ jobs: cmd.exe /c "7z a -ttar -snl -so clang+llvm-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}}.tar clang+llvm-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}} | 7z a -si clang+llvm-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}}.tar.xz" - name: Upload LLVM package - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: clang+llvm-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}} path: clang+llvm-${{matrix.version}}-${{matrix.arch}}-${{matrix.os}}.tar.xz diff --git a/.github/workflows/msvc-bundle.yml b/.github/workflows/msvc-bundle.yml index e4d4d72..5b59831 100644 --- a/.github/workflows/msvc-bundle.yml +++ b/.github/workflows/msvc-bundle.yml @@ -43,13 +43,13 @@ jobs: cmd.exe /c "7z a -ttar -snl -so winsdk.tar winsdk | 7z a -si winsdk.tar.xz" - name: Upload vctools package - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: vctools path: vctools.tar.xz - name: Upload winsdk package - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: winsdk path: winsdk.tar.xz diff --git a/.github/workflows/xcode-bundle.yml b/.github/workflows/xcode-bundle.yml index bc30ace..0d7436a 100644 --- a/.github/workflows/xcode-bundle.yml +++ b/.github/workflows/xcode-bundle.yml @@ -46,7 +46,7 @@ jobs: tar -cJf xcode-bundle.tar.xz ./xcode-bundle - name: Upload xcode-bundle package - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: xcode-bundle path: xcode-bundle.tar.xz diff --git a/.gitignore b/.gitignore index 723ef36..9dd88ec 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,31 @@ -.idea \ No newline at end of file +.idea/ + +# Upstream source checkouts and build trees +llvm-project/ +llvm-host/ +llvm-build/ +llvm-install/ +llvm-host-tools/ +halide/ +halide-host/ +halide-build/ +halide-install/ +xar/ +apple-libtapi/ +apple-libdispatch/ +cctools-port/ + +# Packaged artifacts and extracted bundles +clang+llvm-*/ +halide-*/ +cctools-*/ +xcode-bundle/ +vctools/ +winsdk/ +*.tar +*.tar.xz + +# Ubuntu package extraction output +*.deb +libxml2-dev-*/ +zlib1g-dev-*/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..6ce191e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,154 @@ +# AGENTS.md + +## Project Scope + +This repository builds prebuilt LLVM/Clang, Halide, cctools, MSVC, and Xcode support bundles for release artifacts. Most meaningful behavior lives in GitHub Actions workflows, CMake toolchain/cache files, and version-specific patches applied to external upstream checkouts. + +Treat this as a packaging and cross-compilation repository. Keep changes small, reproducible in CI, and aligned with the existing matrix of: + +- Architectures: `x86_64`, `aarch64` +- Platforms: `windows`, `macos`, `ubuntu-22.04`, `ubuntu-24.04` +- LLVM versions currently built by CI: `18.1.8`, `20.1.8` +- Halide versions currently built by CI: `18.0.0`, `19.0.0` + +## Repository Layout + +- `.github/workflows/llvm-prebuilt.yml` builds and packages LLVM/Clang distributions. +- `.github/workflows/halide-prebuilt.yml` builds Halide packages using LLVM artifacts from a previous LLVM workflow run. +- `.github/workflows/cctools-prebuilt.yml` builds Linux cctools artifacts. +- `.github/workflows/msvc-bundle.yml` and `.github/workflows/xcode-bundle.yml` package platform SDK/toolchain support files. +- `.github/workflows/github-release.yml` downloads workflow artifacts, creates checksums, and publishes a GitHub release. +- `cmake/*.cmake` contains target toolchain files plus `llvm-distribution.cmake`, the LLVM initial cache that defines the installable distribution surface. +- `patches/*.patch` contains version-specific patches applied to upstream LLVM or Halide sources after checkout. +- `scripts/Get-UbuntuPackage.ps1` resolves, downloads, and extracts Ubuntu packages for Linux aarch64 cross-compilation. + +## Build Model + +LLVM builds are two-stage: + +1. Build native host tools from the upstream LLVM checkout with Ninja. +2. Configure the target LLVM build with the matching `cmake/-.cmake` toolchain file and `-C cmake/llvm-distribution.cmake`. + +Important variables exported by CI and consumed by `llvm-distribution.cmake` include: + +- `LLVM_NATIVE_TOOL_DIR` +- `LLVM_TABLEGEN` +- `CLANG_TABLEGEN` +- `LLVM_CONFIG_PATH` +- `LLVM_VERSION` +- `CMAKE_INSTALL_PREFIX` + +Do not remove the host-tools phase when changing cross-compilation behavior. Cross builds depend on native tablegen/config/helper tools. + +## Local Verification Commands + +Prefer PowerShell-compatible commands because the workflows use `pwsh` heavily across platforms. + +Patch application smoke test: + +```powershell +git -C llvm-project apply ../llvm-prebuilt/patches/llvm-20-add-lld-install-targets.patch +git -C llvm-project apply ../llvm-prebuilt/patches/llvm-20-add-llvm-name-prefix-to-llc-lli-opt-tools.patch +``` + +LLVM host tools: + +```powershell +cmake -G Ninja -S llvm-project/llvm -B llvm-host ` + -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" ` + -DCMAKE_BUILD_TYPE=Release -Wno-dev +cmake --build llvm-host --target llvm-tblgen clang-tblgen llvm-config +``` + +LLVM target configuration: + +```powershell +cmake -G Ninja -S llvm-project/llvm -B llvm-build ` + -DCMAKE_INSTALL_PREFIX=llvm-install ` + -DCMAKE_TOOLCHAIN_FILE="$PWD/llvm-prebuilt/cmake/x86_64-ubuntu-24.04.cmake" ` + -C "$PWD/llvm-prebuilt/cmake/llvm-distribution.cmake" -Wno-dev +``` + +LLVM package build/install targets: + +```powershell +cmake --build llvm-build +cmake --build llvm-build --target install-distribution +``` + +When validating a broad change, test Ubuntu x86_64 first because it is the simplest build path, then test aarch64 cross-compilation and both Ubuntu releases separately. + +## Toolchain File Rules + +- Keep platform leaf files small. Ubuntu files should include the matching base Linux file. +- Base Linux files set `CMAKE_SYSTEM_NAME`, `CMAKE_SYSTEM_PROCESSOR`, cross-compilation state, compilers, target architecture, and target triples. +- Windows toolchains set `CMAKE_SYSTEM_NAME` and processor only; CI supplies MSVC target environments with `ilammy/msvc-dev-cmd`. +- macOS toolchains set `CMAKE_SYSTEM_NAME Darwin` and `CMAKE_OSX_ARCHITECTURES`; CI sets `MACOSX_DEPLOYMENT_TARGET`. +- Preserve the `aarch64`/`arm64` distinction: artifact and matrix names use `aarch64`; Apple CMake architecture values use `arm64`; Windows MSVC target architecture values use `arm64`. + +## LLVM Distribution Rules + +`cmake/llvm-distribution.cmake` is the distribution contract. Be careful when changing: + +- `LLVM_ENABLE_PROJECTS` +- `LLVM_TARGETS_TO_BUILD` +- `LLVM_DISTRIBUTION_COMPONENTS` +- `LLVM_DEVELOPMENT_COMPONENTS` +- tool utility/component lists + +The lld development components depend on the `llvm-*-add-lld-install-targets.patch` patch. If lld install components fail, verify the matching lld patch still applies and still creates the expected install targets before changing the component list. + +The `llc`, `lli`, and `opt` tools are patched to install as `llvm-llc`, `llvm-lli`, and `llvm-opt` to avoid system tool conflicts. Account for that naming in tests, package inspections, and docs. + +## Patch Management + +- Patches are version-specific. Add new patch files for new upstream major versions instead of silently reusing an older filename. +- Apply patches after checking out upstream `llvm/llvm-project` or `halide/Halide`, before CMake configuration. +- Keep LLVM patch conditionals in `.github/workflows/llvm-prebuilt.yml` in sync with files under `patches/llvm-*`. +- Keep Halide patch conditionals in `.github/workflows/halide-prebuilt.yml` in sync with files under `patches/halide-*`. +- When updating LLVM, verify all patch categories: lld install targets, LLVM tool name prefixes, and any cross-compilation feature disables still needed for that version. +- When updating Halide, verify all patch categories: host tools directory option, autoscheduler disablement, and verbose warning suppression. + +## Linux Dependency Handling + +Linux aarch64 cross-compilation needs target `arm64` libraries. Do not hardcode Ubuntu `.deb` filenames in workflows. Use `scripts/Get-UbuntuPackage.ps1` so package filenames are resolved from `packages.ubuntu.com`. + +Current workflow dependencies include native `libxml2-dev` and `zlib1g-dev`, plus extracted arm64 `libxml2-dev` and `zlib1g-dev` copied into `/usr/lib/aarch64-linux-gnu/`. + +The helper maps Ubuntu release numbers to codenames, currently including `22.04` -> `jammy` and `24.04` -> `noble`. If adding Ubuntu releases, update this mapping and test both package resolution and extraction formats (`.zst`, `.xz`, `.gz`). + +## Windows and macOS Notes + +Windows LLVM builds use vcpkg static-release packages: + +- `zlib:-windows-static-release` +- `libxml2:-windows-static-release` + +Keep Windows package paths aligned with vcpkg's installed package directory names and use `7z` packaging as in CI. + +macOS LLVM builds use the system SDK/toolchain and set `MACOSX_DEPLOYMENT_TARGET=10.13`. Halide uses `10.13` for 18.x and `10.15` for 19.x because of `std::filesystem` requirements. + +## Artifact Naming + +Keep artifact names stable; downstream workflows and release publishing depend on them. + +- LLVM: `clang+llvm---.tar.xz` +- Halide: `halide---.tar.xz` +- cctools: `cctools--.tar.xz` +- SDK/toolchain bundles: `vctools.tar.xz`, `winsdk.tar.xz`, `xcode-bundle.tar.xz` + +## Release Workflow + +`.github/workflows/github-release.yml` expects successful artifact-producing workflow runs. It accepts `llvm_run_id`, `halide_run_id`, and `cctools_run_id`, each supporting `latest`, plus `dry-run` and `draft-release` controls. + +When changing artifact names, matrix dimensions, or workflow names, update release download logic at the same time. + +## Coding and Review Guidance + +- Favor CI-compatible PowerShell for scripts and workflow snippets. +- Keep CMake cache/toolchain changes explicit and minimal. +- Avoid broad refactors of workflow steps unless required by the requested behavior. +- Do not check in generated upstream source trees or build directories such as `llvm-project`, `llvm-host`, `llvm-build`, `halide`, `halide-host`, or package extraction folders. +- Preserve unrelated user changes in the working tree. +- Before finishing changes that affect builds, run the narrowest practical validation: patch-apply checks, CMake configure, target listing, or package-content inspection. +- If a full LLVM or Halide build is impractical locally, state what was validated and what still requires CI. \ No newline at end of file diff --git a/README.md b/README.md index c90d1b8..6d1e4e0 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,141 @@ -# LLVM prebuilt +# LLVM Prebuilt -This is a repository to create prebuilt clang+llvm distributions for major operating systems and processor architectures. It can be used to create your own custom builds, or you can download the [prebuilt packages](https://github.com/awakecoding/llvm-prebuilt/releases). +This repository builds custom prebuilt toolchain packages for LLVM/Clang, Halide, Linux cctools, and platform SDK/toolchain support bundles. It is intended for reproducible GitHub Actions builds across the major desktop operating systems and CPU architectures used by downstream native projects. -## clang-llvm +You can download published packages from the [GitHub releases page](https://github.com/awakecoding/llvm-prebuilt/releases), or run the workflows in this repository to produce your own artifacts. -Arch | Windows | macOS | Ubuntu 22.04 | Ubuntu 24.04 | -----------------------|--------------------|--------------------|--------------------|--------------------| -x86_64 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | -aarch64 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | +## Packages -## ctools +### LLVM/Clang -Arch | Windows | macOS | Ubuntu 22.04 | Ubuntu 24.04 | -----------------------|--------------------|--------------------|--------------------|--------------------| -x86_64 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | -aarch64 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | +The LLVM workflow builds minimal installable Clang/LLVM distributions with Clang, LLD, selected LLVM tools, development headers, CMake exports, zlib, and libxml2 support. -## Halide +Currently built LLVM versions: `18.1.8`, `20.1.8`. -Arch | Windows | macOS | Ubuntu 22.04 | Ubuntu 24.04 | -----------------------|--------------------|--------------------|--------------------|--------------------| -x86_64 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | -aarch64 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | +| Arch | Windows | macOS | Ubuntu 22.04 | Ubuntu 24.04 | +|---------|---------|-------|--------------|--------------| +| x86_64 | Yes | Yes | Yes | Yes | +| aarch64 | Yes | Yes | Yes | Yes | + +Artifacts are named: + +```text +clang+llvm---.tar.xz +``` + +### Halide + +The Halide workflow builds Halide packages against LLVM artifacts produced by the LLVM workflow. It first installs a host LLVM package, then installs a target LLVM package when cross-compiling for `aarch64`. + +Currently built Halide versions: `18.0.0`, `19.0.0`. + +| Arch | Windows | macOS | Ubuntu 22.04 | Ubuntu 24.04 | +|---------|---------|-------|--------------|--------------| +| x86_64 | Yes | Yes | Yes | Yes | +| aarch64 | Yes | Yes | Yes | Yes | + +Artifacts are named: + +```text +halide---.tar.xz +``` + +### cctools + +The cctools workflow builds Linux packages containing the Apple cctools port and supporting libraries. + +| Arch | Windows | macOS | Ubuntu 22.04 | Ubuntu 24.04 | +|---------|---------|-------|--------------|--------------| +| x86_64 | No | No | Yes | Yes | +| aarch64 | No | No | Yes | Yes | + +Artifacts are named: + +```text +cctools--.tar.xz +``` + +### Platform Bundles + +Additional workflows package host platform support files: + +| Workflow | Artifact | +|----------|----------| +| MSVC bundle | `vctools.tar.xz`, `winsdk.tar.xz` | +| Xcode bundle | `xcode-bundle.tar.xz` | + +## Repository Layout + +| Path | Purpose | +|------|---------| +| `.github/workflows/llvm-prebuilt.yml` | Builds LLVM/Clang packages. | +| `.github/workflows/halide-prebuilt.yml` | Builds Halide packages from LLVM workflow artifacts. | +| `.github/workflows/cctools-prebuilt.yml` | Builds Linux cctools packages. | +| `.github/workflows/github-release.yml` | Downloads workflow artifacts, creates checksums, and publishes a release. | +| `cmake/*.cmake` | LLVM and Halide toolchain files plus LLVM distribution cache settings. | +| `patches/*.patch` | Version-specific LLVM and Halide patches applied before configuration. | +| `scripts/Get-UbuntuPackage.ps1` | Resolves, downloads, and extracts Ubuntu packages for Linux cross-compilation. | + +## Build Overview + +LLVM builds use a two-stage flow: + +1. Build native host tools from `llvm/llvm-project`, including `llvm-tblgen`, `clang-tblgen`, and `llvm-config`. +2. Configure and build the target distribution with the matching `cmake/-.cmake` toolchain file and `cmake/llvm-distribution.cmake` initial cache. + +The distribution surface is controlled by `cmake/llvm-distribution.cmake`. It enables Clang, clang-tools-extra, LLVM, LLD, the selected target backends, development headers/libraries, CMake exports, and selected toolchain utilities. + +Halide builds follow a similar host/target split. Host tools such as `build_halide_h`, `binary2cpp`, and `regexp_replace` are built first, then reused through the `HALIDE_HOST_TOOLS_DIR` CMake option added by the repository patches. + +## Running the Workflows + +All major package workflows are manually triggered with `workflow_dispatch`. + +Recommended order: + +1. Run `LLVM prebuilt`. +2. Run `halide prebuilt` with the LLVM workflow run ID, or use `latest`. +3. Run `cctools prebuilt` if Linux cctools artifacts are needed. +4. Run `GitHub Release` with the desired workflow run IDs to collect artifacts and publish a release. + +The release workflow supports `dry-run` and `draft-release` inputs. Keep `dry-run` enabled while checking artifact collection and checksums. + +## Local Development Notes + +The workflows use PowerShell heavily, including on Linux runners, so PowerShell-compatible commands are preferred when updating scripts or documentation. + +For Linux `aarch64` cross-compilation, do not hardcode Ubuntu `.deb` filenames. Use `scripts/Get-UbuntuPackage.ps1`, which maps Ubuntu releases to codenames, resolves current package filenames through `packages.ubuntu.com`, downloads the package, and extracts the data archive. + +Patch files are version-specific. When adding or updating upstream LLVM or Halide versions, keep workflow conditionals and patch filenames in sync, then verify that patches apply cleanly before testing the build. + +Generated upstream source trees and build directories should remain untracked. Common local directories include `llvm-project`, `llvm-host`, `llvm-build`, `halide`, `halide-host`, and `halide-build`. + +## Example LLVM Configure Commands + +Build native host tools: + +```powershell +cmake -G Ninja -S llvm-project/llvm -B llvm-host ` + -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" ` + -DCMAKE_BUILD_TYPE=Release -Wno-dev + +cmake --build llvm-host --target llvm-tblgen clang-tblgen llvm-config +``` + +Configure a target LLVM build: + +```powershell +cmake -G Ninja -S llvm-project/llvm -B llvm-build ` + -DCMAKE_INSTALL_PREFIX=llvm-install ` + -DCMAKE_TOOLCHAIN_FILE="$PWD/cmake/x86_64-ubuntu-24.04.cmake" ` + -C "$PWD/cmake/llvm-distribution.cmake" -Wno-dev +``` + +Build and install the distribution: + +```powershell +cmake --build llvm-build +cmake --build llvm-build --target install-distribution +``` + +Full local builds are large. For broad changes, start with an Ubuntu `x86_64` LLVM build, then validate `aarch64` cross-compilation and the Ubuntu release variants separately.