chore: replace API Extractor with CodeQL public-API checks#1623
Open
u9g wants to merge 19 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: 83f62bd The changes in this PR will be included in the next version bump. This PR includes changesets to release 33 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
lukasIO
reviewed
May 28, 2026
Contributor
lukasIO
left a comment
lukasIO
left a comment
There was a problem hiding this comment.
Nice. Anything's better than API extractor not working at all for this repo.
Do I understand correctly that this doesn't check for API/method signatures right now?
I think that might be the main thing missing from what API extractor can also check.
Contributor
Author
I've just added api/method signatures tracking, const, type, interface, enum, generics, and class field tracking |
API Extractor's api:check no longer works against the repo's TypeScript version (it cannot represent the "export * as" namespace re-exports used across @livekit/agents and its plugins), so remove the tooling entirely: - drop @microsoft/api-extractor devDep and api:check/api:update scripts from all packages - delete all api-extractor.json configs and api-extractor-shared.json - remove the api:check/api:update turbo pipeline entries - delete the committed plugin API report and its REUSE annotation - drop stale references from CLAUDE.md and the cursor rules
Replace the removed API Extractor with CodeQL-backed api:check/api:update, run from the root via scripts/codeql-api-check.mjs (requires the CodeQL CLI). Two queries in codeql/queries/, each diffed against a committed snapshot so existing surface/debt is tracked and only drift fails the check: - api-surface: names exported from each published package entry point - forgotten-exports: types referenced by the public API but not exported (mirrors API Extractor's ae-forgotten-export)
- Export ~45 types that were referenced by exported declarations but not themselves exported across @livekit/agents and 10 plugins, clearing the forgotten-exports baseline from 100 to 6 (api-surface grows 352 -> 367). - Add the implicit-public-return-types query and make the runner script query-driven so additional checks can be registered.
- Add explicit return types to ~245 exported functions / public methods (return types taken verbatim from the TS-emitted .d.ts), clearing the implicit-public-return-types baseline to 0. - Export the remaining internal types referenced by the public API (forgotten-exports baseline to 0), including the inworld message chain. - Refine implicit-public-return-types to ignore arrow consts that already carry an explicit binding type annotation. - Force-remove the CodeQL DB before each create (stale --overwrite reuse).
Add two snapshot-baselined public-API checks and factor shared logic into codeql/queries/PublicApi.qll: - any-in-public-api: exported functions/public members using `any`/`unknown` in a public signature position (115 baseline entries). - undeclared-dependency-leak: public API referencing a type from a package not in the owning package's dependencies/peerDependencies; reads each package.json via CodeQL's JSON support (7 real findings — avatar plugins re-export pino's Logger without declaring pino). Runner changes (scripts/codeql-api-check.mjs): - run all problem queries in one analyze pass, grouped by rule name - parse CodeQL CSV as whole-file rows (handles quoted commas/newlines) - extract package.json into the database for the dependency check Also route the implicit-return check through the shared inPublishedSource helper, which excludes the private plugins/test package (-4 entries).
The problem queries treated any module-level export as public API, so they flagged internal symbols never re-exported from a package's index.ts — e.g. the avatar plugins' internal log() (pino Logger leak: 7 false positives) and 87 module-only any/unknown annotations. Unify all queries on the entry-point definition already used by api-surface (shared isPackageEntryPoint/isExportedName in PublicApi.qll). This surfaced 4 genuine forgotten exports, now fixed by exporting the types: - baseten, xai: SpeechStream (returned by STT.stream()) - silero: VADOptions (exposed by VAD) Remaining any-in-public-api baseline (33) is intentional: 32 type-safe unknown plus cancelAndWait's Task<any>[] (Task<T> is invariant, so Task<unknown>[] does not type-check at call sites). Also fix a getImportedPath deprecation.
Picked up by rebasing onto main (inworld STT plugin, #1516). No other check drifted — the new plugin introduces no forgotten exports, implicit return types, any/unknown, or undeclared dependencies.
Add an api-check job to build.yml that installs the pinned CodeQL CLI bundle, resolves the query pack, and runs pnpm api:check. Snapshot drift now fails CI, so the committed codeql/*.snapshot.txt can no longer silently fall out of sync (as happened with the inworld STT exports).
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Snapshots the parameter list and return-type annotation of every public callable so signature drift (renames, reorderings, type changes) shows up as a diff. CodeQL's TypeExpr.toString() truncates long types, so the query emits AST locations and the runner slices each annotation out of source to reconstruct full signatures.
Switch from .mjs + JSDoc to a typed .ts file. Bump the CI api-check job to Node 24 so TS type-stripping works natively without tsx.
…ures The signature query now emits a 'generics' slot per public callable (the `<T, E extends Error>` declared on the function/method) and a 'class-generics' row per exported class/interface, which the runner splices into the snapshot. Catches drift in type-parameter constraints and defaults that the previous signature-text-only snapshot missed.
…natures
Extends the signature query to emit rows for:
- type aliases (`type Foo<T> = ...`), incl. RHS source
- enums (`enum Foo { A = 'a', ... }`), members snapshot
- interface/class properties (`interface Foo.bar?: Type`), one per
non-private/computed field
- exported `const NAME: Type` declarations (skips arrow-function consts
already covered by the function branch)
u9g
force-pushed
the
chore/remove-api-extractor
branch
from
2649bef to
0eb4413
Compare
Builds the CodeQL DB once and shares it via an upload-artifact step; each query group (problems, api-surface, api-signatures) then runs on its own runner against the prebuilt DB. The CodeQL CLI tarball is cached across jobs keyed on the bundle version.
The api-check script uses only node built-ins + the codeql CLI, so the pnpm setup + install steps were dead weight (~10s per job, ~40s wall across the 4 codeql jobs).
Each matrix leg currently starts with an empty ~/.codeql/compile-cache, forcing CodeQL to recompile the JS QL standard library from scratch (~55s per query). Restoring the cache between workflow runs cuts ~30s off each leg on the warm path. Cache key invalidates on bundle version or query content change; restore-keys lets a bundle match win even when queries change.
CodeQL's JS QL standard library is ~16 MB of plain source that gets recompiled on every cold runner — ~20s per query, dominating the api-check critical path. Committing the ~1.3 MB compile-cache and pointing codeql at it via `--compilation-cache=codeql/.compile-cache` turns those into instant disk hits. Workflow collapses back to a single job (no DB artifact, no matrix); expected wall-clock ~30-50s in CI, down from ~185s with the split. `pnpm api:update` regenerates both the snapshots and the compile cache, since --compilation-cache is read+write — keep both committed in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
API Extractor no longer works against the repo's TypeScript version (it can't handle the
export * asnamespace re-exports used across@livekit/agentsand its plugins). This removes it and replacespnpm api:check/api:updatewith CodeQL-backed public-API checks.Changes Made
@microsoft/api-extractor(deps, configs,api-extractor-shared.json, turbo pipeline entries, committed report).codeql/queries/(api-surface, forgotten-exports, missing-return-types, any/unknown, undeclared-dependency), run byscripts/codeql-api-check.mjsand diffed against committedcodeql/*.snapshot.txt;api:checkfails only on drift.Pre-Review Checklist
Testing
pnpm api:checkpasses (exit 0); all five snapshots match.pnpm buildpasses.api:check/api:updaterequire the CodeQL CLI onPATH(brew install codeql); CI will need it installed.Additional Notes
SpeechStream/VADOptions) are public-API changes and likely warrant a changeset.any/unknownbaseline (33) is intentional (type-safeunknown+ one invariantTask<any>[]); the baseline blocks new ones.