文法の更新

How to bump a tree-sitter grammar, for contributors maintaining the grammar pins. big-code-analysis parses each language with a tree-sitter grammar. Grammars change over time and carry bugs of their own, so they need updating now and then.

The update scripts are bash, so grammars can be regenerated natively only on Linux and macOS. On Windows, run the scripts under Windows Subsystem for Linux (WSL).

Two kinds of grammar are in use: third-party grammars published on crates.io and maintained by external developers, and internal grammars defined inside the project for the variants of some languages used in Firefox. Each has its own procedure below.

サードパーティの文法

Update the grammar version in Cargo.toml and enums/Cargo.toml. For example, for the tree-sitter-java grammar:

tree-sitter-java = "=x.xx.x"

where x represents a digit. The pin must be exact (=X.Y.Z); make check-excluded-manifests rejects a caret range. The same version also appears as the last field of the language’s mk_langs! tuple in big-code-analysis-ast/src/langs.rs, and the grammar_version_matches_cargo_toml_pin test there fails until the two agree.

Run ./recreate-grammars.sh to regenerate every grammar’s enum:

./recreate-grammars.sh

Once the script finishes, fix any failed tests and problems the grammar changes introduced. make enums-codegen-drift (part of make pre-commit) re-runs the codegen into a temporary directory and fails if the checked-in language_*.rs or c_langs_macros/*.rs files differ from it.

Commit your changes and open a pull request.

内部の文法

Update the version of tree-sitter-cli in the package.json file of the internal grammar, then refresh its committed package-lock.json with npm install --package-lock-only --ignore-scripts in the same directory and commit both files together. The regen scripts install with npm ci, which fails loudly when the lockfile is missing or out of sync with package.json. This keeps every regen hash-verified and byte-reproducible (OpenSSF Scorecard Pinned-Dependencies).

The five vendored grammars publish under the bca-tree-sitter-* namespace (see RELEASING.md for the rename rationale), but consumer call sites still reference them as tree-sitter-<lang> via Cargo’s package = ... alias. A grammar refresh does not bump the leaf’s version on its own: every crate in this repository shares one workspace-wide version, and bumping the leaves out of step with the parent is not allowed (see the “Lockstep version policy” in RELEASING.md). Regenerate the parser tables, accept the resulting test-snapshot drift, and ship the change under the current version. The next workspace release picks up the new grammars at whatever shared version the next tag declares.

再生成に加えて tree-sitter の「ランタイム」依存関係の更新も必要な場合は、リーフの Cargo.toml 内の dev-dependency 行を更新します。

[dev-dependencies]
tree-sitter = "=x.x.x"

Leave [package] name = "bca-tree-sitter-<lang>", [package] version, and [lib] name = "tree_sitter_<lang>" untouched; the rename trick in [lib] is what keeps Rust import paths stable, and the version line is managed by the lockstep bump at release time.

Then run the script for the grammar, which regenerates every file.

For tree-sitter-ccomment and tree-sitter-preproc run ./generate-grammars/generate-grammar.sh followed by the name of the grammar, for example:

./generate-grammars/generate-grammar.sh tree-sitter-ccomment

tree-sitter-mozcpp and tree-sitter-mozjs have their own scripts.

tree-sitter-mozcpp の場合は、次を実行します

./generate-grammars/generate-mozcpp.sh

tree-sitter-mozjs の場合は、次を実行します

./generate-grammars/generate-mozjs.sh

Both of those crates carry a notification-only marker dependency in their Cargo.toml (tree-sitter-cpp for mozcpp, tree-sitter-javascript for mozjs) that Cargo never compiles. make grammar-marker-sync compares each marker against .grammar-marker-baseline.toml, so a Dependabot bump of the marker without a regen fails the gate. After a verified regen, refresh the baseline in the same commit with ./utils/check-grammar-marker-sync.py --update.

5 番目のベンダリング文法である tree-sitter-tcl には再生成スクリプトがありません。事前生成されたパーサーソースのみをベンダリングしている（grammar.js がない）ため、更新する際はローカルで tree-sitter generate を実行するのではなく、アップストリームプロジェクトから生成済みの src/ を再ベンダリングします。

Once the script finishes, fix any failed tests and problems the grammar changes introduced.

Commit your changes and open a pull request.