feat(linter/plugins): Token-related `SourceCode` APIs (TS ESLint implementation) #15861

lilnasy · 2025-11-19T09:29:47Z

"Slow but working" implementation for Linter plugins: Token-related SourceCode APIs #14829 (comment)
Uses the parser from @typescript-eslint/typescript-estree for tokens. The source code is fully parsed and everything but the tokens is discarded.
Snapshots are generated in a typescript-eslint project and copied over with only formatting edits.
Direct commits from maintainers welcome.
Work in progress...

Tasks

getTokens
Move token types to src-js/plugins/tokens.ts
Bundling typescript results in unresolved __filename usage. Uncaught ReferenceError: __filname is not defined.
~~Prevent class name mangling by tsdown. Could be deferred.~~ test(linter/plugins): avoid using scope constructor names in test snapshot #15906 (review)
Test regex-divide ambiguous syntax.
Parse as JSX only if source text is JSX. Could be deferred.
Lazy load the parser on demand. Could be deferred.
Test for conformance directly against eslint and typescript-eslint. Could be deferred.

Future work

Decisions

@typescript-eslint/typescript-estree peer-depends on typescript. How should we package it? Currently, typescript is being made an optional dependency.
- overlookmotel (on discord): ideally bundled, direct runtime dependency otherwise.
Deprecated methods are being removed altogether in ESLint 10: getTokenOrCommentBefore, getTokenOrCommentAfter, and isSpaceBetweenTokens. These are surface level deprecations: the functionality was merged with other methods (the includeComments: true option) and plugins can migrate with a one line change. I'm guessing we are targeting fairly modern, actively developed projects. Should we expose them?
- overlookmotel and camc314: yes.

overlookmotel · 2025-11-19T13:16:36Z

Deprecated methods are being removed altogether in ESLint 10: getTokenOrCommentBefore, getTokenOrCommentAfter, and isSpaceBetweenTokens. These are surface level deprecations: the functionality was merged with other methods (the includeComments: true option) and plugins can migrate with a one line change.

Cameron and I had a bit of an argument about exactly this question!

We concluded in the end to keep all the deprecated methods, to maximize compatibility with older plugins, which may take some time to get updated (or in some cases, will never get updated). Our rationale is that most of these methods are just aliases, so no maintenance burden to keep them. The only one which is slightly different from its non-deprecated "brother" is isSpaceBetweenTokens, and that's pretty simple to implement - it just treats JSXText differently (if I remember right).

I'm guessing we are targeting fairly modern, actively developed projects.

That's true, but even actively developed projects may use old unmaintained plugins.

Note: I added the stubs in #15645. That PR also added tests which illustrate the difference in behavior between isSpaceBetween and isSpaceBetweenTokens.

overlookmotel

Possibly premature review, as it's still marked as draft, but I'm keen to get it merged so I thought I'd go through it now.

Apart from the comments below, this looks good to me. Sorry for the volume of comments - most are pretty small details.

Once we're happy with getTokens impl, I think we should merge this, and we can add more methods in separate PRs. No need to do the whole API in a single PR.

A couple of points which we can also leave to follow-ups:

We should ideally lazy-load @typescript-eslint/typescript-estree package only when getTokens is first called.
I assume TS-ESLint's parser also generates a ScopeManager. If it does, we may as well cache it, to avoid running scope analysis again if plugin does sourceCode.getTokens() followed by getting sourceCode.scopeManager.

apps/oxlint/src-js/plugins/tokens.ts

apps/oxlint/src-js/plugins/types.ts

apps/oxlint/tsdown.config.ts

oxfmtrc.jsonc

npm/oxlint/package.json

apps/oxlint/src-js/plugins/tokens.ts

apps/oxlint/src-js/plugins/types.ts

apps/oxlint/test/fixtures/sourceCode_token_methods/files/index.tsx

apps/oxlint/test/fixtures/sourceCode_token_methods/plugin.ts

lilnasy · 2025-11-20T11:28:28Z

I assume TS-ESLint's parser also generates a ScopeManager. If it does, we may as well cache it, to avoid running scope analysis again if plugin does sourceCode.getTokens() followed by getting sourceCode.scopeManager.

@typescript-eslint/parser does. @typescript-eslint/typescript-estree does not.

The former delegates to the latter for parsing and lexing. I decided to use @typescript-eslint/typescript-estree directly to instantiate scope managers and parsers granularly.

…pshot (#15906) Contents of this test snapshot depends on the constructor names of scope class instances. This is a little fragile, as they can change depending on how the code is bundled (see #15861 (comment)). At some point, we'll full minify the bundle, at which point class names will be come random gibberish like `e`, `t`, `aZ`, etc. Avoid this problem by outputting `type` field of scope objects instead.

overlookmotel · 2025-11-20T14:02:49Z

The former delegates to the latter for parsing and lexing. I decided to use @typescript-eslint/typescript-estree directly to instantiate scope managers and parsers granularly.

Ah ha that makes perfect sense. I had misunderstood.

Copilot

Pull Request Overview

This PR implements token-related SourceCode APIs for the oxlint JavaScript plugin system, specifically the getTokens method. The implementation uses @typescript-eslint/typescript-estree to parse source code and extract tokens, providing compatibility with ESLint's token API.

Key changes:

Implements sourceCode.getTokens() method with support for filters, comment inclusion, and before/after token counts
Defines comprehensive TypeScript token types (13 token types including Boolean, Keyword, Punctuator, JSXIdentifier, etc.)
Adds typescript and @typescript-eslint/typescript-estree as runtime dependencies

Reviewed Changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Adds lockfile entries for @typescript-eslint/typescript-estree and its dependencies
npm/oxlint/package.json	Adds typescript as a production dependency for the published package
apps/oxlint/package.json	Moves typescript to dependencies and adds typescript-estree as devDependency
apps/oxlint/src-js/plugins/tokens.ts	Implements getTokens() with binary search and token type definitions
apps/oxlint/src-js/plugins/types.ts	Removes duplicate Token interface, imports from tokens.ts instead
apps/oxlint/src-js/plugins/source_code.ts	Integrates resetTokens() into cleanup logic
apps/oxlint/src-js/index.ts	Exports all token-related types
apps/oxlint/test/tokens.test.ts	Unit tests for getTokens() based on ESLint test cases
apps/oxlint/test/fixtures/sourceCode_token_methods/*	E2E test fixture demonstrating token API usage
apps/oxlint/test/e2e.test.ts	Adds E2E test for token methods

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

apps/oxlint/src-js/plugins/tokens.ts

apps/oxlint/package.json

npm/oxlint/package.json

apps/oxlint/src-js/plugins/tokens.ts

apps/oxlint/test/fixtures/sourceCode_token_methods/files/eslint-test-case.js

apps/oxlint/src-js/plugins/tokens.ts

overlookmotel

Apart from the small comments below, looks good!

overlookmotel · 2025-11-20T22:36:35Z

apps/oxlint/test/tokens.test.ts

+  // TODO: this test is failing because of a difference in program range between `espree` and `@typescript-eslint/typescript-estree`.
+  // Unskip after integrating `espree`.


This is a really weird one. ESLint is changing it's behavior in ESLint 10 to make the range of Program be [0, <length of source text>] (no matter whether there's leading/trailing comments/whitespace or not). TS-ESLint plans to follow suit.

https://eslint.org/blog/2025/10/whats-coming-in-eslint-10.0.0/#updates-to-program-ast-node-range-coverage
typescript-eslint/typescript-eslint#11026 (comment)

Oxc will make the change too, and everyone will be aligned (at last).

So I suggest modifying these tests and unskipping them by either:

const Program = { range: [5, 36] } as Node; (to match current Espree). or

Manually add the extra tokens to the array below, so the tests pass.

Either is fine to do, I think. Just need to make a comment explaining why.

PS Thanks for putting in the unglamorous work of upping the test coverage. Appreciate it.

apps/oxlint/src-js/plugins/tokens.ts

overlookmotel · 2025-11-20T23:03:41Z

apps/oxlint/src-js/plugins/tokens.ts

+  /**
+   * Number of following tokens to additionally return.
+   */
+  afterCount = typeof afterCount === 'number' ? afterCount : 0;


According to the JSDoc comment on this function (which is lifted from ESLint), params are either:

(countOptions: CountOptions | FilterFn) or

(beforeCount?: number, afterCount?: number)

That would mean that afterCount param should be ignored if countOptions is an object or function.

Does ESLint's implementation follow the JSDoc comment, or not actually do what it says it does? I guess we should do whatever ESLint does, but if behavior is contrary to the doc comment, it'd be good to add a comment explaining why.

Or we could open a bug report on ESLint.

Sorry, it really is insanely anal of me to question this!

Sorry, it really is insanely anal of me to question this!

YOU WORK FULL TIME ON A LINTER! It's part of the job description 😭.

Looking into it.

github-actions bot added A-linter Area - Linter A-cli Area - CLI A-linter-plugins Area - Linter JS plugins labels Nov 19, 2025

lilnasy changed the title ~~Feat/linter/plugins/token methods~~ feat(linter/plugins): Token-related SourceCode APIs (TS ESLint implementation) Nov 19, 2025

github-actions bot added the C-enhancement Category - New feature or request label Nov 19, 2025

lilnasy force-pushed the feat/linter/plugins/token-methods branch 2 times, most recently from 3e3a5bb to dd03ef0 Compare November 19, 2025 12:35

overlookmotel reviewed Nov 19, 2025

View reviewed changes

lilnasy force-pushed the feat/linter/plugins/token-methods branch 2 times, most recently from 176a184 to db8abc2 Compare November 20, 2025 06:37

lilnasy mentioned this pull request Nov 20, 2025

style(oxlint/plugins): update oxfmt config to only format plugins in oxlint fixtures #15862

Closed

lilnasy force-pushed the feat/linter/plugins/token-methods branch from db8abc2 to 0d4164e Compare November 20, 2025 09:14

overlookmotel mentioned this pull request Nov 20, 2025

test(linter/plugins): avoid using scope constructor names in test snapshot #15906

Merged

lilnasy force-pushed the feat/linter/plugins/token-methods branch from 0d4164e to bc17b6d Compare November 20, 2025 11:20

lilnasy force-pushed the feat/linter/plugins/token-methods branch from faf3560 to de1f77e Compare November 20, 2025 12:21

lilnasy force-pushed the feat/linter/plugins/token-methods branch 6 times, most recently from e23e179 to 173e5a9 Compare November 20, 2025 20:05

lilnasy marked this pull request as ready for review November 20, 2025 21:03

lilnasy requested a review from camc314 as a code owner November 20, 2025 21:03

Copilot AI review requested due to automatic review settings November 20, 2025 21:03

lilnasy force-pushed the feat/linter/plugins/token-methods branch from 9fb3811 to ab159d9 Compare November 20, 2025 21:04

Copilot started reviewing on behalf of lilnasy November 20, 2025 21:04 View session

Copilot finished reviewing on behalf of lilnasy November 20, 2025 21:07

Copilot AI reviewed Nov 20, 2025

View reviewed changes

overlookmotel reviewed Nov 20, 2025

View reviewed changes

apps/oxlint/src-js/plugins/tokens.ts Outdated Show resolved Hide resolved

overlookmotel reviewed Nov 20, 2025

View reviewed changes

apps/oxlint/src-js/plugins/tokens.ts Outdated Show resolved Hide resolved

overlookmotel reviewed Nov 20, 2025

View reviewed changes

lilnasy added 19 commits November 21, 2025 09:36

implement sourceCode.getTokens

8886f36

format linted file and test snapshot

7b1fe26

make typescript a full dependency for now

9f064e2

add regex field to RegularExpressionToken

9f33fee

add todo for enabling jsx when needed

7399825

add testing todo

dafb91e

tsTokens -> tokens

c88a955

add todo for countOptions and afterCount for getTokens()

27236b3

move token types to tokens.ts

21e01c8

refactor test

9ed179d

add slash lexing corner case

8720761

implement getTokens()

5de8ffb

test includeComments: true

73f380d

vendor eslint tests

0874ca8

skip tests failing due to incompatibility

44488b6

handle count === 0 edge case

ca07b2e

style

abd74ac

reset comments and tokensWithComments arrays

31e30df

move assertIsNonNull to the beginning of the function

e1202dd

lilnasy force-pushed the feat/linter/plugins/token-methods branch from ab159d9 to e1202dd Compare November 21, 2025 04:18

Update output.snap.md

5a08732

		// TODO: this test is failing because of a difference in program range between `espree` and `@typescript-eslint/typescript-estree`.
		// Unskip after integrating `espree`.

Uh oh!

feat(linter/plugins): Token-related SourceCode APIs (TS ESLint implementation) #15861

Are you sure you want to change the base?

feat(linter/plugins): Token-related SourceCode APIs (TS ESLint implementation) #15861

Conversation

lilnasy commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Future work

Decisions

Uh oh!

overlookmotel commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

overlookmotel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lilnasy commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

overlookmotel commented Nov 20, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

overlookmotel left a comment

Choose a reason for hiding this comment

Uh oh!

overlookmotel Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

overlookmotel Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

overlookmotel Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lilnasy Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(linter/plugins): Token-related `SourceCode` APIs (TS ESLint implementation) #15861

feat(linter/plugins): Token-related `SourceCode` APIs (TS ESLint implementation) #15861

lilnasy commented Nov 19, 2025 •

edited

Loading

overlookmotel commented Nov 19, 2025 •

edited

Loading

overlookmotel left a comment •

edited

Loading

lilnasy commented Nov 20, 2025 •

edited

Loading

overlookmotel Nov 20, 2025 •

edited

Loading