Classify the panel's code column with real token types

The panel guessed that any capitalised identifier was a type, so
`Profiler.Frames.Count` came out as three type-coloured names where the
editor shows a class and two members. The semantic overlay already fetches
per-file tokens, so feed the same data to the webview.

- codeSpans() returns the server's token spans per referenced line in the
  row's own trimmed coordinates; rows carry them and the webview colours from
  them, mapping token type names to its palette. Fields, properties, events
  and methods share the member colour, as they do in the stock themes.
- The regex tokenizer stays as the fallback for servers that serve no
  semantic tokens, and coloredReferences.semanticTokens now gates both views
  rather than just the editor.

Colours are still the approximated Dark+/Light+ palette — a webview is not
given the theme's token colours — so only the editor view can be theme-exact.
The README says so instead of promising this would fix it.

Two things the new tests establish, both assumptions the code was already
making: all files in a result share the origin's legend, so decoding tokens
from every file against one legend is sound; and all 237 relocated tokens in
the Profiler search report the same type name as their source token.

Also relaxes an over-specific assertion: DotRush calls Profiler.Frames a
field, not a property. Either way it is a member, which is what matters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -47,9 +47,10 @@ and no compilation behind it. So the tokens are fetched for each *source* file i
 results-document coordinates — the same line and column mapping the extension already uses to place the
 highlights. The colouring you get is the server's own answer, moved, rather than a guess.
 
-This happens after the results are on screen and only ever changes colors, never text, so nothing waits on it.
-Set `coloredReferences.semanticTokens` to `false` for TextMate-only coloring. Results spanning more than 40
-files skip it.
+In the editor view this happens after the results are on screen and only ever changes colours, never text, so
+nothing waits on it. The panel gets the same token *types* but paints them from its own palette, since a
+webview is not given the theme’s token colours. Set `coloredReferences.semanticTokens` to `false` for
+grammar-only colouring; results spanning more than 40 files skip it.
 
 ## Settings
 
@@ -57,8 +58,8 @@ files skip it.
 - `coloredReferences.panelLocation` — where the panel opens: `bottom` (default, docked next to Terminal /
   Problems and draggable to a side bar), `beside` (editor group to the side), or `below` (editor group
   underneath, so the table is wide and short)
-- `coloredReferences.semanticTokens` — color the editor view with semantic tokens fetched from the source
-  files (default `true`)
+- `coloredReferences.semanticTokens` — colour identifiers by what the language server says they are, in
+  both views (default `true`)
 - `coloredReferences.openBeside` — open results beside the current editor (default `true`)
 - `coloredReferences.showProject` — show the containing project in file headers (default `true`)
 - `coloredReferences.reuseTab` — reuse one results tab/panel instead of opening a new one per search (default `true`)
@@ -87,9 +88,10 @@ the expected results stay deterministic.
 ## Known limitations
 
 - The panel's code column cannot use your theme's token colors: webviews are not given them as CSS variables.
-  It classifies code with a small tokenizer of its own and paints it with approximated Dark+/Light+ hues, so
-  it will not match your theme exactly. Semantic tokens do not fix this — they supply the *classification*
-  ("this is a class"), never the colors — so only the editor view can be theme-exact.
+  *What* each token is comes from the language server, so `Profiler.Frames.Count` is correctly a class then two
+  properties — but the hues are approximated Dark+/Light+ values, so a custom theme will not match exactly.
+  Only the editor view can be theme-exact. Without a server that serves semantic tokens the panel falls back to
+  a small regex tokenizer, which does guess that any capitalised identifier is a type.
 - Read/write kind is not part of the reference request, so it is derived two ways. Every reference is first
   classified from the surrounding text (assignment and compound-assignment operators, `++`/`--`, `ref`/`out`
   arguments), then `textDocument/documentHighlight` is asked per file and its `Read`/`Write` kinds override
@@ -104,8 +106,7 @@ the expected results stay deterministic.
 
 1. ~~Webview panel with resizable, sortable columns~~ — done; missing: virtualized rendering for very large
    result sets, and remembering column layout per workspace rather than per panel
-2. ~~Semantic token overlay~~ — done for the editor view; the panel still uses its own tokenizer
+2. ~~Semantic token overlay~~ — done in both views
 3. Filter by project / exclude tests
 4. ~~Read/write kind~~ — done in both views
-5. Feed the panel's code column the real token types instead of its regex tokenizer (colors would still be
-   the approximated palette, but the classification would be right)
+

media/panel.css

@@ -277,6 +277,7 @@ body.resizing {
 .tok-type    { color: var(--cr-type); }
 .tok-call    { color: var(--cr-call); }
 .tok-punct   { color: var(--cr-fg-default); }
+.tok-variable{ color: var(--cr-variable); }
 
 .hit {
     background: var(--vscode-editor-findMatchHighlightBackground);

media/panel.js

@@ -64,25 +64,44 @@
     // -----------------------------------------------------------------------
 
     // Token colours are not exposed to webviews as CSS variables, so approximate the
-    // stock Dark+/Light+ hues. Roadmap item 2 (semantic tokens) can replace this.
+    // stock Dark+/Light+ hues. What each token *is* comes from the language server
+    // (see TOKEN_CLASS); only the hues are ours, so this cannot match a custom theme.
     const PALETTES = {
         dark: {
             'fg-default': '#d4d4d4', comment: '#6a9955', string: '#ce9178',
             keyword: '#569cd6', control: '#c586c0', number: '#b5cea8',
-            type: '#4ec9b0', call: '#dcdcaa',
+            type: '#4ec9b0', call: '#dcdcaa', variable: '#9cdcfe',
         },
         light: {
             'fg-default': '#000000', comment: '#008000', string: '#a31515',
             keyword: '#0000ff', control: '#af00db', number: '#098658',
-            type: '#267f99', call: '#795e26',
+            type: '#267f99', call: '#795e26', variable: '#001080',
         },
         contrast: {
             'fg-default': '#ffffff', comment: '#7ca668', string: '#ce9178',
             keyword: '#569cd6', control: '#c586c0', number: '#b5cea8',
-            type: '#4ec9b0', call: '#dcdcaa',
+            type: '#4ec9b0', call: '#dcdcaa', variable: '#9cdcfe',
         },
     };
 
+    // Language-server token type -> palette class. Types the server reports but we have
+    // no distinct colour for fall through to the default foreground.
+    const TOKEN_CLASS = {
+        comment: 'tok-comment',
+        string: 'tok-string', verbatimString: 'tok-string', stringEscapeCharacter: 'tok-string',
+        number: 'tok-number',
+        keyword: 'tok-keyword', modifier: 'tok-keyword', preprocessorKeyword: 'tok-keyword',
+        controlKeyword: 'tok-control',
+        operator: 'tok-punct', punctuation: 'tok-punct',
+        class: 'tok-type', struct: 'tok-type', interface: 'tok-type', enum: 'tok-type',
+        delegate: 'tok-type', typeParameter: 'tok-type', type: 'tok-type', record: 'tok-type',
+        recordStruct: 'tok-type', namespace: 'tok-type',
+        method: 'tok-call', function: 'tok-call', extensionMethod: 'tok-call',
+        property: 'tok-variable', field: 'tok-variable', variable: 'tok-variable',
+        parameter: 'tok-variable', enumMember: 'tok-variable', event: 'tok-variable',
+        constant: 'tok-variable', local: 'tok-variable',
+    };
+
     function applyPalette() {
         const cls = document.body.className;
         const palette = cls.indexOf('vscode-high-contrast-light') >= 0 ? PALETTES.light
@@ -164,12 +183,41 @@
         return out;
     }
 
+    /**
+     * Turns the language server's token spans into [className, text] pairs, leaving the
+     * gaps between them (punctuation, whitespace) unclassified.
+     * @param {string} text
+     * @param {{start: number, end: number, type: string}[]} spans
+     * @returns {[string, string][]}
+     */
+    function fromSpans(text, spans) {
+        /** @type {[string, string][]} */
+        const out = [];
+        let at = 0;
+        for (const span of [...spans].sort((a, b) => a.start - b.start)) {
+            const start = Math.max(at, Math.min(span.start, text.length));
+            const end = Math.max(start, Math.min(span.end, text.length));
+            if (start > at) {
+                out.push(['', text.slice(at, start)]);
+            }
+            if (end > start) {
+                out.push([TOKEN_CLASS[span.type] || '', text.slice(start, end)]);
+            }
+            at = end;
+        }
+        if (at < text.length) {
+            out.push(['', text.slice(at)]);
+        }
+        return out;
+    }
+
     /**
      * Renders a code line: tokenized, with the referenced symbol boxed.
      * @param {string} text
      * @param {[number, number][]} hits
+     * @param {{start: number, end: number, type: string}[]} [spans]
      */
-    function renderCode(text, hits) {
+    function renderCode(text, hits, spans) {
         const fragment = document.createDocumentFragment();
         // Split token boundaries on hit boundaries so a hit never straddles two spans.
         const cuts = new Set([0, text.length]);
@@ -177,8 +225,10 @@
             cuts.add(start);
             cuts.add(end);
         }
+        // Real token types when the server served them, the local guess otherwise.
+        const tokens = spans && spans.length > 0 ? fromSpans(text, spans) : tokenize(text);
         let offset = 0;
-        for (const [cls, piece] of tokenize(text)) {
+        for (const [cls, piece] of tokens) {
             let from = offset;
             const to = offset + piece.length;
             const inner = [...cuts].filter(c => c > from && c < to).sort((a, b) => a - b);
@@ -382,7 +432,7 @@
         div.setAttribute('role', 'row');
 
         const code = cell('code');
-        code.appendChild(renderCode(row.code, row.hits));
+        code.appendChild(renderCode(row.code, row.hits, row.spans));
         code.title = row.code;
         div.appendChild(code);
 

package.json

@@ -164,7 +164,7 @@
         "coloredReferences.semanticTokens": {
           "type": "boolean",
           "default": true,
-          "description": "Colour the editor view with semantic tokens fetched from the source files, so identifiers look the way they do in the real file. Turn off to use TextMate grammar colouring only."
+          "description": "Use semantic tokens from the language server so identifiers are coloured by what they are. The editor view gets the theme-exact colouring; the panel gets the classification with an approximated palette. Turn off for grammar-only colouring."
         }
       }
     },

src/panel.ts

@@ -3,10 +3,62 @@ import * as path from 'path';
 import {
     RefKind, ReferenceResults, applyKinds, containingMembers, displayLine, referenceKinds, relativePath,
 } from './references';
+import { SEMANTIC_FILE_LIMIT, sourceLegend, sourceTokens } from './semantic';
 
 /** Where the results table is shown. */
 export type Placement = 'bottom' | 'beside' | 'below';
 
+/** A semantic token inside a row's code text, so the webview can colour by meaning. */
+export interface CodeSpan {
+    start: number;
+    end: number;
+    /** Token type name from the server's legend, e.g. `class`, `property`, `method`. */
+    type: string;
+}
+
+/**
+ * Semantic token spans per referenced line, in the row's own (trimmed) coordinates.
+ *
+ * A webview is not given the theme's token colours, so it cannot paint exactly what the
+ * editor does — but with the server's token *types* it at least stops guessing that every
+ * capitalised identifier is a type.
+ */
+export async function codeSpans(results: ReferenceResults): Promise<Map<string, CodeSpan[]>> {
+    const spans = new Map<string, CodeSpan[]>();
+    const enabled = vscode.workspace.getConfiguration('coloredReferences')
+        .get<boolean>('semanticTokens', true);
+    if (!enabled || results.files.length > SEMANTIC_FILE_LIMIT) {
+        return spans;
+    }
+    const legend = await sourceLegend(results.origin.uri);
+    if (!legend) {
+        return spans;
+    }
+
+    await Promise.all(results.files.map(async file => {
+        const shifts = new Map<number, number>();
+        for (const line of file.lines) {
+            shifts.set(line.line, -(line.text.length - line.text.trimStart().length));
+        }
+        for (const token of await sourceTokens(file.uri)) {
+            const shift = shifts.get(token.line);
+            if (shift === undefined) {
+                continue;
+            }
+            const start = token.char + shift;
+            if (start < 0) {
+                continue;
+            }
+            const key = `${file.uri.toString()}|${token.line}`;
+            const list = spans.get(key) ?? [];
+            list.push({ start, end: start + token.length, type: legend.tokenTypes[token.type] ?? '' });
+            spans.set(key, list);
+        }
+    }));
+
+    return spans;
+}
+
 /** One reference, as shown in a table row. */
 export interface PanelRow {
     id: number;
@@ -26,6 +78,8 @@ export interface PanelRow {
     member: string;
     /** Whether this reference reads or writes the symbol. */
     kind: RefKind;
+    /** Semantic token spans within `code`, empty when the server serves none. */
+    spans: CodeSpan[];
 }
 
 /**
@@ -38,6 +92,7 @@ export function buildRows(
     results: ReferenceResults,
     members: Map<string, string>,
     kinds: Map<string, RefKind> = new Map(),
+    spans: Map<string, CodeSpan[]> = new Map(),
 ): { rows: PanelRow[]; locations: Map<number, vscode.Location> } {
     const rows: PanelRow[] = [];
     const locations = new Map<number, vscode.Location>();
@@ -63,6 +118,7 @@ export function buildRows(
                     project: file.project ?? '',
                     member,
                     kind: symbolRanges[index].kind,
+                    spans: spans.get(`${file.uri.toString()}|${source.line}`) ?? [],
                 });
                 locations.set(id, new vscode.Location(file.uri, range.start));
                 id++;
@@ -134,11 +190,12 @@ export class ResultsView {
         this.results = results;
         this.host.setTitle(`References to ${results.symbol}`);
 
-        const [members, kinds] = await Promise.all([
+        const [members, kinds, spans] = await Promise.all([
             containingMembers(results),
             referenceKinds(results),
+            codeSpans(results),
         ]);
-        const { rows, locations } = buildRows(results, members, kinds);
+        const { rows, locations } = buildRows(results, members, kinds, spans);
         this.locations = locations;
 
         this.post({

src/semantic.ts

@@ -20,7 +20,7 @@ export interface Token {
 }
 
 /** Asking for whole-document tokens per file gets expensive on huge result sets. */
-const SEMANTIC_FILE_LIMIT = 40;
+export const SEMANTIC_FILE_LIMIT = 40;
 
 function timeout<T>(promise: Thenable<T>, ms: number): Promise<T | undefined> {
     return Promise.race([
@@ -48,13 +48,13 @@ export function decodeTokens(data: Uint32Array): Token[] {
     return tokens;
 }
 
-async function sourceLegend(uri: vscode.Uri): Promise<vscode.SemanticTokensLegend | undefined> {
+export async function sourceLegend(uri: vscode.Uri): Promise<vscode.SemanticTokensLegend | undefined> {
     const legend = await timeout(vscode.commands.executeCommand<vscode.SemanticTokensLegend>(
         'vscode.provideDocumentSemanticTokensLegend', uri), 4000);
     return legend?.tokenTypes?.length ? legend : undefined;
 }
 
-async function sourceTokens(uri: vscode.Uri): Promise<Token[]> {
+export async function sourceTokens(uri: vscode.Uri): Promise<Token[]> {
     try {
         // The server needs the document open to answer.
         await vscode.workspace.openTextDocument(uri);

src/test/suite/semantic.test.ts

@@ -3,6 +3,7 @@ import * as path from 'path';
 import * as vscode from 'vscode';
 
 import { SemanticOverlay, decodeTokens } from '../../semantic';
+import { buildRows, codeSpans } from '../../panel';
 import { CODE_INDENT, gather, renderDocument } from '../../references';
 
 const TARGET_FILE = path.join('Nerfed.Runtime', 'Profiler.cs');
@@ -131,6 +132,143 @@ suite('Semantic token overlay', () => {
             `${failures.length} mismatched token(s):\n${failures.slice(0, 10).join('\n')}`);
     });
 
+    test('every result file uses the same legend as the origin', async () => {
+        // The overlay fetches one legend (from the origin) but tokens from every file,
+        // so a file whose provider numbers its token types differently would decode to
+        // the wrong names — and therefore the wrong colours.
+        const locations = await vscode.commands.executeCommand<vscode.Location[]>(
+            'vscode.executeReferenceProvider', sourceUri, position);
+        const results = await gather(
+            TARGET_SYMBOL, 'csharp', { uri: sourceUri, position }, locations ?? [], true);
+        assert.ok(results.files.length > 1, 'need a multi-file result to test this');
+
+        const origin = await vscode.commands.executeCommand<vscode.SemanticTokensLegend>(
+            LEGEND_COMMAND, results.origin.uri);
+        assert.ok(origin);
+
+        const differences: string[] = [];
+        for (const file of results.files) {
+            await vscode.workspace.openTextDocument(file.uri);
+            const legend = await vscode.commands.executeCommand<vscode.SemanticTokensLegend>(
+                LEGEND_COMMAND, file.uri);
+            if (!legend) {
+                differences.push(`${path.basename(file.uri.fsPath)}: no legend`);
+                continue;
+            }
+            if (legend.tokenTypes.join() !== origin.tokenTypes.join()) {
+                differences.push(`${path.basename(file.uri.fsPath)}: token types differ`);
+            }
+            if (legend.tokenModifiers.join() !== origin.tokenModifiers.join()) {
+                differences.push(`${path.basename(file.uri.fsPath)}: token modifiers differ`);
+            }
+        }
+        assert.deepStrictEqual(differences, [], differences.join('\n'));
+    });
+
+    test('a relocated token reports the same type name as the source token', async () => {
+        const locations = await vscode.commands.executeCommand<vscode.Location[]>(
+            'vscode.executeReferenceProvider', sourceUri, position);
+        const results = await gather(
+            TARGET_SYMBOL, 'csharp', { uri: sourceUri, position }, locations ?? [], true);
+        const rendered = renderDocument(results);
+        const legend = await vscode.commands.executeCommand<vscode.SemanticTokensLegend>(
+            LEGEND_COMMAND, sourceUri);
+        assert.ok(legend);
+
+        const overlay = new SemanticOverlay();
+        const uri = vscode.Uri.from({ scheme: 'colored-refs', path: '/semantic-names.cs', query: 't' });
+        assert.ok(await overlay.build(uri, rendered.lines, results));
+        const relocated = decodeTokens(overlay.peek(uri)!.data);
+
+        // Rebuild the source-side answer independently and compare type names.
+        const mismatches: string[] = [];
+        let compared = 0;
+        for (const file of results.files) {
+            await vscode.workspace.openTextDocument(file.uri);
+            const raw = await vscode.commands.executeCommand<vscode.SemanticTokens>(
+                TOKENS_COMMAND, file.uri);
+            if (!raw?.data) {
+                continue;
+            }
+            const sourceTokens = decodeTokens(raw.data);
+            const displayed = new Map<number, number>(); // source line -> results row
+            rendered.lines.forEach((line, row) => {
+                if (line.kind === 'code' && line.file?.toString() === file.uri.toString() &&
+                    line.sourceLine !== undefined) {
+                    displayed.set(line.sourceLine, row);
+                }
+            });
+
+            for (const token of sourceTokens) {
+                const row = displayed.get(token.line);
+                if (row === undefined) {
+                    continue;
+                }
+                const here = relocated.filter(r => r.line === row);
+                const match = here.find(r => r.length === token.length &&
+                    legend.tokenTypes[r.type] === legend.tokenTypes[token.type]);
+                compared++;
+                if (!match) {
+                    mismatches.push(`${path.basename(file.uri.fsPath)}:${token.line + 1} ` +
+                        `${legend.tokenTypes[token.type]} (len ${token.length}) has no ` +
+                        `counterpart on results row ${row}`);
+                }
+            }
+        }
+        console.log(`[test] compared ${compared} source tokens, ${mismatches.length} unmatched`);
+        assert.deepStrictEqual(mismatches.slice(0, 8), [],
+            `${mismatches.length} unmatched:\n${mismatches.slice(0, 8).join('\n')}`);
+    });
+
+    test('panel rows carry token spans that name the real token type', async () => {
+        const locations = await vscode.commands.executeCommand<vscode.Location[]>(
+            'vscode.executeReferenceProvider', sourceUri, position);
+        const results = await gather(
+            TARGET_SYMBOL, 'csharp', { uri: sourceUri, position }, locations ?? [], true);
+
+        const spans = await codeSpans(results);
+        assert.ok(spans.size > 0, 'no token spans for the panel');
+        const { rows } = buildRows(results, new Map(), new Map(), spans);
+        assert.ok(rows.some(r => r.spans.length > 0), 'no row carries token spans');
+
+        // Spans are in the row's own trimmed coordinates, so slicing the code must give
+        // back a real token, and the type must come from the server's legend.
+        const failures: string[] = [];
+        const seen = new Map<string, string>();
+        for (const row of rows) {
+            for (const span of row.spans) {
+                const text = row.code.slice(span.start, span.end);
+                if (span.start < 0 || span.end > row.code.length) {
+                    failures.push(`${row.file}:${row.line} span ${span.start}-${span.end} ` +
+                        `is outside ${JSON.stringify(row.code)}`);
+                } else if (text.trim().length === 0) {
+                    failures.push(`${row.file}:${row.line} span covers whitespace`);
+                }
+                if (text.trim()) {
+                    seen.set(text, span.type);
+                }
+            }
+        }
+        assert.deepStrictEqual(failures.slice(0, 8), [], failures.slice(0, 8).join('\n'));
+
+        // The regression this fixes: `Profiler.Frames.Count` used to render all three as
+        // types because they are capitalised. The server calls them class + property.
+        console.log('[test] sample classifications: ' + ['Profiler', 'Frames', 'Count', 'if']
+            .filter(t => seen.has(t)).map(t => `${t}=${seen.get(t)}`).join(', '));
+        assert.strictEqual(seen.get(TARGET_SYMBOL), 'class');
+        // The point is that these are *members*, not types — which is what the regex
+        // tokenizer called them, since they are capitalised. Field vs property vs method
+        // does not matter here; they share a colour, as they do in the stock themes.
+        const members = ['property', 'field', 'event', 'method'];
+        for (const name of ['Frames', 'Count', 'IsRecording', 'SetActive']) {
+            const type = seen.get(name);
+            if (type) {
+                assert.ok(members.includes(type),
+                    `${name} should be a member, not ${type}`);
+            }
+        }
+    });
+
     test('the searched symbol is classified as a type', async () => {
         const legend = await vscode.commands.executeCommand<vscode.SemanticTokensLegend>(
             LEGEND_COMMAND, sourceUri);