Semantic Highlighting in the PowerShell Preview extension for Visual Studio Code

Semantic Highlighting in the PowerShell Preview extension for Visual Studio Code

Hi everyone! I’m Justin and I am currently an intern on the PowerShell team. One of my projects was to add PowerShell semantic highlighting support in VS Code allowing for more accurate highlighting in the editor. I’m excited to share that the first iteration has been released.

Getting started

Great news! You don’t have to do anything to get this feature except for making sure you have at least the v2020.7.0 version of the PowerShell Preview extension for Visual Studio Code .

For theme authors: Supporting Semantic Highlighting

If you are a theme author, make sure to add {semanticHighlighting: true} to the theme.json file of your VS Code theme.

For a more complete guide into supporting Semantic Highlighting in your theme, please look at:

• The Semantic Highlighting Overview
• The Semantic Highlighting Guide

The rest of this blog post will discuss the shortcomings of the old syntax highlighting mechanism and how semantic highlighting addresses those issues.

Syntax Highlighting

Currently, the syntax highlighting support for PowerShell scripts in VS Code leverages TextMate grammars , which are mappings of regular expressions to tokens. For instance, to identify control keywords, something like the following would be used

{
    name = 'keyword.control.untitled';
    match = 'b(if|while|for|return)b';
}

However, there are some limitations with regular expressions and their ability to recognize different syntax patterns. Since TextMate grammars rely on these expressions, there are many complex and context-dependent tokens these grammars are unable to parse, leading to inconsistent or incorrect highlighting. Just skim through the issues in the EditorSyntax repo , our TextMate grammar.

Here are a few examples where syntax highlighting fails in tokenizing a PowerShell script.

Semantic Highlighting

To solve those cases (and many other ones) we use the PowerShell tokenizer which describes the tokens more accurately than regular expressions can, while also always being up-to-date with the language grammar. The only problem is that the tokens generated by the PowerShell tokenizer do not align perfectly to the semantic token types predefined by VS Code. The semantic token types provided by VS Code are:

• namespace
• type, class, enum, interface, struct, typeParameter
• parameter, variable, property, enumMember, event
• function, member, macro
• label
• comment, string, keyword, number, regexp, operator

On the other hand, there are over 100 PowerShell token kinds and also manytoken flags that can modify those types.

The main task (aside from setting up a semantic tokenization handler) was to create a mapping from PowerShell tokens to VS Code semantic token types. The result of enabling semantic highlighting can be seen below.

If we compare the semantic highlighting to the highlighting in PowerShell ISE, we can see they are quite similar (in tokenization, not color).

Next Steps

Although semantic highlighting does a better job than syntax highlighting in identifying tokens, there remain some cases that can still be improved at the PowerShell layer.

In Example 5, for instance, while the enum does have better highlighting, the name and members of the enums are highlighted identically. This occurs because PowerShell tokenizes them all of them the same way (as identifiers with a token flags denoting that they are member names meaning that the semantic highlighting has no way to differentiate them.

How to Provide Feedback

If you experience any issues or have comments on improvement, please raise an issue in PowerShell/vscode-powershell . Since this was just released, any feedback will be greatly appreciated.

Justin Chen PowerShell Team