Modular•7mo ago

tree-sitter-mojo

I had half an hour today, and Mojo is close enough to Python, so I forked the Python Tree Sitter grammar and got it working for mojo.

49 Replies

lukasOP•7mo ago

https://github.com/lsh/tree-sitter-mojo

GitHub

GitHub - lsh/tree-sitter-mojo: Mojo grammar for tree-sitter

Mojo grammar for tree-sitter. Contribute to lsh/tree-sitter-mojo development by creating an account on GitHub.

Dmitry Salin•7mo ago

Nice, I will try to use your variant with Helix editor. You can also look at https://github.com/garam-kim1/tree-sitter-mojo The idea is to automatically update using GitHub Actions.

GitHub

GitHub - garam-kim1/tree-sitter-mojo: mojo grammar for tree-sitter

mojo grammar for tree-sitter. Contribute to garam-kim1/tree-sitter-mojo development by creating an account on GitHub.

lukasOP•7mo ago

Oh if I had seen that maybe I would have just used that lol

Dmitry Salin•7mo ago

I haven't looked into the details, but perhaps your mojo part of the grammar is more relevant. But without automation it will be difficult to maintain such a fork. https://github.com/garam-kim1/tree-sitter-mojo/blob/master/.github/workflows/tree_sitter_maker.yml

Jack Clayton•7mo ago

Epic thanks both helix works now!

lukasOP•7mo ago

Whoops I forgot to add alias, should do that now

Dmitry Salin•7mo ago

It would be nice to push Helix PR forward at some point. https://github.com/helix-editor/helix/pull/8657

GitHub

feat: adds mojo language support by mabasic · Pull Request #8657 · ...

The same as: #8583 removed tags.scm removed commented line from language.toml applied suggested changes

Jack Clayton•7mo ago

If you raise a PR from your repo @Dmitry Salin it might avoid that test failure?

Dmitry Salin•7mo ago

It's not mine, but it could definitely be used in PR and then maybe the build and tests will be ok. It seems better to create a new PR. The current one is already the second 🙂

ModularBot•7mo ago

Congrats @Dmitry Salin, you just advanced to level 1!

lukasOP•7mo ago

FWIW this is my helix config and it works for me

[language-server.mojo-lsp]
command = "mojo-lsp-server"
config = { provideFormatter = true }

[[grammar]]
name = "mojo"
source = { git = "https://github.com/lsh/tree-sitter-mojo", rev = "e645ad097257a29bed5ee6213fef47e26b280f63" }

[[language]]
name = "mojo"
grammar = "mojo"
scope = "source.mojo"
roots = ["__init__.mojo"]
file-types = ["mojo", "🔥"]
indent = { tab-width = 4, unit = " " }
auto-format = true
language-servers = ["mojo-lsp"]
formatter = { command = "mojo", args = ["format", "-q", "-"]}

[language-server.mojo-lsp]
command = "mojo-lsp-server"
config = { provideFormatter = true }

[[grammar]]
name = "mojo"
source = { git = "https://github.com/lsh/tree-sitter-mojo", rev = "e645ad097257a29bed5ee6213fef47e26b280f63" }

[[language]]
name = "mojo"
grammar = "mojo"
scope = "source.mojo"
roots = ["__init__.mojo"]
file-types = ["mojo", "🔥"]
indent = { tab-width = 4, unit = " " }
auto-format = true
language-servers = ["mojo-lsp"]
formatter = { command = "mojo", args = ["format", "-q", "-"]}

Melody Daniel•7mo ago

This is what I get while trying to compile the parser in Neovim Edit: I just failed to include scanner.c, it is compiled now.

Jack Clayton•7mo ago

Only thing stopping me from using Helix as my main editor is it still doesn't have persistent undo history. I tried building this but doesn't work on mac: https://github.com/helix-editor/helix/pull/9154

lukasOP•7mo ago

Just pushed better support for parameter functions and decorators. Also wrote a Zed extension

Jack Clayton•7mo ago

Awesome cheers @lukas!

lukasOP•7mo ago

I’ll try to publish the zed extension later today Ooh wait, is there a tree sitter grammar for MLIR? We could have it so that things like:

__mlir_op.`1: i1`

__mlir_op.`1: i1`

uses MLIR syntax highlighting injection within the backticks

aurelian•7mo ago

can you share the steps? thx

Melody Daniel•7mo ago

https://github.com/nvim-treesitter/nvim-treesitter?tab=readme-ov-file#adding-parsers

GitHub

GitHub - nvim-treesitter/nvim-treesitter: Nvim Treesitter configura...

Nvim Treesitter configurations and abstraction layer - nvim-treesitter/nvim-treesitter

Dmitry Salin•7mo ago

My PR has been merged and Helix now supports Mojo. https://github.com/helix-editor/helix/pull/10743 For now, users have to build from master, but future releases should have out of the box support.

GitHub

Add support for Mojo by dmitry-salin · Pull Request #10743 · helix-...

Previous PR has not been updated for a long time: #8657 In this PR, I use the most up-to-date Mojo grammar.

lukasOP•7mo ago

We’ll probably have to patch in some of the newer syntax/tree sitter stuff. Lately I’ve been playing with special highlighting for MLIR stuff

Dmitry Salin•7mo ago

Yea, it will be cool to support MLIR. I initially added this:

((identifier) @type.builtin
  (#match?
    @type.builtin
    "^(__mlir_type)$"))

((identifier) @type.builtin
  (#match?
    @type.builtin
    "^(__mlir_type)$"))

But this is a workaround, it's better to have built-in support in tree-sitter. Traits are also need special handling, I will try to add them later.

lukasOP•7mo ago

This is my WIP

lukasOP•7mo ago

I’m not sure how I want to highlight dialect names though. Or builtins I haven’t pushed it yet

Dmitry Salin•7mo ago

Good progress. I thought that from a Helix highlighting perspective, __mlir_attr and __mlir_type should be @type.builtin, and __mlir_op - @function.builtin or @function.special

lukasOP•7mo ago

Yep, that’s what I’m doing (except I’m using Zed in the above which might have slightly different tree sitter setup) I’m probably gonna start on __mlir_op sometimes this weekend or next week. It’s tricky to get right since it requires custom grammar parsing and not just a highlight token. At least that’s what it requires if we want more granular control over things like bracket vs plain backtick

Dmitry Salin•7mo ago

Yes, MLIR parsing is not an easy part. I will try to add traits.

lukasOP•7mo ago

It should just be the same as how I added struct. Pr for that is welcome. I also should add tests

Dmitry Salin•7mo ago

So expect PR from me. It looks like your tree-sitter version will be used in Helix, Vim and Zed. VS Code might someday switch to tree-sitters - https://github.com/microsoft/vscode/issues/50140. This way, the user can have a consistent experience with the major popular editors.

dcow•7mo ago

@lukas can you link me to your zed extension? I was working on adding mojo support to zed too and want to exchange notes

lukasOP•7mo ago

Ah I haven't put it up yet since I'm still tweaking a lot, but I can publish it with a YMMV warning I guess

dcow•7mo ago

no rush whenever you're ready maybe you've just saved me some work XD

lukasOP•7mo ago

It's mostly all working, aside from a few corner cases in the tree sitter grammar

Melody Daniel•7mo ago

Neovim 0.10 just dropped with support recognising Mojo files. This is how Mojo is highlighted

Darin Simmons•7mo ago

I'm gonna make the leap from VSCode to neovim just for this. I feel like typing, "May God have mercy on you" or something. I have no idea what I'm signing up for though I'll eventually have to go linux (because reasons), and this just feels inevitable. Yep, stood on that landmine already. Probably about my sixth time trying (n)vim and it was a much smoother experience this time. From what I can gather, the program parsers are .dll in windows and well, there's no mojo yet. Sigh...

Melody Daniel•7mo ago

Hoping we get Mojo in the remaining major platforms soon

Moosems / Three chickens•7mo ago

Having an OSS LSP for Mojo would be nice

Darin Simmons•7mo ago

I feel dirty (mostly joking), I ended up installing nvim on WSL then open another WSL window to build and I actually (serious surprise) that I was able to write with nvim in one (simple open a file) and run in another. Will continue learning nvim without code completions, hell, will continue to learn to code and learn mojo in the meantime. 💪

Zahhak•4mo ago

Hi Dmitry. I followed your setup instructions and this is the output of hx --health mojo. But autocomplete doesn't work for mojo files and all the diagnostics go away once I enter the insert mode. The only way to bring them back is to re-open the file. Interestingly, I changed the comment token in languages.toml and it takes effect, but can't seem to figure out the autocomplete and diagnostics. Appreciate any feedback.

Dmitry Salin•4mo ago

Hi. These setup instructions are for an older version of tree-sitter. The latest version of helix has out of the box Mojo support https://github.com/helix-editor/helix/blob/master/CHANGELOG.md?plain=1#L134 Can you try it?

Zahhak•4mo ago

This is the output of Helix log when I try with/without language.toml:

2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.918] --> initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.919] --> reply:initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "D[10:33:34.919] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\",\".\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"

2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.918] --> initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.919] --> reply:initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "D[10:33:34.919] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\",\".\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"

I used https://github.com/helix-editor/helix/blob/master/languages.toml#L377C1-L392C1 I also did the grammar fetch/build for mojo.

GitHub

helix/languages.toml at master · helix-editor/helix

A post-modern modal text editor. Contribute to helix-editor/helix development by creating an account on GitHub.

Dmitry Salin•4mo ago

I'm trying these binaries https://github.com/helix-editor/helix/releases/tag/24.07 And on Linux everything works fine. My log:

2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> reply:initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "D[18:08:50.071] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"

2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> reply:initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "D[18:08:50.071] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"

There was a bug with LSP in the old Mojo nightlies, but it has been fixed. https://github.com/modularml/mojo/issues/2835 I use most recent Mojo nightly build.

Zahhak•4mo ago

Most recent nightly build fixed the problem. Thanks much.

ModularBot•4mo ago

Congrats @Zahhak, you just advanced to level 1!

Moosems / Three chickens•4mo ago

Will we ever get any pypi releases of this?

lukasOP•4mo ago

That’s an interesting question. Maybe when magic gets a little better I’ll write a Mojo wrapper around the C and publish

Moosems / Three chickens•4mo ago

For the python bindings?

lukasOP•4mo ago

Ah I hadn't considered publishing the python bindings. Is there a specific use case you have in mind that the bindings would be able to accomplish?