M
Modular•6mo ago
lukas

tree-sitter-mojo

I had half an hour today, and Mojo is close enough to Python, so I forked the Python Tree Sitter grammar and got it working for mojo.
No description
49 Replies
lukas
lukasOP•6mo ago
GitHub
GitHub - lsh/tree-sitter-mojo: Mojo grammar for tree-sitter
Mojo grammar for tree-sitter. Contribute to lsh/tree-sitter-mojo development by creating an account on GitHub.
Dmitry Salin
Dmitry Salin•6mo ago
Nice, I will try to use your variant with Helix editor. You can also look at https://github.com/garam-kim1/tree-sitter-mojo The idea is to automatically update using GitHub Actions.
GitHub
GitHub - garam-kim1/tree-sitter-mojo: mojo grammar for tree-sitter
mojo grammar for tree-sitter. Contribute to garam-kim1/tree-sitter-mojo development by creating an account on GitHub.
lukas
lukasOP•6mo ago
Oh if I had seen that maybe I would have just used that lol
Dmitry Salin
Dmitry Salin•6mo ago
I haven't looked into the details, but perhaps your mojo part of the grammar is more relevant. But without automation it will be difficult to maintain such a fork. https://github.com/garam-kim1/tree-sitter-mojo/blob/master/.github/workflows/tree_sitter_maker.yml
Jack Clayton
Jack Clayton•6mo ago
Epic thanks both helix works now!
No description
lukas
lukasOP•6mo ago
Whoops I forgot to add alias, should do that now
Dmitry Salin
Dmitry Salin•6mo ago
It would be nice to push Helix PR forward at some point. https://github.com/helix-editor/helix/pull/8657
GitHub
feat: adds mojo language support by mabasic · Pull Request #8657 · ...
The same as: #8583 removed tags.scm removed commented line from language.toml applied suggested changes
Jack Clayton
Jack Clayton•6mo ago
If you raise a PR from your repo @Dmitry Salin it might avoid that test failure?
Dmitry Salin
Dmitry Salin•6mo ago
It's not mine, but it could definitely be used in PR and then maybe the build and tests will be ok. It seems better to create a new PR. The current one is already the second 🙂
ModularBot
ModularBot•6mo ago
Congrats @Dmitry Salin, you just advanced to level 1!
lukas
lukasOP•6mo ago
FWIW this is my helix config and it works for me
[language-server.mojo-lsp]
command = "mojo-lsp-server"
config = { provideFormatter = true }

[[grammar]]
name = "mojo"
source = { git = "https://github.com/lsh/tree-sitter-mojo", rev = "e645ad097257a29bed5ee6213fef47e26b280f63" }

[[language]]
name = "mojo"
grammar = "mojo"
scope = "source.mojo"
roots = ["__init__.mojo"]
file-types = ["mojo", "🔥"]
indent = { tab-width = 4, unit = " " }
auto-format = true
language-servers = ["mojo-lsp"]
formatter = { command = "mojo", args = ["format", "-q", "-"]}
[language-server.mojo-lsp]
command = "mojo-lsp-server"
config = { provideFormatter = true }

[[grammar]]
name = "mojo"
source = { git = "https://github.com/lsh/tree-sitter-mojo", rev = "e645ad097257a29bed5ee6213fef47e26b280f63" }

[[language]]
name = "mojo"
grammar = "mojo"
scope = "source.mojo"
roots = ["__init__.mojo"]
file-types = ["mojo", "🔥"]
indent = { tab-width = 4, unit = " " }
auto-format = true
language-servers = ["mojo-lsp"]
formatter = { command = "mojo", args = ["format", "-q", "-"]}
Melody Daniel
Melody Daniel•6mo ago
This is what I get while trying to compile the parser in Neovim Edit: I just failed to include scanner.c, it is compiled now.
No description
Jack Clayton
Jack Clayton•6mo ago
Only thing stopping me from using Helix as my main editor is it still doesn't have persistent undo history. I tried building this but doesn't work on mac: https://github.com/helix-editor/helix/pull/9154
lukas
lukasOP•6mo ago
Just pushed better support for parameter functions and decorators. Also wrote a Zed extension
Jack Clayton
Jack Clayton•6mo ago
Awesome cheers @lukas!
lukas
lukasOP•6mo ago
I’ll try to publish the zed extension later today Ooh wait, is there a tree sitter grammar for MLIR? We could have it so that things like:
__mlir_op.`1: i1`
__mlir_op.`1: i1`
uses MLIR syntax highlighting injection within the backticks
aurelian
aurelian•6mo ago
can you share the steps? thx
Dmitry Salin
Dmitry Salin•6mo ago
My PR has been merged and Helix now supports Mojo. https://github.com/helix-editor/helix/pull/10743 For now, users have to build from master, but future releases should have out of the box support.
GitHub
Add support for Mojo by dmitry-salin · Pull Request #10743 · helix-...
Previous PR has not been updated for a long time: #8657 In this PR, I use the most up-to-date Mojo grammar.
lukas
lukasOP•6mo ago
We’ll probably have to patch in some of the newer syntax/tree sitter stuff. Lately I’ve been playing with special highlighting for MLIR stuff
Dmitry Salin
Dmitry Salin•6mo ago
Yea, it will be cool to support MLIR. I initially added this:
((identifier) @type.builtin
(#match?
@type.builtin
"^(__mlir_type)$"))
((identifier) @type.builtin
(#match?
@type.builtin
"^(__mlir_type)$"))
But this is a workaround, it's better to have built-in support in tree-sitter. Traits are also need special handling, I will try to add them later.
lukas
lukasOP•6mo ago
This is my WIP
No description
lukas
lukasOP•6mo ago
I’m not sure how I want to highlight dialect names though. Or builtins I haven’t pushed it yet
Dmitry Salin
Dmitry Salin•6mo ago
Good progress. I thought that from a Helix highlighting perspective, __mlir_attr and __mlir_type should be @type.builtin, and __mlir_op - @function.builtin or @function.special
lukas
lukasOP•6mo ago
Yep, that’s what I’m doing (except I’m using Zed in the above which might have slightly different tree sitter setup) I’m probably gonna start on __mlir_op sometimes this weekend or next week. It’s tricky to get right since it requires custom grammar parsing and not just a highlight token. At least that’s what it requires if we want more granular control over things like bracket vs plain backtick
Dmitry Salin
Dmitry Salin•6mo ago
Yes, MLIR parsing is not an easy part. I will try to add traits.
lukas
lukasOP•6mo ago
It should just be the same as how I added struct. Pr for that is welcome. I also should add tests
Dmitry Salin
Dmitry Salin•6mo ago
So expect PR from me. It looks like your tree-sitter version will be used in Helix, Vim and Zed. VS Code might someday switch to tree-sitters - https://github.com/microsoft/vscode/issues/50140. This way, the user can have a consistent experience with the major popular editors.
dcow
dcow•6mo ago
@lukas can you link me to your zed extension? I was working on adding mojo support to zed too and want to exchange notes
lukas
lukasOP•6mo ago
Ah I haven't put it up yet since I'm still tweaking a lot, but I can publish it with a YMMV warning I guess
dcow
dcow•6mo ago
no rush whenever you're ready maybe you've just saved me some work XD
lukas
lukasOP•6mo ago
It's mostly all working, aside from a few corner cases in the tree sitter grammar
Melody Daniel
Melody Daniel•6mo ago
Neovim 0.10 just dropped with support recognising Mojo files. This is how Mojo is highlighted
No description
Darin Simmons
Darin Simmons•6mo ago
I'm gonna make the leap from VSCode to neovim just for this. I feel like typing, "May God have mercy on you" or something. I have no idea what I'm signing up for though I'll eventually have to go linux (because reasons), and this just feels inevitable. Yep, stood on that landmine already. Probably about my sixth time trying (n)vim and it was a much smoother experience this time. From what I can gather, the program parsers are .dll in windows and well, there's no mojo yet. Sigh...
Melody Daniel
Melody Daniel•6mo ago
Hoping we get Mojo in the remaining major platforms soon
Moosems / Three chickens
Having an OSS LSP for Mojo would be nice
Darin Simmons
Darin Simmons•6mo ago
I feel dirty (mostly joking), I ended up installing nvim on WSL then open another WSL window to build and I actually (serious surprise) that I was able to write with nvim in one (simple open a file) and run in another. Will continue learning nvim without code completions, hell, will continue to learn to code and learn mojo in the meantime. 💪
Zahhak
Zahhak•4mo ago
Hi Dmitry. I followed your setup instructions and this is the output of hx --health mojo. But autocomplete doesn't work for mojo files and all the diagnostics go away once I enter the insert mode. The only way to bring them back is to re-open the file. Interestingly, I changed the comment token in languages.toml and it takes effect, but can't seem to figure out the autocomplete and diagnostics. Appreciate any feedback.
No description
Dmitry Salin
Dmitry Salin•4mo ago
Hi. These setup instructions are for an older version of tree-sitter. The latest version of helix has out of the box Mojo support https://github.com/helix-editor/helix/blob/master/CHANGELOG.md?plain=1#L134 Can you try it?
Zahhak
Zahhak•4mo ago
This is the output of Helix log when I try with/without language.toml:
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.918] --> initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.919] --> reply:initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "D[10:33:34.919] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\",\".\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.918] --> initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.919] --> reply:initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "D[10:33:34.919] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\",\".\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"
I used https://github.com/helix-editor/helix/blob/master/languages.toml#L377C1-L392C1 I also did the grammar fetch/build for mojo.
GitHub
helix/languages.toml at master · helix-editor/helix
A post-modern modal text editor. Contribute to helix-editor/helix development by creating an account on GitHub.
Dmitry Salin
Dmitry Salin•4mo ago
I'm trying these binaries https://github.com/helix-editor/helix/releases/tag/24.07 And on Linux everything works fine. My log:
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> reply:initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "D[18:08:50.071] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> reply:initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "D[18:08:50.071] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"
There was a bug with LSP in the old Mojo nightlies, but it has been fixed. https://github.com/modularml/mojo/issues/2835 I use most recent Mojo nightly build.
Zahhak
Zahhak•4mo ago
Most recent nightly build fixed the problem. Thanks much.
ModularBot
ModularBot•4mo ago
Congrats @Zahhak, you just advanced to level 1!
Moosems / Three chickens
Will we ever get any pypi releases of this?
lukas
lukasOP•4mo ago
That’s an interesting question. Maybe when magic gets a little better I’ll write a Mojo wrapper around the C and publish
Moosems / Three chickens
For the python bindings?
lukas
lukasOP•4mo ago
Ah I hadn't considered publishing the python bindings. Is there a specific use case you have in mind that the bindings would be able to accomplish?
Moosems / Three chickens
Allowing me to use it with the python tree sitter library?
xentropy
xentropy•4mo ago
You are a gentleman and scholar for this. Thank you!
Want results from more Discord servers?
Add your server