Switch nvim-treesitter to the new main branch

[oriori1703]

As described in nvim-treesitter's README:

The master branch is frozen and provided for backward compatibility only. All future updates happen on the main branch, which will become the default branch in the future.

This is a proactive branch to switch to the new branch.
I have been using it for a couple of month and it seems pretty stable to me, though I'm not sure if this is really ready to be merged right now.
There maybe features that are not yet supported (or will never be) like incremental selection.
Does anyone else have experience with this transition, and can give feedback?

PS, I also added support for treesitter based folds, but left it commented out. Let me know if you think I should enable it in this PR.

[guru245]

I realized that my test was completely wrong and so I deleted my comments. Now that I've found out how to use main branch, I will retest and get back to you.

[dotfrag]

Thank you for the PR. A bit unrelated to kickstart but.. I'm a bit confused as to where all my previous options should go. Could you provide some insight? This is my current setup https://github.com/dotfrag/dotvim/blob/main/lua/plugins/treesitter.lua

I can't see those in the new branch, are they removed?

Also noticed a typo on line 957 (enbales).

[guru245]

I finally succeeded in running main branch. Everything is working as expected.

Installing tree-sitter CLI was a main hurdle. I think others will have the same difficulty and so I leave a how-to-do here:

1. Download a prebuilt tree-sitter CLI.
2. Place it in your bin directory. (For me, ~/.local/bin)
3. Change it to tree-sitter.
4. Apply the PR and :Lazy sync
5. Tada~

My thought:
This transition will cause a lot of trouble for beginners. They must setup tree-sitter CLI even before using neovim just like fd and rg. Therefore, we must describe how to install tree-sitter CLI in README.md.

[oriori1703]

Therefore, we must describe how to install tree-sitter CLI in README.md.

You are probably right 😄 . I will add that.

BTW, as mentioned in their README, you can also install using cargo:

cargo install --locked tree-sitter-cli

or with npm:

npm install tree-sitter-cli

In addition to that, in some distros, you can probably use your package manager. For example, in fedora you can install it using:

sudo dnf install tree-sitter-cli

[guru245]

Jeez. I was wondering how you found the right README.md in the source tree--there are so many of them. 😂

Maybe it would help if I share some of the difficulties I had while trying to get started:

• I haven't used rust before, so I have to figure out how to use cargo install.
• I don't know what npm is--sorry for my ignorance. I tried but I got many errors.
• Ubuntu (Probably the most widely used distro) doesn't provide the package.

💡 Not a request, just thinking out loud
In blink.cmp, when we use fuzzy = { implementation = 'rust' }, it downloads the code, builds it with cargo, puts the binary in a bin directory, and cleans up the code. Maybe doing something similar in the new tree-sitter would make the transition a lot easier.

[dotfrag]

Thank you for the PR. A bit unrelated to kickstart but.. I'm a bit confused as to where all my previous options should go. Could you provide some insight? This is my current setup https://github.com/dotfrag/dotvim/blob/main/lua/plugins/treesitter.lua

I can't see those in the new branch, are they removed?

Also noticed a typo on line 957 (enbales).

Replying to myself, I don't think this is fully ready for everyone. I've tried multiple times to get text objects to work (with main branch) and I haven't been able to. Maybe it's a skill issue on my end but it appears to do nothing.

The context plugin does not have a working version.

And for me the autocmd doesn't work, i.e. I get no treesitter highlighting when opening a file. This is most likely on me as it seems to work for you, so I'm still investigating.

Update: Highlighting wasn't working for me, because not all parsers match filetype names. For example, for a shell script, the parser is bash, while the filetype is sh. You can see my workaround in the link.

[Ktsierra]

I think the auto_install feature is essential for people like me that are new to neovim, i tried to make it work with the main branch but couldn't, maybe the rewrite is still too green?

[dotfrag]

I think the auto_install feature is essential for people like me that are new to neovim, i tried to make it work with the main branch but couldn't, maybe the rewrite is still too green?

Same here, I work in the terminal all day and open all sorts of files, I can't bother manually adding them one by one..

[oriori1703]

@dotfrag

The context plugin does not have a working version.

I works for me. Could you possibly be affected by this breaking change?

[oriori1703]

I think the auto_install feature is essential for people like me that are new to neovim, i tried to make it work with the main branch but couldn't, maybe the rewrite is still too green?

I do agree that the auto_install feature is probably a deal breaker for merging this PR.
Though it could probably be replicated using a simple autocmd. I will take a crack at it when I have time.

For the time being, it is probably best to leave this PR as a draft.

[dotfrag]

@dotfrag

The context plugin does not have a working version.

I works for me. Could you possibly be affected by this breaking change?

Yes! Thank you.

[dotfrag]

I do agree that the auto_install feature is probably a deal breaker for merging this PR. Though it could probably be replicated using a simple autocmd. I will take a crack at it when I have time.

This is a (probably) badly optimised way to do it:

        vim.api.nvim_create_autocmd("FileType", {
          callback = function()
            local ft = vim.bo.filetype
            if not parsers[ft] then
              require("nvim-treesitter").install(ft)
            end
          end,
        })

Updated version with exclude option https://github.com/dotfrag/dotvim/blob/ecdc0ab1dca98c901f1cea2a8cf6d057e596fece/lua/plugins/treesitter.lua#L39-L54.

[MeanderingProgrammer]

I wrote a plugin to fill the gaps between the master and main branches: https://github.com/MeanderingProgrammer/treesitter-modules.nvim

You're welcome to pull any of the implementations from there or use it directly.

Here's a minimal implementation of auto_install:

---@param buf integer
---@param language string
---@return boolean
local function attach(buf, language)
    -- check if parser exists before starting highlighter
    if not vim.treesitter.language.add(language) then
        return false
    end
    vim.treesitter.start(buf, language)
    return true
end

vim.api.nvim_create_autocmd('FileType', {
    callback = function(args)
        local buf, filetype = args.buf, args.match
        local language = vim.treesitter.language.get_lang(filetype)
        if not language then
            return
        end
        if attach(buf, language) then
            return
        end
        -- attempt to start highlighter after installing missing language
        require('nvim-treesitter').install(language):await(function()
            attach(buf, language)
        end)
    end,
})

[dotfrag]

Thanks @MeanderingProgrammer, this works for the most part, except the cases mentioned above, where bash parser does not match sh filetype. I'm guessing a translation table is needed, unless vim.treesitter provides this somewhere. I believe master version did, but maybe it was left out from main.

After using several sophisticated print statement, I see that the translation from sh to bash happens correctly, but it still doesn't work for these cases 🤔. Aaand it seems to create infinite loop excessive calls with snacks notification windows and blink completion windows.

[MeanderingProgrammer]

vim.treesitter.language.get_lang('sh') should return 'bash', which should make all the other logic work correctly.

I don't believe I have anything special going on that makes this associate the filetype with the correct language.

What does this command :lua vim.print(vim.treesitter.language.get_lang('sh')) print for you? And if it prints bash then everything should be working so am unsure what's not working.

[dotfrag]

vim.treesitter.language.add('bash') returns true even if it's uninstalled. Maybe that is one of the reasons. Not entirely sure to be honest. Thanks for spending time on this - I'll figure out the rest :)

[MeanderingProgrammer]

It might be the case that you do have bash installed (even if you didn't expect to), but that is very strange if that's not the case.

If I do :TSUninstall bash then run :lua vim.print(vim.treesitter.language.add('bash')) I get this output:

nil
No parser for language "bash"

[dotfrag]

It might be the case that you do have bash installed (even if you didn't expect to), but that is very strange if that's not the case.

If I do :TSUninstall bash then run :lua vim.print(vim.treesitter.language.add('bash')) I get this output:

nil
No parser for language "bash"

That wasn't my experience, and after a lot of troubleshooting I only managed to get the expected behavior after entirely resetting my setup. I deleted the following dirs and then it was fine:

  "${XDG_CONFIG_HOME:-${HOME}/.config}/nvim"
  "${XDG_DATA_HOME:-${HOME}/.local/share}/nvim"
  "${XDG_STATE_HOME:-${HOME}/.local/state}/nvim"
  "${XDG_CACHE_HOME:-${HOME}/.cache}/nvim"

I suspect some state/cache was stuck, because I was doing :TSUninstall then getting true on the check, but treesitter was obviously not starting. Thanks for your response, it helped me narrow down the issue.