Ignore line endings for testing --exclude-file contents

[mdeweerd]

With --exclude-file, when the file with the line to be ignored has a CRLF line ending, the ignore does not work.

Adding a CR to the exclude-file does not help.

Run codespell -x exclude-file.txt test.html test2.html on the files in the test case.

codespell_tc_eol.zip

The expected result is no issues.

Tested with codespel 2.2.6

[DimitriPapadopoulos]

The exclude-file.txt file is missing from the test case. Also, codespell does not detect any typos in the provided test files. Not sure what I should to reproduce.

[DimitriPapadopoulos]

I tried with this example:

$ file exclude-file1.txt exclude-file2.txt test1.txt test2.txt 
exclude-file1.txt: ASCII text
exclude-file2.txt: ASCII text, with CRLF line terminators
test1.txt:         ASCII text
test2.txt:         ASCII text, with CRLF line terminators
$ 
$ codespell test1.txt test2.txt 
test1.txt:1: accepecting ==> accepting
test2.txt:1: accepecting ==> accepting
$ 
$ codespell -x exclude-file1.txt test1.txt test2.txt 
$ 
$ codespell -x exclude-file2.txt test1.txt test2.txt 
$ 

[mdeweerd]

Hi

Sorry for the missing exclusion file.
Here is a zip with a .codespellrc, and also a doCodespell.sh , result.log has the output.

codespell-exclude-test.zip .

As you can not reproduce it, I tried under WSL where the issue does not appear.
So this seems to be related to cygwin.

[DimitriPapadopoulos]

Interesting. How is codespell (or Python) related to Cygwin? Do you mean that you're starting codespell from a Cygwin shell?

[mdeweerd]

Interesting. How is codespell (or Python) related to Cygwin? Do you mean that you're starting codespell from a Cygwin shell?

Yes, I start codespell from a cygwin shell or through pre-commit running on python from the cygwin bash shell. (I do not find the same comfort in the WSL bash shell yet).

So something in python is surely environment related - for instance the way it reads the exlude file. I know that python can convert line endings implicitally - so maybe something like that is happening in one location, and not in the other.

[mdeweerd]

Probably partly related to this:

https://github.com/codespell-project/codespell/blob/6877de0cd8fdce87f8c6608e0afb267ba36aba32/codespell_lib/_codespell.py#L698C1-L700C58

I have:

    for i, line in enumerate(lines):
        if line in exclude_lines:
            continue

and

def build_exclude_hashes(filename: str, exclude_lines: Set[str]) -> None:
    with open(filename, encoding="utf-8") as f:
        for line in f:
            exclude_lines.add(line)

I have codespell v2.2.6, which is in _version.py in the location where I found the above coe.

So that's a change that is not released yet (?).

Also, rstrip is not just excluding 'line endings", it's excluding all whitespace at the end of the line - not exactly "whole lines that match those" - the description needs an update or only the line ending needs to be stripped.

#!/usr/bin/env python3
my_s = "abc \r\n";
stripped=my_s.rstrip()
print(":".join("{:02x}".format(ord(c)) for c in stripped))

results in: "61:62:63", confirming that spaces are stripped as well.

Possibly, converting the exclude_lines to a compiled regex is more efficient than stripping every line to match the excluded ones.

However, I think that the above does not explain why it works in one setup and not the other - maybe because the files are opened with "encoding" and not the exclude_file ?

[DimitriPapadopoulos]

The purpose of rstrip() call is precisely to ignore line endings, see #1889 / 86b30db. It would help if you could test the master branch.

[DimitriPapadopoulos]

The encoding is a real issue, all files containing text should all be opened using FileOpener:

codespell/codespell_lib/_codespell.py

Line 217 in 6877de0

class FileOpener:

However, let's focus on line endings for now. We open files with newline="", probably because we want to write them back using the same newline characters:

newline determines how to parse newline characters from the stream. It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

• When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

We want both:

• keep newlines so that we can write them back lines in their original form, including the newlines,
• discard newlines so that they don't interfere when matching.

[DimitriPapadopoulos]

This is a duplicate of #1888. Let's follow up there if needed.