scalac doesn't accept Windows notepad or wordpad files saved as UTF-8

[scabug]

scalac doesn't accept Windows notepad or wordpad files saved as
"UTF-8" or "Unicode", even when the -encoding UTF-8 option is
used. When the same file is saved with jEdit as UTF-8, there
is no problem. But notepad and wordpad are probably installed on
more computers than any other editor in the world, and this
could be a nuisance for people just trying out Scala. This
is probably one of the dreaded BOM issues...

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-2109?orig=1
Reporter: Terry Woodruff (twoodruff)
Attachments:

• Test.scala (created on Jun 9, 2011 5:21:19 PM UTC, 13 bytes)

[scabug]

@paulp said:
Can you please attach a sample file which it refused to compile? And specify what version you're trying to compile with? I have the basics of a solution to the encoding issues in place but I don't use windows and would really appreciate not having to do so.

[scabug]

Gábor Bakos (aborg) said:
It contains the following first three bytes: EF BB BF (http://en.wikipedia.org/wiki/Byte_order_mark)

[scabug]

Gábor Bakos (aborg) said:
The attached file fails to compile with Scala 2.9.0.1:

q:\utils\scala-2.9.0.1\bin\scalac -encoding UTF-8 Test.scala
Test.scala:1: error:
illegal character
?class X {}
^
one error found

[scabug]

Gábor Bakos (aborg) said:
As the resolved state has set on 18/May/2011, and 2.9.0.1 released on 25/May/2011, and the resolution is incomplete, I guess it worth reopening.

[scabug]

Gábor Bakos (aborg) said:
I have tried to handle the most common UTF BOMs in the following commit: aborg0/scala@e1ba73d although the tests do not work, as the decoder (encoding) is inconsistent with the BOM. Should the BOM override the encoding to that specific file, or there should be a warning in this case and use the original encoding (with the bytes skipped)? What is your opinion/decision?

[scabug]

@paulp said:
I think by default the BOM should override the default encoding, but an encoding given on the command line should override the bom.

Can you take a look at EncodingHeuristics in the library and see about using that? I can get it moved somewhere more appropriate, I just don't want to duplicate the logic. (I don't know if the logic is right, either, but it should be easy for someone to determine.)

[scabug]

Gábor Bakos (aborg) said:
Thanks for the pointer, I was not aware of EncodingHeuristics. Hmm. I think part of the logic can be extracted, but they have different inputs (java.io.InputStream and java.nio.channels.ReadableByteChannel), so a third implementation might be required for the input (or is there already a Scala container with conversion from both?).
I agree that it would be best to have the common part available only in one place. I am unsure where would best fit.

I see a problem related to the default. I do not know how can I determine whether the decoder is from the default settings, or it was specified previously (to be the same as the default).

[scabug]

Gábor Bakos (aborg) said:
I'm afraid the encoding is always set, at least I could not figure out in Global.scala reader, how to get setting.encoding.isSetByUser as false when no -encode command line option is set.
I could not figure out a way, to easily unify the two implementations (different type of containers (InputStream, ReadableByteChannel), different type of values (Int and Byte)).

[scabug]

Gábor Bakos (aborg) said:
With the last commit (aborg0/scala@6f5b477) it does what you described, although in partest it gives an AssertionError.
I am afraid it is because it uses the same settings objects with multiple threads. If you have an idea how to correct it, that is welcome. (I have to admit this is a really ugly patchwork.)
I could not find an easy way to integrate the two approaches handling the BOMs.

[scabug]

Gábor Bakos (aborg) said:
Here is another attempt to challange this issue: commit (this time it is quite recent, so if it is suitable I guess it can be merged, passes all the tests). I think the important parts are mentioned in the commit message, although it worth noting, that UTF-7 is not supported. (Internals thread)

[scabug]

@jsuereth said:
I realize it's been a while, just getting to this now and reviewing. The commit: aborg0/scala-1@83f358b Doesn't look bad, but it does look like it ignores specific user settings. Perhaps a "DETECT" setting for encoding is needed? Also, this needs tests, including files, to ensure it works appropriately and we maintain the code correctly.

It may be too late for 2.10.0, but I can help you out for 2.10.1 if you need time putting together something with more automated tests and more "general" (in the sense that specifying encodings don't break things).