Support for BiDi in placeables

[zbraniecki]

Since placeables can be of mixed directionality, I'd like to suggest that Fluent's FSI/PDI insertion for string placeholders is added to requirements.

This allows a variable like userName to be inserted in a string with different directionality and inform the layout of the possible direction change.

W3C backlog: https://www.w3.org/International/articles/inline-bidi-markup/
Fluent wiki: https://github.com/projectfluent/fluent/wiki/BiDi-in-Fluent

[mihnita]

+100

What I've seen being useful was marking a placeholder / area as RTL / LTR.
But that is not very often.

A lot more often was "smart detection", basically inspect the value of the parameter and "guess" what the best direction would be.
See for example: https://developer.android.com/reference/android/support/v4/text/BidiFormatter

The android solution is not ideal, the developer should explicitly "wrap" the parameter
Pseudocode:
...loadString(id).format( bidiWrapper(userName))
I think it should be more like
...loadString(id).format(arg)

And the string would be something like ...{userName}...
It would be ALWAYS wrapped, by default, unless explicitly disabled:

• ...{userName, bidi, rtl}...
• ...{userName, bidi, ltr}...
• ...{userName, bidi, none}...
• ...{userName, bidi, auto}... => same as ...{userName}...

The wrapper is smart, does not add bidi control characters if not needed.
So you are not going to see Hello {LRM}John{PDF}! in an English string :-)

[nbouvrette]

I don't know what you think but to me, this type of feature seems related to text transformation (see related thread).

If I'm not mistaken Fluent handles this with function-like wrappers. I'm trying to picture a scenario where you might want to capitalize and change text direction. If we could have a standard way to transform text, it could keep this simple:

# This software is made by {brand}
يتكون هذا البرنامج بواسطة {brand, transform, {rtl, titlecase}}

[zbraniecki]

If I'm not mistaken Fluent handles this with function-like wrappers.

No, Fluent implicitly wraps all placeables of type String in FSI/PDI to reset directionality.

[mihnita]

+100 doing it by default.

Not 100% sure FSI/DPI is the right thing, I would have to spend some time experimenting.
But yes to do the right thing by default, with the ability to turn it off for false positives.

[aphillips]

+1 to providing this by default. Note that when direction metadata is available the FSI should be replaced with the appropriate base-direction isolating control.

Note too that @zbraniecki only mentions placeables of type string, but non-string placeables can have spillover effects (for example currency values).

[zbraniecki]

@mihnita yes! I imagine that we would be able to do sth like:

bundle.formatPattern(pattern, {
  userName: FluentString(user.name, {dir: "rtl"});
});

as an option, and then, if that's provided, we can specify the directionality if it differs from the direction of the translation. If it matches, then we can skip directionality signs. If it is unknown, then we can use FSI/PDI.

@aphillips yes! In particular, we can know if the formatter provided result in the same language as the translation and wrap in marks or not. For most common scenario, where the currency formatter provided formatted text in the same direcitonality as the translation, we could skip it, but if we had to fallback and the currency is in different directionality, we would wrap.

[aharon-lanin]

A few comments:

• FSI / PDI may not be supported in all platforms (browsers and operating systems). It seems to work on Chrome, Firefox, Android, and Linux. I have not tried it on Safari or iOS, and when I last tried it on Edge and Windows, it did not work. If it is not sufficiently supported, one needs to estimate directionality of the placeable in code, and then wrap the placeable with LRMs or RLMs on the outside (depending on message locale) and an LRE or RLE and a PDF on the inside (depending on placeable directionality), as BidiFormatter does, skipping as many of these as can be safely done.
• For estimating directionality, I would recommend the first-strong algorithm, as used by FSI/PDI, for forward compatibility.
• Being able to specify the directionality of the placeable is a great option. However, from experience, 95% of the time the caller has nowhere to get it.

[zbraniecki]

Safari and iOS support FSI/PDI. Edge supports it as well. Windows modern APIs do support it, win32 does not.

[rxaviers]

Cross link globalizejs/globalize#862 globalizejs/globalize#539

[mihnita]

We can also take a look at what Android does.

The BidiFormatter::unicodeWrap takes a TextDirectionHeuristic, with several supported out of the box: android.text.TextDirectionHeuristics

At runtime that method looks at the value of the parameter, and adds the proper BiDi control characters, a bit smarter than just FSI / PDI, or first strong, or any "fixed" approach.

---

Dart has two methods, one "wraps" using BiDi control characters, the other one uses HTML tags:
https://api.flutter.dev/flutter/intl/BidiFormatter-class.html

---

I'm not advocating for any of these "as is", just submitting them as "prior art" and source of inspiration.

But if we go with this direction then I would call the wrappers inside of MessageFormat, not force the developers to wrap parameters by explicitly calling these kind of helper methods.

[zbraniecki]

Can we revisit it now?

It seems like we still didn't add it to the spec. I suggest that we by default wrap any placeable in FSI/PDI marks, just like Fluent does it, in line with W3C recommendation for placeables - https://www.w3.org/International/articles/inline-bidi-markup/

We can introduce evasion logic that allows us to explicitly turn off FSI/PDI for a given message format as an option to communicate request to format a message without inserting FSI/PDI.

Finally, we could start building evading logic for scenarios where the directionality of the surrounding text and the placeable is known to match. For example number/date inserted in the same locale as a surrounding message does not need FSI/PDI. Similarly, a string inserted could be marked with explicit directionality:

let mf = new MessageFormat("en");
mf.format("Hello, { $user }", { user: MFString("John", { dir: "ltr" }) });

or as matching:

let mf = new MessageFormat("en");
mf.format("Hello, { $user }", { user: MFString("John", { dir: "matching" }) });

In the former case the algorithm will detect directionality of "en" and if the directionality of MFString matches it it'll evade FSI/PDI. In the latter it will evade it automatically.

@mihnita @stasm @eemeli

[zbraniecki]

I'd like to suggest making a decision on it very soon. In my experience a lot of API users are not familiar with the problem space of directionality and the body of code starts growing where people expect to be able to match the output to a particular string and are surprised when FSI/PDI shows up in the output.

With Fluent we had to do quite a bit of evangelism - it was always well received, but definitely a paper cut.

I'm concerned that if we wait too long the argument of "too late" will pop up.