Sorting strangeness

[Mark Waddingham]

On 2019-09-16 21:54, Paul Dupuis via use-livecode wrote:
> IF sort lines of <var> ascending text was working correctly for lines
> of mixed ASCII and Unicode, for someone sorting lines of text that can
> be Native text, Unicode text (both RTL and LTR), or mixtures of both,
> is it better to use SORT ... TEXT or SORT ... INTERNATIONAL? I don't
> know enough about what the "international" (using the system locale
> settings)  and Unicode may mean in relation to one another?

You should use 'sort international' when you are displaying a sorted 
list
to a user who is looking through it manually.

The ordering provided by 'sort text' is purely by unicode codepoint, 
which
has no direct relation to 'expected' order when read by a human and 
instead
is determined by technical considerations (structuring a large 21-bit 
space,
frequency of use and, most importantly, round-tripping to legacy 
encodings and
standards).

The core of the sort order provided by 'international' sorting is the
Unicode Collation Algorithm - which provides (at its code) a 
locale-independent
order for all the languages/scripts present in Unicode. e.g. Latin 
European
languages are generally expected to come before Greek which is expected 
to
come before Cyrillic.

This core order is then tailored by locale to enable account to be taken
of the individual expectations of the user of the sorted list. For 
example,
different languages have different sort orders for what you might 
consider
the 'same letters' due to using the same glyphs. For example, a Swedish 
user
would expect 'z' to sort before 'ö'; whereas a German user would expect
'ö' to sort before 'z'.

The engine uses ICU's implementation of Unicode collation, and supports 
a wide
range of locales - the locale used is read on engine startup from the 
user's
system settings.

Hope this helps,

Mark.

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps