feat: implement UTF-8 aware string subscript ranges with new wrenNewStringFromRange helper

Add wrenNewStringFromRange function to wren_value.c that decodes UTF-8 code points from source string and re-encodes them into result, enabling correct range subscripts on multi-byte characters. Rename wrenUtf8NumBytes to wrenUtf8EncodeNumBytes and make wrenUtf8Encode return written byte count for use in range construction. Add wrenUtf8DecodeNumBytes utility to skip continuation bytes when iterating by byte index. Remove all skip directives from test files and add UTF-8 specific test cases verifying correct slicing across code point boundaries.
This commit is contained in:
Bob Nystrom
2015-09-02 05:14:55 +00:00
parent ef93496fd4
commit a619851cf2
15 changed files with 96 additions and 51 deletions
+13 -1
View File
@@ -1,4 +1,3 @@
// skip: Range subscripts for strings don't handle UTF-8.
var string = "abcde"
IO.print(string[0..0]) // expect: a
IO.print(string[1...1] == "") // expect: true
@@ -33,3 +32,16 @@ IO.print(string[3...-6]) // expect: dcba
// An empty range at zero is allowed on an empty string.
IO.print(""[0...0] == "") // expect: true
IO.print(""[0..-1] == "") // expect: true
// Indexes by byte, not code point.
//
// Bytes: 11111
// 012345678901234
// Chars: sø mé ஃ thî ng
IO.print("søméஃthîng"[0..3]) // expect: søm
IO.print("søméஃthîng"[3...10]) // expect: méஃt
// Only includes sequences whose first byte is in the range.
IO.print("søméஃthîng"[2..6]) // expect: méஃ
IO.print("søméஃthîng"[2...6]) // expect: mé
IO.print("søméஃthîng"[2...7]) // expect: méஃ