fix: set token type to TOKEN_ERROR after invalid character to prevent compiler loop

When the lexer encounters an invalid character, it now explicitly sets the
current token's type to TOKEN_ERROR and its length to 0. Previously, the
token retained the previous token's type, which could cause the compiler
to get stuck in an infinite loop in certain code paths.

Also improves the error message for non-ASCII invalid bytes by showing
the raw hex value instead of attempting to display the character, since
the lexer operates on raw bytes without UTF-8 decoding.

Adds regression test for issue #428 that was crashing the compiler with
an out-of-bounds memory access due to this loop.
This commit is contained in:
Bob Nystrom
2017-03-24 04:19:20 +00:00
parent 9b0d2aefb6
commit 184126372c
2 changed files with 21 additions and 5 deletions
+6
View File
@@ -0,0 +1,6 @@
// This was crashing the compiler with an out of bounds memory access.
// expect error line 6
// expect error line 7
Fiber.new {
isDone ["", àààààààààààààààààààààààààààààààààààààààààààààààààà