fix: set token type to TOKEN_ERROR after invalid character to prevent compiler loop
When the lexer encounters an invalid character, it now explicitly sets the current token's type to TOKEN_ERROR and its length to 0. Previously, the token retained the previous token's type, which could cause the compiler to get stuck in an infinite loop in certain code paths. Also improves the error message for non-ASCII invalid bytes by showing the raw hex value instead of attempting to display the character, since the lexer operates on raw bytes without UTF-8 decoding. Adds regression test for issue #428 that was crashing the compiler with an out-of-bounds memory access due to this loop.
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
// This was crashing the compiler with an out of bounds memory access.
|
||||
|
||||
// expect error line 6
|
||||
// expect error line 7
|
||||
Fiber.new {
|
||||
isDone ["", àààààààààààààààààààààààààààààààààààààààààààààààààà
|
||||
Reference in New Issue
Block a user