fix: treat tab character as valid whitespace in tokenizer
Add explicit case for '\t' alongside space character in the nextToken function's whitespace-skipping logic. Previously, only spaces were consumed during whitespace skipping, causing tab characters to be incorrectly treated as unexpected tokens. The fix ensures tabs are properly skipped by first matching the tab case and then falling through to the existing space-handling loop.
This commit is contained in:
@@ -789,6 +789,7 @@ static void nextToken(Parser* parser)
|
||||
makeToken(parser, TOKEN_LINE);
|
||||
return;
|
||||
|
||||
case '\t':
|
||||
case ' ':
|
||||
// Skip forward until we run out of whitespace.
|
||||
while (peekChar(parser) == ' ') nextChar(parser);
|
||||
|
||||
Reference in New Issue
Block a user