fix: add tab character to whitespace skipping in nextToken parser

The nextToken function in wren_compiler.c previously only skipped spaces and carriage returns when advancing past whitespace, causing tab characters to be treated as non-whitespace tokens. This change adds '\t' to the whitespace check in both the initial case statement and the peekChar loop, ensuring tabs are properly skipped like other whitespace characters. A new test file test/whitespace.wren is added to verify correct handling of tabs in various indentation patterns and inline code positions.
This commit is contained in:
Bob Nystrom
2015-01-12 04:01:30 +00:00
2 changed files with 30 additions and 1 deletions
+26
View File
@@ -0,0 +1,26 @@
// No indent
IO.print("ok") // expect: ok
// Indent with space
IO.print("ok") // expect: ok
// Indent with tab
IO.print("ok") // expect: ok
// Indent with space then tab
IO.print("ok") // expect: ok
// Indent with tab then space
IO.print("ok") // expect: ok
// Indent with mixed tab and space
IO.print("ok") // expect: ok
// Space in with code
IO . print ( "ok" ) // expect: ok
// Tab in with code
IO . print ( "ok" ) // expect: ok
// Tab and space mixed in with code
IO . print ( "ok" ) // expect: ok