refactor: move newline handling from lexer into parser grammar rules

This change shifts newline tokenization from the lexer's readRawToken into the parser's nextToken, allowing the grammar to selectively ignore newlines in specific contexts (e.g., after operators like "|", "is", "for", "if", "while") while treating them as errors in others (e.g., after "var", "new", "class", "static", before commas in lists). The lexer now emits TOKEN_LINE for semicolons and newlines uniformly, and the parser decides which to elide. Test files are updated to reflect the new error expectations and to remove tests for newlines that are now grammatically allowed.

This commit is contained in:

Bob Nystrom

2014-04-04 14:51:18 +00:00

parent a1365c4183

commit 093da7befb

23 changed files with 151 additions and 165 deletions

test/while/newline_after_while.wren

View File

@@ -0,0 +1,2 @@
 while // expect error
 (true) IO.print("bad")