Commit Graph

146 Commits

Author SHA1 Message Date
Marco Lizza
a9b9c483cc feat: add bitwise shift and XOR operators to compiler tokenizer and parser
Add TOKEN_LEFT_SHIFT, TOKEN_RIGHT_SHIFT, and TOKEN_CARET to the token enum, implement lexing for '^' as single-char token and '<<'/'>>' as two-char tokens in nextToken(), and register them as infix operators at PREC_BITWISE precedence in the grammar rules table.
2015-02-16 10:28:34 +00:00
Bob Nystrom
8eecc5abe3 feat: add '\0' escape sequence support for null characters in string parsing
Add handling for the null character escape sequence '\0' in the readString function within wren_compiler.c. This extends the existing escape sequence handling to include the null character, allowing strings to contain embedded null bytes when explicitly escaped.
2015-02-04 14:57:42 +00:00
Marco Lizza
23dcd3feb3 refactor: replace strcpy/strncpy with memcpy for safer string handling across compiler, core, utils, value, and vm modules 2015-02-04 12:51:54 +00:00
Marco Lizza
ed38494eba feat: add '\0' escape sequence support in string literal parsing
Add handling for the null character escape sequence '\0' in the
readString function within the Wren compiler. This allows string
literals to include null bytes via the standard C escape syntax,
matching the existing support for other escape sequences like '\a',
'\b', and '\f'.
2015-02-04 12:25:25 +00:00
Bob Nystrom
bee26d2ab2 refactor: remove dedicated CODE_LIST bytecode in favor of new List instantiation with .add() calls
Eliminate the LIST opcode and its interpreter case, debug printing, and enum entry. Instead compile list literals by loading the List class, calling its constructor, then emitting DUP, element expression, CALL_1 for add(), and POP for each element. This removes interpreter loop code and lifts the previous 255-element limit on list literals.
2015-01-26 06:49:30 +00:00
Bob Nystrom
f1d2f2f9c1 fix: add explicit readNumber call for numeric literal tokens in nextToken 2015-01-26 01:51:50 +00:00
Bob Nystrom
7fd1751499 feat: add hex literal parsing with readHexDigit and readHexNumber in wren_compiler.c
Implement hexadecimal number literal support in the Wren compiler by adding a `number` field to the Parser struct, a `readHexDigit` helper that returns -1 for invalid hex chars, and a `readHexNumber` function that skips the 'x' prefix, accumulates hex digits, converts via strtol, and emits a TOKEN_NUMBER. Update `readNumber` to remove the TODO placeholder for hex support. Include test cases for valid hex literals (0xFF, 0x09, negative -0xFF) and an invalid hex literal (2xFF) that triggers a lex error.
2015-01-26 01:49:05 +00:00
Gavin Schulz
992a2aeb96 feat: add hexadecimal number literal parsing to wren compiler
Remove the TOKEN_HEXADECIMAL token type and isHexDigit helper, replacing them with a readHexDigit function and readHexNumber parser that directly converts hex literals to numeric values during lexing. Add a `number` field to the Parser struct to hold the parsed value, and include a test for invalid hex literal error handling.
2015-01-25 21:02:44 +00:00
Bob Nystrom
2ec0a8aef9 feat: implement map literal syntax with curly braces and DUP bytecode
Add support for map literals using `{}` syntax in the compiler, replacing the previous `new Map` constructor pattern. This introduces a new `map()` grammar function that loads the Map class, instantiates it, and compiles key-value pairs using subscript setter calls. A new `CODE_DUP` bytecode is added to duplicate the map reference on the stack for each element insertion. The change includes updated test files to use literal syntax and new error tests for edge cases like EOF after colon, comma, key, or value.
2015-01-25 07:21:50 +00:00
Gavin Schulz
b2c6dfce8d feat: add hexadecimal literal support to lexer and compiler
Add TOKEN_HEXADECIMAL token type and isHexDigit helper function to lexer.
Modify readNumber to detect 'x' prefix after '0' and lex hex digits.
Implement emitNumericConstant for hex values in compiler.
Add test file test/number/hex_literals.wren verifying basic hex parsing,
arithmetic, type checks, and negative hex literals.
2015-01-25 06:07:23 +00:00
Bob Nystrom
2fdb5cddd0 feat: add FLEXIBLE_ARRAY macro and update struct hack usage for C89/C99 portability 2015-01-23 18:38:12 +00:00
Bob Nystrom
2a123400b6 feat: add MSVC 2013 project files and fix C89 compatibility for Windows build
Add Visual Studio 2013 solution and project files under project/msvc2013/ directory, enabling native Windows builds without MinGW. Include Marco Lizza in AUTHORS for the contribution. Fix C89 compatibility issues for MSVC: disable computed goto via _MSC_VER check, define inline as _inline for non-C++ mode, change flexible array members from empty brackets to [0] syntax, and fix char to int type for readHexDigit return value. Refactor single-line if/while/for bodies in wren_compiler.c to use braces to satisfy MSVC stricter parsing.
2015-01-23 18:12:12 +00:00
Marco Lizza
915f403bc7 chore: restore standard header include order and move CRT secure warning to MSVC project
The diff reorders includes across multiple source files to place standard library headers before project headers, removes the `_CRT_SECURE_NO_WARNINGS` define from `wren_common.h`, and adds it as a preprocessor definition in both Debug and Release configurations of the MSVC 2013 project file.
2015-01-23 09:05:10 +00:00
Bob Nystrom
23c67c2e8b feat: remove UTF-8 encoding logic from addStringChar and accept raw bytes in string literals 2015-01-22 23:18:30 +00:00
Marco Lizza
c2f1a962d6 fix: replace WREN_PIN/WREN_UNPIN macros with wrenPushRoot/wrenPopRoot and fix modulo operator precedence 2015-01-22 09:13:16 +00:00
Evan Shaw
ffcdd8ae9e fix: correct operator precedence for % token from PREC_TERM to PREC_FACTOR
The modulo operator was incorrectly assigned PREC_TERM precedence, making it
evaluate at the same level as + and -. This caused expressions like "13 + 1 % 7"
to parse as "(13 + 1) % 7" instead of the correct "13 + (1 % 7)".

Changed the grammar rule for TOKEN_PERCENT to use PREC_FACTOR, matching the
precedence of * and / operators. Added a precedence test case in mod.wren to
verify the fix.
2015-01-21 20:49:43 +00:00
Bob Nystrom
6223bb2342 fix: replace broken WREN_PIN macro with proper wrenPushRoot/wrenPopRoot GC root stack 2015-01-21 15:56:06 +00:00
Marco Lizza
ff9625a80d chore: reorder include directives to place wren_common.h first across all source files
Enforce a consistent include ordering convention across the codebase: wren_common.h (as configuration header) first, followed by the module's own header, then other wren_*.h headers in lexicographic order, and finally standard library includes. Also add missing wren_common.h include to main.c and wren_io.c, and add _CRT_SECURE_NO_WARNINGS define for MSVC in wren_common.h.
2015-01-21 09:30:37 +00:00
Marco Lizza
a70a23e03d fix: avoid returning void function result in for/while/class/var statements 2015-01-21 09:23:46 +00:00
Marco Lizza
19862a2959 fix: change readHexDigit return type from char to int to prevent sign extension bugs 2015-01-21 09:22:56 +00:00
Bob Nystrom
a4baa42d6f feat: add user-defined subscript operators with bracket parameter lists
Refactor parameter list parsing to support bracket-delimited parameters for
subscript operators. Extract finishParameterList from parameterList to handle
the common closing logic, and add TOKEN_RIGHT_BRACKET case for error messages.
Add comprehensive test suite for subscript getters and setters with 1-3
parameters in test/method/subscript_operators.wren.
2015-01-21 02:25:54 +00:00
Bob Nystrom
f99f61376d refactor: remove unused return values from consume() and consumeLine() in wren_compiler.c
The consume() function previously returned a pointer to the consumed token,
and consumeLine() returned a boolean indicating success. Both return values
were never used by any callers. This change converts both functions to void,
eliminating dead code and simplifying the interface.
2015-01-20 22:40:34 +00:00
Bob Nystrom
850d6d1b5d fix: raise precedence of is above equality operators in parser
The `is` operator was placed at `PREC_IS` which had lower precedence than `PREC_EQUALITY`, causing expressions like `true == 10 is Num` to parse incorrectly. Moved `PREC_IS` above `PREC_EQUALITY` in the precedence enum to give `is` higher binding power than `==` and `!=`. Added precedence tests in new `test/precedence.wren` file and removed TODO comments about precedence from `test/is/is.wren`.
2015-01-18 18:20:13 +00:00
Bob Nystrom
ba81988800 fix: cast malloc result and adjust compiler emit functions for C++ compatibility
- Cast malloc return to char* in src/main.c to avoid C++ implicit void* conversion
- Rename emit to emitByteArg and emitShort to emitShortArg in src/wren_compiler.c to avoid conflict with new emit and emitShort helpers
- Add new emit function taking uint8_t byte and emitShort helper for 16-bit big-endian emission
- Remove stray semicolon in script/test.py
2015-01-15 23:59:14 +00:00
Bob Nystrom
ee239f2bb6 feat: add undefined sentinel value and implicit global declaration for forward references
Add a new singleton value `VAL_UNDEFINED` to represent globals that have been implicitly declared via forward reference but not yet explicitly defined. Introduce `wrenDeclareGlobal()` to create such entries in the global table, and modify `wrenDefineGlobal()` to check for and resolve undefined slots. Update the compiler's `findUpvalue()` to stop closure capture at method boundaries, preventing methods from closing over local variables. Refactor `wrenSymbolTableAdd()` to always add symbols without duplicate checking, moving that logic to callers. Adjust error messages in the compiler to handle EOF tokens and improve newline formatting. Fix the delta_blue benchmark by renaming the global `planner` variable to `ThePlanner` to avoid shadowing the forward-declared function.
2015-01-15 07:08:34 +00:00
Bob Nystrom
72291f8b74 feat: implicitly declare nonlocal names as globals for forward references
Add support for forward references to top-level variables by implicitly
declaring unresolved capitalized names as globals during compilation.
If a real definition is not found later, a compile-time error is raised.
This enables mutual recursion at the top level and fixes issues #101 and #106.

Introduce an `UNDEFINED` singleton value to mark implicitly declared globals
that have not yet been explicitly defined. Update `wrenDeclareGlobal` to add
such entries, and modify `wrenDefineGlobal` to check for and replace undefined
placeholders. Adjust the symbol table API to allow duplicate additions for
implicit declarations. Add test cases for mutual recursion, forward references
in functions and methods, and undefined variable errors.
2015-01-15 07:08:25 +00:00
Bob Nystrom
82ef82cdb9 fix: prevent redundant keyword checks after first match in readName
Replace sequential `if` statements with `else if` chains in `readName` so that once a keyword matches (e.g. "break"), the remaining keyword checks are skipped, fixing issue #110 where later keywords could incorrectly override an already-matched token type.
2015-01-14 14:46:02 +00:00
Bob Nystrom
54ef9bbc73 feat: stop closure upvalue search at method boundaries and treat capitalized names as globals
Modify `findUpvalue` in the compiler to return -1 when the enclosing function is a method, preventing closures from capturing outer local variables. Add a new `nonlocal` variable resolution path that treats capitalized identifiers as global variables rather than method receivers. Update existing test files to use capitalized global names and add new test cases for nonlocal assignment, duplicate nonlocal declarations, block scoping, and method-local variable shadowing.
2015-01-14 05:36:42 +00:00
Bob Nystrom
a2c7499e0a fix: add tab character to whitespace skipping in nextToken parser
The nextToken function in wren_compiler.c previously only skipped spaces and carriage returns when advancing past whitespace, causing tab characters to be treated as non-whitespace tokens. This change adds '\t' to the whitespace check in both the initial case statement and the peekChar loop, ensuring tabs are properly skipped like other whitespace characters. A new test file test/whitespace.wren is added to verify correct handling of tabs in various indentation patterns and inline code positions.
2015-01-12 04:01:30 +00:00
Bob Nystrom
cd6d6c1f91 style: reformat while-loop body in nextToken to use braces 2015-01-12 03:52:59 +00:00
Bob Nystrom
34d3a8edb7 fix: open source files in binary mode and skip CR in whitespace
Change `fopen` mode from `"r"` to `"rb"` in `readFile()` to prevent CRLF-to-LF conversion on Windows, and add `'\r'` as a skipped whitespace character in `nextToken()` to handle carriage returns embedded in source files. This fixes parsing errors when Wren source files use Windows-style line endings.
2015-01-12 03:52:08 +00:00
Bob Nystrom
4e461f0b0c chore: clarify comment for super outside method error handling path 2015-01-12 03:44:35 +00:00
Kyle Marek-Spartz
1175d93efa fix: allow compiler to continue after super used outside method body
Previously, when `super` was used at the top level or outside any method, the compiler would error but still attempt to read `enclosingClass->methodLength` and `enclosingClass->methodName` from a NULL pointer, leading to undefined behavior. This restructures the error handling in `super_()` to always call `loadThis()` and guard the unnamed super call path with a NULL check on `enclosingClass`, setting an empty method signature when no enclosing class exists. Two additional test cases for `super.foo("bar")` and `super("foo")` are added to verify the compiler continues past the error without crashing.
2015-01-11 23:43:51 +00:00
Kyle Marek-Spartz
dd0760a54c fix: guard super() compilation against null enclosingClass to prevent segfault
When `super` is called at top-level with no arguments, `enclosingClass` is null, causing a segfault when trying to access its method name. This fix wraps the unnamed super call logic inside an else branch that only executes when `enclosingClass` is non-null, and adds a test case for the unnamed `super` call at top-level.
2015-01-09 21:14:06 +00:00
Evan Shaw
0526cee655 fix: stop lexing negative numbers as single tokens to fix parsing of subtraction
Previously, the lexer greedily consumed a `-` followed by a digit as a single negative number token, causing "1 -1" to be lexed as two number tokens instead of a minus operator and a positive literal. This broke parsing of subtraction expressions without spaces.

Now `-` always emits a TOKEN_MINUS, and the parser handles unary negation. This is a breaking change: `-16.sqrt` is now parsed as `-(16.sqrt)` instead of `(-16).sqrt`, requiring parentheses for negative method receivers. All number method tests (abs, ceil, floor, sqrt, toString) are updated to use explicit parentheses for negative receivers.
2015-01-09 07:06:50 +00:00
Yury Korchemkin
c769f93833 fix: treat carriage return as whitespace instead of line token in lexer 2015-01-06 02:33:49 +00:00
Yury Korchemkin
a137ea8ccd feat: add support for carriage return as line terminator in lexer
- Open source files in binary mode to prevent OS-level newline translation
- Treat '\r' character as equivalent to '\n' in the tokenizer
- Handle '\r' followed by '\n' as a single line break via twoCharToken
2015-01-04 22:25:17 +00:00
Bob Nystrom
b61e55d4c9 fix: skip duplicate lexer error reporting in compiler error handler
The `error()` function in the compiler now returns early when the offending token
is of type `TOKEN_ERROR`, preventing the lexer's already-reported error from
being emitted a second time to stderr. This eliminates redundant error messages
for malformed tokens.
2015-01-04 21:10:46 +00:00
Bob Nystrom
96d893a637 feat: ignore shebang lines starting with #! on first line of source
Add special handling in the lexer's nextToken function to detect a '#' character followed by '!' on the first line (currentLine == 1) and skip the entire line as a comment. If the shebang appears on any other line, or if a '#' is not followed by '!', emit a lex error for invalid character. Include test cases for valid shebang execution, shebang at EOF, shebang on a non-first line (expected error), and an invalid shebang pattern (expected error).
2015-01-04 21:02:42 +00:00
Bob Nystrom
ed31544ead fix: handle empty and comment-only files by removing early skip in repl and compiler
Remove the empty-line check in runRepl that skipped interpretation for blank input, and refactor the main compilation loop in wrenCompile to use a while loop with TOKEN_EOF matching instead of a for loop with redundant EOF break. This allows the compiler to process files containing only comments or whitespace without prematurely terminating, fixing issue #57. Add test fixtures for comment-only source files.
2015-01-04 21:00:53 +00:00
mauvm
6a69382275 refactor: inline unmatchedChar logic into nextToken and add shebang error test 2015-01-04 18:56:16 +00:00
mauvm
685f791fb0 fix: remove duplicate comment in unmatchedChar function in wren_compiler.c 2015-01-04 07:52:40 +00:00
mauvm
9828fafee7 feat: ignore shebang lines starting with #! on first line in compiler lexer 2015-01-03 19:53:12 +00:00
Bob Nystrom
1951059c8f feat: remove semicolon token from lexer and treat it as invalid syntax
Remove the `;` case from the `nextToken` function in `wren_compiler.c` so that semicolons are no longer lexed as `TOKEN_LINE`. Update the documentation in `doc/site/syntax.markdown` to describe newlines as the sole statement separator, and adjust all test files (`test/limit/many_constants.wren`, `test/limit/many_globals.wren`, `test/limit/too_many_constants.wren`, `test/semicolon.wren`) to reflect that semicolons now produce an error instead of being treated as line breaks.
2015-01-03 18:02:45 +00:00
Aleksander
d67036b95d fix: extend whitespace skipping in nextToken to handle tabs and add test coverage 2015-01-03 13:57:34 +00:00
Aleksander
4236d1e2de fix: treat tab character as valid whitespace in tokenizer
Add explicit case for '\t' alongside space character in the nextToken function's whitespace-skipping logic. Previously, only spaces were consumed during whitespace skipping, causing tab characters to be incorrectly treated as unexpected tokens. The fix ensures tabs are properly skipped by first matching the tab case and then falling through to the existing space-handling loop.
2015-01-02 22:22:09 +00:00
Bob Nystrom
520e841044 feat: add specialized LOAD_LOCAL_0..8 instructions to skip arg byte decode
Introduce nine new bytecode instructions (CODE_LOAD_LOCAL_0 through CODE_LOAD_LOCAL_8) that directly encode the local slot number in the opcode itself, eliminating the need to read and decode a separate argument byte for the most frequently accessed local variables. This optimization targets the hottest path in the interpreter, as LOAD_LOCAL is the most executed instruction across benchmarks (e.g., 3.75M times in delta_blue). The compiler's `loadLocal` helper emits the specialized opcode when the slot index is ≤8, falling back to the generic `CODE_LOAD_LOCAL` with an argument byte otherwise. The VM dispatch table and debug printer are updated accordingly. Benchmarks show consistent speedups: fib +7.3%, method_call +5.9%, binary_trees +1.9%, for +2.8%, delta_blue +1.0%.
2014-12-07 03:59:11 +00:00
Bob Nystrom
9b498b4638 fix: ensure all switch and unreachable paths return a value in release builds
In `getNumArguments`, add a `return 0` after the `UNREACHABLE()` default case to prevent
undefined behavior when assertions are disabled. In `wrenGetClass`, reorder the
`UNREACHABLE()` macro before the `return NULL` so the unreachable annotation is
always emitted. In `runInterpreter`, replace the `ASSERT(0, ...)` with an explicit
`UNREACHABLE()` and `return false` to guarantee a return value on all control paths
when the interpreter exits unexpectedly.
2014-12-02 05:36:06 +00:00
Bob Nystrom
3266491cf5 fix: correct break statement instruction size from 2 to 3 in endLoop to prevent code walker from misinterpreting arguments 2014-11-27 02:33:54 +00:00
Bob Nystrom
d9a42f21ba fix: add missing -lm link flag and fix GCC warnings for getline and switch coverage
Add -lm flag to both debug and release linker commands in Makefile to resolve
undefined math library references. Define _GNU_SOURCE in main.c to make getline()
available under GCC. Add default cases to switch statements in runFile and
getNumArguments to suppress -Wswitch warnings. Fix u_int8_t to uint8_t type in
wrenNewFunction declaration and definition for POSIX compliance.
2014-11-26 22:03:05 +00:00