feat: skip UTF-8 BOM in source input and add regression test

Add handling for UTF-8 byte order mark (BOM) at the start of source code
in wrenCompile function by checking for the three-byte sequence \xEF\xBB\xBF
and advancing the source pointer past it. Include a regression test file
(test/regression/520.wren) that contains a BOM to verify the fix works
correctly.
This commit is contained in:
Bob Nystrom
2018-04-28 17:00:59 +00:00
2 changed files with 9 additions and 1 deletions
+7 -1
View File
@@ -3420,6 +3420,12 @@ void definition(Compiler* compiler)
ObjFn* wrenCompile(WrenVM* vm, ObjModule* module, const char* source,
bool isExpression, bool printErrors)
{
// Skip potential UTF-8 BOM
if (strncmp(source, "\xEF\xBB\xBF", 3) == 0)
{
source += 3;
}
Parser parser;
parser.vm = vm;
parser.module = module;
@@ -3447,7 +3453,7 @@ ObjFn* wrenCompile(WrenVM* vm, ObjModule* module, const char* source,
nextToken(&parser);
int numExistingVariables = module->variables.count;
Compiler compiler;
initCompiler(&compiler, &parser, NULL, false);
ignoreNewlines(&compiler);
+2
View File
@@ -0,0 +1,2 @@
// This file should have a UTF-8 byte order mark
System.print("ok") // expect: ok