2026-07-08 10:01:58 +02:00
|
|
|
## Fuzz testing module -- Nimcheck against random, binary, and malicious inputs.
|
feat: add initial validatrix project with multi-language validator framework
Establish the Validatrix source code validation framework in Nim, including core detection, tokenization, and validation infrastructure for 10 languages (bash, HTML, JavaScript, Jinja, JSON, Nim, PHP, Python, TOML, YAML). The initial commit introduces the main validatrix.nim entry point with public API (validateSource, validateFile, inspectSource), a comprehensive test suite with per-language test files, build configuration via Makefile and .nimble, detailed LESSONS_LEARNED.md documenting 14 bug classes found during development, and a .gitignore excluding compiled binaries, logs, and build artifacts. The detector module provides extension-to-flavor mapping for 40+ file extensions and content-based language detection, while the debug module supports conditional compilation with -d:validatrixDebug for detailed tokenization logging.
2026-07-08 06:25:59 +02:00
|
|
|
## Auto-generated. Tests framework robustness, never-crash guarantee.
|
feat: add extended language validators and fix tokenizer crash bugs
Expand nimcheck with validators and tokenizers for C, C++, C#, Go, Rust,
Ruby, CSS, SQL, Markdown, Dockerfile, Makefile, Kotlin, Lua, Swift,
TypeScript, and XML. Register all flavors in the validator factory and
improve auto-detection scoring for the new languages.
Fix infinite tokenizer loops that caused OOM kills: closeBracket now
advances position, finishTokenizeStep guards stalled tokenization, and
Jinja/JS tokenizers no longer double-advance on brackets.
Fix block-balance false positives in Lua (for/do) and Ruby (postfix
unless), SQL trailing-comma detection across whitespace, and Makefile
tab literals in test fixtures.
2026-07-15 06:49:45 +02:00
|
|
|
## Includes false-positive prevention tests.
|
feat: add initial validatrix project with multi-language validator framework
Establish the Validatrix source code validation framework in Nim, including core detection, tokenization, and validation infrastructure for 10 languages (bash, HTML, JavaScript, Jinja, JSON, Nim, PHP, Python, TOML, YAML). The initial commit introduces the main validatrix.nim entry point with public API (validateSource, validateFile, inspectSource), a comprehensive test suite with per-language test files, build configuration via Makefile and .nimble, detailed LESSONS_LEARNED.md documenting 14 bug classes found during development, and a .gitignore excluding compiled binaries, logs, and build artifacts. The detector module provides extension-to-flavor mapping for 40+ file extensions and content-based language detection, while the debug module supports conditional compilation with -d:validatrixDebug for detailed tokenization logging.
2026-07-08 06:25:59 +02:00
|
|
|
|
|
|
|
|
import std/[unittest, strutils, strformat, math, sequtils]
|
2026-07-08 10:01:58 +02:00
|
|
|
import ../src/nimcheck
|
feat: add initial validatrix project with multi-language validator framework
Establish the Validatrix source code validation framework in Nim, including core detection, tokenization, and validation infrastructure for 10 languages (bash, HTML, JavaScript, Jinja, JSON, Nim, PHP, Python, TOML, YAML). The initial commit introduces the main validatrix.nim entry point with public API (validateSource, validateFile, inspectSource), a comprehensive test suite with per-language test files, build configuration via Makefile and .nimble, detailed LESSONS_LEARNED.md documenting 14 bug classes found during development, and a .gitignore excluding compiled binaries, logs, and build artifacts. The detector module provides extension-to-flavor mapping for 40+ file extensions and content-based language detection, while the debug module supports conditional compilation with -d:validatrixDebug for detailed tokenization logging.
2026-07-08 06:25:59 +02:00
|
|
|
|
feat: add extended language validators and fix tokenizer crash bugs
Expand nimcheck with validators and tokenizers for C, C++, C#, Go, Rust,
Ruby, CSS, SQL, Markdown, Dockerfile, Makefile, Kotlin, Lua, Swift,
TypeScript, and XML. Register all flavors in the validator factory and
improve auto-detection scoring for the new languages.
Fix infinite tokenizer loops that caused OOM kills: closeBracket now
advances position, finishTokenizeStep guards stalled tokenization, and
Jinja/JS tokenizers no longer double-advance on brackets.
Fix block-balance false positives in Lua (for/do) and Ruby (postfix
unless), SQL trailing-comma detection across whitespace, and Makefile
tab literals in test fixtures.
2026-07-15 06:49:45 +02:00
|
|
|
const allLangs* = [
|
|
|
|
|
lfNim, lfBash, lfPython, lfJavaScript, lfTypeScript,
|
|
|
|
|
lfPHP, lfHTML, lfXML, lfJinja, lfJSON, lfYAML, lfTOML,
|
|
|
|
|
lfC, lfCpp, lfJava, lfCSharp, lfKotlin, lfLua, lfSwift,
|
|
|
|
|
lfGo, lfRust, lfRuby, lfCSS, lfSQL, lfMarkdown,
|
|
|
|
|
lfDockerfile, lfMakefile
|
|
|
|
|
]
|
|
|
|
|
|
feat: add initial validatrix project with multi-language validator framework
Establish the Validatrix source code validation framework in Nim, including core detection, tokenization, and validation infrastructure for 10 languages (bash, HTML, JavaScript, Jinja, JSON, Nim, PHP, Python, TOML, YAML). The initial commit introduces the main validatrix.nim entry point with public API (validateSource, validateFile, inspectSource), a comprehensive test suite with per-language test files, build configuration via Makefile and .nimble, detailed LESSONS_LEARNED.md documenting 14 bug classes found during development, and a .gitignore excluding compiled binaries, logs, and build artifacts. The detector module provides extension-to-flavor mapping for 40+ file extensions and content-based language detection, while the debug module supports conditional compilation with -d:validatrixDebug for detailed tokenization logging.
2026-07-08 06:25:59 +02:00
|
|
|
const threeLangs* = [lfNim, lfPython, lfJSON]
|
|
|
|
|
|
|
|
|
|
suite "Fuzz and Robustness Tests":
|
|
|
|
|
|
|
|
|
|
test "null bytes in every language":
|
|
|
|
|
let src = "\x00\x01\x02\x03hello\x00world"
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "very long strings (10k chars)":
|
|
|
|
|
let src = "x = \"" & repeat('A', 1000) & "\""
|
|
|
|
|
for lang in [lfNim, lfPython, lfJavaScript]:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "deep bracket nesting (100 levels)":
|
|
|
|
|
let src = "let x = " & repeat('(', 100) & "42" & repeat(')', 100)
|
|
|
|
|
let result = validateSource(src, flavor = lfJavaScript)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "unclosed strings at end of file (every lang)":
|
|
|
|
|
let src = "x = \"unclosed"
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "empty source never crashes":
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let result = validateSource("", flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "whitespace-only source":
|
|
|
|
|
let src = " \t \n \t \n "
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "raw binary data never crashes":
|
|
|
|
|
let src = "\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0A\x0B\x0C\x0D\x0E\x0F\x10ABCDEFGH\x1F\x20\x7F\x80\xFF"
|
|
|
|
|
for lang in threeLangs:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "unicode BOM and weird encodings":
|
|
|
|
|
let src = "\xEF\xBB\xBF\xF0\x9F\x98\x80\xE2\x82\xAC\xC0\xAF"
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "high unicode (surrogate pairs)":
|
|
|
|
|
let src = "let x = '\U0001F600\U0001F601\U0001F602'"
|
|
|
|
|
let result = validateSource(src, flavor = lfJavaScript)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "mixed encoding attack":
|
|
|
|
|
let src = "proc hello() =\x80\x81\x82\x83\n discard\n"
|
|
|
|
|
let result = validateSource(src, flavor = lfNim)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "alternating open/close brackets with garbage":
|
|
|
|
|
let src = "({[({[({[({[)}])}])}])}"
|
|
|
|
|
let result = validateSource(src, flavor = lfJavaScript)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "inspectSource never crashes on any input":
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let json = inspectSource("garbage!@#$%^&*()_+", flavor = lang)
|
|
|
|
|
check json.kind == JObject
|
|
|
|
|
|
|
|
|
|
test "version and supportedFlavors":
|
|
|
|
|
check version().len > 0
|
feat: add extended language validators and fix tokenizer crash bugs
Expand nimcheck with validators and tokenizers for C, C++, C#, Go, Rust,
Ruby, CSS, SQL, Markdown, Dockerfile, Makefile, Kotlin, Lua, Swift,
TypeScript, and XML. Register all flavors in the validator factory and
improve auto-detection scoring for the new languages.
Fix infinite tokenizer loops that caused OOM kills: closeBracket now
advances position, finishTokenizeStep guards stalled tokenization, and
Jinja/JS tokenizers no longer double-advance on brackets.
Fix block-balance false positives in Lua (for/do) and Ruby (postfix
unless), SQL trailing-comma detection across whitespace, and Makefile
tab literals in test fixtures.
2026-07-15 06:49:45 +02:00
|
|
|
check supportedFlavors().len >= 28
|
feat: add initial validatrix project with multi-language validator framework
Establish the Validatrix source code validation framework in Nim, including core detection, tokenization, and validation infrastructure for 10 languages (bash, HTML, JavaScript, Jinja, JSON, Nim, PHP, Python, TOML, YAML). The initial commit introduces the main validatrix.nim entry point with public API (validateSource, validateFile, inspectSource), a comprehensive test suite with per-language test files, build configuration via Makefile and .nimble, detailed LESSONS_LEARNED.md documenting 14 bug classes found during development, and a .gitignore excluding compiled binaries, logs, and build artifacts. The detector module provides extension-to-flavor mapping for 40+ file extensions and content-based language detection, while the debug module supports conditional compilation with -d:validatrixDebug for detailed tokenization logging.
2026-07-08 06:25:59 +02:00
|
|
|
|
|
|
|
|
test "detectFlavor on all good code samples":
|
|
|
|
|
discard detectFlavor("proc x() = discard")
|
|
|
|
|
discard detectFlavor("echo hello")
|
|
|
|
|
discard detectFlavor("def x(): pass")
|
|
|
|
|
discard detectFlavor("function x() {}")
|
|
|
|
|
discard detectFlavor("<?php echo 'hi';")
|
|
|
|
|
discard detectFlavor("<html></html>")
|
|
|
|
|
discard detectFlavor("{% block x %}{% endblock %}")
|
|
|
|
|
discard detectFlavor("{\"a\": 1}")
|
|
|
|
|
discard detectFlavor("a: 1")
|
|
|
|
|
discard detectFlavor("x = 1")
|
feat: add extended language validators and fix tokenizer crash bugs
Expand nimcheck with validators and tokenizers for C, C++, C#, Go, Rust,
Ruby, CSS, SQL, Markdown, Dockerfile, Makefile, Kotlin, Lua, Swift,
TypeScript, and XML. Register all flavors in the validator factory and
improve auto-detection scoring for the new languages.
Fix infinite tokenizer loops that caused OOM kills: closeBracket now
advances position, finishTokenizeStep guards stalled tokenization, and
Jinja/JS tokenizers no longer double-advance on brackets.
Fix block-balance false positives in Lua (for/do) and Ruby (postfix
unless), SQL trailing-comma detection across whitespace, and Makefile
tab literals in test fixtures.
2026-07-15 06:49:45 +02:00
|
|
|
discard detectFlavor("#include <stdio.h>")
|
|
|
|
|
discard detectFlavor("package main")
|
|
|
|
|
discard detectFlavor("fn main() {}")
|
|
|
|
|
discard detectFlavor("fun main() {}")
|
|
|
|
|
discard detectFlavor("import Foundation")
|
|
|
|
|
discard detectFlavor("<?xml version=\"1.0\"?>")
|
|
|
|
|
discard detectFlavor("interface Foo {}")
|
feat: add initial validatrix project with multi-language validator framework
Establish the Validatrix source code validation framework in Nim, including core detection, tokenization, and validation infrastructure for 10 languages (bash, HTML, JavaScript, Jinja, JSON, Nim, PHP, Python, TOML, YAML). The initial commit introduces the main validatrix.nim entry point with public API (validateSource, validateFile, inspectSource), a comprehensive test suite with per-language test files, build configuration via Makefile and .nimble, detailed LESSONS_LEARNED.md documenting 14 bug classes found during development, and a .gitignore excluding compiled binaries, logs, and build artifacts. The detector module provides extension-to-flavor mapping for 40+ file extensions and content-based language detection, while the debug module supports conditional compilation with -d:validatrixDebug for detailed tokenization logging.
2026-07-08 06:25:59 +02:00
|
|
|
|
|
|
|
|
test "source with only newlines":
|
|
|
|
|
let src = "\n\n\n\n\n\n\n\n\n\n"
|
|
|
|
|
for lang in allLangs:
|
|
|
|
|
let result = validateSource(src, flavor = lang)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "very deep comment nesting":
|
|
|
|
|
let src = "proc test() =\n " & repeat("#[ ", 20) & " content " & repeat(" ]# ", 20) & "\n discard\n"
|
|
|
|
|
let result = validateSource(src, flavor = lfNim)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "concurrent validation calls":
|
|
|
|
|
for i in 0..10:
|
|
|
|
|
let src = "proc test" & $i & "() = discard\n"
|
|
|
|
|
let result = validateSource(src, flavor = lfNim)
|
|
|
|
|
check result.errors.len >= 0
|
|
|
|
|
|
|
|
|
|
test "detectFlavor unknown garbage":
|
|
|
|
|
let flavor = detectFlavor("!@#$%^&*()_+{}[]|\\:;\"'<>,.?/~`")
|
|
|
|
|
check $flavor == "unknown" or $flavor == "bash"
|
|
|
|
|
|
|
|
|
|
test "validateFile with nonexistent path":
|
|
|
|
|
let result = validateFile("/nonexistent/path/file.nim")
|
|
|
|
|
check result.errors.len > 0
|
|
|
|
|
check not result.valid
|
feat: add extended language validators and fix tokenizer crash bugs
Expand nimcheck with validators and tokenizers for C, C++, C#, Go, Rust,
Ruby, CSS, SQL, Markdown, Dockerfile, Makefile, Kotlin, Lua, Swift,
TypeScript, and XML. Register all flavors in the validator factory and
improve auto-detection scoring for the new languages.
Fix infinite tokenizer loops that caused OOM kills: closeBracket now
advances position, finishTokenizeStep guards stalled tokenization, and
Jinja/JS tokenizers no longer double-advance on brackets.
Fix block-balance false positives in Lua (for/do) and Ruby (postfix
unless), SQL trailing-comma detection across whitespace, and Makefile
tab literals in test fixtures.
2026-07-15 06:49:45 +02:00
|
|
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
# False Positive Prevention Tests
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
test "false positive: short random text returns unknown":
|
|
|
|
|
let flavor = detectFlavor("abc def ghi")
|
|
|
|
|
check $flavor == "unknown"
|
|
|
|
|
|
|
|
|
|
test "false positive: single token returns unknown":
|
|
|
|
|
let flavor = detectFlavor("hello")
|
|
|
|
|
check $flavor == "unknown"
|
|
|
|
|
|
|
|
|
|
test "false positive: numbers-only returns unknown":
|
|
|
|
|
let flavor = detectFlavor("42 123 456 7890")
|
|
|
|
|
check $flavor == "unknown"
|
|
|
|
|
|
|
|
|
|
test "false positive: punctuation-only returns unknown":
|
|
|
|
|
let flavor = detectFlavor("... --- !!! ??? >>> <<<")
|
|
|
|
|
check $flavor == "unknown"
|
|
|
|
|
|
|
|
|
|
test "false positive: binary-like data returns unknown":
|
|
|
|
|
let flavor = detectFlavor("\x00\x01\x02\xFF\xFE\xFD\x00\x01\x02\x03")
|
|
|
|
|
check $flavor == "unknown"
|