Nimcheck

A multi-language source code validation framework written in Nim. Nimcheck uses tokenizer-based validation to check syntax correctness across 10 languages and config formats, with auto-detection, comprehensive diagnostics, and a never-crash guarantee.

Features

• 10 languages supported: Nim, Bash, Python, JavaScript, PHP, HTML, Jinja, JSON, YAML, TOML
• Tokenizer-based validation -- real tokenizers that track bracket depth, string context, comments, and nesting
• Auto-detection -- detect language from file extension or content analysis
• Comprehensive diagnostics -- structured error reports with position, severity, hints, and debug output
• Never-crash guarantee -- every validation path is wrapped in try/except; invalid input returns errors gracefully, never panics
• JSON output -- machine-readable diagnostics via toJson() on any validation result
• Inspection API -- extract module structure (imports, definitions, functions/classes) as JSON
• Debug mode -- compile with -d:nimcheckDebug for detailed tokenization logging
• No external dependencies -- pure Nim standard library

Quick Start

import nimcheck

# Validate source code from a string
let r = validateSource("proc hello() = echo 'hi'", lfNim)
echo r.errors  # seq[ValidationError]

# Auto-detect language
let r2 = validateSource("print('hello')", lfUnknown)  # detects Python

# Validate a file
let r3 = validateFile("path/to/file.nim", lfNim)

# Get JSON diagnostics
echo r.toJson().pretty()

# Inspect source structure (returns JsonNode)
let info = inspectSource("class Foo: ...", lfPython)
echo info.pretty()  # formatted JSON with class/function info

# Get human-readable report
let report = reportSource("proc foo() =", lfNim)
echo report  # formatted error summary

Building

Using Makefile (recommended)

# Build the library binary
make build

# Run all tests
make test

# Build with debug mode
make build-debug
make debug       # build debug + run all tests

# Run a specific language test
make test-nim
make test-python
make test-bash

# Lint with full warnings
make lint

# Clean build artifacts
make clean

# Show all available targets
make help

Using nimble

# Run all tests
nimble test

# Run per-language tests
nimble test_nim
nimble test_python

# Build the library
nimble build

Manual compilation

# Build the library
nim c src/nimcheck.nim

# Build with debug mode
nim c -d:nimcheckDebug src/nimcheck.nim

# Run all tests
nim c -r tests/test_all.nim

CLI Usage

When compiled as a standalone binary (make build produces bin/nimcheck):

# Validate a file
bin/nimcheck validate --file=source.nim

# Validate source code directly
bin/nimcheck validate --source="print('hi')" --flavor=python

# Read from stdin
echo "var x = 1" | bin/nimcheck validate

# Inspect source structure (returns JSON)
bin/nimcheck inspect --file=source.py

# Detect language
bin/nimcheck detect --file=unknown.txt
bin/nimcheck detect --source="#!/usr/bin/env bash"

# Version and help
bin/nimcheck version
bin/nimcheck help

API Reference

Core Functions

Function	Description
validateSource(source, flavor, options, sourcePath)	Validate source code string; returns ValidationResult
validateFile(filePath, flavor, options)	Validate a file; returns ValidationResult
inspectSource(source, flavor, options, sourcePath)	Return structural diagnostics as JsonNode
inspectFile(filePath, flavor, options)	Return structural diagnostics for a file as JsonNode
validateSourceJson(source, flavor)	Validate source and return result as JSON string
validateFileJson(filePath, flavor)	Validate file and return result as JSON string
reportSource(source, flavor)	Get a human-readable validation report (string)
reportFile(filePath, flavor)	Get a human-readable validation report for a file (string)
detectFlavor(source, filePath)	Auto-detect language flavor
supportedFlavors()	List all registered language flavors (returns seq[string])
version()	Get the Nimcheck version string
enableDebug() / disableDebug()	Toggle debug mode globally

Types

• ValidationResult -- contains errors: seq[ValidationError], valid: bool, moduleInfo, debugOutput, detectionInfo
• ValidationError -- contains code, message, severity, position, hint, context
• ValidationOptions -- contains flavor, debugMode, maxErrors, validateAll
• LanguageFlavor -- enum: lfUnknown, lfNim, lfBash, lfPython, lfJavaScript, lfPHP, lfHTML, lfJinja, lfJSON, lfYAML, lfTOML

JSON Output

Both ValidationResult and ValidationError have a .toJson() method returning a JsonNode. Use .pretty() for formatted output.

Error Codes

Code	Meaning
E1000	Unexpected/unmatched bracket
E1001	Unclosed bracket
E1002	Unmatched closing bracket
E2000	Unclosed string literal
E2001	Invalid string escape
E2002	Unclosed multi-line string
E3000	Unclosed comment
E3001	Invalid comment syntax
E4000	Invalid syntax
E4001	Reserved word misuse
E5000	Unclosed block/control structure
E9999	Internal error (never-crash fallback)

Supported Languages

Language	File Extensions	Tokenizer Features
Nim	.nim, .nims, .nimble	Block comments #[ ]# with nesting, triple-quoted strings, raw strings, doc comments
Bash	.sh, .bash	Shebang detection, heredocs, command substitution, variable expansion, fi/done/esac matching
Python	.py	F-strings with nesting, triple-quoted strings, decorators, async/await, match statements
JavaScript	.js, .jsx, .mjs, .cjs	Template literals with nesting, regex literals, arrow functions, ESM imports
PHP	.php	<?php tags, heredocs/nowdocs, variable variables
HTML	.html, .htm	Tag balancing, self-closing tags, attributes
Jinja	.j2, .jinja, .html.j2	{% %}, {{ }}, {# #} block detection, nested blocks
JSON	.json	Bracket balance, string validation, comma checking
YAML	.yaml, .yml	Indentation tracking, key-value checking
TOML	.toml	Table headers, bracket balance, string validation

Testing

The test suite covers 250+ test cases across all supported languages:

• Per-language tests -- valid and invalid source for each of the 10 languages
• Exhaustive tests -- edge cases: unclosed strings, unclosed comments, mismatched brackets, binary data, unicode bombs
• Fuzz tests -- null bytes, binary data, deep nesting (100 levels), empty sources, concurrent validation
• Config format tests -- JSON, YAML, TOML validation with invalid syntax checks
• Mixed-file tests -- HTML+Jinja combined files
• Never-crash guarantee -- every test verifies invalid input returns graceful errors, not crashes
• Debug mode tests -- make debug runs all tests with debug logging enabled

Run all tests with:

make test         # or
nimble test       # or
nim c -r tests/test_all.nim

Development

Prerequisites

• Nim >= 2.0.0
• make (optional, for Makefile targets)

Project structure

src/
  nimcheck.nim          Public API entry point and CLI (isMainModule)
  nimcheck/
    core/
      types.nim           Core types: ValidationResult, ValidationError, LanguageFlavor
      tokenizerbase.nim   Abstract tokenizer base class
      validatorbase.nim   Abstract validator base class
      detector.nim        Language auto-detection
      debug.nim           Debug logging infrastructure
    tokenizers/
      nim_tokenizer.nim
      bash_tokenizer.nim
      python_tokenizer.nim
      javascript_tokenizer.nim
      php_tokenizer.nim
      html_tokenizer.nim
      jinja_tokenizer.nim
    languages/
      nim_validator.nim
      bash_validator.nim
      python_validator.nim
      javascript_validator.nim
      php_validator.nim
      html_validator.nim
      jinja_validator.nim
      config_validators.nim   JSON, YAML, TOML validators
    reporting/
      errors.nim          Error formatting and JSON output
tests/
  test_all.nim            Master test runner (imports all test modules)
  test_<lang>.nim         Per-language tests
  test_<lang>_exhaustive.nim  Exhaustive edge-case tests per language
  test_config.nim         JSON/YAML/TOML tests
  test_fuzz.nim           Fuzz and boundary tests
  test_common.nim         Shared test utilities
  test_mixed.nim          Mixed-language file tests
  fixtures/               Test data files per language

Building with warnings

make lint
# or manually:
nim c --hints:on --warnings:on src/nimcheck.nim

Adding a new language

See the detailed checklist in LESSONS_LEARNED.md (Section 14).

Architecture

The framework uses a layered architecture:

1. Tokenizers -- convert raw source into a stream of tokens, tracking bracket depth, string context, and comments
2. Validators -- analyze token streams for language-specific syntax rules
3. Detector -- identifies language from file extension, shebang, or content heuristics
4. Reporting -- formats errors as human-readable strings or structured JSON

Validators register themselves at import time via a side-effect factory pattern. The public API (nimcheck.nim) imports all validators and re-exports core types for consumer convenience.

Versioning

Current version: 0.1.0

Contributing

Contributions are welcome. Please open an issue to discuss your idea before submitting a pull request. For detailed instructions on adding support for a new language, see LESSONS_LEARNED.md Section 16.

License

This project is licensed under the MIT License -- see the LICENSE file for details.

Author

molodetz