All source listed below is under MIT license if no LICENSE file stating different is available.

Nimcheck

Universal source code validation framework written in Nim. Tokenizer-based syntax validation across 10 languages and config formats, with auto-detection, structured JSON diagnostics, structural inspection, and a never-crash guarantee.

Version: 0.1.0 | Author: molodetz | License: MIT | Requires: Nim >= 2.0.0

Architecture
Quick Start
API Reference
Data Types
JSON Output Structures
Error Code Reference
IO Table: Supported Languages and Per-Language Errors
CLI Usage
Building and Testing
Project Structure
Contributing

Architecture

Nimcheck uses a four-layer pipeline architecture. Each layer is independent and communicates through well-defined interfaces.

Source Code (string or file)
        |
        v
+-------------------+
|     Detector      |  LanguageFlavor from extension, shebang, or content analysis
+-------------------+
        |
        v
+-------------------+
|    Tokenizer      |  Converts raw source into a seq[Token] stream
+-------------------+
        |
        v
+-------------------+
|    Validator      |  Analyzes token stream for syntax errors and structure
+-------------------+
        |
        v
+-------------------+
|    Reporting      |  Formats errors as human-readable text or structured JSON
+-------------------+
        |
        v
  ValidationResult

Layer 1: Detector (`src/nimcheck/core/detector.nim`)

Language auto-detection with cascading priority:

Priority	Method	Source	Confidence
1 (highest)	Explicit	User-provided `LanguageFlavor`	1.0
2	File extension	70+ extension-to-flavor mappings	0.9
3	Shebang line	`#!` line interpreter matching (bash, python, node, ruby, lua, swift)	0.95
4	Content analysis	Keyword scoring, structural patterns, bracket usage	0.6-0.9

The detectFlavor() function returns a (LanguageFlavor, FlavorDetectionMethod, confidence: float) tuple.

Layer 2: Tokenizer (`src/nimcheck/core/tokenizerbase.nim`, per-language impls)

Each tokenizer converts raw source into a seq[Token]. Tokenizers track:

Position: line, column, and byte offset for every token
Bracket stack: (, [, {, < pushed and popped with position tracking
String context: escape sequences, multi-line delimiters, interpolation
Comment context: line comments, block comments with nesting support

Tokenizers emit errors for:

Unclosed strings (ErrUnclosedString)
Unclosed comments (ErrUnclosedComment)
Unexpected characters (ErrUnexpectedToken)
Unclosed block constructs (language-specific, e.g., ${}, $())

Layer 3: Validator (`src/nimcheck/core/validatorbase.nim`, per-language impls)

Each validator:

Creates its tokenizer and runs tokenize()
Collects tokenizer errors
Runs analyzeTokens() - language-specific structural analysis
Runs buildModuleInfo() - populates ModuleInfo with imports, functions, classes
Computes validity, warning/info counts, duration

Validators register themselves via a factory pattern at import time. The registry maps flavor strings (e.g., "python", "bash") to constructors.

Layer 4: Reporting (`src/nimcheck/reporting/errors.nim`)

Produces:

Human-readable reports: formatReport() with position, severity, code, message, source context with carets, and hints
JSON output: every type has a toJson() method producing JsonNode
Error categorization: countBySeverity(), countByCode(), filterBySeverity()

Quick Start

Nim library usage

import nimcheck

# Validate source code from a string
let r = validateSource("proc hello() = echo 'hi'", lfNim)
echo r.valid           # bool
echo r.errors          # seq[ValidationError]
echo r.toJson().pretty()  # Full JSON output

# Validate a file (auto-detects language from extension)
let r2 = validateFile("src/main.nim")

# Auto-detect language
let r3 = validateSource("print('hello')", lfUnknown)
echo r3.flavor         # lfPython

# Inspect source structure (returns JsonNode with functions, classes, imports)
let info = inspectSource("class Foo:\n    def bar(self): pass", lfPython)
echo info.pretty()

# Human-readable report
echo reportSource("proc foo() =\n  let x = \"unclosed", lfNim)

CLI usage

# Build the binary
make build

# Validate a file
bin/nimcheck validate --file=source.nim

# Validate source directly
bin/nimcheck validate --source="print('hi')" --flavor=python

# Read from stdin
echo "var x = 1" | bin/nimcheck validate

# Inspect structure (JSON output)
bin/nimcheck inspect --file=source.py

# Detect language
bin/nimcheck detect --file=unknown.txt
bin/nimcheck detect --source="#!/usr/bin/env bash"

# Version and help
bin/nimcheck version
bin/nimcheck help

API Reference

Core Functions

All functions are exported from nimcheck.nim. Import once with import nimcheck.

Validation

Function	Signature	Returns	Description
`validateSource`	`(source: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions(), sourcePath: string = "")`	`ValidationResult`	Validate source code string. Auto-detects flavor if `lfUnknown`.
`validateFile`	`(filePath: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions())`	`ValidationResult`	Validate a file from disk. Auto-detects flavor from file extension.
`validateSourceJson`	`(source: string, flavor: LanguageFlavor = lfUnknown)`	`string`	Validate and return pretty-printed JSON string.
`validateFileJson`	`(filePath: string, flavor: LanguageFlavor = lfUnknown)`	`string`	Validate file and return pretty-printed JSON string.

Inspection

Function	Signature	Returns	Description
`inspectSource`	`(source: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions(), sourcePath: string = "")`	`JsonNode`	Validate and return structural diagnostics (functions, classes, imports).
`inspectFile`	`(filePath: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions())`	`JsonNode`	Inspect a file and return structural diagnostics.

Reporting

Function	Signature	Returns	Description
`reportSource`	`(source: string, flavor: LanguageFlavor = lfUnknown)`	`string`	Human-readable validation report for source code.
`reportFile`	`(filePath: string, flavor: LanguageFlavor = lfUnknown)`	`string`	Human-readable validation report for a file.

Detection and Utility

Function	Signature	Returns	Description
`detectFlavor`	`(source: string, filePath: string = "")`	`LanguageFlavor`	Auto-detect language flavor (convenience wrapper).
`supportedFlavors`	`()`	`seq[string]`	List all registered language flavors sorted alphabetically.
`version`	`()`	`string`	Version string (`"0.1.0"`).
`enableDebug`	`()`	`void`	Enable debug mode globally.
`disableDebug`	`()`	`void`	Disable debug mode globally.

Core Detector (lower-level)

Function	Signature	Returns	Description
`detectFlavor` (core)	`(source: string, filePath: string = "", explicitFlavor: LanguageFlavor = lfUnknown)`	`(LanguageFlavor, FlavorDetectionMethod, float)`	Full detection with confidence score.

Factory

Function	Signature	Returns	Description
`createValidator`	`(flavor: LanguageFlavor, source: string, options: ValidationOptions, sourcePath: string)`	`ValidatorBase`	Factory: create the correct validator for a flavor.
`registerValidator`	`(flavor: string, constructor: ValidatorConstructor)`	`void`	Register a new validator constructor.

Data Types

LanguageFlavor (enum, pure)

All 28 flavors defined in src/nimcheck/core/types.nim. Flavors with validators (bold) are fully supported; the rest are detectable but have no dedicated validator.

Enum Value	String	Validator	File Extensions (auto-detect)
`lfUnknown`	`"unknown"`	No	--
`lfNim`	`"nim"`	NimValidator	`.nim`, `.nims`, `.nimble`
`lfPython`	`"python"`	PythonValidator	`.py`, `.pyw`, `.pyx`, `.pxd`
`lfBash`	`"bash"`	BashValidator	`.sh`, `.bash`
`lfShell`	`"shell"`	BashValidator (shared)	`.zsh`, `.fish`
`lfJavaScript`	`"javascript"`	JavaScriptValidator	`.js`, `.mjs`, `.cjs`
`lfTypeScript`	`"typescript"`	JavaScriptValidator (shared)	`.ts`, `.tsx`
`lfPHP`	`"php"`	PHPValidator	`.php`, `.phtml`
`lfHTML`	`"html"`	HTMLValidator	`.html`, `.htm`, `.xhtml`
`lfXML`	`"xml"`	HTMLValidator (shared)	`.xml`, `.svg`
`lfJinja`	`"jinja"`	JinjaValidator	`.jinja`, `.jinja2`, `.j2`
`lfJSON`	`"json"`	JSONValidator	`.json`, `.jsonc`
`lfYAML`	`"yaml"`	YAMLValidator	`.yaml`, `.yml`
`lfTOML`	`"toml"`	TOMLValidator	`.toml`
`lfCSS`	`"css"`	No	`.css`
`lfSQL`	`"sql"`	No	`.sql`
`lfMarkdown`	`"markdown"`	No	`.md`, `.markdown`
`lfDockerfile`	`"dockerfile"`	No	`Dockerfile`
`lfMakefile`	`"makefile"`	No	`Makefile`, `makefile`
`lfRuby`	`"ruby"`	No	`.rb`
`lfRust`	`"rust"`	No	`.rs`
`lfGo`	`"go"`	No	`.go`
`lfLua`	`"lua"`	No	`.lua`
`lfC`	`"c"`	No	`.c`, `.h`
`lfCpp`	`"cpp"`	No	`.cpp`, `.cxx`, `.hpp`
`lfCSharp`	`"csharp"`	No	`.cs`
`lfJava`	`"java"`	No	`.java`
`lfSwift`	`"swift"`	No	`.swift`
`lfKotlin`	`"kotlin"`	No	`.kt`, `.kts`

FlavorDetectionMethod (enum)

Value	Meaning
`fdmExplicit`	User explicitly provided the flavor
`fdmExtension`	Detected from file extension
`fdmShebang`	Detected from `#!` shebang line
`fdmContent`	Detected via content keyword analysis
`fdmUnknown`	Could not determine

TokenKind (enum)

Lexical token classifications used by all tokenizers.

Kind	Description
`tkWhitespace`	Spaces, tabs, carriage returns
`tkNewline`	`\n`
`tkComment`	Line (`#`, `//`) or block (`/* */`) comment
`tkDocComment`	Documentation comment (`##`, `///`, `"""`)
`tkString`	Single or double quoted string literal
`tkRawString`	Raw string (`r"..."`)
`tkMultilineString`	Triple-quoted string (`"""..."""`)
`tkNumber`	Integer, float, hex, octal, binary
`tkIdentifier`	Variable, function, type name
`tkKeyword`	Language-specific reserved word
`tkOperator`	`+`, `-`, `*`, `/`, `&&`, etc.
`tkAssignment`	`=`, `:=`, `+=`, etc.
`tkPunctuation`	`.`, `,`, `;`, `:`
`kOpenParen`	`(`
`kCloseParen`	`)`
`kOpenBracket`	`[`
`kCloseBracket`	`]`
`kOpenBrace`	`{`
`kCloseBrace`	`}`
`kAngleOpen`	`<`
`kAngleClose`	`>`
`tkInterpolation`	String interpolation (`${var}`, f-string `{expr}`)
`tkDirective`	Preprocessor / compiler directive (`#include`, shebang)
`tkTemplateTag`	Template language tag (`{{ }}`, `{% %}`)
`tkSpecial`	Language-specific special token
`tkError`	Malformed or unrecognizable character
`tkEndOfFile`	End of input marker

Token

Token = object
  kind: TokenKind
  value: string
  position: SourcePosition
  range: SourceRange

SourcePosition

SourcePosition = object
  line: int      # 1-indexed line number
  column: int    # 1-indexed column number
  offset: int    # 0-indexed byte offset from start

SourceRange

SourceRange = object
  startPos: SourcePosition
  endPos: SourcePosition

ErrorSeverity (enum)

Value	String	Description
`esInfo`	`"info"`	Informational note
`esWarning`	`"warning"`	Non-fatal issue
`esError`	`"error"`	Validation failure
`esCritical`	`"critical"`	Internal framework failure

ValidationError

ValidationError = object
  severity: ErrorSeverity
  message: string
  code: string           # Error code, e.g. "E0002"
  position: SourcePosition
  range: SourceRange
  context: string        # Surrounding source context (optional)
  hint: string           # Fix suggestion (optional)

ValidationResult

ValidationResult = object
  valid: bool
  flavor: LanguageFlavor
  detectionMethod: FlavorDetectionMethod
  errors: seq[ValidationError]
  warnings: int
  infos: int
  moduleInfo: ModuleInfo
  durationMs: float
  debugOutput: string     # Empty unless compiled with -d:nimcheckDebug

ValidationOptions

ValidationOptions = object
  flavor: LanguageFlavor        # lfUnknown = auto-detect (default)
  debugMode: bool               # Enable per-validation debug logging (default: false)
  strictMode: bool              # Warnings become errors (default: false)
  maxErrors: int                # Stop after N errors; 0 = unlimited (default: 0)
  includeContext: bool          # Include surrounding source lines in error output (default: true)
  includeHints: bool            # Include fix suggestions (default: true)

Structural Types (populated by `analyzeTokens()`)

VariableInfo = object
  name: string
  kind: string             # "var", "let", "const", "global", "local"
  typeAnnotation: string
  defaultValue: string
  position: SourcePosition
  mutable: bool
  exported: bool

FunctionInfo = object
  name: string
  kind: string             # "function", "method", "proc", "func", "lambda"
  parameters: seq[ParameterInfo]
  returnType: string
  isAsync: bool
  isExported: bool
  isGenerator: bool
  visibility: string       # "public", "private", "protected"
  position: SourcePosition
  docComment: string

ParameterInfo = object
  name: string
  typeAnnotation: string
  defaultValue: string
  isOptional: bool
  isVarargs: bool

ClassInfo = object
  name: string
  kind: string             # "class", "object", "struct", "interface"
  baseTypes: seq[string]
  fields: seq[VariableInfo]
  methods: seq[FunctionInfo]
  generics: seq[string]
  visibility: string
  position: SourcePosition
  docComment: string

ImportInfo = object
  module: string
  symbols: seq[string]     # Specific imported symbols
  isRelative: bool
  alias: string
  position: SourcePosition

ModuleInfo = object
  flavor: LanguageFlavor
  detectionMethod: FlavorDetectionMethod
  imports: seq[ImportInfo]
  variables: seq[VariableInfo]
  functions: seq[FunctionInfo]
  classes: seq[ClassInfo]
  linesOfCode: int
  totalTokens: int
  comments: int
  debugInfo: JsonNode      # Language-specific extra info

JSON Output Structures

Every type has a .toJson() method. The ValidationResult.toJson() aggregates all nested structures.

ValidationResult JSON

{
  "valid": true,
  "flavor": "python",
  "detectionMethod": "by_extension",
  "errors": [
    {
      "severity": "error",
      "message": "Unclosed string literal starting with '\"'",
      "code": "E0002",
      "position": {"line": 2, "column": 13, "offset": 42},
      "range": {
        "start": {"line": 2, "column": 13, "offset": 42},
        "end":   {"line": 2, "column": 13, "offset": 42}
      },
      "context": "",
      "hint": "Add a closing quote before end of file"
    }
  ],
  "warnings": 0,
  "infos": 0,
  "moduleInfo": {
    "flavor": "python",
    "detectionMethod": "by_extension",
    "imports": [],
    "variables": [],
    "functions": [
      {
        "name": "hello",
        "kind": "def",
        "parameters": [],
        "returnType": "",
        "isAsync": false,
        "isExported": false,
        "isGenerator": false,
        "visibility": "",
        "position": {"line": 1, "column": 5, "offset": 4},
        "docComment": ""
      }
    ],
    "classes": [],
    "linesOfCode": 3,
    "totalTokens": 12,
    "comments": 0,
    "debugInfo": null
  },
  "durationMs": 0.342,
  "debugOutput": ""
}

ValidationError JSON

{
  "severity": "error",
  "message": "Unclosed string literal starting with '\"'",
  "code": "E0002",
  "position": {"line": 2, "column": 13, "offset": 42},
  "range": {
    "start": {"line": 2, "column": 13, "offset": 42},
    "end":   {"line": 2, "column": 13, "offset": 42}
  },
  "context": "",
  "hint": "Add a closing quote before end of file"
}

ModuleInfo JSON

{
  "flavor": "nim",
  "detectionMethod": "by_extension",
  "imports": [
    {
      "module": "strutils",
      "symbols": [],
      "isRelative": false,
      "alias": "",
      "position": {"line": 1, "column": 8, "offset": 7}
    }
  ],
  "variables": [],
  "functions": [
    {
      "name": "add",
      "kind": "proc",
      "parameters": [],
      "returnType": "",
      "isAsync": false,
      "isExported": false,
      "isGenerator": false,
      "visibility": "",
      "position": {"line": 3, "column": 6, "offset": 28},
      "docComment": ""
    }
  ],
  "classes": [
    {
      "name": "MyType",
      "kind": "type",
      "baseTypes": [],
      "fields": [],
      "methods": [],
      "generics": [],
      "visibility": "",
      "position": {"line": 5, "column": 6, "offset": 52},
      "docComment": ""
    }
  ],
  "linesOfCode": 8,
  "totalTokens": 45,
  "comments": 2,
  "debugInfo": null
}

Human-Readable Report Format

=== Validation Report ===
  Flavor: python
  Detection: by_extension
  Valid: false
  Errors: 1
  Warnings: 0
  Duration: 0.34ms
  Lines of code: 3
  Tokens: 12

  --- Findings ---
  [E0002] (error) L2:13: Unclosed string literal starting with '"'
         |     x = "
         |            ^
         hint: Add a closing quote before end of file

  --- Summary ---
  INVALID - source has issues that need attention.

Error Formatter Output (single error)

  [E0002] (error) L2:13: Unclosed string literal starting with '"'
         |     x = "
         |            ^
         hint: Add a closing quote before end of file

Error Code Reference

Error codes follow a prefixed numbering scheme. The prefix encodes the category and severity.

General Errors (E0xxx)

Code	Constant	Description	Triggered By
`E0001`	`ErrUnexpectedToken`	Unexpected or unrecognizable character	Tokenizer encounter of non-ASCII, non-operator char
`E0002`	`ErrUnclosedString`	Unclosed string literal	Missing closing quote at EOF
`E0003`	`ErrUnclosedComment`	Unclosed block comment	Missing `*/`, `]#`, `#}` etc. at EOF
`E0004`	`ErrUnclosedBlock`	Unclosed block/control structure	Language-specific: unclosed `if`/`for`/`class`/etc.
`E0005`	`ErrMismatchedBracket`	Mismatched or unexpected closing bracket	`)` without `(`, `</div>` without `<div>`
`E0006`	`ErrUnclosedBracket`	Unclosed bracket at EOF	`validateBrackets()` finds remaining items on bracket stack
`E0007`	`ErrInvalidSyntax`	General invalid syntax	JSON parse error, TOML missing `=`, YAML empty key
`E0008`	`ErrUnexpectedEOF`	Unexpected end of file	Reserved for incomplete constructs not covered by other codes
`E0009`	`ErrEmptySource`	Source is empty (zero-length string)	`validate()` with `source.len == 0`
`E0010`	`ErrEncodingError`	Encoding error	Reserved for non-UTF-8 input
`E0011`	--	File not found	`validateFile()` with non-existent path

Language-Specific Errors (E1xxx)

Code	Constant	Description	Languages
`E1001`	`ErrUnexpectedIndent`	Unexpected or inconsistent indentation	Python
`E1002`	`ErrMissingIndent`	Missing indentation after block opener	Python
`E1003`	`ErrDuplicateParam`	Duplicate parameter name	(reserved)
`E1004`	`ErrUndefinedVariable`	Reference to undefined variable	(reserved)
`E1005`	`ErrTypeMismatch`	Type mismatch	(reserved)
`E1006`	`ErrInvalidAssignment`	Invalid assignment target	(reserved)
`E1007`	`ErrMissingReturn`	Function missing return statement	(reserved)
`E1008`	`ErrInvalidDecorator`	Invalid decorator usage	Python
`E1009`	`ErrInvalidImport`	Invalid import statement	(reserved)

Template Errors (E2xxx)

Code	Constant	Description	Languages
`E2001`	`ErrUnclosedTemplateTag`	Unclosed template expression/block tag	Jinja
`E2002`	`ErrUnclosedBlockTag`	Unclosed block tag with no matching end tag	Jinja
`E2003`	`ErrInvalidFilter`	Invalid filter in template expression	Jinja

Info Codes (I0xxx)

Code	Constant	Description
`I0001`	`InfoLongLine`	Line exceeds recommended length
`I0002`	`InfoTrailingWhitespace`	Trailing whitespace detected

Warning Codes (W0xxx / W1xxx)

Code	Constant	Description
`W0001`	`WarnUnusedVariable`	Variable declared but never used
`W0002`	`WarnInconsistentNaming`	Naming convention inconsistency
`W0003`	`WarnMissingDoc`	Public symbol missing documentation
`W1001`	--	Tab characters in YAML indentation

Internal Error

Code	Description	Triggered By
`E9999`	Internal framework error (never-crash fallback)	Any unhandled `Exception` in `validate()` or `validateSource()`

IO Table: Supported Languages and Per-Language Errors

Input/Output Contract

Every validation function follows the same contract:

Input	Output
`source: string` (required)	`ValidationResult` with `valid: bool`
`flavor: LanguageFlavor = lfUnknown`	`ValidationResult.errors: seq[ValidationError]`
`options: ValidationOptions` (optional)	`ValidationResult.moduleInfo: ModuleInfo`
`sourcePath: string = ""` (optional)	`ValidationResult.durationMs: float`

Guarantees:

Never throws - all exceptions are caught and returned as E9999 errors
Empty source returns E0009 (or valid: true for YAML/TOML where empty is valid)
File not found returns E0011
Unknown flavor with no detection possible returns an E0007 error
Every error has a code, message, severity, position, and optional hint

Per-Language Tokenizer, Validator, and Error Table

Nim (`.nim`, `.nims`, `.nimble`)

Aspect	Details
Tokenizer	`nim_tokenizer.nim` - `NimTokenizer`
Validator	`nim_validator.nim` - `NimValidator`
Keywords	80+ (addr, and, as, block, break, case, const, converter, defer, discard, elif, else, enum, except, export, finally, for, from, func, if, import, in, include, iterator, let, macro, method, mixin, mod, nil, not, object, of, or, proc, ptr, raise, ref, return, template, try, tuple, type, using, var, when, while, with, yield, etc.)

Error Code	Condition	Severity	Hint
`E0001`	Unexpected character (non-ASCII, unrecognizable)	`esWarning`	Shows hex `U+XXXX` value
`E0002`	Unclosed string (`"..."` at EOF)	`esError`	"Add a closing quote"
`E0002`	Unclosed multi-line string (`"""` at EOF)	`esError`	--
`E0002`	Newline inside string literal	`esError`	--
`E0002`	Unclosed raw string (`r"..."` at EOF)	`esError`	--
`E0002`	Unclosed character literal (`'x` at EOF)	`esError`	--
`E0003`	Unclosed block comment (`#[` without `]#`)	`esError`	"Add closing `]#`"
`E0006`	Unclosed bracket (`(`, `[`, `{` at EOF)	`esError`	"Add matching closing bracket"

Structural extraction: imports (from X import Y), functions (proc, func, method, template, macro, iterator, converter), types (type Foo).

Bash / Shell (`.sh`, `.bash`, `.zsh`, `.fish`)

Aspect	Details
Tokenizer	`bash_tokenizer.nim` - `BashTokenizer`
Validator	`bash_validator.nim` - `BashValidator`
Keywords	38+ (if, then, else, elif, fi, case, esac, for, while, until, do, done, in, function, select, time, return, break, continue, exit, declare, local, export, readonly, unset, eval, exec, let, source, shift, trap, type, typeset, ulimit, umask, alias, read, echo)

Error Code	Condition	Severity	Hint
`E0002`	Unclosed single-quoted string	`esError`	--
`E0002`	Unclosed double-quoted string	`esError`	--
`E0004`	Unclosed `${var}` expansion	`esError`	--
`E0004`	Unclosed `$()` command substitution	`esError`	--
`E0004`	Unclosed `if` block (`if` without matching `fi`)	`esError`	"Add missing 'fi'"
`E0004`	Unclosed loop (`for`/`while`/`until` without `done`)	`esError`	"Add missing 'done'"
`E0004`	Unclosed `case` block (`case` without `esac`)	`esError`	"Add missing 'esac'"
`E0006`	Unclosed bracket (`(`, `[`, `{` at EOF)	`esError`	--

Special features: shebang detection (#!), heredoc detection (<<EOF), variable expansion ($var, ${var}), backtick command substitution, arithmetic expansion ((( ))).

Python (`.py`, `.pyw`, `.pyx`, `.pxd`)

Aspect	Details
Tokenizer	`python_tokenizer.nim` - `PythonTokenizer`
Validator	`python_validator.nim` - `PythonValidator`
Keywords	33 (False, None, True, and, as, assert, async, await, break, class, continue, def, del, elif, else, except, finally, for, from, global, if, import, in, is, lambda, nonlocal, not, or, pass, raise, return, try, while, with, yield)

Error Code	Condition	Severity	Hint
`E0002`	Unclosed string (`"..."` or `'...'` at EOF)	`esError`	--
`E0002`	Unclosed triple-quoted string (`"""` or `'''` at EOF)	`esError`	--
`E0002`	Unclosed f-string (`f"..."` at EOF)	`esError`	--
`E0002`	Newline inside string literal	`esError`	--
`E1001`	Indentation mismatch at block boundary	`esError`	--
`E0006`	Unclosed bracket (`(`, `[`, `{` at EOF)	`esError`	--

Special features: f-string interpolation with nested braces, indentation tracking (INDENT/DEDENT tokens), decorator detection (@), line continuation (\).

Structural extraction: imports (import X, from X import Y), functions (def), classes (class).

JavaScript / TypeScript (`.js`, `.mjs`, `.cjs`, `.ts`, `.tsx`)

Aspect	Details
Tokenizer	`javascript_tokenizer.nim` - `JavaScriptTokenizer`
Validator	`javascript_validator.nim` - `JavaScriptValidator`
Keywords	60+ (async, await, break, case, catch, class, const, continue, debugger, default, delete, do, else, export, extends, false, finally, for, function, if, import, in, instanceof, let, new, null, of, return, super, switch, static, this, throw, true, try, typeof, var, void, while, with, yield + TypeScript: interface, type, declare, module, namespace, abstract, readonly, as, any, boolean, number, string, never, keyof, infer, unknown)

Error Code	Condition	Severity	Hint
`E0002`	Unclosed string (`"..."` or `'...'` at EOF)	`esError`	--
`E0002`	Unclosed template literal (`...` at EOF)	`esError`	--
`E0002`	Newline inside string literal	`esError`	--
`E0006`	Unclosed bracket (`(`, `[`, `{` at EOF)	`esError`	--

Special features: template literals with ${} interpolation nesting, regex literal detection, arrow function detection (=>), ESM import/export.

Structural extraction: imports/exports, functions (function, async), classes.

PHP (`.php`, `.phtml`)

Aspect	Details
Tokenizer	`php_tokenizer.nim` - `PHPTokenizer`
Validator	`php_validator.nim` - `PHPValidator`
Mode switching	Tracks `inPhpMode` flag - toggles on `<?php`/`<?` and `?>`

Error Code	Condition	Severity	Hint
`E0002`	Unclosed string	`esError`	--
`E0006`	Unclosed bracket (`(`, `[`, `{` at EOF)	`esError`	--

Special features: PHP/HTML mode switching, <?php, <?=, ?> detection, variable names ($var, $obj->prop).

Structural extraction: functions (function, fn), classes (class), imports/requires (include, require, use, namespace).

HTML / XML (`.html`, `.htm`, `.xhtml`, `.xml`, `.svg`)

Aspect	Details
Tokenizer	`html_tokenizer.nim` - `HTMLTokenizer`
Validator	`html_validator.nim` - `HTMLValidator`
Void elements	14: area, base, br, col, embed, hr, img, input, link, meta, param, source, track, wbr

Error Code	Condition	Severity	Hint
`E0003`	Unclosed HTML comment (`<!--` without `-->`)	`esError`	--
`E0004`	Unclosed HTML tag (e.g., `<div>` without `</div>`)	`esError`	"Add closing `</tagname>`"
`E0005`	Unexpected closing tag (mismatched nesting)	`esWarning`	"Check that tags are properly nested"

Special features: tag stack balancing, void element awareness (no closing tag needed for self-closing elements), attribute parsing with quoted values, <!DOCTYPE>, <?xml?> directives.

Jinja2 (`.jinja`, `.jinja2`, `.j2`, `.html.j2`)

Aspect	Details
Tokenizer	`jinja_tokenizer.nim` - `JinjaTokenizer`
Validator	`jinja_validator.nim` - `JinjaValidator`

Error Code	Condition	Severity	Hint
`E0003`	Unclosed Jinja comment (`{#` without `#}`)	`esError`	--
`E2001`	Unclosed expression tag (`{{` without `}}`)	Implicit via tokenizer	--
`E2001`	Unclosed block tag (`{%` without `%}`)	Implicit via tokenizer	--
`E2002`	Unclosed `for` block (no matching `endfor`)	`esError`	"Add `{% endfor %}`"
`E2002`	Unclosed `if` block (no matching `endif`)	`esError`	"Add `{% endif %}`"
`E2002`	Unclosed `block` (no matching `endblock`)	`esError`	"Add `{% endblock %}`"
`E2002`	Unclosed `macro`, `filter`, `raw`, `autoescape`, `call`, `with`	`esError`	"Add matching `end{tag}`"

Special features: tracks 12 block tag types (for, if, block, macro, filter, raw, autoescape, call, with), validates matching end tags, handles content between template syntax and plain text.

JSON (`.json`, `.jsonc`)

Aspect	Details
Validator	`config_validators.nim` - `JSONValidator`
Method	Uses Nim's built-in `parseJson()` for full validation

Error Code	Condition	Severity	Hint
`E0007`	JSON parse error (any `JsonParsingError`)	`esError`	Error message from parser
`E0007`	Any other exception during parse	`esError`	"JSON error: {message}"
`E0009`	Empty JSON source	`esError`	--

YAML (`.yaml`, `.yml`)

Aspect	Details
Validator	`config_validators.nim` - `YAMLValidator`
Method	Line-by-line structural analysis

Error Code	Condition	Severity	Hint
`E0007`	Empty mapping key (no key before `:`)	`esError`	--
`E0007`	General YAML structural error	`esError`	--
`E0009`	No content after stripping comments/blanks	`esError`	--
`W1001`	Tab characters used for indentation	`esWarning`	"Use spaces for YAML indentation"

Note: Empty YAML (only whitespace/comments) is valid: true.

TOML (`.toml`)

Aspect	Details
Validator	`config_validators.nim` - `TOMLValidator`
Method	Line-by-line structural analysis

Error Code	Condition	Severity	Hint
`E0007`	Missing `=` in key-value pair	`esError`	"TOML uses `key = value` format"
`E0007`	Missing key before `=`	`esError`	--
`E0007`	General TOML structural error	`esError`	--
`E0009`	No content after stripping comments/blanks	`esError`	--

CLI Usage

Full CLI Reference

Nimcheck v0.1.0 - Universal Source Code Validation Framework

Usage:
  nimcheck validate [--file=<path> | --source=<code>] [--flavor=<lang>]
  nimcheck inspect  [--file=<path> | --source=<code>] [--flavor=<lang>]
  nimcheck detect   [--file=<path> | --source=<code>]
  nimcheck help

Options:
  --file=<path>     Source file to process
  --source=<code>   Source code string to process
  --flavor=<lang>   Language flavor (auto-detect if omitted)
  --debug           Enable debug mode
  --json            Output as JSON (default for inspect)

Supported flavors:
  - bash
  - html
  - jinja
  - jinja2
  - js
  - json
  - jsonc
  - nim
  - nimble
  - nims
  - php
  - phtml
  - python
  - py
  - shell
  - sh
  - toml
  - ts
  - typescript
  - yaml
  - yml

Exit Codes

Exit Code	Meaning
0	Validation successful (or help/version requested)
Non-zero (Nim default)	Runtime error

Note: The validate command always exits 0 on the Nim level; the valid field in the output indicates pass/fail. This is by design - Nimcheck never crashes.

Building and Testing

Prerequisites

Nim >= 2.0.0
make (optional for Makefile targets)

Build

# Using Makefile
make build                  # Release build -> bin/nimcheck
make build-debug            # Debug build with -d:nimcheckDebug

# Using nimble
nimble build

# Manual
nim c src/nimcheck.nim
nim c -d:nimcheckDebug src/nimcheck.nim   # With debug output

Test

The test suite covers 250+ test cases across all supported languages.

make test                   # Run all tests
make test-nim               # Nim-specific tests only
make test-python            # Python-specific tests only
make test-bash              # Bash-specific tests only
make test-js                # JavaScript/TypeScript tests
make test-php               # PHP tests
make test-html              # HTML/XML tests
make test-jinja             # Jinja2 tests
make test-json              # JSON tests
make test-yaml              # YAML tests
make test-toml              # TOML tests
make test-mixed             # Mixed-language file tests
make debug                  # Build debug + run all tests

Test Categories

Suite	Files	Coverage
Per-language basic	`test_<lang>.nim`	Valid + invalid source for each language
Per-language exhaustive	`test_<lang>_exhaustive.nim`	Edge cases: unclosed strings, unclosed comments, mismatched brackets, binary data, Unicode
Config formats	`test_config.nim` + `test_<json	yaml
Fuzz tests	`test_fuzz.nim`	Null bytes, binary data, deep nesting (100 levels), empty sources, concurrent validation
Mixed files	`test_mixed.nim`	HTML+Jinja combined, other mixed-format files
Common	`test_common.nim`	Shared test fixtures for valid and invalid code per language

Lint

make lint
# or
nim c --hints:on --warnings:on src/nimcheck.nim

Clean

make clean
make rebuild    # clean + build

Project Structure

.
|-- Makefile
|-- nimcheck.nimble
|-- README.md
|-- LICENSE
|-- bin/                          # Build output
|   `-- nimcheck                  # Compiled binary
|-- src/
|   |-- nimcheck.nim              # Public API entry point + CLI (isMainModule)
|   `-- nimcheck/
|       |-- core/
|       |   |-- types.nim         # All core types, toJson() methods, error code constants
|       |   |-- tokenizerbase.nim # Abstract TokenizerBase with position tracking, bracket stack, error recording
|       |   |-- validatorbase.nim # Abstract ValidatorBase with pipeline, factory, registry
|       |   |-- detector.nim      # Language auto-detection (extension, shebang, content scoring)
|       |   |-- debug.nim         # Debug logging (compiled only with -d:nimcheckDebug)
|       |   `-- init_validators.nim
|       |-- tokenizers/
|       |   |-- nim_tokenizer.nim       # Nim: """ strings, r"..." raw strings, #[ ]# nested comments, ## doc comments
|       |   |-- bash_tokenizer.nim      # Bash: heredocs, $var/${var}, $(), backticks, (( ))
|       |   |-- python_tokenizer.nim    # Python: f-strings, """ strings, decorators, indentation tracking
|       |   |-- javascript_tokenizer.nim # JS/TS: template literals, regex literals, arrow functions
|       |   |-- php_tokenizer.nim       # PHP: <?php/?> mode switching, $variables
|       |   |-- html_tokenizer.nim      # HTML/XML: tag balancing, void elements, attributes
|       |   `-- jinja_tokenizer.nim     # Jinja2: {{ }}, {% %}, {# #} block detection
|       |-- languages/
|       |   |-- nim_validator.nim       # Analyzes Nim: imports, procs, types
|       |   |-- bash_validator.nim      # Analyzes Bash: if/fi, for/done, case/esac matching
|       |   |-- python_validator.nim    # Analyzes Python: imports, def, class
|       |   |-- javascript_validator.nim # Analyzes JS/TS: imports, functions, classes
|       |   |-- php_validator.nim       # Analyzes PHP: functions, classes, includes
|       |   |-- html_validator.nim      # Analyzes HTML/XML: tag balancing
|       |   |-- jinja_validator.nim     # Analyzes Jinja2: block tag matching (12 tag types)
|       |   `-- config_validators.nim   # JSON (via parseJson), YAML, TOML validators
|       `-- reporting/
|           `-- errors.nim       # Human-readable and JSON error formatting
`-- tests/
    |-- test_all.nim             # Master test runner (imports all)
    |-- test_nim.nim             # Nim basic tests
    |-- test_nim_exhaustive.nim  # Nim edge cases
    |-- test_bash.nim
    |-- test_bash_exhaustive.nim
    |-- test_python.nim
    |-- test_python_exhaustive.nim
    |-- test_javascript.nim
    |-- test_javascript_exhaustive.nim
    |-- test_php.nim
    |-- test_php_exhaustive.nim
    |-- test_html.nim
    |-- test_html_exhaustive.nim
    |-- test_jinja.nim
    |-- test_jinja_exhaustive.nim
    |-- test_json_exhaustive.nim
    |-- test_yaml_exhaustive.nim
    |-- test_toml_exhaustive.nim
    |-- test_config.nim          # Config format tests
    |-- test_mixed.nim           # Mixed-language file tests
    |-- test_fuzz.nim            # Fuzz and boundary tests
    |-- test_common.nim          # Shared test fixtures
    `-- fixtures/                # Per-language test data files
        |-- nim/
        |-- bash/
        |-- python/
        |-- javascript/
        |-- php/
        |-- html/
        |-- jinja/
        |-- json/
        |-- yaml/
        |-- toml/
        `-- mixed/

Contributing

Contributions are welcome. Open an issue to discuss your idea before submitting a pull request.

Adding a New Language

See LESSONS_LEARNED.md for the complete checklist (Section 14). The minimum steps are:

Create src/nimcheck/tokenizers/<lang>_tokenizer.nim inheriting from TokenizerBase
Implement the tokenize() method with language-specific lexing
Create src/nimcheck/languages/<lang>_validator.nim inheriting from ValidatorBase
Implement createTokenizer(), analyzeTokens(), and buildModuleInfo()
Add registerValidator("flavor", constructor) in the validator's init() proc
Import the validator in src/nimcheck.nim
Add extension mappings to detector.nim
Add test fixtures and test files under tests/
Update the test master runner tests/test_all.nim

Design Principles

KISS: Keep validators simple - tokenize, analyze brackets, report
Never crash: Every validation path in a try/except
No external deps: Pure Nim standard library
DRY: Common logic in tokenizerbase.nim and validatorbase.nim

License

MIT -- see LICENSE for details.

Author

molodetz

.gitea/workflows
src
tests
.gitignore
build.tmp
LICENSE
Makefile
nimcheck.nimble
README.md