# Nimcheck

Universal source code validation framework written in Nim. Tokenizer-based syntax validation across 10 languages and config formats, with auto-detection, structured JSON diagnostics, structural inspection, and a never-crash guarantee.

Version: **0.1.0** | Author: **molodetz** | License: **MIT** | Requires: **Nim >= 2.0.0**

---

## Table of Contents

1. [Architecture](#architecture)
2. [Quick Start](#quick-start)
3. [API Reference](#api-reference)
4. [Data Types](#data-types)
5. [JSON Output Structures](#json-output-structures)
6. [Error Code Reference](#error-code-reference)
7. [IO Table: Supported Languages and Per-Language Errors](#io-table-supported-languages-and-per-language-errors)
8. [CLI Usage](#cli-usage)
9. [Building and Testing](#building-and-testing)
10. [Project Structure](#project-structure)
11. [Contributing](#contributing)

---

## Architecture

Nimcheck uses a four-layer pipeline architecture. Each layer is independent and communicates through well-defined interfaces.

```
Source Code (string or file)
        |
        v
+-------------------+
|     Detector      |  LanguageFlavor from extension, shebang, or content analysis
+-------------------+
        |
        v
+-------------------+
|    Tokenizer      |  Converts raw source into a seq[Token] stream
+-------------------+
        |
        v
+-------------------+
|    Validator      |  Analyzes token stream for syntax errors and structure
+-------------------+
        |
        v
+-------------------+
|    Reporting      |  Formats errors as human-readable text or structured JSON
+-------------------+
        |
        v
  ValidationResult
```

### Layer 1: Detector (`src/nimcheck/core/detector.nim`)

Language auto-detection with cascading priority:

| Priority | Method | Source | Confidence |
|----------|--------|--------|------------|
| 1 (highest) | Explicit | User-provided `LanguageFlavor` | 1.0 |
| 2 | File extension | 70+ extension-to-flavor mappings | 0.9 |
| 3 | Shebang line | `#!` line interpreter matching (bash, python, node, ruby, lua, swift) | 0.95 |
| 4 | Content analysis | Keyword scoring, structural patterns, bracket usage | 0.6-0.9 |

The `detectFlavor()` function returns a `(LanguageFlavor, FlavorDetectionMethod, confidence: float)` tuple.

### Layer 2: Tokenizer (`src/nimcheck/core/tokenizerbase.nim`, per-language impls)

Each tokenizer converts raw source into a `seq[Token]`. Tokenizers track:

- **Position**: line, column, and byte offset for every token
- **Bracket stack**: `(`, `[`, `{`, `<` pushed and popped with position tracking
- **String context**: escape sequences, multi-line delimiters, interpolation
- **Comment context**: line comments, block comments with nesting support

Tokenizers emit errors for:
- Unclosed strings (`ErrUnclosedString`)
- Unclosed comments (`ErrUnclosedComment`)
- Unexpected characters (`ErrUnexpectedToken`)
- Unclosed block constructs (language-specific, e.g., `${}`, `$()`)

### Layer 3: Validator (`src/nimcheck/core/validatorbase.nim`, per-language impls)

Each validator:

1. Creates its tokenizer and runs `tokenize()`
2. Collects tokenizer errors
3. Runs `analyzeTokens()` — language-specific structural analysis
4. Runs `buildModuleInfo()` — populates `ModuleInfo` with imports, functions, classes
5. Computes validity, warning/info counts, duration

Validators register themselves via a factory pattern at import time. The registry maps flavor strings (e.g., `"python"`, `"bash"`) to constructors.

### Layer 4: Reporting (`src/nimcheck/reporting/errors.nim`)

Produces:
- **Human-readable reports**: `formatReport()` with position, severity, code, message, source context with carets, and hints
- **JSON output**: every type has a `toJson()` method producing `JsonNode`
- **Error categorization**: `countBySeverity()`, `countByCode()`, `filterBySeverity()`

---

## Quick Start

### Nim library usage

```nim
import nimcheck

# Validate source code from a string
let r = validateSource("proc hello() = echo 'hi'", lfNim)
echo r.valid           # bool
echo r.errors          # seq[ValidationError]
echo r.toJson().pretty()  # Full JSON output

# Validate a file (auto-detects language from extension)
let r2 = validateFile("src/main.nim")

# Auto-detect language
let r3 = validateSource("print('hello')", lfUnknown)
echo r3.flavor         # lfPython

# Inspect source structure (returns JsonNode with functions, classes, imports)
let info = inspectSource("class Foo:\n    def bar(self): pass", lfPython)
echo info.pretty()

# Human-readable report
echo reportSource("proc foo() =\n  let x = \"unclosed", lfNim)
```

### CLI usage

```bash
# Build the binary
make build

# Validate a file
bin/nimcheck validate --file=source.nim

# Validate source directly
bin/nimcheck validate --source="print('hi')" --flavor=python

# Read from stdin
echo "var x = 1" | bin/nimcheck validate

# Inspect structure (JSON output)
bin/nimcheck inspect --file=source.py

# Detect language
bin/nimcheck detect --file=unknown.txt
bin/nimcheck detect --source="#!/usr/bin/env bash"

# Version and help
bin/nimcheck version
bin/nimcheck help
```

---

## API Reference

### Core Functions

All functions are exported from `nimcheck.nim`. Import once with `import nimcheck`.

#### Validation

| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `validateSource` | `(source: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions(), sourcePath: string = "")` | `ValidationResult` | Validate source code string. Auto-detects flavor if `lfUnknown`. |
| `validateFile` | `(filePath: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions())` | `ValidationResult` | Validate a file from disk. Auto-detects flavor from file extension. |
| `validateSourceJson` | `(source: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Validate and return pretty-printed JSON string. |
| `validateFileJson` | `(filePath: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Validate file and return pretty-printed JSON string. |

#### Inspection

| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `inspectSource` | `(source: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions(), sourcePath: string = "")` | `JsonNode` | Validate and return structural diagnostics (functions, classes, imports). |
| `inspectFile` | `(filePath: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions())` | `JsonNode` | Inspect a file and return structural diagnostics. |

#### Reporting

| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `reportSource` | `(source: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Human-readable validation report for source code. |
| `reportFile` | `(filePath: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Human-readable validation report for a file. |

#### Detection and Utility

| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `detectFlavor` | `(source: string, filePath: string = "")` | `LanguageFlavor` | Auto-detect language flavor (convenience wrapper). |
| `supportedFlavors` | `()` | `seq[string]` | List all registered language flavors sorted alphabetically. |
| `version` | `()` | `string` | Version string (`"0.1.0"`). |
| `enableDebug` | `()` | `void` | Enable debug mode globally. |
| `disableDebug` | `()` | `void` | Disable debug mode globally. |

#### Core Detector (lower-level)

| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `detectFlavor` (core) | `(source: string, filePath: string = "", explicitFlavor: LanguageFlavor = lfUnknown)` | `(LanguageFlavor, FlavorDetectionMethod, float)` | Full detection with confidence score. |

#### Factory

| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `createValidator` | `(flavor: LanguageFlavor, source: string, options: ValidationOptions, sourcePath: string)` | `ValidatorBase` | Factory: create the correct validator for a flavor. |
| `registerValidator` | `(flavor: string, constructor: ValidatorConstructor)` | `void` | Register a new validator constructor. |

---

## Data Types

### LanguageFlavor (enum, pure)

All 28 flavors defined in `src/nimcheck/core/types.nim`. Flavors with validators (bold) are fully supported; the rest are detectable but have no dedicated validator.

| Enum Value | String | Validator | File Extensions (auto-detect) |
|------------|--------|-----------|-------------------------------|
| `lfUnknown` | `"unknown"` | No | -- |
| **`lfNim`** | `"nim"` | NimValidator | `.nim`, `.nims`, `.nimble` |
| **`lfPython`** | `"python"` | PythonValidator | `.py`, `.pyw`, `.pyx`, `.pxd` |
| **`lfBash`** | `"bash"` | BashValidator | `.sh`, `.bash` |
| `lfShell` | `"shell"` | BashValidator (shared) | `.zsh`, `.fish` |
| **`lfJavaScript`** | `"javascript"` | JavaScriptValidator | `.js`, `.mjs`, `.cjs` |
| `lfTypeScript` | `"typescript"` | JavaScriptValidator (shared) | `.ts`, `.tsx` |
| **`lfPHP`** | `"php"` | PHPValidator | `.php`, `.phtml` |
| **`lfHTML`** | `"html"` | HTMLValidator | `.html`, `.htm`, `.xhtml` |
| `lfXML` | `"xml"` | HTMLValidator (shared) | `.xml`, `.svg` |
| **`lfJinja`** | `"jinja"` | JinjaValidator | `.jinja`, `.jinja2`, `.j2` |
| **`lfJSON`** | `"json"` | JSONValidator | `.json`, `.jsonc` |
| **`lfYAML`** | `"yaml"` | YAMLValidator | `.yaml`, `.yml` |
| **`lfTOML`** | `"toml"` | TOMLValidator | `.toml` |
| `lfCSS` | `"css"` | No | `.css` |
| `lfSQL` | `"sql"` | No | `.sql` |
| `lfMarkdown` | `"markdown"` | No | `.md`, `.markdown` |
| `lfDockerfile` | `"dockerfile"` | No | `Dockerfile` |
| `lfMakefile` | `"makefile"` | No | `Makefile`, `makefile` |
| `lfRuby` | `"ruby"` | No | `.rb` |
| `lfRust` | `"rust"` | No | `.rs` |
| `lfGo` | `"go"` | No | `.go` |
| `lfLua` | `"lua"` | No | `.lua` |
| `lfC` | `"c"` | No | `.c`, `.h` |
| `lfCpp` | `"cpp"` | No | `.cpp`, `.cxx`, `.hpp` |
| `lfCSharp` | `"csharp"` | No | `.cs` |
| `lfJava` | `"java"` | No | `.java` |
| `lfSwift` | `"swift"` | No | `.swift` |
| `lfKotlin` | `"kotlin"` | No | `.kt`, `.kts` |

### FlavorDetectionMethod (enum)

| Value | Meaning |
|-------|---------|
| `fdmExplicit` | User explicitly provided the flavor |
| `fdmExtension` | Detected from file extension |
| `fdmShebang` | Detected from `#!` shebang line |
| `fdmContent` | Detected via content keyword analysis |
| `fdmUnknown` | Could not determine |

### TokenKind (enum)

Lexical token classifications used by all tokenizers.

| Kind | Description |
|------|-------------|
| `tkWhitespace` | Spaces, tabs, carriage returns |
| `tkNewline` | `\n` |
| `tkComment` | Line (`#`, `//`) or block (`/* */`) comment |
| `tkDocComment` | Documentation comment (`##`, `///`, `"""`) |
| `tkString` | Single or double quoted string literal |
| `tkRawString` | Raw string (`r"..."`) |
| `tkMultilineString` | Triple-quoted string (`"""..."""`) |
| `tkNumber` | Integer, float, hex, octal, binary |
| `tkIdentifier` | Variable, function, type name |
| `tkKeyword` | Language-specific reserved word |
| `tkOperator` | `+`, `-`, `*`, `/`, `&&`, etc. |
| `tkAssignment` | `=`, `:=`, `+=`, etc. |
| `tkPunctuation` | `.`, `,`, `;`, `:` |
| `kOpenParen` | `(` |
| `kCloseParen` | `)` |
| `kOpenBracket` | `[` |
| `kCloseBracket` | `]` |
| `kOpenBrace` | `{` |
| `kCloseBrace` | `}` |
| `kAngleOpen` | `<` |
| `kAngleClose` | `>` |
| `tkInterpolation` | String interpolation (`${var}`, f-string `{expr}`) |
| `tkDirective` | Preprocessor / compiler directive (`#include`, shebang) |
| `tkTemplateTag` | Template language tag (`{{ }}`, `{% %}`) |
| `tkSpecial` | Language-specific special token |
| `tkError` | Malformed or unrecognizable character |
| `tkEndOfFile` | End of input marker |

### Token

```
Token = object
  kind: TokenKind
  value: string
  position: SourcePosition
  range: SourceRange
```

### SourcePosition

```
SourcePosition = object
  line: int      # 1-indexed line number
  column: int    # 1-indexed column number
  offset: int    # 0-indexed byte offset from start
```

### SourceRange

```
SourceRange = object
  startPos: SourcePosition
  endPos: SourcePosition
```

### ErrorSeverity (enum)

| Value | String | Description |
|-------|--------|-------------|
| `esInfo` | `"info"` | Informational note |
| `esWarning` | `"warning"` | Non-fatal issue |
| `esError` | `"error"` | Validation failure |
| `esCritical` | `"critical"` | Internal framework failure |

### ValidationError

```
ValidationError = object
  severity: ErrorSeverity
  message: string
  code: string           # Error code, e.g. "E0002"
  position: SourcePosition
  range: SourceRange
  context: string        # Surrounding source context (optional)
  hint: string           # Fix suggestion (optional)
```

### ValidationResult

```
ValidationResult = object
  valid: bool
  flavor: LanguageFlavor
  detectionMethod: FlavorDetectionMethod
  errors: seq[ValidationError]
  warnings: int
  infos: int
  moduleInfo: ModuleInfo
  durationMs: float
  debugOutput: string     # Empty unless compiled with -d:nimcheckDebug
```

### ValidationOptions

```
ValidationOptions = object
  flavor: LanguageFlavor        # lfUnknown = auto-detect (default)
  debugMode: bool               # Enable per-validation debug logging (default: false)
  strictMode: bool              # Warnings become errors (default: false)
  maxErrors: int                # Stop after N errors; 0 = unlimited (default: 0)
  includeContext: bool          # Include surrounding source lines in error output (default: true)
  includeHints: bool            # Include fix suggestions (default: true)
```

### Structural Types (populated by `analyzeTokens()`)

```
VariableInfo = object
  name: string
  kind: string             # "var", "let", "const", "global", "local"
  typeAnnotation: string
  defaultValue: string
  position: SourcePosition
  mutable: bool
  exported: bool

FunctionInfo = object
  name: string
  kind: string             # "function", "method", "proc", "func", "lambda"
  parameters: seq[ParameterInfo]
  returnType: string
  isAsync: bool
  isExported: bool
  isGenerator: bool
  visibility: string       # "public", "private", "protected"
  position: SourcePosition
  docComment: string

ParameterInfo = object
  name: string
  typeAnnotation: string
  defaultValue: string
  isOptional: bool
  isVarargs: bool

ClassInfo = object
  name: string
  kind: string             # "class", "object", "struct", "interface"
  baseTypes: seq[string]
  fields: seq[VariableInfo]
  methods: seq[FunctionInfo]
  generics: seq[string]
  visibility: string
  position: SourcePosition
  docComment: string

ImportInfo = object
  module: string
  symbols: seq[string]     # Specific imported symbols
  isRelative: bool
  alias: string
  position: SourcePosition

ModuleInfo = object
  flavor: LanguageFlavor
  detectionMethod: FlavorDetectionMethod
  imports: seq[ImportInfo]
  variables: seq[VariableInfo]
  functions: seq[FunctionInfo]
  classes: seq[ClassInfo]
  linesOfCode: int
  totalTokens: int
  comments: int
  debugInfo: JsonNode      # Language-specific extra info
```

---

## JSON Output Structures

Every type has a `.toJson()` method. The `ValidationResult.toJson()` aggregates all nested structures.

### ValidationResult JSON

```json
{
  "valid": true,
  "flavor": "python",
  "detectionMethod": "by_extension",
  "errors": [
    {
      "severity": "error",
      "message": "Unclosed string literal starting with '\"'",
      "code": "E0002",
      "position": {"line": 2, "column": 13, "offset": 42},
      "range": {
        "start": {"line": 2, "column": 13, "offset": 42},
        "end":   {"line": 2, "column": 13, "offset": 42}
      },
      "context": "",
      "hint": "Add a closing quote before end of file"
    }
  ],
  "warnings": 0,
  "infos": 0,
  "moduleInfo": {
    "flavor": "python",
    "detectionMethod": "by_extension",
    "imports": [],
    "variables": [],
    "functions": [
      {
        "name": "hello",
        "kind": "def",
        "parameters": [],
        "returnType": "",
        "isAsync": false,
        "isExported": false,
        "isGenerator": false,
        "visibility": "",
        "position": {"line": 1, "column": 5, "offset": 4},
        "docComment": ""
      }
    ],
    "classes": [],
    "linesOfCode": 3,
    "totalTokens": 12,
    "comments": 0,
    "debugInfo": null
  },
  "durationMs": 0.342,
  "debugOutput": ""
}
```

### ValidationError JSON

```json
{
  "severity": "error",
  "message": "Unclosed string literal starting with '\"'",
  "code": "E0002",
  "position": {"line": 2, "column": 13, "offset": 42},
  "range": {
    "start": {"line": 2, "column": 13, "offset": 42},
    "end":   {"line": 2, "column": 13, "offset": 42}
  },
  "context": "",
  "hint": "Add a closing quote before end of file"
}
```

### ModuleInfo JSON

```json
{
  "flavor": "nim",
  "detectionMethod": "by_extension",
  "imports": [
    {
      "module": "strutils",
      "symbols": [],
      "isRelative": false,
      "alias": "",
      "position": {"line": 1, "column": 8, "offset": 7}
    }
  ],
  "variables": [],
  "functions": [
    {
      "name": "add",
      "kind": "proc",
      "parameters": [],
      "returnType": "",
      "isAsync": false,
      "isExported": false,
      "isGenerator": false,
      "visibility": "",
      "position": {"line": 3, "column": 6, "offset": 28},
      "docComment": ""
    }
  ],
  "classes": [
    {
      "name": "MyType",
      "kind": "type",
      "baseTypes": [],
      "fields": [],
      "methods": [],
      "generics": [],
      "visibility": "",
      "position": {"line": 5, "column": 6, "offset": 52},
      "docComment": ""
    }
  ],
  "linesOfCode": 8,
  "totalTokens": 45,
  "comments": 2,
  "debugInfo": null
}
```

### Human-Readable Report Format

```
=== Validation Report ===
  Flavor: python
  Detection: by_extension
  Valid: false
  Errors: 1
  Warnings: 0
  Duration: 0.34ms
  Lines of code: 3
  Tokens: 12

  --- Findings ---
  [E0002] (error) L2:13: Unclosed string literal starting with '"'
         |     x = "
         |            ^
         hint: Add a closing quote before end of file

  --- Summary ---
  INVALID - source has issues that need attention.
```

### Error Formatter Output (single error)

```
  [E0002] (error) L2:13: Unclosed string literal starting with '"'
         |     x = "
         |            ^
         hint: Add a closing quote before end of file
```

---

## Error Code Reference

Error codes follow a prefixed numbering scheme. The prefix encodes the category and severity.

### General Errors (E0xxx)

| Code | Constant | Description | Triggered By |
|------|----------|-------------|-------------|
| `E0001` | `ErrUnexpectedToken` | Unexpected or unrecognizable character | Tokenizer encounter of non-ASCII, non-operator char |
| `E0002` | `ErrUnclosedString` | Unclosed string literal | Missing closing quote at EOF |
| `E0003` | `ErrUnclosedComment` | Unclosed block comment | Missing `*/`, `]#`, `#}` etc. at EOF |
| `E0004` | `ErrUnclosedBlock` | Unclosed block/control structure | Language-specific: unclosed `if`/`for`/`class`/etc. |
| `E0005` | `ErrMismatchedBracket` | Mismatched or unexpected closing bracket | `)` without `(`, `</div>` without `<div>` |
| `E0006` | `ErrUnclosedBracket` | Unclosed bracket at EOF | `validateBrackets()` finds remaining items on bracket stack |
| `E0007` | `ErrInvalidSyntax` | General invalid syntax | JSON parse error, TOML missing `=`, YAML empty key |
| `E0008` | `ErrUnexpectedEOF` | Unexpected end of file | Reserved for incomplete constructs not covered by other codes |
| `E0009` | `ErrEmptySource` | Source is empty (zero-length string) | `validate()` with `source.len == 0` |
| `E0010` | `ErrEncodingError` | Encoding error | Reserved for non-UTF-8 input |
| `E0011` | -- | File not found | `validateFile()` with non-existent path |

### Language-Specific Errors (E1xxx)

| Code | Constant | Description | Languages |
|------|----------|-------------|-----------|
| `E1001` | `ErrUnexpectedIndent` | Unexpected or inconsistent indentation | Python |
| `E1002` | `ErrMissingIndent` | Missing indentation after block opener | Python |
| `E1003` | `ErrDuplicateParam` | Duplicate parameter name | (reserved) |
| `E1004` | `ErrUndefinedVariable` | Reference to undefined variable | (reserved) |
| `E1005` | `ErrTypeMismatch` | Type mismatch | (reserved) |
| `E1006` | `ErrInvalidAssignment` | Invalid assignment target | (reserved) |
| `E1007` | `ErrMissingReturn` | Function missing return statement | (reserved) |
| `E1008` | `ErrInvalidDecorator` | Invalid decorator usage | Python |
| `E1009` | `ErrInvalidImport` | Invalid import statement | (reserved) |

### Template Errors (E2xxx)

| Code | Constant | Description | Languages |
|------|----------|-------------|-----------|
| `E2001` | `ErrUnclosedTemplateTag` | Unclosed template expression/block tag | Jinja |
| `E2002` | `ErrUnclosedBlockTag` | Unclosed block tag with no matching end tag | Jinja |
| `E2003` | `ErrInvalidFilter` | Invalid filter in template expression | Jinja |

### Info Codes (I0xxx)

| Code | Constant | Description |
|------|----------|-------------|
| `I0001` | `InfoLongLine` | Line exceeds recommended length |
| `I0002` | `InfoTrailingWhitespace` | Trailing whitespace detected |

### Warning Codes (W0xxx / W1xxx)

| Code | Constant | Description |
|------|----------|-------------|
| `W0001` | `WarnUnusedVariable` | Variable declared but never used |
| `W0002` | `WarnInconsistentNaming` | Naming convention inconsistency |
| `W0003` | `WarnMissingDoc` | Public symbol missing documentation |
| `W1001` | -- | Tab characters in YAML indentation |

### Internal Error

| Code | Description | Triggered By |
|------|-------------|-------------|
| `E9999` | Internal framework error (never-crash fallback) | Any unhandled `Exception` in `validate()` or `validateSource()` |

---

## IO Table: Supported Languages and Per-Language Errors

### Input/Output Contract

Every validation function follows the same contract:

| Input | Output |
|-------|--------|
| `source: string` (required) | `ValidationResult` with `valid: bool` |
| `flavor: LanguageFlavor = lfUnknown` | `ValidationResult.errors: seq[ValidationError]` |
| `options: ValidationOptions` (optional) | `ValidationResult.moduleInfo: ModuleInfo` |
| `sourcePath: string = ""` (optional) | `ValidationResult.durationMs: float` |

**Guarantees**:
- Never throws — all exceptions are caught and returned as `E9999` errors
- Empty source returns `E0009` (or `valid: true` for YAML/TOML where empty is valid)
- File not found returns `E0011`
- Unknown flavor with no detection possible returns an `E0007` error
- Every error has a `code`, `message`, `severity`, `position`, and optional `hint`

### Per-Language Tokenizer, Validator, and Error Table

#### Nim (`.nim`, `.nims`, `.nimble`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `nim_tokenizer.nim` — `NimTokenizer` |
| Validator | `nim_validator.nim` — `NimValidator` |
| Keywords | 80+ (addr, and, as, block, break, case, const, converter, defer, discard, elif, else, enum, except, export, finally, for, from, func, if, import, in, include, iterator, let, macro, method, mixin, mod, nil, not, object, of, or, proc, ptr, raise, ref, return, template, try, tuple, type, using, var, when, while, with, yield, etc.) |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0001` | Unexpected character (non-ASCII, unrecognizable) | `esWarning` | Shows hex `U+XXXX` value |
| `E0002` | Unclosed string (`"..."` at EOF) | `esError` | "Add a closing quote" |
| `E0002` | Unclosed multi-line string (`"""` at EOF) | `esError` | -- |
| `E0002` | Newline inside string literal | `esError` | -- |
| `E0002` | Unclosed raw string (`r"..."` at EOF) | `esError` | -- |
| `E0002` | Unclosed character literal (`'x` at EOF) | `esError` | -- |
| `E0003` | Unclosed block comment (`#[` without `]#`) | `esError` | "Add closing `]#`" |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | "Add matching closing bracket" |

Structural extraction: imports (`from X import Y`), functions (`proc`, `func`, `method`, `template`, `macro`, `iterator`, `converter`), types (`type Foo`).

#### Bash / Shell (`.sh`, `.bash`, `.zsh`, `.fish`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `bash_tokenizer.nim` — `BashTokenizer` |
| Validator | `bash_validator.nim` — `BashValidator` |
| Keywords | 38+ (if, then, else, elif, fi, case, esac, for, while, until, do, done, in, function, select, time, return, break, continue, exit, declare, local, export, readonly, unset, eval, exec, let, source, shift, trap, type, typeset, ulimit, umask, alias, read, echo) |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed single-quoted string | `esError` | -- |
| `E0002` | Unclosed double-quoted string | `esError` | -- |
| `E0004` | Unclosed `${var}` expansion | `esError` | -- |
| `E0004` | Unclosed `$()` command substitution | `esError` | -- |
| `E0004` | Unclosed `if` block (`if` without matching `fi`) | `esError` | "Add missing 'fi'" |
| `E0004` | Unclosed loop (`for`/`while`/`until` without `done`) | `esError` | "Add missing 'done'" |
| `E0004` | Unclosed `case` block (`case` without `esac`) | `esError` | "Add missing 'esac'" |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |

Special features: shebang detection (`#!`), heredoc detection (`<<EOF`), variable expansion (`$var`, `${var}`), backtick command substitution, arithmetic expansion (`(( ))`).

#### Python (`.py`, `.pyw`, `.pyx`, `.pxd`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `python_tokenizer.nim` — `PythonTokenizer` |
| Validator | `python_validator.nim` — `PythonValidator` |
| Keywords | 33 (False, None, True, and, as, assert, async, await, break, class, continue, def, del, elif, else, except, finally, for, from, global, if, import, in, is, lambda, nonlocal, not, or, pass, raise, return, try, while, with, yield) |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed string (`"..."` or `'...'` at EOF) | `esError` | -- |
| `E0002` | Unclosed triple-quoted string (`"""` or `'''` at EOF) | `esError` | -- |
| `E0002` | Unclosed f-string (`f"..."` at EOF) | `esError` | -- |
| `E0002` | Newline inside string literal | `esError` | -- |
| `E1001` | Indentation mismatch at block boundary | `esError` | -- |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |

Special features: f-string interpolation with nested braces, indentation tracking (INDENT/DEDENT tokens), decorator detection (`@`), line continuation (`\`).

Structural extraction: imports (`import X`, `from X import Y`), functions (`def`), classes (`class`).

#### JavaScript / TypeScript (`.js`, `.mjs`, `.cjs`, `.ts`, `.tsx`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `javascript_tokenizer.nim` — `JavaScriptTokenizer` |
| Validator | `javascript_validator.nim` — `JavaScriptValidator` |
| Keywords | 60+ (async, await, break, case, catch, class, const, continue, debugger, default, delete, do, else, export, extends, false, finally, for, function, if, import, in, instanceof, let, new, null, of, return, super, switch, static, this, throw, true, try, typeof, var, void, while, with, yield + TypeScript: interface, type, declare, module, namespace, abstract, readonly, as, any, boolean, number, string, never, keyof, infer, unknown) |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed string (`"..."` or `'...'` at EOF) | `esError` | -- |
| `E0002` | Unclosed template literal (`` `...` `` at EOF) | `esError` | -- |
| `E0002` | Newline inside string literal | `esError` | -- |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |

Special features: template literals with `${}` interpolation nesting, regex literal detection, arrow function detection (`=>`), ESM import/export.

Structural extraction: imports/exports, functions (`function`, `async`), classes.

#### PHP (`.php`, `.phtml`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `php_tokenizer.nim` — `PHPTokenizer` |
| Validator | `php_validator.nim` — `PHPValidator` |
| Mode switching | Tracks `inPhpMode` flag — toggles on `<?php`/`<?` and `?>` |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed string | `esError` | -- |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |

Special features: PHP/HTML mode switching, `<?php`, `<?=`, `?>` detection, variable names (`$var`, `$obj->prop`).

Structural extraction: functions (`function`, `fn`), classes (`class`), imports/requires (`include`, `require`, `use`, `namespace`).

#### HTML / XML (`.html`, `.htm`, `.xhtml`, `.xml`, `.svg`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `html_tokenizer.nim` — `HTMLTokenizer` |
| Validator | `html_validator.nim` — `HTMLValidator` |
| Void elements | 14: area, base, br, col, embed, hr, img, input, link, meta, param, source, track, wbr |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0003` | Unclosed HTML comment (`<!--` without `-->`) | `esError` | -- |
| `E0004` | Unclosed HTML tag (e.g., `<div>` without `</div>`) | `esError` | "Add closing `</tagname>`" |
| `E0005` | Unexpected closing tag (mismatched nesting) | `esWarning` | "Check that tags are properly nested" |

Special features: tag stack balancing, void element awareness (no closing tag needed for self-closing elements), attribute parsing with quoted values, `<!DOCTYPE>`, `<?xml?>` directives.

#### Jinja2 (`.jinja`, `.jinja2`, `.j2`, `.html.j2`)

| Aspect | Details |
|--------|---------|
| Tokenizer | `jinja_tokenizer.nim` — `JinjaTokenizer` |
| Validator | `jinja_validator.nim` — `JinjaValidator` |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0003` | Unclosed Jinja comment (`{#` without `#}`) | `esError` | -- |
| `E2001` | Unclosed expression tag (`{{` without `}}`) | Implicit via tokenizer | -- |
| `E2001` | Unclosed block tag (`{%` without `%}`) | Implicit via tokenizer | -- |
| `E2002` | Unclosed `for` block (no matching `endfor`) | `esError` | "Add `{% endfor %}`" |
| `E2002` | Unclosed `if` block (no matching `endif`) | `esError` | "Add `{% endif %}`" |
| `E2002` | Unclosed `block` (no matching `endblock`) | `esError` | "Add `{% endblock %}`" |
| `E2002` | Unclosed `macro`, `filter`, `raw`, `autoescape`, `call`, `with` | `esError` | "Add matching `end{tag}`" |

Special features: tracks 12 block tag types (`for`, `if`, `block`, `macro`, `filter`, `raw`, `autoescape`, `call`, `with`), validates matching end tags, handles content between template syntax and plain text.

#### JSON (`.json`, `.jsonc`)

| Aspect | Details |
|--------|---------|
| Validator | `config_validators.nim` — `JSONValidator` |
| Method | Uses Nim's built-in `parseJson()` for full validation |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0007` | JSON parse error (any `JsonParsingError`) | `esError` | Error message from parser |
| `E0007` | Any other exception during parse | `esError` | "JSON error: {message}" |
| `E0009` | Empty JSON source | `esError` | -- |

#### YAML (`.yaml`, `.yml`)

| Aspect | Details |
|--------|---------|
| Validator | `config_validators.nim` — `YAMLValidator` |
| Method | Line-by-line structural analysis |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0007` | Empty mapping key (no key before `:`) | `esError` | -- |
| `E0007` | General YAML structural error | `esError` | -- |
| `E0009` | No content after stripping comments/blanks | `esError` | -- |
| `W1001` | Tab characters used for indentation | `esWarning` | "Use spaces for YAML indentation" |

Note: Empty YAML (only whitespace/comments) is `valid: true`.

#### TOML (`.toml`)

| Aspect | Details |
|--------|---------|
| Validator | `config_validators.nim` — `TOMLValidator` |
| Method | Line-by-line structural analysis |

| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0007` | Missing `=` in key-value pair | `esError` | "TOML uses `key = value` format" |
| `E0007` | Missing key before `=` | `esError` | -- |
| `E0007` | General TOML structural error | `esError` | -- |
| `E0009` | No content after stripping comments/blanks | `esError` | -- |

---

## CLI Usage

### Full CLI Reference

```
Nimcheck v0.1.0 - Universal Source Code Validation Framework

Usage:
  nimcheck validate [--file=<path> | --source=<code>] [--flavor=<lang>]
  nimcheck inspect  [--file=<path> | --source=<code>] [--flavor=<lang>]
  nimcheck detect   [--file=<path> | --source=<code>]
  nimcheck help

Options:
  --file=<path>     Source file to process
  --source=<code>   Source code string to process
  --flavor=<lang>   Language flavor (auto-detect if omitted)
  --debug           Enable debug mode
  --json            Output as JSON (default for inspect)

Supported flavors:
  - bash
  - html
  - jinja
  - jinja2
  - js
  - json
  - jsonc
  - nim
  - nimble
  - nims
  - php
  - phtml
  - python
  - py
  - shell
  - sh
  - toml
  - ts
  - typescript
  - yaml
  - yml
```

### Exit Codes

| Exit Code | Meaning |
|-----------|---------|
| 0 | Validation successful (or help/version requested) |
| Non-zero (Nim default) | Runtime error |

Note: The `validate` command always exits 0 on the Nim level; the `valid` field in the output indicates pass/fail. This is by design — Nimcheck never crashes.

---

## Building and Testing

### Prerequisites

- Nim >= 2.0.0
- `make` (optional for Makefile targets)

### Build

```bash
# Using Makefile
make build                  # Release build -> bin/nimcheck
make build-debug            # Debug build with -d:nimcheckDebug

# Using nimble
nimble build

# Manual
nim c src/nimcheck.nim
nim c -d:nimcheckDebug src/nimcheck.nim   # With debug output
```

### Test

The test suite covers **250+ test cases** across all supported languages.

```bash
make test                   # Run all tests
make test-nim               # Nim-specific tests only
make test-python            # Python-specific tests only
make test-bash              # Bash-specific tests only
make test-js                # JavaScript/TypeScript tests
make test-php               # PHP tests
make test-html              # HTML/XML tests
make test-jinja             # Jinja2 tests
make test-json              # JSON tests
make test-yaml              # YAML tests
make test-toml              # TOML tests
make test-mixed             # Mixed-language file tests
make debug                  # Build debug + run all tests
```

### Test Categories

| Suite | Files | Coverage |
|-------|-------|----------|
| Per-language basic | `test_<lang>.nim` | Valid + invalid source for each language |
| Per-language exhaustive | `test_<lang>_exhaustive.nim` | Edge cases: unclosed strings, unclosed comments, mismatched brackets, binary data, Unicode |
| Config formats | `test_config.nim` + `test_<json|yaml|toml>_exhaustive.nim` | JSON/YAML/TOML structural validation |
| Fuzz tests | `test_fuzz.nim` | Null bytes, binary data, deep nesting (100 levels), empty sources, concurrent validation |
| Mixed files | `test_mixed.nim` | HTML+Jinja combined, other mixed-format files |
| Common | `test_common.nim` | Shared test fixtures for valid and invalid code per language |

### Lint

```bash
make lint
# or
nim c --hints:on --warnings:on src/nimcheck.nim
```

### Clean

```bash
make clean
make rebuild    # clean + build
```

---

## Project Structure

```
.
|-- Makefile
|-- nimcheck.nimble
|-- README.md
|-- LICENSE
|-- bin/                          # Build output
|   `-- nimcheck                  # Compiled binary
|-- src/
|   |-- nimcheck.nim              # Public API entry point + CLI (isMainModule)
|   `-- nimcheck/
|       |-- core/
|       |   |-- types.nim         # All core types, toJson() methods, error code constants
|       |   |-- tokenizerbase.nim # Abstract TokenizerBase with position tracking, bracket stack, error recording
|       |   |-- validatorbase.nim # Abstract ValidatorBase with pipeline, factory, registry
|       |   |-- detector.nim      # Language auto-detection (extension, shebang, content scoring)
|       |   |-- debug.nim         # Debug logging (compiled only with -d:nimcheckDebug)
|       |   `-- init_validators.nim
|       |-- tokenizers/
|       |   |-- nim_tokenizer.nim       # Nim: """ strings, r"..." raw strings, #[ ]# nested comments, ## doc comments
|       |   |-- bash_tokenizer.nim      # Bash: heredocs, $var/${var}, $(), backticks, (( ))
|       |   |-- python_tokenizer.nim    # Python: f-strings, """ strings, decorators, indentation tracking
|       |   |-- javascript_tokenizer.nim # JS/TS: template literals, regex literals, arrow functions
|       |   |-- php_tokenizer.nim       # PHP: <?php/?> mode switching, $variables
|       |   |-- html_tokenizer.nim      # HTML/XML: tag balancing, void elements, attributes
|       |   `-- jinja_tokenizer.nim     # Jinja2: {{ }}, {% %}, {# #} block detection
|       |-- languages/
|       |   |-- nim_validator.nim       # Analyzes Nim: imports, procs, types
|       |   |-- bash_validator.nim      # Analyzes Bash: if/fi, for/done, case/esac matching
|       |   |-- python_validator.nim    # Analyzes Python: imports, def, class
|       |   |-- javascript_validator.nim # Analyzes JS/TS: imports, functions, classes
|       |   |-- php_validator.nim       # Analyzes PHP: functions, classes, includes
|       |   |-- html_validator.nim      # Analyzes HTML/XML: tag balancing
|       |   |-- jinja_validator.nim     # Analyzes Jinja2: block tag matching (12 tag types)
|       |   `-- config_validators.nim   # JSON (via parseJson), YAML, TOML validators
|       `-- reporting/
|           `-- errors.nim       # Human-readable and JSON error formatting
`-- tests/
    |-- test_all.nim             # Master test runner (imports all)
    |-- test_nim.nim             # Nim basic tests
    |-- test_nim_exhaustive.nim  # Nim edge cases
    |-- test_bash.nim
    |-- test_bash_exhaustive.nim
    |-- test_python.nim
    |-- test_python_exhaustive.nim
    |-- test_javascript.nim
    |-- test_javascript_exhaustive.nim
    |-- test_php.nim
    |-- test_php_exhaustive.nim
    |-- test_html.nim
    |-- test_html_exhaustive.nim
    |-- test_jinja.nim
    |-- test_jinja_exhaustive.nim
    |-- test_json_exhaustive.nim
    |-- test_yaml_exhaustive.nim
    |-- test_toml_exhaustive.nim
    |-- test_config.nim          # Config format tests
    |-- test_mixed.nim           # Mixed-language file tests
    |-- test_fuzz.nim            # Fuzz and boundary tests
    |-- test_common.nim          # Shared test fixtures
    `-- fixtures/                # Per-language test data files
        |-- nim/
        |-- bash/
        |-- python/
        |-- javascript/
        |-- php/
        |-- html/
        |-- jinja/
        |-- json/
        |-- yaml/
        |-- toml/
        `-- mixed/
```

---

## Contributing

Contributions are welcome. Open an issue to discuss your idea before submitting a pull request.

### Adding a New Language

See `LESSONS_LEARNED.md` for the complete checklist (Section 14). The minimum steps are:

1. Create `src/nimcheck/tokenizers/<lang>_tokenizer.nim` inheriting from `TokenizerBase`
2. Implement the `tokenize()` method with language-specific lexing
3. Create `src/nimcheck/languages/<lang>_validator.nim` inheriting from `ValidatorBase`
4. Implement `createTokenizer()`, `analyzeTokens()`, and `buildModuleInfo()`
5. Add `registerValidator("flavor", constructor)` in the validator's `init()` proc
6. Import the validator in `src/nimcheck.nim`
7. Add extension mappings to `detector.nim`
8. Add test fixtures and test files under `tests/`
9. Update the test master runner `tests/test_all.nim`

### Design Principles

- **KISS**: Keep validators simple — tokenize, analyze brackets, report
- **Never crash**: Every validation path in a try/except
- **No external deps**: Pure Nim standard library
- **DRY**: Common logic in `tokenizerbase.nim` and `validatorbase.nim`

---

## License

MIT -- see [LICENSE](LICENSE) for details.

## Author

molodetz