# Nimcheck
Universal source code validation framework written in Nim. Tokenizer-based syntax validation across 10 languages and config formats, with auto-detection, structured JSON diagnostics, structural inspection, and a never-crash guarantee.
Version: **0.1.0** | Author: **molodetz** | License: **MIT** | Requires: **Nim >= 2.0.0**
---
## Table of Contents
1. [Architecture](#architecture)
2. [Quick Start](#quick-start)
3. [API Reference](#api-reference)
4. [Data Types](#data-types)
5. [JSON Output Structures](#json-output-structures)
6. [Error Code Reference](#error-code-reference)
7. [IO Table: Supported Languages and Per-Language Errors](#io-table-supported-languages-and-per-language-errors)
8. [CLI Usage](#cli-usage)
9. [Building and Testing](#building-and-testing)
10. [Project Structure](#project-structure)
11. [Contributing](#contributing)
---
## Architecture
Nimcheck uses a four-layer pipeline architecture. Each layer is independent and communicates through well-defined interfaces.
```
Source Code (string or file)
|
v
+-------------------+
| Detector | LanguageFlavor from extension, shebang, or content analysis
+-------------------+
|
v
+-------------------+
| Tokenizer | Converts raw source into a seq[Token] stream
+-------------------+
|
v
+-------------------+
| Validator | Analyzes token stream for syntax errors and structure
+-------------------+
|
v
+-------------------+
| Reporting | Formats errors as human-readable text or structured JSON
+-------------------+
|
v
ValidationResult
```
### Layer 1: Detector (`src/nimcheck/core/detector.nim`)
Language auto-detection with cascading priority:
| Priority | Method | Source | Confidence |
|----------|--------|--------|------------|
| 1 (highest) | Explicit | User-provided `LanguageFlavor` | 1.0 |
| 2 | File extension | 70+ extension-to-flavor mappings | 0.9 |
| 3 | Shebang line | `#!` line interpreter matching (bash, python, node, ruby, lua, swift) | 0.95 |
| 4 | Content analysis | Keyword scoring, structural patterns, bracket usage | 0.6-0.9 |
The `detectFlavor()` function returns a `(LanguageFlavor, FlavorDetectionMethod, confidence: float)` tuple.
### Layer 2: Tokenizer (`src/nimcheck/core/tokenizerbase.nim`, per-language impls)
Each tokenizer converts raw source into a `seq[Token]`. Tokenizers track:
- **Position**: line, column, and byte offset for every token
- **Bracket stack**: `(`, `[`, `{`, `<` pushed and popped with position tracking
- **String context**: escape sequences, multi-line delimiters, interpolation
- **Comment context**: line comments, block comments with nesting support
Tokenizers emit errors for:
- Unclosed strings (`ErrUnclosedString`)
- Unclosed comments (`ErrUnclosedComment`)
- Unexpected characters (`ErrUnexpectedToken`)
- Unclosed block constructs (language-specific, e.g., `${}`, `$()`)
### Layer 3: Validator (`src/nimcheck/core/validatorbase.nim`, per-language impls)
Each validator:
1. Creates its tokenizer and runs `tokenize()`
2. Collects tokenizer errors
3. Runs `analyzeTokens()` — language-specific structural analysis
4. Runs `buildModuleInfo()` — populates `ModuleInfo` with imports, functions, classes
5. Computes validity, warning/info counts, duration
Validators register themselves via a factory pattern at import time. The registry maps flavor strings (e.g., `"python"`, `"bash"`) to constructors.
### Layer 4: Reporting (`src/nimcheck/reporting/errors.nim`)
Produces:
- **Human-readable reports**: `formatReport()` with position, severity, code, message, source context with carets, and hints
- **JSON output**: every type has a `toJson()` method producing `JsonNode`
- **Error categorization**: `countBySeverity()`, `countByCode()`, `filterBySeverity()`
---
## Quick Start
### Nim library usage
```nim
import nimcheck
# Validate source code from a string
let r = validateSource("proc hello() = echo 'hi'", lfNim)
echo r.valid # bool
echo r.errors # seq[ValidationError]
echo r.toJson().pretty() # Full JSON output
# Validate a file (auto-detects language from extension)
let r2 = validateFile("src/main.nim")
# Auto-detect language
let r3 = validateSource("print('hello')", lfUnknown)
echo r3.flavor # lfPython
# Inspect source structure (returns JsonNode with functions, classes, imports)
let info = inspectSource("class Foo:\n def bar(self): pass", lfPython)
echo info.pretty()
# Human-readable report
echo reportSource("proc foo() =\n let x = \"unclosed", lfNim)
```
### CLI usage
```bash
# Build the binary
make build
# Validate a file
bin/nimcheck validate --file=source.nim
# Validate source directly
bin/nimcheck validate --source="print('hi')" --flavor=python
# Read from stdin
echo "var x = 1" | bin/nimcheck validate
# Inspect structure (JSON output)
bin/nimcheck inspect --file=source.py
# Detect language
bin/nimcheck detect --file=unknown.txt
bin/nimcheck detect --source="#!/usr/bin/env bash"
# Version and help
bin/nimcheck version
bin/nimcheck help
```
---
## API Reference
### Core Functions
All functions are exported from `nimcheck.nim`. Import once with `import nimcheck`.
#### Validation
| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `validateSource` | `(source: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions(), sourcePath: string = "")` | `ValidationResult` | Validate source code string. Auto-detects flavor if `lfUnknown`. |
| `validateFile` | `(filePath: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions())` | `ValidationResult` | Validate a file from disk. Auto-detects flavor from file extension. |
| `validateSourceJson` | `(source: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Validate and return pretty-printed JSON string. |
| `validateFileJson` | `(filePath: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Validate file and return pretty-printed JSON string. |
#### Inspection
| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `inspectSource` | `(source: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions(), sourcePath: string = "")` | `JsonNode` | Validate and return structural diagnostics (functions, classes, imports). |
| `inspectFile` | `(filePath: string, flavor: LanguageFlavor = lfUnknown, options: ValidationOptions = newValidationOptions())` | `JsonNode` | Inspect a file and return structural diagnostics. |
#### Reporting
| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `reportSource` | `(source: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Human-readable validation report for source code. |
| `reportFile` | `(filePath: string, flavor: LanguageFlavor = lfUnknown)` | `string` | Human-readable validation report for a file. |
#### Detection and Utility
| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `detectFlavor` | `(source: string, filePath: string = "")` | `LanguageFlavor` | Auto-detect language flavor (convenience wrapper). |
| `supportedFlavors` | `()` | `seq[string]` | List all registered language flavors sorted alphabetically. |
| `version` | `()` | `string` | Version string (`"0.1.0"`). |
| `enableDebug` | `()` | `void` | Enable debug mode globally. |
| `disableDebug` | `()` | `void` | Disable debug mode globally. |
#### Core Detector (lower-level)
| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `detectFlavor` (core) | `(source: string, filePath: string = "", explicitFlavor: LanguageFlavor = lfUnknown)` | `(LanguageFlavor, FlavorDetectionMethod, float)` | Full detection with confidence score. |
#### Factory
| Function | Signature | Returns | Description |
|----------|-----------|---------|-------------|
| `createValidator` | `(flavor: LanguageFlavor, source: string, options: ValidationOptions, sourcePath: string)` | `ValidatorBase` | Factory: create the correct validator for a flavor. |
| `registerValidator` | `(flavor: string, constructor: ValidatorConstructor)` | `void` | Register a new validator constructor. |
---
## Data Types
### LanguageFlavor (enum, pure)
All 28 flavors defined in `src/nimcheck/core/types.nim`. Flavors with validators (bold) are fully supported; the rest are detectable but have no dedicated validator.
| Enum Value | String | Validator | File Extensions (auto-detect) |
|------------|--------|-----------|-------------------------------|
| `lfUnknown` | `"unknown"` | No | -- |
| **`lfNim`** | `"nim"` | NimValidator | `.nim`, `.nims`, `.nimble` |
| **`lfPython`** | `"python"` | PythonValidator | `.py`, `.pyw`, `.pyx`, `.pxd` |
| **`lfBash`** | `"bash"` | BashValidator | `.sh`, `.bash` |
| `lfShell` | `"shell"` | BashValidator (shared) | `.zsh`, `.fish` |
| **`lfJavaScript`** | `"javascript"` | JavaScriptValidator | `.js`, `.mjs`, `.cjs` |
| `lfTypeScript` | `"typescript"` | JavaScriptValidator (shared) | `.ts`, `.tsx` |
| **`lfPHP`** | `"php"` | PHPValidator | `.php`, `.phtml` |
| **`lfHTML`** | `"html"` | HTMLValidator | `.html`, `.htm`, `.xhtml` |
| `lfXML` | `"xml"` | HTMLValidator (shared) | `.xml`, `.svg` |
| **`lfJinja`** | `"jinja"` | JinjaValidator | `.jinja`, `.jinja2`, `.j2` |
| **`lfJSON`** | `"json"` | JSONValidator | `.json`, `.jsonc` |
| **`lfYAML`** | `"yaml"` | YAMLValidator | `.yaml`, `.yml` |
| **`lfTOML`** | `"toml"` | TOMLValidator | `.toml` |
| `lfCSS` | `"css"` | No | `.css` |
| `lfSQL` | `"sql"` | No | `.sql` |
| `lfMarkdown` | `"markdown"` | No | `.md`, `.markdown` |
| `lfDockerfile` | `"dockerfile"` | No | `Dockerfile` |
| `lfMakefile` | `"makefile"` | No | `Makefile`, `makefile` |
| `lfRuby` | `"ruby"` | No | `.rb` |
| `lfRust` | `"rust"` | No | `.rs` |
| `lfGo` | `"go"` | No | `.go` |
| `lfLua` | `"lua"` | No | `.lua` |
| `lfC` | `"c"` | No | `.c`, `.h` |
| `lfCpp` | `"cpp"` | No | `.cpp`, `.cxx`, `.hpp` |
| `lfCSharp` | `"csharp"` | No | `.cs` |
| `lfJava` | `"java"` | No | `.java` |
| `lfSwift` | `"swift"` | No | `.swift` |
| `lfKotlin` | `"kotlin"` | No | `.kt`, `.kts` |
### FlavorDetectionMethod (enum)
| Value | Meaning |
|-------|---------|
| `fdmExplicit` | User explicitly provided the flavor |
| `fdmExtension` | Detected from file extension |
| `fdmShebang` | Detected from `#!` shebang line |
| `fdmContent` | Detected via content keyword analysis |
| `fdmUnknown` | Could not determine |
### TokenKind (enum)
Lexical token classifications used by all tokenizers.
| Kind | Description |
|------|-------------|
| `tkWhitespace` | Spaces, tabs, carriage returns |
| `tkNewline` | `\n` |
| `tkComment` | Line (`#`, `//`) or block (`/* */`) comment |
| `tkDocComment` | Documentation comment (`##`, `///`, `"""`) |
| `tkString` | Single or double quoted string literal |
| `tkRawString` | Raw string (`r"..."`) |
| `tkMultilineString` | Triple-quoted string (`"""..."""`) |
| `tkNumber` | Integer, float, hex, octal, binary |
| `tkIdentifier` | Variable, function, type name |
| `tkKeyword` | Language-specific reserved word |
| `tkOperator` | `+`, `-`, `*`, `/`, `&&`, etc. |
| `tkAssignment` | `=`, `:=`, `+=`, etc. |
| `tkPunctuation` | `.`, `,`, `;`, `:` |
| `kOpenParen` | `(` |
| `kCloseParen` | `)` |
| `kOpenBracket` | `[` |
| `kCloseBracket` | `]` |
| `kOpenBrace` | `{` |
| `kCloseBrace` | `}` |
| `kAngleOpen` | `<` |
| `kAngleClose` | `>` |
| `tkInterpolation` | String interpolation (`${var}`, f-string `{expr}`) |
| `tkDirective` | Preprocessor / compiler directive (`#include`, shebang) |
| `tkTemplateTag` | Template language tag (`{{ }}`, `{% %}`) |
| `tkSpecial` | Language-specific special token |
| `tkError` | Malformed or unrecognizable character |
| `tkEndOfFile` | End of input marker |
### Token
```
Token = object
kind: TokenKind
value: string
position: SourcePosition
range: SourceRange
```
### SourcePosition
```
SourcePosition = object
line: int # 1-indexed line number
column: int # 1-indexed column number
offset: int # 0-indexed byte offset from start
```
### SourceRange
```
SourceRange = object
startPos: SourcePosition
endPos: SourcePosition
```
### ErrorSeverity (enum)
| Value | String | Description |
|-------|--------|-------------|
| `esInfo` | `"info"` | Informational note |
| `esWarning` | `"warning"` | Non-fatal issue |
| `esError` | `"error"` | Validation failure |
| `esCritical` | `"critical"` | Internal framework failure |
### ValidationError
```
ValidationError = object
severity: ErrorSeverity
message: string
code: string # Error code, e.g. "E0002"
position: SourcePosition
range: SourceRange
context: string # Surrounding source context (optional)
hint: string # Fix suggestion (optional)
```
### ValidationResult
```
ValidationResult = object
valid: bool
flavor: LanguageFlavor
detectionMethod: FlavorDetectionMethod
errors: seq[ValidationError]
warnings: int
infos: int
moduleInfo: ModuleInfo
durationMs: float
debugOutput: string # Empty unless compiled with -d:nimcheckDebug
```
### ValidationOptions
```
ValidationOptions = object
flavor: LanguageFlavor # lfUnknown = auto-detect (default)
debugMode: bool # Enable per-validation debug logging (default: false)
strictMode: bool # Warnings become errors (default: false)
maxErrors: int # Stop after N errors; 0 = unlimited (default: 0)
includeContext: bool # Include surrounding source lines in error output (default: true)
includeHints: bool # Include fix suggestions (default: true)
```
### Structural Types (populated by `analyzeTokens()`)
```
VariableInfo = object
name: string
kind: string # "var", "let", "const", "global", "local"
typeAnnotation: string
defaultValue: string
position: SourcePosition
mutable: bool
exported: bool
FunctionInfo = object
name: string
kind: string # "function", "method", "proc", "func", "lambda"
parameters: seq[ParameterInfo]
returnType: string
isAsync: bool
isExported: bool
isGenerator: bool
visibility: string # "public", "private", "protected"
position: SourcePosition
docComment: string
ParameterInfo = object
name: string
typeAnnotation: string
defaultValue: string
isOptional: bool
isVarargs: bool
ClassInfo = object
name: string
kind: string # "class", "object", "struct", "interface"
baseTypes: seq[string]
fields: seq[VariableInfo]
methods: seq[FunctionInfo]
generics: seq[string]
visibility: string
position: SourcePosition
docComment: string
ImportInfo = object
module: string
symbols: seq[string] # Specific imported symbols
isRelative: bool
alias: string
position: SourcePosition
ModuleInfo = object
flavor: LanguageFlavor
detectionMethod: FlavorDetectionMethod
imports: seq[ImportInfo]
variables: seq[VariableInfo]
functions: seq[FunctionInfo]
classes: seq[ClassInfo]
linesOfCode: int
totalTokens: int
comments: int
debugInfo: JsonNode # Language-specific extra info
```
---
## JSON Output Structures
Every type has a `.toJson()` method. The `ValidationResult.toJson()` aggregates all nested structures.
### ValidationResult JSON
```json
{
"valid": true,
"flavor": "python",
"detectionMethod": "by_extension",
"errors": [
{
"severity": "error",
"message": "Unclosed string literal starting with '\"'",
"code": "E0002",
"position": {"line": 2, "column": 13, "offset": 42},
"range": {
"start": {"line": 2, "column": 13, "offset": 42},
"end": {"line": 2, "column": 13, "offset": 42}
},
"context": "",
"hint": "Add a closing quote before end of file"
}
],
"warnings": 0,
"infos": 0,
"moduleInfo": {
"flavor": "python",
"detectionMethod": "by_extension",
"imports": [],
"variables": [],
"functions": [
{
"name": "hello",
"kind": "def",
"parameters": [],
"returnType": "",
"isAsync": false,
"isExported": false,
"isGenerator": false,
"visibility": "",
"position": {"line": 1, "column": 5, "offset": 4},
"docComment": ""
}
],
"classes": [],
"linesOfCode": 3,
"totalTokens": 12,
"comments": 0,
"debugInfo": null
},
"durationMs": 0.342,
"debugOutput": ""
}
```
### ValidationError JSON
```json
{
"severity": "error",
"message": "Unclosed string literal starting with '\"'",
"code": "E0002",
"position": {"line": 2, "column": 13, "offset": 42},
"range": {
"start": {"line": 2, "column": 13, "offset": 42},
"end": {"line": 2, "column": 13, "offset": 42}
},
"context": "",
"hint": "Add a closing quote before end of file"
}
```
### ModuleInfo JSON
```json
{
"flavor": "nim",
"detectionMethod": "by_extension",
"imports": [
{
"module": "strutils",
"symbols": [],
"isRelative": false,
"alias": "",
"position": {"line": 1, "column": 8, "offset": 7}
}
],
"variables": [],
"functions": [
{
"name": "add",
"kind": "proc",
"parameters": [],
"returnType": "",
"isAsync": false,
"isExported": false,
"isGenerator": false,
"visibility": "",
"position": {"line": 3, "column": 6, "offset": 28},
"docComment": ""
}
],
"classes": [
{
"name": "MyType",
"kind": "type",
"baseTypes": [],
"fields": [],
"methods": [],
"generics": [],
"visibility": "",
"position": {"line": 5, "column": 6, "offset": 52},
"docComment": ""
}
],
"linesOfCode": 8,
"totalTokens": 45,
"comments": 2,
"debugInfo": null
}
```
### Human-Readable Report Format
```
=== Validation Report ===
Flavor: python
Detection: by_extension
Valid: false
Errors: 1
Warnings: 0
Duration: 0.34ms
Lines of code: 3
Tokens: 12
--- Findings ---
[E0002] (error) L2:13: Unclosed string literal starting with '"'
| x = "
| ^
hint: Add a closing quote before end of file
--- Summary ---
INVALID - source has issues that need attention.
```
### Error Formatter Output (single error)
```
[E0002] (error) L2:13: Unclosed string literal starting with '"'
| x = "
| ^
hint: Add a closing quote before end of file
```
---
## Error Code Reference
Error codes follow a prefixed numbering scheme. The prefix encodes the category and severity.
### General Errors (E0xxx)
| Code | Constant | Description | Triggered By |
|------|----------|-------------|-------------|
| `E0001` | `ErrUnexpectedToken` | Unexpected or unrecognizable character | Tokenizer encounter of non-ASCII, non-operator char |
| `E0002` | `ErrUnclosedString` | Unclosed string literal | Missing closing quote at EOF |
| `E0003` | `ErrUnclosedComment` | Unclosed block comment | Missing `*/`, `]#`, `#}` etc. at EOF |
| `E0004` | `ErrUnclosedBlock` | Unclosed block/control structure | Language-specific: unclosed `if`/`for`/`class`/etc. |
| `E0005` | `ErrMismatchedBracket` | Mismatched or unexpected closing bracket | `)` without `(`, `</div>` without `<div>` |
| `E0006` | `ErrUnclosedBracket` | Unclosed bracket at EOF | `validateBrackets()` finds remaining items on bracket stack |
| `E0007` | `ErrInvalidSyntax` | General invalid syntax | JSON parse error, TOML missing `=`, YAML empty key |
| `E0008` | `ErrUnexpectedEOF` | Unexpected end of file | Reserved for incomplete constructs not covered by other codes |
| `E0009` | `ErrEmptySource` | Source is empty (zero-length string) | `validate()` with `source.len == 0` |
| `E0010` | `ErrEncodingError` | Encoding error | Reserved for non-UTF-8 input |
| `E0011` | -- | File not found | `validateFile()` with non-existent path |
### Language-Specific Errors (E1xxx)
| Code | Constant | Description | Languages |
|------|----------|-------------|-----------|
| `E1001` | `ErrUnexpectedIndent` | Unexpected or inconsistent indentation | Python |
| `E1002` | `ErrMissingIndent` | Missing indentation after block opener | Python |
| `E1003` | `ErrDuplicateParam` | Duplicate parameter name | (reserved) |
| `E1004` | `ErrUndefinedVariable` | Reference to undefined variable | (reserved) |
| `E1005` | `ErrTypeMismatch` | Type mismatch | (reserved) |
| `E1006` | `ErrInvalidAssignment` | Invalid assignment target | (reserved) |
| `E1007` | `ErrMissingReturn` | Function missing return statement | (reserved) |
| `E1008` | `ErrInvalidDecorator` | Invalid decorator usage | Python |
| `E1009` | `ErrInvalidImport` | Invalid import statement | (reserved) |
### Template Errors (E2xxx)
| Code | Constant | Description | Languages |
|------|----------|-------------|-----------|
| `E2001` | `ErrUnclosedTemplateTag` | Unclosed template expression/block tag | Jinja |
| `E2002` | `ErrUnclosedBlockTag` | Unclosed block tag with no matching end tag | Jinja |
| `E2003` | `ErrInvalidFilter` | Invalid filter in template expression | Jinja |
### Info Codes (I0xxx)
| Code | Constant | Description |
|------|----------|-------------|
| `I0001` | `InfoLongLine` | Line exceeds recommended length |
| `I0002` | `InfoTrailingWhitespace` | Trailing whitespace detected |
### Warning Codes (W0xxx / W1xxx)
| Code | Constant | Description |
|------|----------|-------------|
| `W0001` | `WarnUnusedVariable` | Variable declared but never used |
| `W0002` | `WarnInconsistentNaming` | Naming convention inconsistency |
| `W0003` | `WarnMissingDoc` | Public symbol missing documentation |
| `W1001` | -- | Tab characters in YAML indentation |
### Internal Error
| Code | Description | Triggered By |
|------|-------------|-------------|
| `E9999` | Internal framework error (never-crash fallback) | Any unhandled `Exception` in `validate()` or `validateSource()` |
---
## IO Table: Supported Languages and Per-Language Errors
### Input/Output Contract
Every validation function follows the same contract:
| Input | Output |
|-------|--------|
| `source: string` (required) | `ValidationResult` with `valid: bool` |
| `flavor: LanguageFlavor = lfUnknown` | `ValidationResult.errors: seq[ValidationError]` |
| `options: ValidationOptions` (optional) | `ValidationResult.moduleInfo: ModuleInfo` |
| `sourcePath: string = ""` (optional) | `ValidationResult.durationMs: float` |
**Guarantees**:
- Never throws — all exceptions are caught and returned as `E9999` errors
- Empty source returns `E0009` (or `valid: true` for YAML/TOML where empty is valid)
- File not found returns `E0011`
- Unknown flavor with no detection possible returns an `E0007` error
- Every error has a `code`, `message`, `severity`, `position`, and optional `hint`
### Per-Language Tokenizer, Validator, and Error Table
#### Nim (`.nim`, `.nims`, `.nimble`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `nim_tokenizer.nim` — `NimTokenizer` |
| Validator | `nim_validator.nim` — `NimValidator` |
| Keywords | 80+ (addr, and, as, block, break, case, const, converter, defer, discard, elif, else, enum, except, export, finally, for, from, func, if, import, in, include, iterator, let, macro, method, mixin, mod, nil, not, object, of, or, proc, ptr, raise, ref, return, template, try, tuple, type, using, var, when, while, with, yield, etc.) |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0001` | Unexpected character (non-ASCII, unrecognizable) | `esWarning` | Shows hex `U+XXXX` value |
| `E0002` | Unclosed string (`"..."` at EOF) | `esError` | "Add a closing quote" |
| `E0002` | Unclosed multi-line string (`"""` at EOF) | `esError` | -- |
| `E0002` | Newline inside string literal | `esError` | -- |
| `E0002` | Unclosed raw string (`r"..."` at EOF) | `esError` | -- |
| `E0002` | Unclosed character literal (`'x` at EOF) | `esError` | -- |
| `E0003` | Unclosed block comment (`#[` without `]#`) | `esError` | "Add closing `]#`" |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | "Add matching closing bracket" |
Structural extraction: imports (`from X import Y`), functions (`proc`, `func`, `method`, `template`, `macro`, `iterator`, `converter`), types (`type Foo`).
#### Bash / Shell (`.sh`, `.bash`, `.zsh`, `.fish`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `bash_tokenizer.nim` — `BashTokenizer` |
| Validator | `bash_validator.nim` — `BashValidator` |
| Keywords | 38+ (if, then, else, elif, fi, case, esac, for, while, until, do, done, in, function, select, time, return, break, continue, exit, declare, local, export, readonly, unset, eval, exec, let, source, shift, trap, type, typeset, ulimit, umask, alias, read, echo) |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed single-quoted string | `esError` | -- |
| `E0002` | Unclosed double-quoted string | `esError` | -- |
| `E0004` | Unclosed `${var}` expansion | `esError` | -- |
| `E0004` | Unclosed `$()` command substitution | `esError` | -- |
| `E0004` | Unclosed `if` block (`if` without matching `fi`) | `esError` | "Add missing 'fi'" |
| `E0004` | Unclosed loop (`for`/`while`/`until` without `done`) | `esError` | "Add missing 'done'" |
| `E0004` | Unclosed `case` block (`case` without `esac`) | `esError` | "Add missing 'esac'" |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |
Special features: shebang detection (`#!`), heredoc detection (`<<EOF`), variable expansion (`$var`, `${var}`), backtick command substitution, arithmetic expansion (`(( ))`).
#### Python (`.py`, `.pyw`, `.pyx`, `.pxd`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `python_tokenizer.nim` — `PythonTokenizer` |
| Validator | `python_validator.nim` — `PythonValidator` |
| Keywords | 33 (False, None, True, and, as, assert, async, await, break, class, continue, def, del, elif, else, except, finally, for, from, global, if, import, in, is, lambda, nonlocal, not, or, pass, raise, return, try, while, with, yield) |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed string (`"..."` or `'...'` at EOF) | `esError` | -- |
| `E0002` | Unclosed triple-quoted string (`"""` or `'''` at EOF) | `esError` | -- |
| `E0002` | Unclosed f-string (`f"..."` at EOF) | `esError` | -- |
| `E0002` | Newline inside string literal | `esError` | -- |
| `E1001` | Indentation mismatch at block boundary | `esError` | -- |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |
Special features: f-string interpolation with nested braces, indentation tracking (INDENT/DEDENT tokens), decorator detection (`@`), line continuation (`\`).
Structural extraction: imports (`import X`, `from X import Y`), functions (`def`), classes (`class`).
#### JavaScript / TypeScript (`.js`, `.mjs`, `.cjs`, `.ts`, `.tsx`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `javascript_tokenizer.nim` — `JavaScriptTokenizer` |
| Validator | `javascript_validator.nim` — `JavaScriptValidator` |
| Keywords | 60+ (async, await, break, case, catch, class, const, continue, debugger, default, delete, do, else, export, extends, false, finally, for, function, if, import, in, instanceof, let, new, null, of, return, super, switch, static, this, throw, true, try, typeof, var, void, while, with, yield + TypeScript: interface, type, declare, module, namespace, abstract, readonly, as, any, boolean, number, string, never, keyof, infer, unknown) |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed string (`"..."` or `'...'` at EOF) | `esError` | -- |
| `E0002` | Unclosed template literal (`` `...` `` at EOF) | `esError` | -- |
| `E0002` | Newline inside string literal | `esError` | -- |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |
Special features: template literals with `${}` interpolation nesting, regex literal detection, arrow function detection (`=>`), ESM import/export.
Structural extraction: imports/exports, functions (`function`, `async`), classes.
#### PHP (`.php`, `.phtml`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `php_tokenizer.nim` — `PHPTokenizer` |
| Validator | `php_validator.nim` — `PHPValidator` |
| Mode switching | Tracks `inPhpMode` flag — toggles on `<?php`/`<?` and `?>` |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0002` | Unclosed string | `esError` | -- |
| `E0006` | Unclosed bracket (`(`, `[`, `{` at EOF) | `esError` | -- |
Special features: PHP/HTML mode switching, `<?php`, `<?=`, `?>` detection, variable names (`$var`, `$obj->prop`).
Structural extraction: functions (`function`, `fn`), classes (`class`), imports/requires (`include`, `require`, `use`, `namespace`).
#### HTML / XML (`.html`, `.htm`, `.xhtml`, `.xml`, `.svg`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `html_tokenizer.nim` — `HTMLTokenizer` |
| Validator | `html_validator.nim` — `HTMLValidator` |
| Void elements | 14: area, base, br, col, embed, hr, img, input, link, meta, param, source, track, wbr |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0003` | Unclosed HTML comment (`<!--` without `-->`) | `esError` | -- |
| `E0004` | Unclosed HTML tag (e.g., `<div>` without `</div>`) | `esError` | "Add closing `</tagname>`" |
| `E0005` | Unexpected closing tag (mismatched nesting) | `esWarning` | "Check that tags are properly nested" |
Special features: tag stack balancing, void element awareness (no closing tag needed for self-closing elements), attribute parsing with quoted values, `<!DOCTYPE>`, `<?xml?>` directives.
#### Jinja2 (`.jinja`, `.jinja2`, `.j2`, `.html.j2`)
| Aspect | Details |
|--------|---------|
| Tokenizer | `jinja_tokenizer.nim` — `JinjaTokenizer` |
| Validator | `jinja_validator.nim` — `JinjaValidator` |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0003` | Unclosed Jinja comment (`{#` without `#}`) | `esError` | -- |
| `E2001` | Unclosed expression tag (`{{` without `}}`) | Implicit via tokenizer | -- |
| `E2001` | Unclosed block tag (`{%` without `%}`) | Implicit via tokenizer | -- |
| `E2002` | Unclosed `for` block (no matching `endfor`) | `esError` | "Add `{% endfor %}`" |
| `E2002` | Unclosed `if` block (no matching `endif`) | `esError` | "Add `{% endif %}`" |
| `E2002` | Unclosed `block` (no matching `endblock`) | `esError` | "Add `{% endblock %}`" |
| `E2002` | Unclosed `macro`, `filter`, `raw`, `autoescape`, `call`, `with` | `esError` | "Add matching `end{tag}`" |
Special features: tracks 12 block tag types (`for`, `if`, `block`, `macro`, `filter`, `raw`, `autoescape`, `call`, `with`), validates matching end tags, handles content between template syntax and plain text.
#### JSON (`.json`, `.jsonc`)
| Aspect | Details |
|--------|---------|
| Validator | `config_validators.nim` — `JSONValidator` |
| Method | Uses Nim's built-in `parseJson()` for full validation |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0007` | JSON parse error (any `JsonParsingError`) | `esError` | Error message from parser |
| `E0007` | Any other exception during parse | `esError` | "JSON error: {message}" |
| `E0009` | Empty JSON source | `esError` | -- |
#### YAML (`.yaml`, `.yml`)
| Aspect | Details |
|--------|---------|
| Validator | `config_validators.nim` — `YAMLValidator` |
| Method | Line-by-line structural analysis |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0007` | Empty mapping key (no key before `:`) | `esError` | -- |
| `E0007` | General YAML structural error | `esError` | -- |
| `E0009` | No content after stripping comments/blanks | `esError` | -- |
| `W1001` | Tab characters used for indentation | `esWarning` | "Use spaces for YAML indentation" |
Note: Empty YAML (only whitespace/comments) is `valid: true`.
#### TOML (`.toml`)
| Aspect | Details |
|--------|---------|
| Validator | `config_validators.nim` — `TOMLValidator` |
| Method | Line-by-line structural analysis |
| Error Code | Condition | Severity | Hint |
|------------|-----------|----------|------|
| `E0007` | Missing `=` in key-value pair | `esError` | "TOML uses `key = value` format" |
| `E0007` | Missing key before `=` | `esError` | -- |
| `E0007` | General TOML structural error | `esError` | -- |
| `E0009` | No content after stripping comments/blanks | `esError` | -- |
---
## CLI Usage
### Full CLI Reference
```
Nimcheck v0.1.0 - Universal Source Code Validation Framework
Usage:
nimcheck validate [--file=<path> | --source=<code>] [--flavor=<lang>]
nimcheck inspect [--file=<path> | --source=<code>] [--flavor=<lang>]
nimcheck detect [--file=<path> | --source=<code>]
nimcheck help
Options:
--file=<path> Source file to process
--source=<code> Source code string to process
--flavor=<lang> Language flavor (auto-detect if omitted)
--debug Enable debug mode
--json Output as JSON (default for inspect)
Supported flavors:
- bash
- html
- jinja
- jinja2
- js
- json
- jsonc
- nim
- nimble
- nims
- php
- phtml
- python
- py
- shell
- sh
- toml
- ts
- typescript
- yaml
- yml
```
### Exit Codes
| Exit Code | Meaning |
|-----------|---------|
| 0 | Validation successful (or help/version requested) |
| Non-zero (Nim default) | Runtime error |
Note: The `validate` command always exits 0 on the Nim level; the `valid` field in the output indicates pass/fail. This is by design — Nimcheck never crashes.
---
## Building and Testing
### Prerequisites
- Nim >= 2.0.0
- `make` (optional for Makefile targets)
### Build
```bash
# Using Makefile
make build # Release build -> bin/nimcheck
make build-debug # Debug build with -d:nimcheckDebug
# Using nimble
nimble build
# Manual
nim c src/nimcheck.nim
nim c -d:nimcheckDebug src/nimcheck.nim # With debug output
```
### Test
The test suite covers **250+ test cases** across all supported languages.
```bash
make test # Run all tests
make test-nim # Nim-specific tests only
make test-python # Python-specific tests only
make test-bash # Bash-specific tests only
make test-js # JavaScript/TypeScript tests
make test-php # PHP tests
make test-html # HTML/XML tests
make test-jinja # Jinja2 tests
make test-json # JSON tests
make test-yaml # YAML tests
make test-toml # TOML tests
make test-mixed # Mixed-language file tests
make debug # Build debug + run all tests
```
### Test Categories
| Suite | Files | Coverage |
|-------|-------|----------|
| Per-language basic | `test_<lang>.nim` | Valid + invalid source for each language |
| Per-language exhaustive | `test_<lang>_exhaustive.nim` | Edge cases: unclosed strings, unclosed comments, mismatched brackets, binary data, Unicode |
| Config formats | `test_config.nim` + `test_<json|yaml|toml>_exhaustive.nim` | JSON/YAML/TOML structural validation |
| Fuzz tests | `test_fuzz.nim` | Null bytes, binary data, deep nesting (100 levels), empty sources, concurrent validation |
| Mixed files | `test_mixed.nim` | HTML+Jinja combined, other mixed-format files |
| Common | `test_common.nim` | Shared test fixtures for valid and invalid code per language |
### Lint
```bash
make lint
# or
nim c --hints:on --warnings:on src/nimcheck.nim
```
### Clean
```bash
make clean
make rebuild # clean + build
```
---
## Project Structure
```
.
|-- Makefile
|-- nimcheck.nimble
|-- README.md
|-- LICENSE
|-- bin/ # Build output
| `-- nimcheck # Compiled binary
|-- src/
| |-- nimcheck.nim # Public API entry point + CLI (isMainModule)
| `-- nimcheck/
| |-- core/
| | |-- types.nim # All core types, toJson() methods, error code constants
| | |-- tokenizerbase.nim # Abstract TokenizerBase with position tracking, bracket stack, error recording
| | |-- validatorbase.nim # Abstract ValidatorBase with pipeline, factory, registry
| | |-- detector.nim # Language auto-detection (extension, shebang, content scoring)
| | |-- debug.nim # Debug logging (compiled only with -d:nimcheckDebug)
| | `-- init_validators.nim
| |-- tokenizers/
| | |-- nim_tokenizer.nim # Nim: """ strings, r"..." raw strings, #[ ]# nested comments, ## doc comments
| | |-- bash_tokenizer.nim # Bash: heredocs, $var/${var}, $(), backticks, (( ))
| | |-- python_tokenizer.nim # Python: f-strings, """ strings, decorators, indentation tracking
| | |-- javascript_tokenizer.nim # JS/TS: template literals, regex literals, arrow functions
| | |-- php_tokenizer.nim # PHP: <?php/?> mode switching, $variables
| | |-- html_tokenizer.nim # HTML/XML: tag balancing, void elements, attributes
| | `-- jinja_tokenizer.nim # Jinja2: {{ }}, {% %}, {# #} block detection
| |-- languages/
| | |-- nim_validator.nim # Analyzes Nim: imports, procs, types
| | |-- bash_validator.nim # Analyzes Bash: if/fi, for/done, case/esac matching
| | |-- python_validator.nim # Analyzes Python: imports, def, class
| | |-- javascript_validator.nim # Analyzes JS/TS: imports, functions, classes
| | |-- php_validator.nim # Analyzes PHP: functions, classes, includes
| | |-- html_validator.nim # Analyzes HTML/XML: tag balancing
| | |-- jinja_validator.nim # Analyzes Jinja2: block tag matching (12 tag types)
| | `-- config_validators.nim # JSON (via parseJson), YAML, TOML validators
| `-- reporting/
| `-- errors.nim # Human-readable and JSON error formatting
`-- tests/
|-- test_all.nim # Master test runner (imports all)
|-- test_nim.nim # Nim basic tests
|-- test_nim_exhaustive.nim # Nim edge cases
|-- test_bash.nim
|-- test_bash_exhaustive.nim
|-- test_python.nim
|-- test_python_exhaustive.nim
|-- test_javascript.nim
|-- test_javascript_exhaustive.nim
|-- test_php.nim
|-- test_php_exhaustive.nim
|-- test_html.nim
|-- test_html_exhaustive.nim
|-- test_jinja.nim
|-- test_jinja_exhaustive.nim
|-- test_json_exhaustive.nim
|-- test_yaml_exhaustive.nim
|-- test_toml_exhaustive.nim
|-- test_config.nim # Config format tests
|-- test_mixed.nim # Mixed-language file tests
|-- test_fuzz.nim # Fuzz and boundary tests
|-- test_common.nim # Shared test fixtures
`-- fixtures/ # Per-language test data files
|-- nim/
|-- bash/
|-- python/
|-- javascript/
|-- php/
|-- html/
|-- jinja/
|-- json/
|-- yaml/
|-- toml/
`-- mixed/
```
---
## Contributing
Contributions are welcome. Open an issue to discuss your idea before submitting a pull request.
### Adding a New Language
See `LESSONS_LEARNED.md` for the complete checklist (Section 14). The minimum steps are:
1. Create `src/nimcheck/tokenizers/<lang>_tokenizer.nim` inheriting from `TokenizerBase`
2. Implement the `tokenize()` method with language-specific lexing
3. Create `src/nimcheck/languages/<lang>_validator.nim` inheriting from `ValidatorBase`
4. Implement `createTokenizer()`, `analyzeTokens()`, and `buildModuleInfo()`
5. Add `registerValidator("flavor", constructor)` in the validator's `init()` proc
6. Import the validator in `src/nimcheck.nim`
7. Add extension mappings to `detector.nim`
8. Add test fixtures and test files under `tests/`
9. Update the test master runner `tests/test_all.nim`
### Design Principles
- **KISS**: Keep validators simple — tokenize, analyze brackets, report
- **Never crash**: Every validation path in a try/except
- **No external deps**: Pure Nim standard library
- **DRY**: Common logic in `tokenizerbase.nim` and `validatorbase.nim`
---
## License
MIT -- see [LICENSE](LICENSE) for details.
## Author
molodetz
Reference in New Issue View Git Blame Copy Permalink