nimcheck/README.md

# Nimcheck

A multi-language source code validation framework written in Nim. Nimcheck uses tokenizer-based validation to check syntax correctness across 10 languages and config formats, with auto-detection, comprehensive diagnostics, and a never-crash guarantee.

## Features

- **10 languages supported**: Nim, Bash, Python, JavaScript, PHP, HTML, Jinja, JSON, YAML, TOML
- **Tokenizer-based validation** -- real tokenizers that track bracket depth, string context, comments, and nesting
- **Auto-detection** -- detect language from file extension or content analysis
- **Comprehensive diagnostics** -- structured error reports with position, severity, hints, and debug output
- **Never-crash guarantee** -- every validation path is wrapped in try/except; invalid input returns errors gracefully, never panics
- **JSON output** -- machine-readable diagnostics via `toJson()` on any validation result
- **Inspection API** -- extract module structure (imports, definitions, functions/classes) as JSON
- **Debug mode** -- compile with `-d:nimcheckDebug` for detailed tokenization logging
- **No external dependencies** -- pure Nim standard library

## Quick Start

```nim
import nimcheck

# Validate source code from a string
let r = validateSource("proc hello() = echo 'hi'", lfNim)
echo r.errors  # seq[ValidationError]

# Auto-detect language
let r2 = validateSource("print('hello')", lfUnknown)  # detects Python

# Validate a file
let r3 = validateFile("path/to/file.nim", lfNim)

# Get JSON diagnostics
echo r.toJson().pretty()

# Inspect source structure (returns JsonNode)
let info = inspectSource("class Foo: ...", lfPython)
echo info.pretty()  # formatted JSON with class/function info

# Get human-readable report
let report = reportSource("proc foo() =", lfNim)
echo report  # formatted error summary
```

## Building

### Using Makefile (recommended)

```bash
# Build the library binary
make build

# Run all tests
make test

# Build with debug mode
make build-debug
make debug       # build debug + run all tests

# Run a specific language test
make test-nim
make test-python
make test-bash

# Lint with full warnings
make lint

# Clean build artifacts
make clean

# Show all available targets
make help
```

### Using nimble

```bash
# Run all tests
nimble test

# Run per-language tests
nimble test_nim
nimble test_python

# Build the library
nimble build
```

### Manual compilation

```bash
# Build the library
nim c src/nimcheck.nim

# Build with debug mode
nim c -d:nimcheckDebug src/nimcheck.nim

# Run all tests
nim c -r tests/test_all.nim
```

## CLI Usage

When compiled as a standalone binary (`make build` produces `bin/nimcheck`):

```bash
# Validate a file
bin/nimcheck validate --file=source.nim

# Validate source code directly
bin/nimcheck validate --source="print('hi')" --flavor=python

# Read from stdin
echo "var x = 1" | bin/nimcheck validate

# Inspect source structure (returns JSON)
bin/nimcheck inspect --file=source.py

# Detect language
bin/nimcheck detect --file=unknown.txt
bin/nimcheck detect --source="#!/usr/bin/env bash"

# Version and help
bin/nimcheck version
bin/nimcheck help
```

## API Reference

### Core Functions

| Function | Description |
|---|---|
| `validateSource(source, flavor, options, sourcePath)` | Validate source code string; returns `ValidationResult` |
| `validateFile(filePath, flavor, options)` | Validate a file; returns `ValidationResult` |
| `inspectSource(source, flavor, options, sourcePath)` | Return structural diagnostics as `JsonNode` |
| `inspectFile(filePath, flavor, options)` | Return structural diagnostics for a file as `JsonNode` |
| `validateSourceJson(source, flavor)` | Validate source and return result as JSON string |
| `validateFileJson(filePath, flavor)` | Validate file and return result as JSON string |
| `reportSource(source, flavor)` | Get a human-readable validation report (string) |
| `reportFile(filePath, flavor)` | Get a human-readable validation report for a file (string) |
| `detectFlavor(source, filePath)` | Auto-detect language flavor |
| `supportedFlavors()` | List all registered language flavors (returns `seq[string]`) |
| `version()` | Get the Nimcheck version string |
| `enableDebug()` / `disableDebug()` | Toggle debug mode globally |

### Types

- **`ValidationResult`** -- contains `errors: seq[ValidationError]`, `valid: bool`, `moduleInfo`, `debugOutput`, `detectionInfo`
- **`ValidationError`** -- contains `code`, `message`, `severity`, `position`, `hint`, `context`
- **`ValidationOptions`** -- contains `flavor`, `debugMode`, `maxErrors`, `validateAll`
- **`LanguageFlavor`** -- enum: `lfUnknown`, `lfNim`, `lfBash`, `lfPython`, `lfJavaScript`, `lfPHP`, `lfHTML`, `lfJinja`, `lfJSON`, `lfYAML`, `lfTOML`

### JSON Output

Both `ValidationResult` and `ValidationError` have a `.toJson()` method returning a `JsonNode`. Use `.pretty()` for formatted output.

## Error Codes

| Code | Meaning |
|---|---|
| `E1000` | Unexpected/unmatched bracket |
| `E1001` | Unclosed bracket |
| `E1002` | Unmatched closing bracket |
| `E2000` | Unclosed string literal |
| `E2001` | Invalid string escape |
| `E2002` | Unclosed multi-line string |
| `E3000` | Unclosed comment |
| `E3001` | Invalid comment syntax |
| `E4000` | Invalid syntax |
| `E4001` | Reserved word misuse |
| `E5000` | Unclosed block/control structure |
| `E9999` | Internal error (never-crash fallback) |

## Supported Languages

| Language | File Extensions | Tokenizer Features |
|---|---|---|
| Nim | `.nim`, `.nims`, `.nimble` | Block comments `#[ ]#` with nesting, triple-quoted strings, raw strings, doc comments |
| Bash | `.sh`, `.bash` | Shebang detection, heredocs, command substitution, variable expansion, fi/done/esac matching |
| Python | `.py` | F-strings with nesting, triple-quoted strings, decorators, async/await, match statements |
| JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | Template literals with nesting, regex literals, arrow functions, ESM imports |
| PHP | `.php` | `<?php` tags, heredocs/nowdocs, variable variables |
| HTML | `.html`, `.htm` | Tag balancing, self-closing tags, attributes |
| Jinja | `.j2`, `.jinja`, `.html.j2` | `{% %}`, `{{ }}`, `{# #}` block detection, nested blocks |
| JSON | `.json` | Bracket balance, string validation, comma checking |
| YAML | `.yaml`, `.yml` | Indentation tracking, key-value checking |
| TOML | `.toml` | Table headers, bracket balance, string validation |

## Testing

The test suite covers **250+ test cases** across all supported languages:

- **Per-language tests** -- valid and invalid source for each of the 10 languages
- **Exhaustive tests** -- edge cases: unclosed strings, unclosed comments, mismatched brackets, binary data, unicode bombs
- **Fuzz tests** -- null bytes, binary data, deep nesting (100 levels), empty sources, concurrent validation
- **Config format tests** -- JSON, YAML, TOML validation with invalid syntax checks
- **Mixed-file tests** -- HTML+Jinja combined files
- **Never-crash guarantee** -- every test verifies invalid input returns graceful errors, not crashes
- **Debug mode tests** -- `make debug` runs all tests with debug logging enabled

Run all tests with:

```bash
make test         # or
nimble test       # or
nim c -r tests/test_all.nim
```

## Development

### Prerequisites

- Nim >= 2.0.0
- `make` (optional, for Makefile targets)

### Project structure

```
src/
  nimcheck.nim          Public API entry point and CLI (isMainModule)
  nimcheck/
    core/
      types.nim           Core types: ValidationResult, ValidationError, LanguageFlavor
      tokenizerbase.nim   Abstract tokenizer base class
      validatorbase.nim   Abstract validator base class
      detector.nim        Language auto-detection
      debug.nim           Debug logging infrastructure
    tokenizers/
      nim_tokenizer.nim
      bash_tokenizer.nim
      python_tokenizer.nim
      javascript_tokenizer.nim
      php_tokenizer.nim
      html_tokenizer.nim
      jinja_tokenizer.nim
    languages/
      nim_validator.nim
      bash_validator.nim
      python_validator.nim
      javascript_validator.nim
      php_validator.nim
      html_validator.nim
      jinja_validator.nim
      config_validators.nim   JSON, YAML, TOML validators
    reporting/
      errors.nim          Error formatting and JSON output
tests/
  test_all.nim            Master test runner (imports all test modules)
  test_<lang>.nim         Per-language tests
  test_<lang>_exhaustive.nim  Exhaustive edge-case tests per language
  test_config.nim         JSON/YAML/TOML tests
  test_fuzz.nim           Fuzz and boundary tests
  test_common.nim         Shared test utilities
  test_mixed.nim          Mixed-language file tests
  fixtures/               Test data files per language
```

### Building with warnings

```bash
make lint
# or manually:
nim c --hints:on --warnings:on src/nimcheck.nim
```

### Adding a new language

See the detailed checklist in [LESSONS_LEARNED.md](LESSONS_LEARNED.md) (Section 14).

## Architecture

The framework uses a layered architecture:

1. **Tokenizers** -- convert raw source into a stream of tokens, tracking bracket depth, string context, and comments
2. **Validators** -- analyze token streams for language-specific syntax rules
3. **Detector** -- identifies language from file extension, shebang, or content heuristics
4. **Reporting** -- formats errors as human-readable strings or structured JSON

Validators register themselves at import time via a side-effect factory pattern. The public API (`nimcheck.nim`) imports all validators and re-exports core types for consumer convenience.

## Versioning

Current version: **0.1.0**

## Contributing

Contributions are welcome. Please open an issue to discuss your idea before
submitting a pull request. For detailed instructions on adding support for a new
language, see [LESSONS_LEARNED.md](LESSONS_LEARNED.md) Section 16.

## License

This project is licensed under the MIT License -- see the [LICENSE](LICENSE) file
for details.

## Author

molodetz