wren/manual_src/pages/contributing/testing.html

{# retoor <retoor@molodetz.nl> #}
{% extends 'page.html' %}

{% set page_title = "Writing Tests" %}
{% set breadcrumb = [{"url": "contributing/index.html", "title": "Contributing"}, {"title": "Writing Tests"}] %}
{% set prev_page = {"url": "contributing/async-patterns.html", "title": "Async Patterns"} %}
{% set next_page = {"url": "contributing/documentation.html", "title": "Documentation"} %}

{% block article %}
                <h1>Writing Tests</h1>

                <p>Wren-CLI uses inline annotations in test files to specify expected behavior. The test runner <code>util/test.py</code> parses these annotations and verifies the output.</p>

                <h2>Test Directory Structure</h2>

                <pre><code>test/
  &lt;modulename&gt;/
    &lt;feature&gt;.wren           # Functional tests
    error_&lt;scenario&gt;.wren    # Error case tests (one runtime error each)</code></pre>

                <p>Each module has its own directory under <code>test/</code>. Test files are named descriptively based on what they test.</p>

                <h2>Running Tests</h2>

                <pre><code># Build and run all tests
make tests

# Run tests for a specific module
python3 util/test.py json

# Run a specific test file (prefix match)
python3 util/test.py json/parse

# Run with debug binary
python3 util/test.py --suffix=_d</code></pre>

                <h2>Test Annotations</h2>

                <table>
                    <tr>
                        <th>Annotation</th>
                        <th>Purpose</th>
                        <th>Expected Exit Code</th>
                    </tr>
                    <tr>
                        <td><code>// expect: output</code></td>
                        <td>Assert stdout line (matched in order)</td>
                        <td>0</td>
                    </tr>
                    <tr>
                        <td><code>// expect error</code></td>
                        <td>Assert compile error on this line</td>
                        <td>65</td>
                    </tr>
                    <tr>
                        <td><code>// expect error line N</code></td>
                        <td>Assert compile error on line N</td>
                        <td>65</td>
                    </tr>
                    <tr>
                        <td><code>// expect runtime error: msg</code></td>
                        <td>Assert runtime error with exact message</td>
                        <td>70</td>
                    </tr>
                    <tr>
                        <td><code>// expect handled runtime error: msg</code></td>
                        <td>Assert caught runtime error</td>
                        <td>0</td>
                    </tr>
                    <tr>
                        <td><code>// stdin: text</code></td>
                        <td>Feed text to stdin (repeatable)</td>
                        <td>-</td>
                    </tr>
                    <tr>
                        <td><code>// skip: reason</code></td>
                        <td>Skip this test file</td>
                        <td>-</td>
                    </tr>
                    <tr>
                        <td><code>// nontest</code></td>
                        <td>Ignore this file</td>
                        <td>-</td>
                    </tr>
                </table>

                <h2>Basic Test Example</h2>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "json" for Json

var obj = {"name": "test", "value": 42}
var str = Json.stringify(obj)
var parsed = Json.parse(str)

System.print(parsed["name"])  // expect: test
System.print(parsed["value"]) // expect: 42</code></pre>

                <p>Each <code>// expect:</code> annotation asserts that the corresponding line of output matches exactly.</p>

                <h2>Multiple Expect Annotations</h2>

                <p>Multiple expectations are matched in order:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "utils" for Utils

System.print(Utils.capitalize("hello")) // expect: Hello
System.print(Utils.capitalize("world")) // expect: World
System.print(Utils.capitalize("UPPER")) // expect: UPPER
System.print(Utils.capitalize(""))      // expect:</code></pre>

                <p>The empty <code>// expect:</code> matches an empty line.</p>

                <h2>Runtime Error Tests</h2>

                <p>For testing error conditions, create a separate file per error case:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "json" for Json

Json.parse("invalid json") // expect runtime error: Invalid JSON.</code></pre>

                <div class="admonition warning">
                    <div class="admonition-title">Warning</div>
                    <p>Only one runtime error per test file. The test runner tracks a single expected error. Split multiple error cases into separate files.</p>
                </div>

                <h3>Error Line Matching</h3>

                <p>The runtime error annotation must be on the line that calls the aborting code. The runner checks the stack trace for a <code>test/...</code> path and matches the line number.</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "io" for File

var content = File.read("/nonexistent/path") // expect runtime error: Cannot open file.</code></pre>

                <h2>Compile Error Tests</h2>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

class Test {
    foo( // expect error
}</code></pre>

                <p>Or specify a different line number:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

// This is valid
var x = 1

class Broken {
    // expect error line 7
    foo(</code></pre>

                <h2>Stdin Input</h2>

                <p>Use <code>// stdin:</code> to provide input:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "io" for Stdin

// stdin: hello
// stdin: world

var line1 = Stdin.readLine()
var line2 = Stdin.readLine()

System.print(line1) // expect: hello
System.print(line2) // expect: world</code></pre>

                <p>Multiple <code>// stdin:</code> lines are concatenated with newlines.</p>

                <h2>Skipping Tests</h2>

                <p>Use <code>// skip:</code> for tests that cannot run in all environments:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;
// skip: Requires network access

import "http" for Http

var response = Http.get("https://example.com")
System.print(response.status) // expect: 200</code></pre>

                <h2>Non-Test Files</h2>

                <p>Use <code>// nontest</code> for helper files that should not be run as tests:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;
// nontest

class TestHelper {
    static setup() { ... }
}</code></pre>

                <h2>Test File Organization</h2>

                <h3>Feature Tests</h3>

                <p>Test each feature in a dedicated file:</p>

                <pre><code>test/json/
  parse.wren           # Basic parsing
  parse_nested.wren    # Nested objects/arrays
  stringify.wren       # JSON serialization
  stringify_pretty.wren # Pretty printing
  types.wren           # Type handling</code></pre>

                <h3>Error Tests</h3>

                <p>One error per file, named with <code>error_</code> prefix:</p>

                <pre><code>test/json/
  error_invalid_syntax.wren
  error_unexpected_eof.wren
  error_invalid_escape.wren</code></pre>

                <h2>Handled Runtime Error</h2>

                <p>For testing error handling where errors are caught:</p>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "json" for Json

var result = Fiber.new {
    Json.parse("bad")
}.try()

if (result.error) {
    System.print("Caught error") // expect: Caught error
} // expect handled runtime error: Invalid JSON.</code></pre>

                <h2>Test Timeout</h2>

                <p>Each test file has a 15-second timeout. If a test hangs (e.g., waiting for input that never comes), it will be killed.</p>

                <h2>Test Discovery</h2>

                <p>The test runner discovers tests by:</p>

                <ol>
                    <li>Walking the <code>test/</code> directory recursively</li>
                    <li>Filtering by <code>.wren</code> extension</li>
                    <li>Converting paths to relative paths from <code>test/</code></li>
                    <li>Checking if path starts with the filter argument</li>
                </ol>

                <p>This means:</p>
                <ul>
                    <li><code>python3 util/test.py json</code> runs everything in <code>test/json/</code></li>
                    <li><code>python3 util/test.py io/file</code> runs everything in <code>test/io/file/</code></li>
                    <li>Subdirectories are supported for organizing large test suites</li>
                </ul>

                <h2>Example Test Suite</h2>

                <pre><code>test/mymodule/
  basic.wren
  advanced.wren
  edge_cases.wren
  error_null_input.wren
  error_invalid_type.wren
  error_overflow.wren</code></pre>

                <h3>basic.wren</h3>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "mymodule" for MyClass

System.print(MyClass.process("hello"))  // expect: HELLO
System.print(MyClass.process("world"))  // expect: WORLD
System.print(MyClass.length("test"))    // expect: 4</code></pre>

                <h3>error_null_input.wren</h3>

                <pre><code>// retoor &lt;retoor@molodetz.nl&gt;

import "mymodule" for MyClass

MyClass.process(null) // expect runtime error: Input cannot be null.</code></pre>

                <h2>Debugging Failing Tests</h2>

                <ol>
                    <li>Run the test manually: <code>bin/wren_cli test/mymodule/failing.wren</code></li>
                    <li>Check the actual output versus expected</li>
                    <li>Verify annotations are on correct lines</li>
                    <li>For runtime errors, ensure annotation is on the calling line</li>
                </ol>

                <h2>Best Practices</h2>

                <ul>
                    <li>Test one concept per file when possible</li>
                    <li>Use descriptive file names</li>
                    <li>Always include the author comment</li>
                    <li>Test edge cases (empty strings, null, large values)</li>
                    <li>Test error conditions in separate files</li>
                    <li>Keep tests simple and focused</li>
                </ul>

                <h2>Next Steps</h2>

                <p>After writing tests, add documentation in <a href="documentation.html">Documentation</a>.</p>
{% endblock %}