2026-03-22-lazy-evaluation.md (40247B)
1 # Lazy Evaluation Implementation Plan 2 3 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. 4 5 **Goal:** Add `nickel_open` / `NickelValue` API for lazy, on-demand evaluation of Nickel configs via the C API's shallow eval. 6 7 **Architecture:** `NickelSession` owns the `nickel_context` and tracks all expr allocations. `NickelValue` is an immutable wrapper around a single expr + session back-reference. `nickel_open` evaluates shallowly; navigation (`getproperty`/`getindex`) evaluates sub-expressions on demand. `collect` materializes an entire subtree eagerly. 8 9 **Tech Stack:** Julia, Nickel C API (ccall via LibNickel module in `src/libnickel.jl`) 10 11 **Spec:** `docs/superpowers/specs/2026-03-21-lazy-evaluation-design.md` 12 13 **Design deviation:** `NickelSession` has no `root` field (avoids circular type reference). `nickel_open` returns `NickelValue` directly in both do-block and manual modes. `close(::NickelValue)` delegates to `close(session)`. 14 15 --- 16 17 ### File Structure 18 19 - **Modify:** `src/NickelEval.jl` — add `NickelSession` and `NickelValue` type definitions, new exports 20 - **Modify:** `src/ffi.jl` — add `nickel_open`, navigation, `collect`, inspection, iteration, `show` 21 - **Create:** `test/test_lazy.jl` — all lazy evaluation tests 22 - **Modify:** `test/runtests.jl` — include `test_lazy.jl` 23 - **Modify:** `docs/src/lib/public.md` — add new exports to docs 24 25 --- 26 27 ### Task 1: Define types and wire up test file 28 29 Define the two new types in `NickelEval.jl` and create the test file skeleton. 30 31 **Files:** 32 - Modify: `src/NickelEval.jl` (lines 1-6 for exports, add types before `include("ffi.jl")`) 33 - Create: `test/test_lazy.jl` 34 - Modify: `test/runtests.jl` (add include for test_lazy.jl) 35 36 - [ ] **Step 1: Add types to `src/NickelEval.jl`** 37 38 Add after the `NickelEnum` definition (after line 69), before `include("ffi.jl")`: 39 40 ```julia 41 # ── Lazy evaluation types ───────────────────────────────────────────────────── 42 43 """ 44 NickelSession 45 46 Owns a Nickel evaluation context for lazy (shallow) evaluation. 47 Tracks all allocated expressions and frees them on `close`. 48 49 Not thread-safe. All access must occur on a single thread. 50 """ 51 mutable struct NickelSession 52 ctx::Ptr{Cvoid} # Ptr{LibNickel.nickel_context} — Cvoid avoids forward ref 53 exprs::Vector{Ptr{Cvoid}} # tracked allocations, freed on close 54 closed::Bool 55 end 56 57 """ 58 NickelValue 59 60 A lazy reference to a Nickel expression. Accessing fields (`.field` or `["field"]`) 61 evaluates only the requested sub-expression. Use `collect` to materialize the 62 full subtree into plain Julia types. 63 64 # Examples 65 ```julia 66 nickel_open("{ x = 1, y = { z = 2 } }") do cfg 67 cfg.x # => 1 68 cfg.y.z # => 2 69 collect(cfg) # => Dict("x" => 1, "y" => Dict("z" => 2)) 70 end 71 ``` 72 """ 73 struct NickelValue 74 session::NickelSession 75 expr::Ptr{Cvoid} # Ptr{LibNickel.nickel_expr} 76 end 77 ``` 78 79 Note: We use `Ptr{Cvoid}` here because the `LibNickel` module hasn't been loaded yet at this point in the file. The actual C API calls in `ffi.jl` will `reinterpret` these pointers to the correct types. This is safe because all Nickel opaque types are just pointer-sized handles. 80 81 - [ ] **Step 2: Add exports to `src/NickelEval.jl`** 82 83 Add a new export line after line 5: 84 85 ```julia 86 export nickel_open, NickelValue, NickelSession, nickel_kind 87 ``` 88 89 - [ ] **Step 3: Create `test/test_lazy.jl` skeleton** 90 91 ```julia 92 @testset "Lazy Evaluation" begin 93 @testset "nickel_open returns NickelValue" begin 94 result = nickel_open("{ x = 1 }") do cfg 95 cfg 96 end 97 @test result isa NickelValue 98 end 99 end 100 ``` 101 102 - [ ] **Step 4: Add include to `test/runtests.jl`** 103 104 After the `include("test_eval.jl")` line (line 6), add: 105 106 ```julia 107 include("test_lazy.jl") 108 ``` 109 110 - [ ] **Step 5: Run tests — expect failure** 111 112 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 113 114 Expected: FAIL — `nickel_open` is not defined yet. This confirms the test is wired up correctly. 115 116 - [ ] **Step 6: Commit types and test skeleton** 117 118 ```bash 119 git add src/NickelEval.jl test/test_lazy.jl test/runtests.jl 120 git commit -m "feat: add NickelSession/NickelValue types and test skeleton" 121 ``` 122 123 --- 124 125 ### Task 2: Implement `nickel_open` (code string, do-block) 126 127 The core function: shallow-eval a code string and return a `NickelValue`. 128 129 **Files:** 130 - Modify: `src/ffi.jl` (append after line 384) 131 132 - [ ] **Step 1: Add session helpers to `src/ffi.jl`** 133 134 Append to end of `src/ffi.jl`: 135 136 ```julia 137 # ── Lazy evaluation ────────────────────────────────────────────────────────── 138 139 function _check_session_open(session::NickelSession) 140 session.closed && throw(ArgumentError("NickelSession is closed")) 141 end 142 143 # Allocate a new expr tracked by the session 144 function _tracked_expr_alloc(session::NickelSession) 145 expr = L.nickel_expr_alloc() 146 push!(session.exprs, Ptr{Cvoid}(expr)) 147 return expr 148 end 149 150 function Base.close(session::NickelSession) 151 session.closed && return 152 session.closed = true 153 for expr_ptr in session.exprs 154 L.nickel_expr_free(Ptr{L.nickel_expr}(expr_ptr)) 155 end 156 empty!(session.exprs) 157 L.nickel_context_free(Ptr{L.nickel_context}(session.ctx)) 158 return nothing 159 end 160 161 Base.close(v::NickelValue) = close(getfield(v, :session)) 162 ``` 163 164 - [ ] **Step 2: Add `nickel_open` for code strings** 165 166 Append to `src/ffi.jl`: 167 168 ```julia 169 """ 170 nickel_open(f, code::String) 171 nickel_open(code::String) -> NickelValue 172 173 Evaluate Nickel code shallowly and return a lazy `NickelValue`. 174 Sub-expressions are evaluated on demand when accessed via `.field` or `["field"]`. 175 176 # Do-block (preferred) 177 ```julia 178 nickel_open("{ x = 1, y = 2 }") do cfg 179 cfg.x # => 1 (only evaluates x) 180 end 181 ``` 182 183 # Manual 184 ```julia 185 cfg = nickel_open("{ x = 1, y = 2 }") 186 cfg.x # => 1 187 close(cfg) 188 ``` 189 """ 190 function nickel_open(f::Function, code::String) 191 val = nickel_open(code) 192 try 193 return f(val) 194 finally 195 close(val) 196 end 197 end 198 199 function nickel_open(code::String) 200 _check_ffi_available() 201 ctx = L.nickel_context_alloc() 202 session = NickelSession(Ptr{Cvoid}(ctx), Ptr{Cvoid}[], false) 203 finalizer(close, session) # safety net: free resources if user forgets close() 204 expr = _tracked_expr_alloc(session) 205 err = L.nickel_error_alloc() 206 try 207 result = L.nickel_context_eval_shallow(ctx, code, expr, err) 208 if result == L.NICKEL_RESULT_ERR 209 _throw_nickel_error(err) 210 end 211 return NickelValue(session, Ptr{Cvoid}(expr)) 212 catch 213 close(session) 214 rethrow() 215 finally 216 L.nickel_error_free(err) 217 end 218 end 219 ``` 220 221 Key logic: 222 - `nickel_context_eval_shallow` evaluates to WHNF — the top-level structure is known (it's a record, array, etc.) but its children remain unevaluated. 223 - The `err` is freed immediately (it's only needed during eval). The `ctx` and `expr` live on inside the session. 224 - On error, the session is closed to free the context. 225 - The do-block variant uses `try/finally` to guarantee cleanup. 226 - A GC `finalizer` on the session is a safety net for manual mode; users should still call `close` explicitly. 227 228 - [ ] **Step 3: Run tests** 229 230 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 231 232 Expected: The test from Task 1 should now... actually fail because we return `cfg` from the do-block but the session is closed at that point. Update the test: 233 234 - [ ] **Step 4: Fix test to check type inside do-block** 235 236 Update `test/test_lazy.jl`: 237 238 ```julia 239 @testset "Lazy Evaluation" begin 240 @testset "nickel_open returns NickelValue" begin 241 is_nickel_value = nickel_open("{ x = 1 }") do cfg 242 cfg isa NickelValue 243 end 244 @test is_nickel_value 245 end 246 end 247 ``` 248 249 - [ ] **Step 5: Run tests — expect pass** 250 251 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 252 253 Expected: PASS 254 255 - [ ] **Step 6: Commit** 256 257 ```bash 258 git add src/ffi.jl test/test_lazy.jl 259 git commit -m "feat: implement nickel_open with shallow evaluation" 260 ``` 261 262 --- 263 264 ### Task 3: Implement `nickel_kind` and `show` 265 266 Before we can navigate, we need to inspect what kind of value we have. 267 268 **Files:** 269 - Modify: `src/ffi.jl` 270 - Modify: `test/test_lazy.jl` 271 272 - [ ] **Step 1: Write failing tests** 273 274 Add to `test/test_lazy.jl`: 275 276 ```julia 277 @testset "nickel_kind" begin 278 nickel_open("{ x = 1 }") do cfg 279 @test nickel_kind(cfg) == :record 280 end 281 nickel_open("[1, 2, 3]") do cfg 282 @test nickel_kind(cfg) == :array 283 end 284 nickel_open("42") do cfg 285 @test nickel_kind(cfg) == :number 286 end 287 nickel_open("\"hello\"") do cfg 288 @test nickel_kind(cfg) == :string 289 end 290 nickel_open("true") do cfg 291 @test nickel_kind(cfg) == :bool 292 end 293 nickel_open("null") do cfg 294 @test nickel_kind(cfg) == :null 295 end 296 end 297 298 @testset "show" begin 299 nickel_open("{ x = 1, y = 2, z = 3 }") do cfg 300 s = repr(cfg) 301 @test occursin("record", s) 302 @test occursin("3", s) 303 end 304 nickel_open("[1, 2]") do cfg 305 s = repr(cfg) 306 @test occursin("array", s) 307 @test occursin("2", s) 308 end 309 end 310 ``` 311 312 - [ ] **Step 2: Run tests to verify failure** 313 314 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 315 316 Expected: FAIL — `nickel_kind` not defined 317 318 - [ ] **Step 3: Implement `nickel_kind`** 319 320 Append to `src/ffi.jl`: 321 322 ```julia 323 """ 324 nickel_kind(v::NickelValue) -> Symbol 325 326 Return the kind of a lazy Nickel value without evaluating its children. 327 328 Returns one of: `:record`, `:array`, `:number`, `:string`, `:bool`, `:null`, `:enum`. 329 """ 330 function nickel_kind(v::NickelValue) 331 _check_session_open(getfield(v, :session)) 332 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 333 if L.nickel_expr_is_null(expr) != 0 334 return :null 335 elseif L.nickel_expr_is_bool(expr) != 0 336 return :bool 337 elseif L.nickel_expr_is_number(expr) != 0 338 return :number 339 elseif L.nickel_expr_is_str(expr) != 0 340 return :string 341 elseif L.nickel_expr_is_array(expr) != 0 342 return :array 343 elseif L.nickel_expr_is_record(expr) != 0 344 return :record 345 elseif L.nickel_expr_is_enum_variant(expr) != 0 || L.nickel_expr_is_enum_tag(expr) != 0 346 return :enum 347 else 348 error("Unknown Nickel expression type") 349 end 350 end 351 ``` 352 353 Note: `getfield(v, :expr)` is used instead of `v.expr` because `getproperty` will be overridden later to navigate Nickel records. Same for `getfield(v, :session)`. 354 355 - [ ] **Step 4: Implement `show`** 356 357 Append to `src/ffi.jl`: 358 359 ```julia 360 function Base.show(io::IO, v::NickelValue) 361 session = getfield(v, :session) 362 if session.closed 363 print(io, "NickelValue(<closed>)") 364 return 365 end 366 k = nickel_kind(v) 367 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 368 if k == :record 369 rec = L.nickel_expr_as_record(expr) 370 n = Int(L.nickel_record_len(rec)) 371 print(io, "NickelValue(:record, $n field", n == 1 ? "" : "s", ")") 372 elseif k == :array 373 arr = L.nickel_expr_as_array(expr) 374 n = Int(L.nickel_array_len(arr)) 375 print(io, "NickelValue(:array, $n element", n == 1 ? "" : "s", ")") 376 else 377 print(io, "NickelValue(:$k)") 378 end 379 end 380 ``` 381 382 - [ ] **Step 5: Run tests — expect pass** 383 384 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 385 386 Expected: PASS 387 388 - [ ] **Step 6: Commit** 389 390 ```bash 391 git add src/ffi.jl test/test_lazy.jl 392 git commit -m "feat: add nickel_kind and show for NickelValue" 393 ``` 394 395 --- 396 397 ### Task 4: Implement record navigation (`getproperty` / `getindex`) 398 399 This is the core lazy behavior: `cfg.database.port` evaluates only the path you walk. 400 401 **Files:** 402 - Modify: `src/ffi.jl` 403 - Modify: `test/test_lazy.jl` 404 405 - [ ] **Step 1: Write failing tests** 406 407 Add to `test/test_lazy.jl`: 408 409 ```julia 410 @testset "Record field access" begin 411 # getproperty (dot syntax) 412 nickel_open("{ x = 42 }") do cfg 413 @test cfg.x === Int64(42) 414 end 415 416 # getindex (bracket syntax) 417 nickel_open("{ x = 42 }") do cfg 418 @test cfg["x"] === Int64(42) 419 end 420 421 # Nested navigation 422 nickel_open("{ a = { b = { c = 99 } } }") do cfg 423 @test cfg.a.b.c === Int64(99) 424 end 425 426 # Mixed types in record 427 nickel_open("{ name = \"test\", count = 42, flag = true }") do cfg 428 @test cfg.name == "test" 429 @test cfg.count === Int64(42) 430 @test cfg.flag === true 431 end 432 433 # Null field 434 nickel_open("{ x = null }") do cfg 435 @test cfg.x === nothing 436 end 437 end 438 ``` 439 440 - [ ] **Step 2: Run tests to verify failure** 441 442 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 443 444 Expected: FAIL 445 446 - [ ] **Step 3: Implement `_resolve_value` helper** 447 448 This helper takes a shallow-evaluated expr and returns either a Julia primitive or a new `NickelValue`. It's used by `getproperty`, `getindex`, and `iterate`. 449 450 Append to `src/ffi.jl` (before `nickel_kind`): 451 452 ```julia 453 # Given a shallow-eval'd expr, return a Julia value (primitive) or NickelValue (compound). 454 # The expr must already be tracked by the session. 455 function _resolve_value(session::NickelSession, expr::Ptr{L.nickel_expr}) 456 if L.nickel_expr_is_null(expr) != 0 457 return nothing 458 elseif L.nickel_expr_is_bool(expr) != 0 459 return L.nickel_expr_as_bool(expr) != 0 460 elseif L.nickel_expr_is_number(expr) != 0 461 num = L.nickel_expr_as_number(expr) 462 if L.nickel_number_is_i64(num) != 0 463 return L.nickel_number_as_i64(num) 464 else 465 return Float64(L.nickel_number_as_f64(num)) 466 end 467 elseif L.nickel_expr_is_str(expr) != 0 468 out_ptr = Ref{Ptr{Cchar}}(C_NULL) 469 len = L.nickel_expr_as_str(expr, out_ptr) 470 return unsafe_string(out_ptr[], len) 471 elseif L.nickel_expr_is_enum_tag(expr) != 0 472 out_ptr = Ref{Ptr{Cchar}}(C_NULL) 473 len = L.nickel_expr_as_enum_tag(expr, out_ptr) 474 tag = Symbol(unsafe_string(out_ptr[], len)) 475 return NickelEnum(tag, nothing) 476 else 477 # record, array, or enum variant — stay lazy 478 return NickelValue(session, Ptr{Cvoid}(expr)) 479 end 480 end 481 ``` 482 483 Key logic: primitives (null, bool, number, string, bare enum tag) are converted to Julia values immediately. Compound types (record, array, enum variant) are wrapped in a new `NickelValue` for lazy access. 484 485 - [ ] **Step 4: Implement `_eval_and_resolve` helper** 486 487 This combines "shallow-eval a sub-expression" and "resolve to Julia value or NickelValue": 488 489 ```julia 490 # Evaluate a sub-expression shallowly, then resolve. 491 function _eval_and_resolve(session::NickelSession, sub_expr::Ptr{L.nickel_expr}) 492 ctx = Ptr{L.nickel_context}(session.ctx) 493 out_expr = _tracked_expr_alloc(session) 494 err = L.nickel_error_alloc() 495 try 496 result = L.nickel_context_eval_expr_shallow(ctx, sub_expr, out_expr, err) 497 if result == L.NICKEL_RESULT_ERR 498 _throw_nickel_error(err) 499 end 500 return _resolve_value(session, out_expr) 501 catch 502 rethrow() 503 finally 504 L.nickel_error_free(err) 505 end 506 end 507 ``` 508 509 - [ ] **Step 5: Implement `getproperty` and string `getindex`** 510 511 ```julia 512 function Base.getproperty(v::NickelValue, name::Symbol) 513 return _lazy_field_access(v, String(name)) 514 end 515 516 function Base.getindex(v::NickelValue, key::String) 517 return _lazy_field_access(v, key) 518 end 519 520 function _lazy_field_access(v::NickelValue, key::String) 521 session = getfield(v, :session) 522 _check_session_open(session) 523 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 524 if L.nickel_expr_is_record(expr) == 0 525 throw(ArgumentError("Cannot access field '$key': NickelValue is not a record")) 526 end 527 rec = L.nickel_expr_as_record(expr) 528 out_expr = _tracked_expr_alloc(session) 529 has_value = L.nickel_record_value_by_name(rec, key, out_expr) 530 if has_value == 0 531 # Field not found or has no value (contract-only field in shallow eval). 532 # Check whether the key exists at all by scanning keys. 533 n = Int(L.nickel_record_len(rec)) 534 found = false 535 key_ptr = Ref{Ptr{Cchar}}(C_NULL) 536 key_len = Ref{Csize_t}(0) 537 for i in 0:(n-1) 538 L.nickel_record_key_value_by_index(rec, Csize_t(i), key_ptr, key_len, 539 Ptr{L.nickel_expr}(C_NULL)) 540 if unsafe_string(key_ptr[], key_len[]) == key 541 found = true 542 break 543 end 544 end 545 if !found 546 throw(NickelError("Field '$key' not found in record")) 547 end 548 # Key exists but has no value — this can happen with contract-only fields 549 # in shallow eval. Return nothing as the value is not available. 550 throw(NickelError("Field '$key' has no value (contract-only or unevaluated)")) 551 end 552 return _eval_and_resolve(session, out_expr) 553 end 554 ``` 555 556 Note: `nickel_record_value_by_name` looks up the field by name directly — O(1) for hash-based records. Much faster than iterating all fields with `nickel_record_key_value_by_index`. 557 558 - [ ] **Step 6: Run tests — expect pass** 559 560 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 561 562 Expected: PASS 563 564 - [ ] **Step 7: Commit** 565 566 ```bash 567 git add src/ffi.jl test/test_lazy.jl 568 git commit -m "feat: add lazy record navigation via getproperty/getindex" 569 ``` 570 571 --- 572 573 ### Task 5: Implement array indexing 574 575 **Files:** 576 - Modify: `src/ffi.jl` 577 - Modify: `test/test_lazy.jl` 578 579 - [ ] **Step 1: Write failing tests** 580 581 Add to `test/test_lazy.jl`: 582 583 ```julia 584 @testset "Array access" begin 585 nickel_open("[10, 20, 30]") do cfg 586 @test cfg[1] === Int64(10) 587 @test cfg[2] === Int64(20) 588 @test cfg[3] === Int64(30) 589 end 590 591 # Array of records (lazy) 592 nickel_open("[{ x = 1 }, { x = 2 }]") do cfg 593 @test cfg[1].x === Int64(1) 594 @test cfg[2].x === Int64(2) 595 end 596 597 # Nested: record containing array 598 nickel_open("{ items = [10, 20, 30] }") do cfg 599 @test cfg.items[2] === Int64(20) 600 end 601 end 602 ``` 603 604 - [ ] **Step 2: Run tests to verify failure** 605 606 Expected: FAIL — integer `getindex` not defined for `NickelValue` 607 608 - [ ] **Step 3: Implement integer `getindex`** 609 610 Append to `src/ffi.jl`: 611 612 ```julia 613 function Base.getindex(v::NickelValue, idx::Integer) 614 session = getfield(v, :session) 615 _check_session_open(session) 616 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 617 if L.nickel_expr_is_array(expr) == 0 618 throw(ArgumentError("Cannot index with integer: NickelValue is not an array")) 619 end 620 arr = L.nickel_expr_as_array(expr) 621 n = Int(L.nickel_array_len(arr)) 622 if idx < 1 || idx > n 623 throw(BoundsError(v, idx)) 624 end 625 out_expr = _tracked_expr_alloc(session) 626 L.nickel_array_get(arr, Csize_t(idx - 1), out_expr) # 0-based C API 627 return _eval_and_resolve(session, out_expr) 628 end 629 ``` 630 631 Note: Julia uses 1-based indexing, C API uses 0-based. We subtract 1. 632 633 - [ ] **Step 4: Run tests — expect pass** 634 635 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 636 637 Expected: PASS 638 639 - [ ] **Step 5: Commit** 640 641 ```bash 642 git add src/ffi.jl test/test_lazy.jl 643 git commit -m "feat: add lazy array indexing" 644 ``` 645 646 --- 647 648 ### Task 6: Implement `collect` (materialization) 649 650 Convert a lazy `NickelValue` subtree into plain Julia types, matching `nickel_eval` output. 651 652 **Files:** 653 - Modify: `src/ffi.jl` 654 - Modify: `test/test_lazy.jl` 655 656 - [ ] **Step 1: Write failing tests** 657 658 Add to `test/test_lazy.jl`: 659 660 ```julia 661 @testset "collect" begin 662 # Record 663 nickel_open("{ a = 1, b = \"two\", c = true }") do cfg 664 result = collect(cfg) 665 @test result isa Dict{String, Any} 666 @test result == nickel_eval("{ a = 1, b = \"two\", c = true }") 667 end 668 669 # Nested record 670 nickel_open("{ x = { y = 42 } }") do cfg 671 result = collect(cfg) 672 @test result["x"]["y"] === Int64(42) 673 end 674 675 # Array 676 nickel_open("[1, 2, 3]") do cfg 677 result = collect(cfg) 678 @test result == Any[1, 2, 3] 679 end 680 681 # Collect a sub-tree 682 nickel_open("{ a = 1, b = { c = 2, d = 3 } }") do cfg 683 sub = collect(cfg.b) 684 @test sub == Dict{String, Any}("c" => 2, "d" => 3) 685 end 686 687 # Primitive passthrough 688 nickel_open("42") do cfg 689 @test collect(cfg) === Int64(42) 690 end 691 end 692 ``` 693 694 - [ ] **Step 2: Run tests to verify failure** 695 696 Expected: FAIL — no `collect` method for `NickelValue` 697 698 - [ ] **Step 3: Implement `collect`** 699 700 Append to `src/ffi.jl`: 701 702 ```julia 703 """ 704 collect(v::NickelValue) -> Any 705 706 Recursively evaluate and materialize the entire subtree rooted at `v`. 707 Returns the same types as `nickel_eval`: Dict, Vector, Int64, Float64, etc. 708 """ 709 function Base.collect(v::NickelValue) 710 session = getfield(v, :session) 711 _check_session_open(session) 712 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 713 return _collect_expr(session, expr) 714 end 715 716 # Recursive collect: shallow-eval each sub-expression, then convert. 717 function _collect_expr(session::NickelSession, expr::Ptr{L.nickel_expr}) 718 ctx = Ptr{L.nickel_context}(session.ctx) 719 720 if L.nickel_expr_is_null(expr) != 0 721 return nothing 722 elseif L.nickel_expr_is_bool(expr) != 0 723 return L.nickel_expr_as_bool(expr) != 0 724 elseif L.nickel_expr_is_number(expr) != 0 725 num = L.nickel_expr_as_number(expr) 726 if L.nickel_number_is_i64(num) != 0 727 return L.nickel_number_as_i64(num) 728 else 729 return Float64(L.nickel_number_as_f64(num)) 730 end 731 elseif L.nickel_expr_is_str(expr) != 0 732 out_ptr = Ref{Ptr{Cchar}}(C_NULL) 733 len = L.nickel_expr_as_str(expr, out_ptr) 734 return unsafe_string(out_ptr[], len) 735 elseif L.nickel_expr_is_array(expr) != 0 736 arr = L.nickel_expr_as_array(expr) 737 n = Int(L.nickel_array_len(arr)) 738 result = Vector{Any}(undef, n) 739 for i in 0:(n-1) 740 elem = _tracked_expr_alloc(session) 741 L.nickel_array_get(arr, Csize_t(i), elem) 742 # Shallow-eval the element before collecting 743 evaled = _tracked_expr_alloc(session) 744 err = L.nickel_error_alloc() 745 try 746 r = L.nickel_context_eval_expr_shallow(ctx, elem, evaled, err) 747 if r == L.NICKEL_RESULT_ERR 748 _throw_nickel_error(err) 749 end 750 finally 751 L.nickel_error_free(err) 752 end 753 result[i+1] = _collect_expr(session, evaled) 754 end 755 return result 756 elseif L.nickel_expr_is_record(expr) != 0 757 rec = L.nickel_expr_as_record(expr) 758 n = Int(L.nickel_record_len(rec)) 759 result = Dict{String, Any}() 760 key_ptr = Ref{Ptr{Cchar}}(C_NULL) 761 key_len = Ref{Csize_t}(0) 762 for i in 0:(n-1) 763 val_expr = _tracked_expr_alloc(session) 764 L.nickel_record_key_value_by_index(rec, Csize_t(i), key_ptr, key_len, val_expr) 765 key = unsafe_string(key_ptr[], key_len[]) 766 # Shallow-eval the value before collecting 767 evaled = _tracked_expr_alloc(session) 768 err = L.nickel_error_alloc() 769 try 770 r = L.nickel_context_eval_expr_shallow(ctx, val_expr, evaled, err) 771 if r == L.NICKEL_RESULT_ERR 772 _throw_nickel_error(err) 773 end 774 finally 775 L.nickel_error_free(err) 776 end 777 result[key] = _collect_expr(session, evaled) 778 end 779 return result 780 elseif L.nickel_expr_is_enum_variant(expr) != 0 781 out_ptr = Ref{Ptr{Cchar}}(C_NULL) 782 arg_expr = _tracked_expr_alloc(session) 783 len = L.nickel_expr_as_enum_variant(expr, out_ptr, arg_expr) 784 tag = Symbol(unsafe_string(out_ptr[], len)) 785 # Shallow-eval the arg before collecting 786 evaled = _tracked_expr_alloc(session) 787 err = L.nickel_error_alloc() 788 try 789 r = L.nickel_context_eval_expr_shallow(ctx, arg_expr, evaled, err) 790 if r == L.NICKEL_RESULT_ERR 791 _throw_nickel_error(err) 792 end 793 finally 794 L.nickel_error_free(err) 795 end 796 return NickelEnum(tag, _collect_expr(session, evaled)) 797 elseif L.nickel_expr_is_enum_tag(expr) != 0 798 out_ptr = Ref{Ptr{Cchar}}(C_NULL) 799 len = L.nickel_expr_as_enum_tag(expr, out_ptr) 800 return NickelEnum(Symbol(unsafe_string(out_ptr[], len)), nothing) 801 else 802 error("Unknown Nickel expression type") 803 end 804 end 805 ``` 806 807 Key logic: Unlike `_walk_expr` (which assumes everything is already deeply evaluated), `_collect_expr` calls `nickel_context_eval_expr_shallow` on each sub-expression before inspecting its type. This forces lazy thunks to evaluate one level at a time, recursing until the entire tree is materialized. 808 809 - [ ] **Step 4: Run tests — expect pass** 810 811 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 812 813 Expected: PASS 814 815 - [ ] **Step 5: Commit** 816 817 ```bash 818 git add src/ffi.jl test/test_lazy.jl 819 git commit -m "feat: add collect for NickelValue materialization" 820 ``` 821 822 --- 823 824 ### Task 7: Implement `keys` and `length` 825 826 **Files:** 827 - Modify: `src/ffi.jl` 828 - Modify: `test/test_lazy.jl` 829 830 - [ ] **Step 1: Write failing tests** 831 832 Add to `test/test_lazy.jl`: 833 834 ```julia 835 @testset "keys and length" begin 836 nickel_open("{ a = 1, b = 2, c = 3 }") do cfg 837 k = keys(cfg) 838 @test k isa Vector{String} 839 @test sort(k) == ["a", "b", "c"] 840 @test length(cfg) == 3 841 end 842 843 nickel_open("[10, 20, 30, 40]") do cfg 844 @test length(cfg) == 4 845 end 846 847 # keys on non-record throws 848 nickel_open("[1, 2]") do cfg 849 @test_throws ArgumentError keys(cfg) 850 end 851 end 852 ``` 853 854 - [ ] **Step 2: Run tests to verify failure** 855 856 Expected: FAIL 857 858 - [ ] **Step 3: Implement `keys` and `length`** 859 860 Append to `src/ffi.jl`: 861 862 ```julia 863 function Base.keys(v::NickelValue) 864 session = getfield(v, :session) 865 _check_session_open(session) 866 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 867 if L.nickel_expr_is_record(expr) == 0 868 throw(ArgumentError("Cannot get keys: NickelValue is not a record")) 869 end 870 rec = L.nickel_expr_as_record(expr) 871 n = Int(L.nickel_record_len(rec)) 872 result = Vector{String}(undef, n) 873 key_ptr = Ref{Ptr{Cchar}}(C_NULL) 874 key_len = Ref{Csize_t}(0) 875 for i in 0:(n-1) 876 # Pass C_NULL for out_expr to skip value extraction 877 L.nickel_record_key_value_by_index(rec, Csize_t(i), key_ptr, key_len, 878 Ptr{L.nickel_expr}(C_NULL)) 879 result[i+1] = unsafe_string(key_ptr[], key_len[]) 880 end 881 return result 882 end 883 884 function Base.length(v::NickelValue) 885 session = getfield(v, :session) 886 _check_session_open(session) 887 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 888 if L.nickel_expr_is_record(expr) != 0 889 return Int(L.nickel_record_len(L.nickel_expr_as_record(expr))) 890 elseif L.nickel_expr_is_array(expr) != 0 891 return Int(L.nickel_array_len(L.nickel_expr_as_array(expr))) 892 else 893 throw(ArgumentError("Cannot get length: NickelValue is not a record or array")) 894 end 895 end 896 ``` 897 898 Note: `keys` passes `C_NULL` as the out-expression to `nickel_record_key_value_by_index`. The C API explicitly supports this — it skips writing the value, giving us just the field names without evaluating anything. 899 900 - [ ] **Step 4: Run tests — expect pass** 901 902 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 903 904 Expected: PASS 905 906 - [ ] **Step 5: Commit** 907 908 ```bash 909 git add src/ffi.jl test/test_lazy.jl 910 git commit -m "feat: add keys and length for NickelValue" 911 ``` 912 913 --- 914 915 ### Task 8: Implement file path support for `nickel_open` 916 917 **Files:** 918 - Modify: `src/ffi.jl` 919 - Modify: `test/test_lazy.jl` 920 921 - [ ] **Step 1: Write failing tests** 922 923 Add to `test/test_lazy.jl`: 924 925 ```julia 926 @testset "File evaluation" begin 927 mktempdir() do dir 928 # Simple file 929 f = joinpath(dir, "config.ncl") 930 write(f, "{ host = \"localhost\", port = 8080 }") 931 nickel_open(f) do cfg 932 @test cfg.host == "localhost" 933 @test cfg.port === Int64(8080) 934 end 935 936 # File with imports 937 shared = joinpath(dir, "shared.ncl") 938 write(shared, "{ version = \"1.0\" }") 939 main = joinpath(dir, "main.ncl") 940 write(main, """ 941 let s = import "shared.ncl" in 942 { app_version = s.version, name = "myapp" } 943 """) 944 nickel_open(main) do cfg 945 @test cfg.app_version == "1.0" 946 @test cfg.name == "myapp" 947 end 948 end 949 end 950 ``` 951 952 - [ ] **Step 2: Run tests to verify failure** 953 954 Expected: FAIL — `nickel_open` doesn't detect file paths 955 956 - [ ] **Step 3: Implement file path variant** 957 958 Add a `nickel_open` method that detects file paths. Insert above the existing `nickel_open(code::String)` in `src/ffi.jl`: 959 960 ```julia 961 function nickel_open(f::Function, path_or_code::String) 962 val = nickel_open(path_or_code) 963 try 964 return f(val) 965 finally 966 close(val) 967 end 968 end 969 970 function nickel_open(path_or_code::String) 971 _check_ffi_available() 972 # Detect file path: ends with .ncl AND exists on disk 973 if endswith(path_or_code, ".ncl") && isfile(abspath(path_or_code)) 974 return _nickel_open_file(path_or_code) 975 end 976 return _nickel_open_code(path_or_code) 977 end 978 979 function _nickel_open_file(path::String) 980 abs_path = abspath(path) 981 if !isfile(abs_path) 982 throw(NickelError("File not found: $abs_path")) 983 end 984 code = read(abs_path, String) 985 ctx = L.nickel_context_alloc() 986 session = NickelSession(Ptr{Cvoid}(ctx), Ptr{Cvoid}[], false) 987 finalizer(close, session) 988 expr = _tracked_expr_alloc(session) 989 err = L.nickel_error_alloc() 990 try 991 GC.@preserve abs_path begin 992 L.nickel_context_set_source_name(ctx, Base.unsafe_convert(Ptr{Cchar}, abs_path)) 993 end 994 result = L.nickel_context_eval_shallow(ctx, code, expr, err) 995 if result == L.NICKEL_RESULT_ERR 996 _throw_nickel_error(err) 997 end 998 return NickelValue(session, Ptr{Cvoid}(expr)) 999 catch 1000 close(session) 1001 rethrow() 1002 finally 1003 L.nickel_error_free(err) 1004 end 1005 end 1006 1007 function _nickel_open_code(code::String) 1008 ctx = L.nickel_context_alloc() 1009 session = NickelSession(Ptr{Cvoid}(ctx), Ptr{Cvoid}[], false) 1010 finalizer(close, session) 1011 expr = _tracked_expr_alloc(session) 1012 err = L.nickel_error_alloc() 1013 try 1014 result = L.nickel_context_eval_shallow(ctx, code, expr, err) 1015 if result == L.NICKEL_RESULT_ERR 1016 _throw_nickel_error(err) 1017 end 1018 return NickelValue(session, Ptr{Cvoid}(expr)) 1019 catch 1020 close(session) 1021 rethrow() 1022 finally 1023 L.nickel_error_free(err) 1024 end 1025 end 1026 ``` 1027 1028 This replaces the original `nickel_open(code::String)` and `nickel_open(f::Function, code::String)`. The routing logic is simple: if the string ends in `.ncl`, treat as file path (mirrors `nickel_eval_file` pattern with source name for import resolution). Otherwise treat as inline code. 1029 1030 - [ ] **Step 4: Run tests — expect pass** 1031 1032 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 1033 1034 Expected: PASS (all tests including earlier ones) 1035 1036 - [ ] **Step 5: Commit** 1037 1038 ```bash 1039 git add src/ffi.jl test/test_lazy.jl 1040 git commit -m "feat: add file path support to nickel_open" 1041 ``` 1042 1043 --- 1044 1045 ### Task 9: Implement `iterate` protocol 1046 1047 **Files:** 1048 - Modify: `src/ffi.jl` 1049 - Modify: `test/test_lazy.jl` 1050 1051 - [ ] **Step 1: Write failing tests** 1052 1053 Add to `test/test_lazy.jl`: 1054 1055 ```julia 1056 @testset "Iteration" begin 1057 # Record iteration yields pairs 1058 nickel_open("{ a = 1, b = 2 }") do cfg 1059 pairs = Dict(k => v for (k, v) in cfg) 1060 @test pairs["a"] === Int64(1) 1061 @test pairs["b"] === Int64(2) 1062 end 1063 1064 # Array iteration 1065 nickel_open("[10, 20, 30]") do cfg 1066 values = [x for x in cfg] 1067 @test values == Any[10, 20, 30] 1068 end 1069 end 1070 ``` 1071 1072 - [ ] **Step 2: Run tests to verify failure** 1073 1074 Expected: FAIL 1075 1076 - [ ] **Step 3: Implement `iterate`** 1077 1078 Append to `src/ffi.jl`: 1079 1080 ```julia 1081 function Base.iterate(v::NickelValue, state=1) 1082 session = getfield(v, :session) 1083 _check_session_open(session) 1084 expr = Ptr{L.nickel_expr}(getfield(v, :expr)) 1085 1086 if L.nickel_expr_is_record(expr) != 0 1087 rec = L.nickel_expr_as_record(expr) 1088 n = Int(L.nickel_record_len(rec)) 1089 state > n && return nothing 1090 key_ptr = Ref{Ptr{Cchar}}(C_NULL) 1091 key_len = Ref{Csize_t}(0) 1092 val_expr = _tracked_expr_alloc(session) 1093 L.nickel_record_key_value_by_index(rec, Csize_t(state - 1), key_ptr, key_len, val_expr) 1094 key = unsafe_string(key_ptr[], key_len[]) 1095 val = _eval_and_resolve(session, val_expr) 1096 return (key => val, state + 1) 1097 elseif L.nickel_expr_is_array(expr) != 0 1098 arr = L.nickel_expr_as_array(expr) 1099 n = Int(L.nickel_array_len(arr)) 1100 state > n && return nothing 1101 elem = _tracked_expr_alloc(session) 1102 L.nickel_array_get(arr, Csize_t(state - 1), elem) 1103 val = _eval_and_resolve(session, elem) 1104 return (val, state + 1) 1105 else 1106 throw(ArgumentError("Cannot iterate: NickelValue is not a record or array")) 1107 end 1108 end 1109 ``` 1110 1111 Records yield `Pair{String, Any}`, arrays yield elements. Both use 1-based state internally, converting to 0-based for the C API. 1112 1113 - [ ] **Step 4: Run tests — expect pass** 1114 1115 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 1116 1117 Expected: PASS 1118 1119 - [ ] **Step 5: Commit** 1120 1121 ```bash 1122 git add src/ffi.jl test/test_lazy.jl 1123 git commit -m "feat: add iterate protocol for NickelValue" 1124 ``` 1125 1126 --- 1127 1128 ### Task 10: Error handling and edge cases 1129 1130 **Files:** 1131 - Modify: `test/test_lazy.jl` 1132 1133 - [ ] **Step 1: Write error handling tests** 1134 1135 Add to `test/test_lazy.jl`: 1136 1137 ```julia 1138 @testset "Error handling" begin 1139 # Closed session 1140 local stale_ref 1141 nickel_open("{ x = 1 }") do cfg 1142 stale_ref = cfg 1143 end 1144 @test_throws ArgumentError stale_ref.x 1145 1146 # Missing field 1147 nickel_open("{ x = 1 }") do cfg 1148 @test_throws Union{NickelError, ArgumentError} cfg.nonexistent 1149 end 1150 1151 # Wrong access type: dot on array 1152 nickel_open("[1, 2, 3]") do cfg 1153 @test_throws ArgumentError cfg.x 1154 end 1155 1156 # Wrong access type: integer index on record 1157 nickel_open("{ x = 1 }") do cfg 1158 @test_throws ArgumentError cfg[1] 1159 end 1160 1161 # Out of bounds 1162 nickel_open("[1, 2]") do cfg 1163 @test_throws BoundsError cfg[3] 1164 end 1165 1166 # Syntax error in code 1167 @test_throws NickelError nickel_open("{ x = }") 1168 end 1169 ``` 1170 1171 - [ ] **Step 2: Run tests — expect pass** 1172 1173 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 1174 1175 Expected: PASS — all error paths should already work from the implementations above. If any fail, fix and re-run. 1176 1177 - [ ] **Step 3: Commit** 1178 1179 ```bash 1180 git add test/test_lazy.jl 1181 git commit -m "test: add error handling tests for lazy evaluation" 1182 ``` 1183 1184 --- 1185 1186 ### Task 11: Enum handling 1187 1188 **Files:** 1189 - Modify: `test/test_lazy.jl` 1190 1191 - [ ] **Step 1: Write enum tests** 1192 1193 Add to `test/test_lazy.jl`: 1194 1195 ```julia 1196 @testset "Enum handling" begin 1197 # Bare enum tag returns NickelEnum immediately 1198 nickel_open("let x = 'Foo in x") do cfg 1199 @test cfg isa NickelEnum 1200 @test cfg.tag == :Foo 1201 end 1202 1203 # Enum variant with primitive arg 1204 nickel_open("let x = 'Some 42 in x") do cfg 1205 @test cfg isa NickelEnum 1206 @test cfg.tag == :Some 1207 @test cfg.arg === Int64(42) 1208 end 1209 1210 # Enum variant with record arg stays lazy 1211 nickel_open("{ status = 'Ok { value = 123 } }") do cfg 1212 status = cfg.status 1213 @test status isa NickelValue # enum variant is compound, stays lazy 1214 result = collect(status) 1215 @test result isa NickelEnum 1216 @test result.tag == :Ok 1217 @test result.arg["value"] === Int64(123) 1218 end 1219 1220 # Enum in collect 1221 nickel_open("{ x = 'None, y = 'Some 42 }") do cfg 1222 result = collect(cfg) 1223 @test result["x"] isa NickelEnum 1224 @test result["x"].tag == :None 1225 @test result["y"] isa NickelEnum 1226 @test result["y"].tag == :Some 1227 end 1228 end 1229 ``` 1230 1231 - [ ] **Step 2: Run tests** 1232 1233 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 1234 1235 Expected: PASS — or reveals edge cases in `_resolve_value`/`_collect_expr` that need fixing. Fix and re-run if needed. 1236 1237 - [ ] **Step 3: Commit** 1238 1239 ```bash 1240 git add test/test_lazy.jl 1241 git commit -m "test: add enum handling tests for lazy evaluation" 1242 ``` 1243 1244 --- 1245 1246 ### Task 12: Manual session usage 1247 1248 **Files:** 1249 - Modify: `test/test_lazy.jl` 1250 1251 - [ ] **Step 1: Write manual session tests** 1252 1253 Add to `test/test_lazy.jl`: 1254 1255 ```julia 1256 @testset "Manual session" begin 1257 cfg = nickel_open("{ x = 42, y = \"hello\" }") 1258 @test cfg.x === Int64(42) 1259 @test cfg.y == "hello" 1260 close(cfg) 1261 1262 # Double close is safe 1263 close(cfg) 1264 1265 # Access after close throws 1266 @test_throws ArgumentError cfg.x 1267 end 1268 ``` 1269 1270 - [ ] **Step 2: Run tests — expect pass** 1271 1272 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 1273 1274 Expected: PASS 1275 1276 - [ ] **Step 3: Commit** 1277 1278 ```bash 1279 git add test/test_lazy.jl 1280 git commit -m "test: add manual session tests for lazy evaluation" 1281 ``` 1282 1283 --- 1284 1285 ### Task 13: Update documentation 1286 1287 **Files:** 1288 - Modify: `docs/src/lib/public.md` 1289 1290 - [ ] **Step 1: Add new exports to docs** 1291 1292 Add a new section to `docs/src/lib/public.md`: 1293 1294 ```markdown 1295 ## Lazy Evaluation 1296 1297 ```@docs 1298 nickel_open 1299 NickelValue 1300 NickelSession 1301 nickel_kind 1302 ``` 1303 ``` 1304 1305 - [ ] **Step 2: Run full test suite one final time** 1306 1307 Run: `cd /Users/loulou/Dropbox/projects_claude/NickelEval && julia --project=. -e 'using Pkg; Pkg.test()'` 1308 1309 Expected: All tests PASS 1310 1311 - [ ] **Step 3: Commit** 1312 1313 ```bash 1314 git add docs/src/lib/public.md 1315 git commit -m "docs: add lazy evaluation API to public docs" 1316 ``` 1317 1318 --- 1319 1320 ### Task 14: Update spec with design deviation 1321 1322 **Files:** 1323 - Modify: `docs/superpowers/specs/2026-03-21-lazy-evaluation-design.md` 1324 1325 - [ ] **Step 1: Add deviation note to spec** 1326 1327 Add to the top of the spec, after the Problem section: 1328 1329 ```markdown 1330 ## Design Deviations 1331 1332 - `NickelSession` has no `root` field (avoids circular type reference). `nickel_open` returns `NickelValue` directly. `close(::NickelValue)` delegates to `close(session)`. 1333 - Types use `Ptr{Cvoid}` instead of `Ptr{LibNickel.nickel_expr}` to avoid forward reference to `LibNickel` module. 1334 - Bare enum tags and enum variants with primitive args are resolved immediately by `_resolve_value` (not wrapped in `NickelValue`). Only enum variants with compound args stay lazy. 1335 ``` 1336 1337 - [ ] **Step 2: Commit** 1338 1339 ```bash 1340 git add docs/superpowers/specs/2026-03-21-lazy-evaluation-design.md 1341 git commit -m "docs: note design deviations in spec" 1342 ```