git » homepage.git » commit 508041c

Add unlisted article previews and the Blacktide introduction

author Alan Dipert
2026-11-10 15:00:17 UTC
committer Alan Dipert
2026-11-10 15:00:17 UTC
parent 91752af3dd186e899e6e7beeb3f09bfdc605e478

Add unlisted article previews and the Blacktide introduction

AGENTS.md +2 -1
README.md +16 -0
md/_MutableLocals.md +208 -0
tests/test_sitegen.py +101 -0
tools/sitegen.py +43 -10

diff --git a/AGENTS.md b/AGENTS.md
index 31f0ce8..460ca6e 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -29,7 +29,8 @@ When drafting or editing Markdown articles under `md/`:
   `OpenSource.md` may begin with `# Open Source`).
 - Use short, conversational paragraphs aimed at technical readers; prefer present tense and active voice.
 - Link to related wiki pages using relative Markdown links (`./PageName.md`) so the generator can rewrite them.
-- When publishing or adding a new article, add a dated entry near the top of the `md/Home.md` Updates table with a relative Markdown link to the article.
+- When publishing a listed article, add a dated entry near the top of the `md/Home.md` Updates table with a relative Markdown link to the article.
+- For a shareable unlisted preview, use `md/_PageName.md` (or an underscore-prefixed directory). It renders at the corresponding HTML URL with `noindex` metadata, outside the index, feed, and backlinks. Do not link to it from listed pages; the build rejects those links. Keep source-only drafts in `drafts/`.
 - Store article-specific assets in a sibling directory named after the article stem (e.g., `md/PageName/image.png` for `md/PageName.md`) and reference them relatively (`./PageName/image.png`) so `make assets` copies them correctly.
 - For historical notes, include dates or context in plain prose rather than YAML front matter.
 - Tables should follow GitHub-style pipe syntax; the build pipeline will handle them.
diff --git a/README.md b/README.md
index 7bce3f3..316b9b4 100644
--- a/README.md
+++ b/README.md
@@ -38,6 +38,22 @@ make verify-reproducible
 
 The `Containerfile` pins both the Debian base-image digest and a dated Debian package snapshot. Update those two values together only when intentionally refreshing the toolchain.
 
+## Unlisted previews
+
+| Source | Generated page | Discovery |
+| --- | --- | --- |
+| `drafts/Page.md` | None | Kept outside the website |
+| `md/_Page.md` | `_Page.html` | Shareable by URL; omitted from the index, feed, and backlinks |
+| `md/Page.md` | `Page.html` | Listed in the index; announce it in Home's Updates table when ready |
+
+An underscore-prefixed directory also makes its pages unlisted: `md/_previews/Page.md` becomes `_previews/Page.html`. Keep the normal level-one title; the underscore belongs only in the path. Editor backups and lock files remain excluded.
+
+Build with `make container-build`, inspect the HTML in `out/`, and deploy through the normal `make deploy` workflow. For example, `md/_MutableLocals.md` will be available at `https://tailrecursion.com/~alan/_MutableLocals.html` after deployment. This uses the ordinary static-site build and deployment, with no separate preview server. Forgejo's source-publishing job does not deploy the website.
+
+Previews can link to published pages and to one another. They do not add backlinks to those pages. The build rejects links from listed pages to previews, including links in Home's Updates table, so previews cannot accidentally appear in the updates feed or Explore tiles. To publish a preview normally, remove the underscore and add its new path to the Updates table.
+
+Preview pages include `noindex` metadata for search engines. They are unlisted, not private: anyone with the URL can read them. Keep work that should not be deployed in `drafts/`.
+
 ## Continuous integration and public source
 
 Forgejo CI uses the pinned builder to run the tests, compare two independent builds, and build the website. It downloads only the LFS media needed by the website. Successful pushes to `master` request publication of the source at [tailrecursion.com/repos/](https://tailrecursion.com/repos/r/homepage.git/).
diff --git a/md/_MutableLocals.md b/md/_MutableLocals.md
new file mode 100644
index 0000000..4d0f99d
--- /dev/null
+++ b/md/_MutableLocals.md
@@ -0,0 +1,208 @@
+# Mutable Locals in Clojure, the Hard Way
+
+Hey, so, I've been thinking about mutable local variables in Clojure. Is that OK? Hear me out.
+
+The Clojure situation has been explained to me. I'm totally down with the philosophy. Stable identities, immutable values, first-class state management. A more civilized, decomplected age. But I gotta level with you. Sometimes I have these intrusive thoughts about mutating local variables. I'll be reading an imperative algorithm in pseudocode, Java or C, and—please don't tell anyone about this—I'll think: man, it would be pretty sweet to port this directly without translating it into atoms or `loop`/`recur` and whatnot.
+
+I mean, under the covers, the JVM certainly has no inherent moral objection to overwriting a [local-variable slot](https://docs.oracle.com/javase/specs/jvms/se25/html/jvms-2.html#jvms-2.6.1). Clojure's own compiler even reuses those slots to _implement_ `loop` and `recur`. But I'm not allowed to assign to an ordinary lexical local myself? That heady stuff is only for the elite class of compiler writers, I guess? Is it ungrateful of me, as a mere user, to not be entirely satisfied with the privilege of supplying replacement values to approved recursion points?
+
+Here's an example of the kind of stuff I'd like to write on occasion. This function counts inversions in a vector: pairs where an earlier element is greater than a later one.
+
+```clojure
+(require '[com.tailrecursion.blacktide :refer [loophole setq label goto]])
+
+(defn inversions [xs]
+  (loophole [n 0]
+    (dotimes [i (count xs)]
+      (dotimes [j i]
+        (when (> (nth xs j) (nth xs i))
+          (setq n (inc n)))))
+    n))
+
+(inversions [3 1 2 1])
+;; => 4
+```
+
+Nothing crazy, just setting a happy little variable `n` inside nested loops. I'm still here, by the way. Nothing terrible happened.
+
+If you find yourself warming to these shenanigans, then I have some good news. This is a real thing that can be done, all thanks to the `loop`/`recur` constructs I just finished griping about. You see, [`recur`](https://clojure.org/reference/special_forms#recur) evaluates its arguments, rebinds the loop locals, and starts the loop body again. In [Clojure's compiler](https://github.com/clojure/clojure/blob/clojure-1.12.6/src/jvm/clojure/lang/Compiler.java#L7156-L7227), that becomes argument evaluation followed by local-variable stores and a jump. A jump. A freaking `GOTO` of yore! We'll be wanting that too, more in a sec.
+
+Anyway, those local-variable stores under the hood are the keys to the kingdom. We can write assignments where we want them, then compile them into ordinary Clojure that does the updating for us, via `recur`. We can automate the mental gymnastics we usually perform when translating an imperative algorithm.
+
+A macro can turn our assignments and jumps into the right combination of lexical bindings, loops, and `recur` forms, and Clojure can do the rest. No atoms, arrays or other mutable reference cells to hold our locals, and no changes to Clojure's compiler.
+
+Doing this for a useful subset of Clojure turns out to involve writing a small compiler. Apparently I'm willing to do that. The result is Blacktide and its `loophole` macro. **Mutable local variables (and goto!) to the people!** Allow me to give you the tour.
+
+## A place to put the next instruction
+
+Let's use a smaller program so we can see the machinery:
+
+```clojure
+(loophole [sum 0]
+  (dotimes [i 10]
+    (setq sum (+ sum i)))
+  sum)
+;; => 45
+```
+
+`loophole` starts `sum` at zero and returns the final `sum`. Each `setq` replaces its value. First we expand `dotimes` and the `when` it introduces to expose the loop. The mutable binding and assignment remain for our compiler to handle. I've shortened the generated name and omitted `clojure.core/` prefixes:
+
+```clojure
+(loophole [sum 0]
+  (let [limit (long 10)]
+    (loop [i 0]
+      (if (< i limit)
+        (do
+          (setq sum (+ sum i))
+          (recur (unchecked-inc i))))))
+  sum)
+;; => 45
+```
+
+With the loop exposed, we can turn the program into operations and arrows: `setq` becomes a write, `if` chooses an edge, and `recur` leads back to the test. This is a [control-flow graph](https://www.cs.cornell.edu/courses/cs4120/2026sp/notes.html?id=traces). Each node is a _basic block_: a straight-line sequence of operations ending in a jump, a choice of successors, or an exit. Here is a simplified view, omitting temporary copies and the loop's discarded nil result:
+
+```mermaid
+flowchart TD
+    init["sum = 0<br/>limit = long(10)<br/>i = 0"] --> test{"0: i < limit?"}
+    test -->|yes| body["1: sum = sum + i<br/>i = unchecked-inc(i)"]
+    body --> test
+    test -->|no| done["2: return sum"]
+```
+
+Now we can execute this graph with ordinary Clojure, using a loop of our own. A loop variable `pc`, for program counter, holds the next block's number. A `case` acts as the _dispatcher_, selecting that block.
+
+Here is a hand-written translation, using the diagram's block numbers:
+
+```clojure
+(let [limit (long 10)]
+  (loop [pc 0 i 0 sum 0]
+    (case pc
+      0 (if (< i limit)
+          (recur 1 i sum)
+          (recur 2 i sum))
+      1 (let [sum (+ sum i)
+              i (unchecked-inc i)]
+          (recur 0 i sum))
+      2 sum)))
+;; => 45
+```
+
+`setq` is gone. In block 1, `let` names the updated sum and index. `recur` carries those values into the loop's bindings and continues at block 0. Clojure supplies the local-variable stores and the jump.
+
+The sum is _live_ across that jump: a later block may read it before it is overwritten. Liveness tells the compiler which values must travel between blocks. Temporaries used entirely within one block can stay in its `let`.
+
+[core.async's `go` macro](https://clojure.org/news/2013/06/28/clojure-clore-async-channels#_go_blocks_and_ioc_threads) also turns code into a state machine, allowing channel operations to park and resume it. Our machine implements assignment and jumps, and runs synchronously on the calling thread.
+
+With the next block under our control, an explicit `goto` is a small step. Here it leaves the `dotimes` early:
+
+```clojure
+(loophole [sum 0]
+  (dotimes [i 10]
+    (when (= i 5) (goto done))
+    (setq sum (+ sum i)))
+  (label done)
+  sum)
+;; => 10
+```
+
+Here the label marks the return block. Labels belong to the region's outer statement sequence, so jumps can leave nested loops but cannot enter a nested scope halfway through its initialization. Every destination must have an explicit label.
+
+`loophole` supports a [bounded subset of Clojure](https://tailrecursion.com/repos/r/blacktide.git/b/master/t/doc/f=language.md.html); function creation, exception handlers and resource scopes stay outside the region. The revolution will respect cleanup.
+
+## Ten plus twenty should remain thirty
+
+So far, we've used `setq` to update a variable and discarded its result. But `setq` also returns the value it assigns. Put two assignments in the arguments to a function, and the first result has to survive the second write:
+
+```clojure
+(loophole [x 0]
+  (+ (setq x 10) (setq x 20)))
+;; => 30
+```
+
+The first argument to `+` is 10; the second is 20. If a transformation performs both writes and then reads `x` twice, it produces a very convincing 40.
+
+The first argument therefore needs a name of its own. The essential transformation looks like this, still written in the source language:
+
+```clojure
+(loophole [x 0]
+  (let [left  (setq x 10)
+        right (setq x 20)]
+    (+ left right)))
+;; => 30
+```
+
+The compiler does this systematically before building the graph. It saves operands in evaluation order, and subsequent operations consume those saved values. This is the purpose of _A-normalization_: making intermediate computations and their order explicit. Matt Might's [A-Normalization: Why and How](https://matt.might.net/articles/a-normalization/) explains the technique with small Scheme examples.
+
+Introducing those temporary names brings another obligation: each [binding](https://craftinginterpreters.com/resolving-and-binding.html#static-scope) needs its own identity, even when two locals share a name. The source `recur` inside `dotimes` must also keep its original loop target. The [walkthrough](https://tailrecursion.com/repos/r/blacktide.git/b/master/t/doc/f=tutorial.md.html) follows these details.
+
+## Many small promises
+
+We've now expanded macros, preserved evaluation order, kept bindings distinct, and connected blocks. Combining all of that in one enormous transformation would make for an exciting debugging experience.
+
+To keep it manageable, I use [nanopass](https://andykeep.com/pubs/np-preprint.pdf), the approach Andrew Keep and R. Kent Dybvig used to rebuild Chez Scheme's compiler. Each pass performs a small transformation between explicitly defined intermediate languages. Keep's [Writing a Nanopass Compiler](https://www.youtube.com/watch?v=Os7FE3J-U5Q), from Clojure/conj, is a good introduction.
+
+Saving operands is one of those passes. Its input language allows expressions in call arguments; its output requires references to saved locals, like `left` and `right`. Later passes can consume those arguments without having to untangle their evaluation order.
+
+These languages describe trees of immutable Clojure maps and vectors. The [nanopass machinery](https://tailrecursion.com/repos/r/blacktide.git/b/master/t/doc/f=nanopass.md.html) generates traversal code for unchanged forms and derives [Typed Clojure](https://github.com/typedclojure/typedclojure) types from the language descriptions. The type checker can reject a pass that leaves an assignment in a call argument. It runs during development, without being loaded when somebody uses `loophole`.
+
+Naturally, a pass can still produce a beautifully well-typed wrong answer. Saving the wrong value for `left` could give us that convincing 40 again.
+
+So our [generative tests](https://tailrecursion.com/repos/r/blacktide.git/b/master/t/test/com/tailrecursion/blacktide/f=generated_test.clj.html) build valid, bounded programs combining assignments, branches, loops, and jumps. We compare their results and recorded execution traces with an independent state model, then repeat the comparison with renamed locals, different control-flow formulations, and different ways of emitting the final Clojure. When a case fails, the generator shrinks it to a smaller valid counterexample.
+
+## Having dismantled the loop, we would like a loop
+
+The `case` machine gives us a general way to execute the graph. But look at the graph for our sum: test the index, update the sum, go back to the test. We have painstakingly described a loop using numbered blocks. Perhaps we could, in fact, just emit a loop?
+
+That removes the need to carry a program counter and repeatedly ask `case` which block comes next. The test becomes an `if`, the backward edge becomes a `recur`, and the exit returns the sum. Even our added `goto` fits: reaching five simply takes another exit from the loop.
+
+This is where the earlier compiler work pays off. A later pass can inspect the graph with bindings, evaluation order and jump destinations already settled. Its job is to recover ordinary loops and conditionals. It doesn't have to understand `dotimes` or redo assignment handling to attempt that optimization. Nanopass gives it a defined input and output, so we can improve this part without reopening every question about what the source means.
+
+The compiler makes this choice before emitting Clojure, keeping the dispatcher when it can't safely reconstruct the control flow.
+
+For our sum with the early exit, it emits the loop below, shown with shorter local names. `loop*` and `let*` are Clojure's underlying loop and binding forms. The annotated [`RT.box(Object)`](https://github.com/clojure/clojure/blob/clojure-1.12.6/src/jvm/clojure/lang/RT.java#L1037-L1039) calls return their arguments unchanged, keeping `sum` typed as an object; arithmetic can still allocate boxed values. A surrounding `let*` initializes `sum` to zero and sets `limit` to ten:
+
+```clojure
+(loop* [sum (^{:param-tags [java.lang.Object]} clojure.lang.RT/box sum)
+        i '0]
+  (if (clojure.lang.Numbers/lt i limit)
+    (let* [done? (clojure.lang.Util/equiv i '5)]
+      (if done?
+        sum
+        (let* [next-sum
+               (^{:param-tags [java.lang.Object]} clojure.lang.RT/box
+                 (clojure.lang.Numbers/add ^java.lang.Object sum i))]
+          (recur next-sum (clojure.lang.Numbers/unchecked_inc i)))))
+    sum))
+```
+
+When the index reaches five, the `done?` branch returns the sum: 10. That was our `goto`. Having gone to considerable trouble to accommodate it, we have compiled it out of existence.
+
+## Back to our little variable
+
+Back at the inversion counter, we can write `(setq n (inc n))` where the counting happens, inside the nested loops. [Blacktide](https://tailrecursion.com/repos/r/blacktide.git/b/master/t/f=README.md.html) figures out the bindings and `recur` arguments needed to carry that value forward. Our assignment becomes the kind of update Clojure already knows how to compile.
+
+To think, all this time, mutation was sitting inside `recur`, wearing a little hat that said FUNCTIONAL PROGRAMMING...
+
+## Appendix: Other stuff like this
+
+Truth be told, this isn't my first rodeo. [HyperTURBO](https://github.com/alandipert/hyperturbo) is my earlier BASIC-inspired Clojure macro. Each numbered line becomes a function; an atom holds the program counter and variables. `GOTO` updates that counter, which the dispatcher consults after the line's function returns. Line numbers included at no extra charge.
+
+To interrupt a function before it returns, you can throw something. [flambard/tagbody](https://github.com/flambard/tagbody) uses this to implement Common Lisp-style `tagbody`: a sequence of forms with labels you can jump to. Its [implementation](https://github.com/flambard/tagbody/blob/05352cd75cc218814cd98c3676f8cac26b0fe4e9/src/tagbody/core.clj) puts each tagged section in a function. `goto` throws a custom `Throwable` carrying the destination; the dispatcher catches it and resumes there.
+
+[Farolero](https://github.com/IGJoshua/farolero) uses thrown signals too. A callback can jump out through intervening function calls, running `finally` cleanup along the way. This is a _nonlocal exit_. Farolero's [`tagbody`/`go` implementation](https://github.com/IGJoshua/farolero/blob/6dbd299d25b37d22370a3134541bc49bb5c41273/src/cljc/farolero/core.cljc#L144-L196) feeds those signals into a `loop`/`case` dispatcher and checks that an escape targets the right, still-active execution of `tagbody`. It also supplies `block`/`return-from`, all within a larger system of Common Lisp-style conditions and restarts: tools for reporting problems and choosing how to recover.
+
+A jump that stays in one function gives the compiler another option. My [2020 JACL paper, §3.3.3](https://tailrecursion.com/~alan/documents/jacl-els-2020-draft.pdf#page=4) describes compiling Common Lisp's local `GO` operations into JavaScript's labeled `continue`. If all jumps to a `TAGBODY` are local, its generated `try`/`catch` disappears too.
+
+That's the division of labor I'd want in a fuller `tagbody`: use Blacktide-style compilation wherever a jump can become ordinary control flow, and reserve exception unwinding for exits through function calls. Both paths still owe us our `finally` blocks. Blacktide currently sticks to its compiled region.
+
+Back in Clojure, the practical differences are:
+
+| Library | Where changing state lives | Where a jump can go |
+| --- | --- | --- |
+| Blacktide | Lexical locals assigned with `setq`/`psetq` | An explicit label in the same compiled region |
+| flambard/tagbody | Separate storage; its example uses `with-local-vars` | Through function calls to an enclosing `tagbody`, matched by label name |
+| Farolero | Separate storage, such as atoms or volatiles | Through function calls to a particular active `tagbody` |
+
+Blacktide spends its compiler machinery on the problem we started with: updating our own little variables inside an algorithm.
+
+Both [tagbody](https://github.com/flambard/tagbody/blob/05352cd75cc218814cd98c3676f8cac26b0fe4e9/src/tagbody/Goto.clj) and [Farolero](https://github.com/IGJoshua/farolero/blob/6dbd299d25b37d22370a3134541bc49bb5c41273/src/java/farolero/signal/Signal.java) suppress stack-trace construction for their signals. A speed ranking needs benchmarks; “exceptions are slow” doesn't get to count as one.
diff --git a/tests/test_sitegen.py b/tests/test_sitegen.py
index b946ae9..9baa53c 100644
--- a/tests/test_sitegen.py
+++ b/tests/test_sitegen.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import os
 import tempfile
 import unittest
 from datetime import datetime, timezone
@@ -195,5 +196,105 @@ class SitegenTests(unittest.TestCase):
                 normalize_output.normalize(root, 123)
 
 
+class PreviewTests(unittest.TestCase):
+    def setUp(self) -> None:
+        directory = tempfile.TemporaryDirectory()
+        self.addCleanup(directory.cleanup)
+        self.root = Path(directory.name)
+        self.src = self.root / "md"
+        self.out = Path(os.path.relpath(self.root / "out"))
+        self.src.mkdir()
+        self.head = self.root / "head.html"
+        self.foot = self.root / "foot.html"
+        self.head.write_text('<html><head>@META@</head><body>', encoding="utf-8")
+        self.foot.write_text('</body></html>', encoding="utf-8")
+        self.home = (
+            "# Home\n\n<!-- latest-works:6 -->\n\n## Updates\n\n"
+            "| Date | Note |\n| --- | --- |\n"
+            "| 2026-01-01 | Added [Published](./Published.md). |\n"
+        )
+        for name, content in {
+            "Home.md": self.home,
+            "Published.md": "# Published\n",
+            "Other.md": "# Other\n\n[Published](./Published.md)\n",
+            "_Preview.md": "# Readable Preview Title\n\n[Published](./Published.md)\n\n[Part](./_series/Part)\n",
+            "_series/Part.md": "# Preview Part\n\n[Preview](../_Preview.md)\n",
+            ".#Locked.md": "# Editor lock\n",
+            "Backup.md~": "# Editor backup\n",
+        }.items():
+            path = self.src / name
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text(content, encoding="utf-8")
+
+    def build(self) -> None:
+        sitegen.build(self.src, self.out, self.head, self.foot, "https://example.com/blog/")
+
+    def test_previews_render_without_automatic_discovery(self) -> None:
+        self.build()
+        preview = (self.out / "_Preview.html").read_text()
+        self.assertIn('<h1>Readable Preview Title</h1>', preview)
+        self.assertIn('<meta name="robots" content="noindex">', preview)
+        self.assertIn('href="https://example.com/blog/_Preview.html"', preview)
+        self.assertIn('href="./_series/Part.html"', preview)
+        self.assertIn('href="../_Preview.html"', (self.out / "_series/Part.html").read_text())
+        self.assertFalse((self.out / ".#Locked.html").exists())
+        self.assertFalse((self.out / "Backup.html").exists())
+        for name in ("Home.html", "Index.html", "Published.html", "Other.html", "updates.xml"):
+            with self.subTest(page=name):
+                rendered = (self.out / name).read_text()
+                self.assertNotIn("_Preview", rendered)
+                self.assertNotIn("_series", rendered)
+                self.assertNotIn("Readable Preview Title", rendered)
+                self.assertNotIn('content="noindex"', rendered)
+        published = (self.out / "Published.html").read_text()
+        self.assertIn('href="Other.html"', published)
+        self.assertIn('href="Home.html"', published)
+
+    def test_listed_pages_cannot_link_to_previews(self) -> None:
+        self.build()
+        listed = self.out / "Published.html"
+        for reference in (
+            "./_Preview.html#section",
+            "./%5fPreview.html?review=1",
+            "_series/Part.html",
+            "/blog/_Preview.html",
+            "https://example.com/blog/_Preview.html",
+            "http://EXAMPLE.COM/blog/_Preview.html",
+            "//example.com/blog/_Preview.html",
+        ):
+            with self.subTest(reference=reference):
+                listed.write_text(f'<a href="{reference}">preview</a>', encoding="utf-8")
+                with self.assertRaisesRegex(RuntimeError, "links to unlisted preview"):
+                    sitegen.validate_internal_references(self.out, "https://example.com/blog/")
+        listed.write_text(
+            '<a href="https://elsewhere.example/_Preview.html">external</a>'
+            '<a href="https://example.com/blog/legacy.pdf">server-managed file</a>',
+            encoding="utf-8",
+        )
+        sitegen.validate_internal_references(self.out, "https://example.com/blog/")
+
+    def test_updates_cannot_announce_previews(self) -> None:
+        (self.src / "Home.md").write_text(
+            self.home + "| 2026-01-02 | Added [Preview](./_Preview.md). |\n",
+            encoding="utf-8",
+        )
+        with self.assertRaisesRegex(RuntimeError, "links to unlisted preview"):
+            self.build()
+
+    def test_removing_prefix_publishes_page_normally(self) -> None:
+        (self.src / "_Preview.md").rename(self.src / "Preview.md")
+        (self.src / "Preview.md").write_text("# Readable Preview Title\n\n[Published](./Published.md)\n")
+        (self.src / "_series/Part.md").write_text("# Preview Part\n\n[Preview](../Preview.md)\n")
+        (self.src / "Home.md").write_text(
+            self.home + "| 2026-01-02 | Added [Preview](./Preview.md). |\n",
+            encoding="utf-8",
+        )
+        self.build()
+        self.assertNotIn('content="noindex"', (self.out / "Preview.html").read_text())
+        for name in ("Home.html", "Index.html", "Published.html", "updates.xml"):
+            with self.subTest(page=name):
+                self.assertIn("Preview.html", (self.out / name).read_text())
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/tools/sitegen.py b/tools/sitegen.py
index 3992b24..62cb9a1 100644
--- a/tools/sitegen.py
+++ b/tools/sitegen.py
@@ -59,7 +59,11 @@ def parse_args() -> argparse.Namespace:
 
 
 def is_temp_path(path: Path) -> bool:
-    return any(part.startswith((".#", "_")) or part.endswith("~") for part in path.parts)
+    return any(part.startswith(".#") or part.endswith("~") for part in path.parts)
+
+
+def is_unlisted(path: Path) -> bool:
+    return any(part.startswith("_") for part in path.parts)
 
 
 def markdown_files(src: Path) -> list[Path]:
@@ -307,6 +311,8 @@ def write_page(
     )
     canonical = urljoin(base_url, output_relative.as_posix())
     meta = metadata_html(title, description, canonical, home=relative == Path("Home.md"))
+    if is_unlisted(relative):
+        meta += '\n  <meta name="robots" content="noindex">'
     info = buildinfo.build_info(source.as_posix())
     body = rewrite_html_links(
         run_cmark(output="html", source=source),
@@ -620,23 +626,44 @@ class LocalReferenceParser(HTMLParser):
         )
 
 
-def validate_internal_references(out: Path) -> None:
+def validate_internal_references(out: Path, base_url: str = "") -> None:
     root = out.resolve()
+    base = urlsplit(base_url)
     failures: list[str] = []
-    for page in sorted(out.rglob("*.html")):
+    for page in sorted(root.rglob("*.html")):
         parser = LocalReferenceParser()
         parser.feed(page.read_text(encoding="utf-8"))
         for reference in parser.references:
             parsed = urlsplit(reference)
-            if parsed.scheme or parsed.netloc or not parsed.path:
+            if not parsed.path:
                 continue
-            target = (page.parent / unquote(parsed.path)).resolve()
+            absolute_reference = bool(
+                parsed.scheme or parsed.netloc or (base_url and parsed.path.startswith("/"))
+            )
+            if absolute_reference:
+                resolved = urlsplit(urljoin(base_url, reference))
+                if (
+                    not base_url
+                    or resolved.scheme not in {"http", "https"}
+                    or resolved.netloc.lower() != base.netloc.lower()
+                    or not resolved.path.startswith(base.path)
+                ):
+                    continue
+                target = (root / unquote(resolved.path[len(base.path):])).resolve()
+            else:
+                target = (page.parent / unquote(parsed.path)).resolve()
             try:
-                target.relative_to(root)
+                relative = target.relative_to(root)
             except ValueError:
                 failures.append(f"{page}: {reference} escapes the output directory")
                 continue
-            if not target.exists():
+            if (
+                target.suffix == ".html" and is_unlisted(relative)
+                and not is_unlisted(page.relative_to(root))
+            ):
+                failures.append(f"{page}: links to unlisted preview {reference}")
+            elif not absolute_reference and not target.exists():
+                # Absolute URLs may name server-managed files outside this build.
                 failures.append(f"{page}: {reference} does not exist")
     if failures:
         raise RuntimeError("Invalid internal references:\n" + "\n".join(failures))
@@ -656,7 +683,11 @@ def build(src: Path, out: Path, head_path: Path, foot_path: Path, base_url: str)
         path.relative_to(src).as_posix(): document_title(document, path.stem)
         for path, document in documents.items()
     }
-    backlinks = backlink_map(src, documents)
+    listed = {
+        path: document for path, document in documents.items()
+        if not is_unlisted(path.relative_to(src))
+    }
+    backlinks = backlink_map(src, listed)
     relatives = [path.relative_to(src) for path in paths]
     home = src / "Home.md"
     if home not in documents:
@@ -668,9 +699,11 @@ def build(src: Path, out: Path, head_path: Path, foot_path: Path, base_url: str)
             source, relative, out, head, foot, backlinks, updates, src,
             documents[source], titles, normalized_base,
         )
-    write_index(out, src, relatives, titles, head, foot, normalized_base)
+    write_index(
+        out, src, [path.relative_to(src) for path in listed], titles, head, foot, normalized_base
+    )
     write_rss(out, updates, normalized_base)
-    validate_internal_references(out)
+    validate_internal_references(out, normalized_base)
 
 
 def main() -> None: