jerboa-websearch

A port of SearXNG to Jerboa Scheme. Two related binaries ship from this repo:

• bin/jerbsearch — a SearXNG-style metasearch HTTP server. Aggregates results from upstream engines (DuckDuckGo today, more pluggable), serves a search UI and a JSON/CSV/RSS API on port 8889.
• bin/jerbsearch-mcp — a Model Context Protocol (MCP) server over stdio, exposing a single tool, web_fetch_extracted, that fetches a URL from this machine's local IP and returns cleaned readable text. It exists so that MCP clients running on networks blocked by some sites (e.g. Claude Code from Anthropic IPs hitting reddit, certain news sites) can read those pages through the user's own network instead.

Both share the same source tree and runtime; the MCP server reuses the HTML parsing and HTTP client from the metasearch core.

---

Repository layout

bin/
  jerbsearch              # metasearch web server launcher (bash)
  jerbsearch-engine-worker # fresh-process engine callback launcher
  jerbsearch-mcp          # MCP-over-stdio launcher (bash)
  jerbsearch-mcp-boot.ss  # MCP boot script (real file so stdin stays free)
src/jerbsearch/
  webapp.ss               # HTTP routes, query parsing, response shaping
  search.ss               # engine orchestrator, result merging
  engine.ss               # engine registry + base type
  engines/duckduckgo.ss   # one concrete upstream engine
  query.ss                # parsed-query record + helpers
  result.ss               # result record + helpers
  bangs.ss                # !bang redirects (DuckDuckGo-style)
  answerers.ss            # built-in answer providers
  metrics.ss              # request/engine counters
  settings.ss             # YAML settings loader
  mcp.ss                  # MCP JSON-RPC server (run-mcp!)
  net-policy.ss           # bounded, pinned HTTPS + SSRF checks
  sandbox/
    html-parse.ss         # SXML parser wrapper around the wasm sandbox
    html_parser.wasm      # built artifact (created by `make wasm`)
  templates/              # web UI templates
  static/                 # web UI static assets
wasm/
  src/lib.rs              # html5ever, compiled to wasm32-unknown-unknown
  Cargo.toml
  build.sh                # cargo build + copy into src/jerbsearch/sandbox/
settings/settings.yml     # default metasearch config
data/external_bangs.json  # bang definitions
tests/
  integration.sh                    # spins server up, hits every endpoint
  test-html-sandbox-smoke.ss        # parser smoke
  test-html-sandbox-fuzz.ss         # parser fuzz
  test-html-sandbox-fallback.ss     # fail-closed/dev-fallback startup policy
  test-engine-worker-failclosed.ss  # fresh-worker identity rejection
  test-search-limits.ss             # global admission + deadline cancellation
  test-net-policy.ss                # SSRF/network policy regressions
notes/searxng-architecture.md       # design notes
docs/                              # threat model, sandbox, release evidence
Makefile
.mcp.json                 # project-scoped MCP config (Claude Code)

---

Prerequisites

For make binary, the build uses jerbuild. If jerbuild is not on PATH, make downloads the matching Jerboa release toolchain into .jerboa/bin automatically. A separate Chez install or Jerboa source checkout is not needed for native binary builds.

Interpreter-mode launchers (bin/jerbsearch, bin/jerbsearch-mcp) still use JERBOA_HOME, defaulting to ~/mine/jerboa, because they run from source and load the runtime library dynamically.

1. Rust + cargo with the wasm32-unknown-unknown target — only needed to (re)build src/jerbsearch/sandbox/html_parser.wasm:

   rustup target add wasm32-unknown-unknown
   

   If the wasm file or runtime is missing, parsing fails closed. An in-process parser exists only behind the explicit development-only startup flag; see tests/test-html-sandbox-fallback.ss.

Platform notes:

• macOS: native lib is libjerboa_native.dylib, resolved via DYLD_LIBRARY_PATH.
• Linux: native lib is libjerboa_native.so, resolved via LD_LIBRARY_PATH. Both launchers export both variables, so the same scripts work on either OS provided the runtime is built for that platform.

---

Build

make binary       # build standalone jerbsearch-mcp via jerbuild
make wasm          # build sandbox/html_parser.wasm
make test          # network policy + sandbox smoke/fuzz/fallback
make parser-soak   # deterministic offline HTML/query/URL-policy parser soak
make verify        # security gate + RustSec audit + tests + parser-soak + binary
make sbom          # write SBOM/toolchain/provenance evidence
make reproducibility-report # compare two clean standalone builds
make release-evidence
make integration   # end-to-end HTTP tests (starts a server on a free port)
make clean

Cross targets:

make linux-amd64
make linux-arm64
make freebsd-amd64

Cross builds use jerbuild for the project build and JERBOA_HOME for target Chez boot/kernel files. If the matching cross Chez output is missing (.chez-cross-ta6le, .chez-cross-tarm64le, or .chez-cross-ta6fb), the Makefile asks that checkout to build it first. You still need the corresponding C toolchain, Rust target, and FreeBSD sysroot for freebsd-amd64.

---

Run the metasearch server

bin/jerbsearch                          # default settings/settings.yml
bin/jerbsearch /path/to/custom.yml      # custom settings file

Then visit http://127.0.0.1:8889/. Useful endpoints:

Path	Description
/	Search UI
/search?q=…	Search (HTML / JSON / CSV / RSS via format=)
/autocompleter?q=…	Autocomplete suggestions
/preferences	Per-cookie preferences
/stats	Engine + request metrics
/healthz	Liveness

Environment overrides honoured by the launcher:

• JERBOA_HOME — where the Jerboa runtime lives (default ~/mine/jerboa)
• SCHEME — scheme binary (default scheme)
• BANGS_PATH — external bangs JSON (default data/external_bangs.json)
• JERBSEARCH_ENGINE_WORKER — absolute path to the executable engine worker. bin/jerbsearch pins the bundled launcher; embedders must ship and pin it too. Missing, relative, non-executable, unexpectedly owned, or writable provenance paths fail closed.
• JERBSEARCH_REQUIRE_SANDBOX=1 — fail closed if the wasm HTML parser sandbox is unavailable (the default is already fail closed).
• JERBSEARCH_ALLOW_UNSANDBOXED_HTML=1 — development-only startup fallback; never use with untrusted input or production deployments.

---

Run as an MCP server

bin/jerbsearch-mcp speaks newline-delimited JSON-RPC 2.0 on stdio (MCP protocol version 2025-06-18). Logs go to stderr. It exposes one tool:

web_fetch_extracted

Argument	Type	Default	Range	Description
url	string	—	public https://… (required)	Absolute URL to fetch
timeout_ms	integer	15000	1000–60000	Per-request timeout
max_chars	integer	200000	1000–1000000	Truncate extracted text to N chars

The fetch runs on this machine's network, the response body is capped at 8 MiB while reading and parsed through wasm-sandboxed html5ever. Scripts/ styles/etc. are stripped, block-level tags inject paragraph breaks, and the result is returned as MCP text content. HTTP errors and bad URLs come back as isError: true tool results (not protocol errors).

Outbound MCP fetches are guarded by src/jerbsearch/net-policy.ss. By default it rejects plaintext HTTP, userinfo, IP literals, loopback, RFC1918, link-local, metadata-style addresses, and nonstandard ports. Every redirect is resolved and revalidated, and the socket connects to the exact approved IP while TLS authenticates the original host. Set JERBSEARCH_ALLOW_NONSTANDARD_PORTS=1 only for explicit local deployments with a separate allowlist or network sandbox.

Smoke-test from a shell

(
  printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"t","version":"0"}}}'
  printf '%s\n' '{"jsonrpc":"2.0","method":"notifications/initialized"}'
  printf '%s\n' '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'
  printf '%s\n' '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"web_fetch_extracted","arguments":{"url":"https://example.com"}}}'
) | bin/jerbsearch-mcp

A successful run prints three JSON-RPC responses to stdout and jerbsearch-mcp 0.1.0 starting on stdio to stderr.

Wire it into a client

Project-scoped (this repo's .mcp.json, already committed):

{
  "mcpServers": {
    "jerbsearch": {
      "command": "bin/jerbsearch-mcp",
      "args": [],
      "env": {}
    }
  }
}

Picked up automatically by Claude Code when launched from the repo root.

Global (Claude Code's ~/.claude.json): add an entry under "mcpServers" with an absolute path to the launcher:

{
  "mcpServers": {
    "jerbsearch": {
      "type": "stdio",
      "command": "/absolute/path/to/jerboa-websearch/bin/jerbsearch-mcp",
      "args": [],
      "env": {}
    }
  }
}

Concrete examples:

• macOS: "command": "/Users/<you>/mine/jerboa-websearch/bin/jerbsearch-mcp"
• Linux: "command": "/home/<you>/mine/jerboa-websearch/bin/jerbsearch-mcp"

If ~/mine/jerboa is somewhere else on that machine, set JERBOA_HOME on the entry's env:

"jerbsearch": {
  "type": "stdio",
  "command": "/home/me/code/jerboa-websearch/bin/jerbsearch-mcp",
  "args": [],
  "env": { "JERBOA_HOME": "/home/me/code/jerboa" }
}

After editing, restart the client and run /mcp (in Claude Code) to approve the server. Tools become callable as mcp__jerbsearch__web_fetch_extracted.

Troubleshooting

Symptoms reported by an MCP client and likely causes:

• "server failed to start" / immediate exit — scheme not on PATH, or $JERBOA_HOME/lib doesn't contain the right libjerboa_native.* for this OS. Run bin/jerbsearch-mcp directly and check stderr.
• "tools list empty" / "incomplete" — the boot script failed to import (jerbsearch mcp). Almost always a --libdirs / JERBOA_HOME mismatch. Re-run the smoke test above.
• web_fetch_extracted returns isError: true with "fetch failed" — network-level failure (DNS, timeout, TLS). Bump timeout_ms or try the URL with curl to confirm reachability.
• Truncation — output ends with […truncated at N chars of M total]; pass a larger max_chars (up to 1000000).

---

How the metasearch core works

webapp.ss routes requests to search.ss, which fans out to registered engines in src/jerbsearch/engines/. Each engine returns a list of result records (result.ss); search.ss deduplicates by URL and ranks. Bangs (bangs.ss) are checked before the search runs and may short-circuit into a redirect. Answerers (answerers.ss) run alongside the engine fan-out and can inject inline answers.

See notes/searxng-architecture.md for the deeper port-mapping notes against upstream SearXNG.

The HTML parser used by both the engine result extraction and the MCP tool is sandboxed: wasm/src/lib.rs wraps html5ever and is built to wasm32, then loaded under wasmi from (jerbsearch sandbox html-parse). The sandbox eliminates a class of parser CVEs from untrusted upstream HTML. If the wasm artifact is missing or the runtime can't load it, the parser fails closed. It does not reparse sandbox failures in-process. The explicit development fallback is selected once before hostile input is handled.

Search-engine callbacks do not run inside the threaded HTTP process. The parent uses Jerboa's native argv spawn to exec a fresh worker, sends a bounded JSON request over stdin, caps both output streams, enforces the absolute search deadline, kills the worker process group, and reaps it. Query text never enters argv, and no Scheme callback resumes after raw fork().

Security and release status

This repository is not yet public-production supported. Before any public release, run:

make verify
make release-evidence

The gate enforces direct-Chez import denial, raw Scheme FFI/shell/read/write sink denial, raw template-render denial, guarded outbound fetches only, RustSec audit for the wasm parser crate, parser fuzz/smoke/fail-closed tests, MCP SSRF policy tests, standalone binary build evidence, SBOM output, and a two-clean-build reproducibility report.

Security documentation lives in:

• SECURITY.md
• docs/threat-model.md
• docs/network-policy.md
• docs/parser-sandbox.md
• docs/security-hardening.md
• docs/release-evidence.md

---

Conventions

This repo is a jerboa-* project, so all source lives in .ss files written in Jerboa style — (import (except (jerboa prelude) ...)), def, defstruct, [...] list literals, {...} hash literals, keyword: args. The first line is still #!chezscheme and the body is wrapped in (library …) (that's how Jerboa libraries look — see ~/mine/jerboa-gitsafe/gitsafe/*.ss for the canonical pattern); the toolchain compiles .ss → .sls. Do not commit hand-written R6RS .sls library files in this repo.

Upstream dependencies in ~/mine/jerboa

This repo uses two modules from the jerboa stdlib that were only recently promoted from untracked working copies into the repo:

• (std net http) — keyword-arg HTTP fetch with redirect-following and http-final-url, on top of (std net tcp) + (std net tls-rustls).
• (std net thread-httpd) — cross-platform thread-per-connection HTTP/1.1 server. Use this rather than (std net fiber-httpd) on macOS / BSD because fiber-httpd is epoll-backed (Linux only).

If a fresh clone of jerboa fails to import (std net http) or (std net thread-httpd), your ~/mine/jerboa checkout is older than commit b3d4dfc ("std/net: add http + thread-httpd modules with BSD support") — git -C ~/mine/jerboa pull and rebuild.