jerboa-search

jerboa-search is a Jerboa-native crawler and full-text search engine. It is an equivalent product to the useful single-node core of YaCy, not a protocol or data-format port.

It composes existing local projects:

• jsqlite stores authoritative document metadata, crawl state, links, content references, and the index generation.
• jerboa-websearch provides SSRF-safe HTTPS fetching and sandboxed HTML5 parsing.
• jerboa-sinatra serves the search UI and JSON crawl/search APIs.
• Jerboa supplies native threads, HTTP, JSON, WASM isolation, and build tooling.

The new search-specific pieces are a BM25 inverted index and a content-addressed body store. The body store deduplicates immutable page text outside SQLite. Generation-bound, checksummed immutable index segments make checkpoints proportional to changed documents and make ordinary restarts fast. A missing, stale, discontinuous, or corrupt segment chain is discarded and rebuilt from authoritative metadata and content. There is no YaCy/Solr compatibility layer.

The mutable core maintains document count through its document table and total document length as an incremental invariant. BM25 average length and hybrid overlay statistics are therefore constant-time metadata reads rather than an unconditional document-table traversal on every query. Single-term core queries stream final BM25 scores directly into the bounded deterministic top-k heap; only multi-term queries need a score accumulator.

The qix version-4 sidecar stores posting document references as validated ordinals into its fixed-width mapped document table. IDs and lengths are read directly from that table, so the immutable base needs no corpus-sized Scheme metadata hash; hot WAND cursors cache only their current posting. Its length-bound hierarchical SHA-256 checksum validates fixed 1 MiB chunks, so opening a large mapping never copies the complete payload into the Scheme heap. The build produces a small dependency-free native QIX shim that validates fixed-width records, posting invariants, and block summaries in one bounded pass. Jerboa's existing native digest hashes each chunk, while Jerboa retains mapping ownership, layout checks, error handling, and the final length-prefixed manifest hash. The checksum and QIX v4 bytes are unchanged and regression-checked against the pure Jerboa hash. Publication uses the same bounded chunks as its sequential output buffer and seeks back only to install the final checksum before atomic rename; it does not materialize a second sidecar-sized bytevector. A mutex-scoped sorted core view streams documents and one term's postings at a time. Dense postings reuse the already sorted document table; sparse postings sort only their own IDs and use forward galloping lookup. Fixed-width posting fields are stored directly into the checksum chunk with Jerboa's endian-aware bytevector operations, avoiding millions of tiny intermediate copies without changing QIX v4 bytes.

All query engines use a bounded deterministic top-k heap rather than sorting every match. Ranking selection scales with log(limit) and retains only the requested candidates. Single-term core, mapped, and hybrid queries stream scores directly into that heap. Mapped and hybrid traversal additionally uses validated 128-posting block maxima to skip blocks whose BM25 upper bound cannot enter top-k. Multi-term mapped queries use exact weak-AND (WAND) cursors with binary advance-to seeks and shared hybrid statistics, so they score only pivot candidates whose safe term-bound sum can beat top-k. Participating cursors then sum validated block-local bounds before exact BM25 scoring; when that sum cannot win, binary seeks advance every cursor past the complete half-open range covered by those bounds. Block-max-aware pivot scheduling across virtual block boundaries remains a possible research refinement; it requires separate evidence because substituting a current-block bound directly into ordinary WAND pivot selection is not safe. Immutable term-wide maxima are reconstructed once from block summaries and retained in a mutex-protected 4,096-slot direct-mapped cache. Cache entries hold only raw QIX statistics, so hybrid document counts and frequencies remain live; collisions merely trigger another safe scan and memory stays corpus-independent.

Mapped queries share a per-index Jerboa read lock and do not hold the SQLite store mutex while scoring postings. Writers and atomic index publication use the exclusive side, while result metadata hydration happens after the read lease is released. Retained result rows are fetched with prepared primary-key point lookups and replayed in rank order.

Closing a search store automatically unloads its managed index and mmap. Explicit unload-search-index! remains useful for restart/fallback tests but is not required for ordinary ownership cleanup.

Crawler ingestion commits each page, generation, outgoing links, and newly discovered frontier URLs atomically before updating the disposable text index. Controlled bulk import paths can use index-documents! to commit a document batch with one generation advance, while ordinary crawler writes retain per-page durability.

Run

Building requires a C compiler for the hardened QIX validation shim. No external crypto library is required. Every Scheme test converts a false run-tests! result into process status 1, so assertion failures stop the aggregate target instead of merely printing a failure summary. Migration fixtures also remove a same-name interrupted-run database before setup and pass cleanup procedures—not invoked return values—to with-resource.

make test
make integration
N=1000 make bench
N=100000 ABSENT_ROUNDS=1000 COMMON_ROUNDS=10 make core-query-bench
ROUNDS=100000 REPEATED=8 make query-terms-bench
LINKS=600 ROUNDS=100 make link-replacement-bench
URLS=200 ROUNDS=20 make frontier-enqueue-bench
N=1000 make restart-bench
N=1000 make segment-bench
N=1000 make mmap-bench
make mmap-capacity-bench CAPACITY_N=1000000 CAPACITY_ROUNDS=1000
N=100000 ROUNDS=10 LIMIT=20 make ranking-bench
N=1000 WORKERS=4 QUERIES=500 make concurrency-bench
N=1000 LIMIT=20 ROUNDS=1000 make hydration-bench
N=1000 LIMIT=20 OFFSET=100 ROUNDS=1000 make search-page-bench
N=1000 LIMIT=20 OFFSET=100 BODY_PREFIX=10000 ROUNDS=1000 make search-page-bench
N=10000 OVERRIDES=100 ROUNDS=1000 make document-frequency-bench
N=1000 make ingest-transaction-bench
MODE=file-store-batch N=2000 make memory-scaling-bench
make run

Open http://127.0.0.1:8890. The first crawl must use a public HTTPS URL. Crawler fetches reject loopback, private networks, IP literals, unsafe ports, and unsafe redirects.

SIGINT and SIGTERM request an ordered shutdown. The listener stops accepting connections, active HTTP requests drain, crawler workers join, the current index delta checkpoints, background compaction quiesces, and mapped-index plus SQLite resources close. Repeated shutdown calls are harmless. A drain or compaction failure is reported only after every remaining cleanup stage has been attempted, causing a non-zero service exit instead of claiming a clean shutdown.

Environment overrides:

• JERBOA_RUNTIME_HOME — one coherent Jerboa runtime/library home (defaults to jerbuild --jerboa-home; do not mix it with a source checkout containing separately compiled .so files)
• JERBOA_SEARCH_DB — database path (default data/search.db)
• JERBOA_SEARCH_PORT — listen port (default 8890)
• JERBOA_SEARCH_BIND — listen address (default 127.0.0.1)
• JERBOA_SEARCH_ADMIN_TOKEN — optional bearer token protecting every state-changing HTTP route; it must contain 32–1024 characters. A non-loopback bind is refused unless this is configured.
• JERBOA_SEARCH_WORKERS — crawler workers (default 4)
• JERBOA_SEARCH_MAX_DEPTH — maximum discovered-link depth (default 3, capped at 32; 0 fetches only the seed)
• JERBOA_SEARCH_MAX_PAGES — hard page-attempt budget per crawl (default 1000, capped at 10000000; 0 disables fetching)
• JERBOA_SEARCH_DELAY_MS — minimum delay between requests to the same host (default 500, capped at ten minutes; 0 is allowed). Crawler-local waits are interruptible in 25 ms slices, so stop does not wait out the interval.
• JERBOA_SEARCH_SAME_ORIGIN_ONLY — whether discovered links must share the seed's scheme/host/port (default true; accepts true/false, yes/no, on/off, or 1/0)
• JERBOA_SEARCH_CHECKPOINT_EVERY — processed pages between incremental index checkpoints (default 1000, capped at 1000000)
• JERBOA_SEARCH_MAX_OVERLAY_DOCUMENTS — changed-document threshold for scheduling background segment compaction (default 1000, capped at 1000000)
• JERBOA_SEARCH_PAGE_UPSERT_BATCH_SIZE — maximum rows per bulk page-upsert statement for controlled import/reindex paths (default 500, capped at 500; ordinary crawler writes remain per page)
• JERBOA_SEARCH_MAX_ATTEMPTS — total attempts allowed per URL, including the initial request (default 3, capped at 16)
• JERBOA_SEARCH_RETRY_BASE_MS — initial delay before retrying a transient fetch failure (default 1000, capped at ten minutes; 0 is allowed)
• JERBOA_SEARCH_RETRY_MAX_MS — exponential-backoff ceiling (default 60000, capped at one hour and never lower than the configured base delay)
• JERBOA_SEARCH_MAX_CONNECTIONS — global concurrent HTTP connections (default 128, capped at 4096)
• JERBOA_SEARCH_MAX_CONNECTIONS_PER_IP — concurrent connections per client IP (default 16, capped by the global limit)
• JERBOA_SEARCH_ACCEPTS_PER_SECOND_PER_IP — accepted connections per second per client IP (default 32, capped at 10000)
• JERBOA_SEARCH_IDLE_TIMEOUT_MS — idle connection timeout (default 5000)
• JERBOA_SEARCH_HEADER_TIMEOUT_MS — request-header deadline (default 15000)
• JERBOA_SEARCH_BODY_TIMEOUT_MS — request-body deadline (default 30000); all three timeout settings are capped at five minutes
• JERBSEARCH_HTML_PARSER_WASM — shared sandbox parser artifact

Scope

The service implements document crawling, durable frontier recovery and retry deadlines, link graph capture, document replacement, BM25 search, HTML results, JSON APIs, crawl controls, and health/status endpoints. It deliberately excludes YaCy peer protocols, DHT/RWI exchange, Solr compatibility, legacy templates, and YaCy data migration.

For a database at data/search.db, durable page bodies live below data/search.db.content/ and the derived index cache is data/search.db.jerboa-index (manifest) plus data/search.db.jerboa-index.segments/ (immutable segments). A validated data/search.db.jerboa-index.qix binary sidecar serves mmap-hybrid queries. Back up the database and content directory together while the service is stopped. The manifest, segments, and binary sidecar are optional derived data: deleting them only makes the next startup rebuild the index.

Operational endpoints are:

• GET /api/search?q=...&limit=20&offset=0 — deterministic BM25 results. limit is capped at 100 and offset at 1000. Responses include the effective limit, offset, returned count, has_more, and bounded previous/next offsets. The HTML /search route accepts the same parameters and renders encoded Previous/Next navigation. Ranked metadata hydration uses prepared primary-key point lookups, so every supported offset avoids compound-select and bind-variable ceilings.
• GET /api/status — crawler, generation, checkpoint generation, segment count, changed-document overlay size/threshold, query engine, full-core residency, compaction activity/failure, source, and dirty status. Documents are exact; terms_exact distinguishes an exact term count from the safe upper bound used while base documents are replaced or tombstoned. Its http object reports authentication state, effective connection/rate/timeout settings, and application input ceilings without exposing the bearer token, seed URL, local filesystem paths, or internal exception text. The crawl object includes the durable crawl-id, its lifecycle state, and frontier counts and max-frontier ceiling scoped to that run. not-modified-this-run counts accepted conditional 304 responses without exposing page validators, while sitemaps-this-run counts successfully processed sitemap documents.
• GET /healthz — process liveness only; it does not touch SQLite or the index.
• GET /readyz — readiness for search traffic. It opens/validates the current index view and returns 503 when the store is unavailable or background compaction has failed. Successful responses include generation and query engine; failures do not expose exception text or filesystem paths.
• GET /metrics — Prometheus text exposition with fixed, label-free gauges for readiness, index documents/terms/generation/segments/overlay/dirty/compaction, and crawler running/workers/retry-policy/frontier/conditional-fetch, robots-cache occupancy, and host-deadline cleanup state. It returns 503 with jerboa_search_up 0 when metric refresh cannot read the store/index.

Cross-origin discovery does not relax fetch security. When JERBOA_SEARCH_SAME_ORIGIN_ONLY=false, every discovered URL still passes the public-HTTPS, DNS/IP, safe-port, redirect, robots, and per-host politeness checks. The page budget remains global across all discovered origins. With same-origin policy enabled, the final URL after redirects must also remain on the seed origin; a cross-origin final response is rejected before storage.

Robots handling follows RFC 9309 group selection and longest-match behavior. Exact, case-insensitive product-token groups are merged; wildcard rules are used only when no exact group exists, and blank lines do not accidentally end a group. A missing robots.txt reported by a permanent 4xx permits crawling. Server/network failures, throttling responses, and unresolved redirects do not fall through to allow-all: the page remains unfetched and enters the bounded durable retry policy. Robots wildcard matching is linear in the rule and URL length, avoiding recursive backtracking on untrusted policy text. Matching normalizes percent-encoded ASCII unreserved bytes, uppercases retained reserved encodings, and percent-encodes raw UTF-8. Encoded literal * and $ bytes remain data and cannot become rule metacharacters.

Global Sitemap: directives in a successful robots.txt are discovered once per origin and entered as durable frontier work. Oversized Sitemap: directive lines share the same 8,192-character robots line ceiling and are ignored. Both URL-set and sitemap-index documents are supported. Fetches are capped at 4 MiB, each document yields at most 50,000 unique locations, indexes nest at most three levels, and redirects and emitted locations must remain on the sitemap's origin. DTDs and nested markup inside ordinary <loc> text are rejected; the narrow parser resolves only the five predefined XML entities and valid numeric character references. Sitemap attempts share the crawl's hard request-attempt budget, durable retry policy, politeness schedule, and run isolation. /metrics publishes jerboa_search_crawl_sitemaps.

Robots responses are read under a 524,288-byte transport ceiling. Their per-crawl in-memory cache uses a fixed 128-entry FIFO ring, bounding worst-case retained policy state even when cross-origin crawling encounters an unbounded stream of hosts. Policies are parsed and normalized once per origin and cached as compiled matchers instead of reparsing raw text for every page. Parsing retains at most 10,000 rules, 256 agents in one group, and 8,192 characters per directive line. Cache occupancy, capacity, response bytes, and parser ceilings are reported as robots-cache-entries, robots-cache-capacity, robots-body-bytes, robots-rule-limit, robots-agent-group-limit, and robots-line-chars in crawl status. Page and sitemap fetch ceilings are likewise reported as page-body-bytes, sitemap-body-bytes, sitemap-location-limit, and sitemap-depth-limit.

Discovery growth is independently bounded. Each crawl run persists a frontier ceiling equal to 10× max-pages, with a 1,000-row floor for nonzero crawls and a 1,000,000-row hard cap. Every transactional HTML-link, conditional-replay, and sitemap enqueue shares the remaining capacity; duplicate URLs consume no additional slot. This prevents a small request budget from generating an unbounded SQLite queue. The effective ceiling is exposed as max-frontier and jerboa_search_crawl_max_frontier. Discovery attempts rejected after that ceiling is full are persisted per crawl run, exposed as discovery-dropped in status, and published as jerboa_search_crawl_discovery_dropped. The counter measures attempts rather than unique URLs; rediscoveries already assigned to the current run are free.

Workers acquire a non-blocking per-host lease before processing a claimed row. If another worker owns that host, the row is durably deferred without consuming a fetch attempt and the worker can claim a ready URL from another origin. This prevents a same-host queue prefix from occupying every worker while preserving the configured request delay and one-at-a-time robots/page processing per host. Per-origin delay deadlines are indexed by a min-heap and removed from the host map as soon as they expire. Stale heap entries from a superseded deadline are discarded without deleting its newer value, so cross-origin crawls retain only the active politeness window instead of every host ever observed. host-deadlines and host-expiry-events expose both live structures in crawl status; jerboa_search_crawl_host_deadlines and jerboa_search_crawl_host_expiry_events publish them as fixed-name gauges. jerboa_search_crawl_robots_cache_entries and jerboa_search_crawl_robots_cache_capacity likewise expose bounded compiled robots-policy cache occupancy and its ceiling.

Successful page fetches have a second, document-level admission policy. Case-insensitive HTML robots and JerboaSearch meta directives are combined with generic and JerboaSearch-scoped X-Robots-Tag response directives; the most restrictive noindex/nofollow result wins. none means both. A noindex page retains its real HTTP status, metadata, body, and outgoing graph in authoritative storage, but is removed from the live BM25 view and excluded from every rebuild. nofollow retains outgoing links in the graph but adds none of them to the crawl frontier. An individual rel=nofollow link is likewise kept in the graph but excluded from discovery. This separates durable fetch history and graph analysis from search-result and crawl admission.

Relative HTML links use the first <base href> in tree order, as defined by the HTML document-base algorithm. An invalid or non-web first base falls back to the response URL and later base elements remain ignored. URL canonicalization removes dot segments without collapsing repeated or trailing slashes, query-only references retain the current document path, and any explicit non-HTTP(S) URI scheme is rejected before frontier admission. Canonical identity also decodes percent-encoded ASCII unreserved bytes and uppercases retained percent triplets. Encoded reserved delimiters stay encoded, malformed triplets are rejected, and dot segments are removed after safe unreserved decoding. Raw whitespace, including leading or trailing whitespace, control characters, and backslashes are rejected before any trimming or resolution. Canonical URL identity is capped at 4,096 characters at the shared URL-module boundary, including relative-resolution output. The same ceiling therefore applies to seeds, redirects, HTML links, canonical hints, robots sitemap directives, and sitemap locations before hashing or durable storage. Callers should provide percent-encoded URL data.

The first link[rel~=canonical][href] is resolved against the effective HTML base. A valid non-self target is accepted only on the response origin, retained in the outgoing graph, and admitted through the ordinary bounded frontier. When follow policy and depth allow that admission, the variant page remains in durable fetch history but is excluded from live and rebuilt search results. Malformed, cross-origin, self, nofollow, and depth-boundary hints do not suppress the fetched page.

Transport conditions and HTTP 408, 425, 429, and 5xx responses are retried with bounded exponential backoff. Retry deadlines and attempt counts are stored in SQLite, so a restart neither forgets nor resets them. Other HTTP responses and failures in parsing, persistence, or indexing are terminal. Every request attempt, including one that is rescheduled, consumes the crawl's hard page-attempt budget. /api/status exposes the effective retry policy and the number of queued URLs currently waiting for a deadline; /metrics publishes the same values as fixed, label-free gauges. Persisted retry/failure diagnostics are truncated at 4,096 characters at the store boundary, preventing an unusually verbose network or parser condition from amplifying durable frontier storage. Status exposes frontier_error_chars. Valid Retry-After delta-seconds and each of the three RFC 9110 HTTP-date wire formats are honored for page and sitemap responses. The server value can extend but never shorten exponential backoff, and the configured retry ceiling still caps it. Malformed or over-128-character values are ignored.

Only exact text/html, application/xhtml+xml, and text/plain success responses enter the text index; content-type parameters are allowed, but media type substrings are not guessed. Unsupported successful media is completed as skipped work rather than falsely incrementing the indexed count. Robots exclusions use the same skipped counter; jerboa_search_crawl_skipped exposes it to Prometheus.

Index amplification is separately bounded: each document contributes at most 50,000 tokens, consuming title tokens before body tokens, and alphanumeric runs longer than 128 characters are ignored. The tokenizer scans string indices without first allocating a character list. Live ingestion, persisted token_count, rebuilds, segments, and mmap sidecars all use this same bounded logical document. /api/status exposes document_tokens and term_chars. Queries likewise use at most the first 32 distinct terms found within a bounded 256-token prefix. Core, mmap, hybrid overlay ranking, result hydration, and snippet selection share this parser, preventing query text from multiplying term-stat lookups or WAND cursors without bound. Status exposes query_terms and query_token_scan. Direct paged search windows are bounded to the first 100,000 ranked candidates, while the HTTP API clamps offsets lower for interactive use; status exposes search_page_window. Preview localization scans at most the first 65,536 body characters and first eight query terms. It reuses the parsed query term list and avoids lowercasing the body window when that window is already lowercase, while retaining case-insensitive matching for mixed-case content. Ranking still uses the full bounded token view; when a match lies outside the preview window, the snippet safely falls back to the document start. Status exposes snippet_scan_chars and snippet_terms.

HTML extraction has independent post-sandbox ceilings. A representation may contain at most 100,000 parsed nodes with nesting no deeper than 256, retain at most 2,097,152 readable characters, and contribute at most 10,000 unique links including an accepted canonical target. The canonical slot is reserved before ordinary-link traversal, and traversal stops as soon as its graph budget is full. Whitespace normalization scans by string index instead of materializing a second character list. /api/status exposes document_nodes, document_depth, document_text_chars, and document_links. Titles are collected directly under a 1,024-character ceiling rather than materializing an unbounded title subtree. HTML/header robots values and link rel values use an index-based 8,192-character directive scanner; oversized directives fail closed as noindex,nofollow and cannot produce a canonical hint. Status exposes document_title_chars and directive_chars.

Each accepted seed belongs to a durable numeric crawl run. Starting the same canonical seed resumes its latest paused run, retaining queued URLs, attempts, and retry deadlines. Starting a different seed creates an isolated run, so it cannot claim frontier rows left by another seed. An explicit stop, process restart, or exhausted page-attempt budget leaves unfinished work paused; a fully drained frontier becomes completed, and starting that seed again creates a new run. Newly rediscovered URLs may move from an older run into the current run, but duplicate links within one run remain no-ops. Stopping interrupts local politeness and frontier-poll waits within one 25 ms slice. A row interrupted before its request is durably requeued and its claim attempt is reversed. Synchronous network activity is allowed to finish under its existing 10/20-second request deadline.

Successful responses persist bounded ETag and Last-Modified validators. Later crawl runs send them as If-None-Match and If-Modified-Since; validator values containing HTTP control characters or more than 1,024 characters are never reused as request headers. An exact-URL 304 Not Modified refreshes fetch time and any returned validators without replacing content, advancing the search generation, or perturbing live rankings. The same transaction replays only the page's previously approved crawl edges into the new run, so unchanged seeds do not truncate traversal and nofollow links remain graph-only. Legacy link rows default to non-crawlable until a fresh representation classifies them.

An observed 404 Not Found or 410 Gone retires the final URL immediately. The authoritative row keeps its prior title, body reference, and link history, but records the new status, clears cache validators, becomes non-indexable, and advances the search generation. Live postings are tombstoned and rebuilds also exclude the row, so stale content cannot reappear after restart or compaction. A later successful response replaces the preserved state and restores the URL normally. Retryable responses such as 408, 429, and 5xx never retire the last good representation. /api/status exposes retired-this-run, and /metrics publishes jerboa_search_crawl_retired.

• POST /api/index/checkpoint — append changes as an immutable segment and atomically publish its manifest; reaching eight segments or 1,000 changed document IDs since the mapped base schedules background compaction without delaying the checkpoint response. Set JERBOA_SEARCH_MAX_OVERLAY_DOCUMENTS to tune the latter threshold. Responses include ok, generation, and checkpoint_generation, not local manifest paths.
• POST /api/index/compact — replace the current chain with one full immutable segment and delete only obsolete files named by the old validated manifest. Responses include ok, generation, and checkpoint_generation, not local manifest paths.
• POST /api/content/gc — remove unreferenced managed content blobs and abandoned managed blob temporaries; the crawler must be stopped so newly fetched content cannot race collection. The stop endpoint joins all fetch workers before returning, so a successful stop is a quiescent GC boundary.

All state-changing routes are administrative: POST /crawl, POST /api/crawl, POST /api/crawl/stop, POST /api/index/checkpoint, POST /api/index/compact, and POST /api/content/gc. When JERBOA_SEARCH_ADMIN_TOKEN is set, clients must send Authorization: Bearer <token> on each of them. Missing or incorrect credentials receive 401, a WWW-Authenticate challenge, and a JSON error; the HTML home page hides its crawl form because browser forms cannot attach the header. Search, status, and health endpoints remain public. Token comparison uses Jerboa's timing-safe crypto primitive, and the token is retained only in the application closure—not exposed in settings or responses.

The application rejects bodies above 65,536 bytes with 413 before form decoding, search queries above 1,024 characters with 414, and crawl seeds above 4,096 characters with 400. Discovered URLs enforce that same ceiling inside canonicalization and are simply ineligible for admission. The underlying Jerboa transport independently bounds request headers at 16 KiB and bodies at 4 MiB, so malformed traffic is bounded before application policy runs.

Without a token, the administrative routes remain available for the default local-only workflow, but the server accepts only 127.0.0.1, ::1, or localhost as its bind address. Set the token even on a loopback bind when a reverse proxy makes the service reachable by other machines. Supply the token through the environment or a service manager's secret facility; do not place it in URLs, request bodies, or command-line arguments. Bearer credentials are replayable, so any non-local client connection must use TLS. The current launcher serves plain HTTP and is intended to sit behind a TLS-terminating reverse proxy on a private or loopback connection.

See ARCHITECTURE.md for scaling boundaries and package extraction points.