AgentSearch launches self-hosted search API for AI agents

AgentSearch is a self-hosted search API for AI agents that adds a larger set of retrieval and filtering tools around SearXNG, according to the project’s Show HN post on Hacker News. The project runs locally with Docker and exposes 16 endpoints for search, reading, batch extraction, source discovery, job search, policy search, news, and adaptation metrics.

The author said the standard setup starts with three commands: clone the repository, run docker compose up -d, and send a request to http://localhost:3939/search. That default instance provides a deduplicated, multi-engine search API on port 3939, without API keys, per-query fees, or vendor lock-in, the post said.

AgentSearch is built as a FastAPI layer on top of SearXNG. According to the project description, it adds deduplication, cross-engine scoring, content extraction, query expansion, domain trust scoring, prompt injection scrubbing, caching, and a self-improvement loop that analyzes failures and tunes configuration.

The project also includes a more private deployment option. The author said users can run a second stack with docker compose -f docker-compose.yml -f examples/compose.private.yml up, which exposes an anonymized instance on port 3940 that routes traffic through Tor with Snowflake obfuscation. The private setup uses CoreDNS with Cloudflare DNS-over-TLS and keeps the private SearXNG container on an internal Docker network with no direct internet egress except through Tor, according to the post.

The /search endpoint performs multi-engine web search with deduplication and scoring. The /search/deep endpoint expands a query server-side, runs multiple variations in parallel, and fuses the results. The /search/extract endpoint combines search and inline content extraction in one call, while /search/jobs, /search/policy, /search/sources, /search/stats, and /news target specific use cases such as job listings, regulatory documents, source discovery, metrics, and multi-source news.

For reading pages, AgentSearch exposes /read and /read/batch. The project says /read uses a nine-step extraction chain that escalates through strategies until one succeeds, including direct fetch, readability scoring, user-agent rotation, Wayback Machine snapshots, Google Cache, fallback search for coverage elsewhere, custom adapters, PDF extraction with pdfplumber, and YouTube transcript extraction with yt-dlp.

Every request also gets SSRF protection, prompt injection detection, paywall detection, and content length caps, according to the post. SSRF, or server-side request forgery, is a class of attack where a server is tricked into making unwanted internal or external requests.

The project lists several supporting modules, including source provenance tracking, domain trust scoring, a content cache, and pluggable adapters for cases like Cloudflare bypass, Medium pages, 403 responses, parse errors, and empty content fallbacks. It also includes a Python SDK, an MCP server for Claude Desktop, and a test suite.

The author claims the system is already useful in an autonomous research workflow. In one example, a real autonomous research agent called “the wolf” went from finding zero frameworks on some hunts to 17 frameworks per hunt after AgentSearch replaced a hand-rolled SearXNG client that had been silently returning 401 errors on three of four engines, according to the post.

AgentSearch is published under the MIT license, and the repository invites forks, branches, commits, and pull requests.