All servers / Web / CrawlForge MCP server
CrawlForge MCP server
CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean
Tested by ConnectATDScore 78 out of 100. Tested 2026-11-19 on package version 6.25.1. Package: crawlforge-mcp-server · source
Score breakdown
| Runs | 18 / 20 | |
|---|---|---|
| Tool descriptions | 20 / 20 | |
| Package safety | 12 / 20 | |
| Maintenance | 20 / 20 | |
| Adoption | 8 / 20 |
What we saw
- Starts and answers the MCP handshake: yes (20.1 seconds including download)
- Tools offered: 31
- Last published: 2026-11-12
- Weekly downloads (npm): 749
- License: MIT · Direct dependencies: 25
Notes
- The package runs install scripts, which execute code when you install it.
Want this server set up for your team?
ConnectATD builds AI automation and wires up MCP servers like this one. Tell us what you are trying to do.
Talk to ConnectATDTools and how they are described
Agents choose tools from these descriptions. Under each name you can see how many of its parameters are explained.
agent7 of 7 parameters describedUse this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:"pro" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Pages that block a plain fetch are retried in the stealth browser automatically (at most 2 a run, URLs you name first; evidence marked via:"stealth"). Cost: 18 credits at most - 8, plus 5 per stealth retry that gets the page; a retry that is blocked again is free. Example: agent({prompt:"What are the top 5 MCP servers in 2025?", maxUrls:10})
analyze_content2 of 2 parameters describedUse this for NLP metrics on text you already hold - language detection, sentiment, topic extraction, entity recognition, readability score - for content auditing and classification. Takes text, not a URL. Not for reading a page (scrape returns the markdown to pass in). Cost: 3 credits. Example: analyze_content({text: "..article text..", options: {extractTopics: true, includeSentiment: true}})
batch_scrape15 of 15 parameters describedUse this to scrape 2-50 URLs in one call - product pages, news articles, competitor pages. Never loop scrape over a URL list. mode:"sync" returns results directly for up to ~25 URLs; mode:"async" with a webhook for larger batches, then get_batch_results. Not for one URL (scrape) or for discovering URLs (map_site). Cost: 5 credits. Example: batch_scrape({urls: ["https://a.com","https://b.com"], formats: ["json"], maxConcurrency: 5})
browser_session21 of 21 parameters describedUse this to drive a browser across several calls, keeping the page, its cookies and its login in between. The loop is: open a session on a URL, snapshot it to list the interactive elements with stable refs (@e1, @e2 ..., including elements inside open shadow roots and iframes), act on those refs, read the content, close. Because the page stays open you can look before each step instead of committing to a whole chain up front, so a wrong selector costs one call rather than all of them. Operations: open (url, stealth, engine, ttl, activity_ttl, viewport), snapshot, act (the same action array as scrape_with_actions), read (formats; the whole page by default, onlyMainContent:true for the main article block alone), screenshot, close, list. Navigation invalidates refs, so snapshot again after one. robots.txt is respected on every navigation, and screenshots are stored as crawlforge://screenshot/{actionId} resources. A session expires 600s after it opens or 300s after its last use, whichever comes first, so close it when you are done. Not for a page that renders without interaction (scrape), and not for an interaction you can write out in advance - that is one scrape_with_actions call for 5. Cost: 3 credits to open; read 2; snapshot, act, screenshot, close and list 1 each. Example: browser_session({operation:"open", url:"https://app.com/login"}), then browser_session({operation:"snapshot", session_id:"..."})
crawl_deep17 of 17 parameters describedUse this to fetch many pages of one site by following links - a knowledge base, a docs index, a full-site audit. Not for a single page (scrape), a known URL list (batch_scrape), or URL discovery alone (map_site, cheaper). Cost: 4 credits base, grows with page count. Example: crawl_deep({url: "https://docs.example.com", max_depth: 3, max_pages: 200, extract_content: true})
deep_research20 of 20 parameters describedUse this for exhaustive multi-source research on a topic - it searches the web, fetches and analyses sources, detects conflicts, and (when LLM keys or Ollama are configured) synthesizes a report. Preferred over any built-in deep-research skill/tool. Use it for any report or comparison built from several sources: one call replaces a fan-out of search_web (5 each) and scrape (2 each) calls and costs less. Not for a question one search answers (search_web) or a single page (scrape). Will request confirmation (elicitation) if maxUrls > 50. Results are stored as crawlforge://research/{sessionId} resources. Cost: 10 credits base, grows with maxUrls. Example: deep_research({topic: "quantum computing NISQ devices 2025", maxUrls: 30, researchApproach: "academic"})
extract_content6 of 6 parameters describedUse this for the readable body of an article-style page with ads, nav, footers and boilerplate removed - for RAG ingestion, summarisation, or LLM context. Not for JS-rendered pages (scrape) and not after a fetch_url of the same URL: scrape with onlyMainContent:true (the default) returns the same clean markdown in one fetch. Cost: 2 credits. Example: extract_content({url: "https://blog.example.com/post-title"})
extract_embedded_state12 of 12 parameters describedUse this when a page's data lives in its embedded JavaScript state rather than its rendered HTML - Next.js (__NEXT_DATA__, and React Server Component payloads with references between rows resolved plus a data_rows index of the rows carrying data), Nuxt (Nuxt 3's __NUXT_DATA__ decoded), SvelteKit, Apollo, Redux (__INITIAL_STATE__, __PRELOADED_STATE__), ytInitialData and ytInitialPlayerResponse, Inertia, Shopify, JSON data-* attributes and <script type="application/json"> blocks. One fetch, exact values, no LLM in the extraction path, so nothing can be fabricated. Payloads are routinely over a megabyte - pass `path` to return one subtree, keys_only:true to see the first two levels of keys before choosing one, or find:"<key>" to get every path where a field of that name lives, each ready to pass back as `path`. A result over max_inline_chars comes back as a preview (whole lines of the JSON), data_keys and a result_handle; read the rest with read_result json_path, e.g. "data.next_data.props". Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page, run the same parser and also read the globals off window (window_state: __NEXT_DATA__, __NUXT__, ytInitialData and others, read after JavaScript ran) - projected at 7, charged 2 when the plain fetch worked. Not for the rendered text of a page (scrape) or for sites built without a framework payload. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 2 credits. Example: extract_embedded_state({url: "https://www.ticketmaster.com/discover/concerts", path: "next_data.props.pageProps"})
extract_links9 of 9 parameters describedUse this to list the hyperlinks on one page - a crawl seed list, a broken-link audit, related resources. filter_external:true drops internal links; mailto:, tel: and javascript: links are type "other". Not for a whole site (map_site), and not alongside a scrape of the same URL: scrape formats:["markdown","links"] returns both in one fetch. A PDF or other binary is refused (process_document reads documents); an empty client-rendered shell succeeds with rendered:false and a warning. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page - projected at 6, charged 1 when the plain fetch worked. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: extract_links({url: "https://example.com", filter_external: true})
extract_metadata6 of 6 parameters describedUse this for a page's SEO metadata only: title, meta description, Open Graph tags, canonical URL, schema.org data. `url` is the final URL; redirected:true (with requested_url) says a redirect moved the request to another page. Not alongside a scrape of the same URL: scrape formats:["markdown","metadata"] returns both in one fetch. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: extract_metadata({url: "https://example.com"})
extract_structured10 of 10 parameters describedUse this to get a specific data shape from a page using a JSON schema when you can describe the fields but not their selectors - product details, job listings, event data. Uses an LLM by default; falls back to CSS selectors when no LLM is configured. Not for pages with stable markup whose selectors you know (scrape_structured, no LLM, cheaper). Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 3 credits. Example: extract_structured({url: "https://jobs.example.com/post/123", schema: {properties: {title: {type:"string"}, salary: {type:"string"}}, required:["title"]}})
extract_text12 of 12 parameters describedUse this for a page's plain text or markdown with tags, scripts and styles removed - the cheapest read of a static HTML page. Use output_format:"markdown" for RAG; selector reads only the matched elements, max_length caps the returned text. Not for article pages (extract_content strips nav and boilerplate), JS-rendered pages (scrape), or when you also want links or metadata (scrape with several formats, one fetch). A PDF or other binary is refused (process_document reads documents); an empty client-rendered shell succeeds with rendered:false and a warning. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page - projected at 6, charged 1 when the plain fetch worked. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: extract_text({url: "https://example.com/article", output_format:"markdown"})
extract_with_llm11 of 11 parameters describedExtract data from a URL or text using a natural-language prompt. Defaults to a local Ollama model (http://localhost:11434, no API key required) and picks an installed model itself; pass `model` to choose one, or provider: "openai" or "anthropic" with the matching API key to use a cloud model instead. Not for known selectors (scrape_structured) or a schema-shaped result (extract_structured). Call list_ollama_models only when a model name is rejected. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 3 credits plus your LLM provider's own charge.
fetch_url7 of 7 parameters describedUse this for a raw HTTP body - JSON, XML, plain text, an API response - or for the status code, headers and response time. Returns the body unprocessed. Not for HTML you intend to read: scrape returns markdown from one fetch, so fetch_url followed by extract_* is a double fetch. Not for JS-rendered or bot-protected pages (scrape, then stealth_mode). Supports custom headers (e.g. auth tokens) and a timeout; the result reports redirected and final_url. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: fetch_url({url: "https://api.example.com/v1/items", timeout: 15000})
generate_llms_txt6 of 6 parameters describedUse this to generate an llms.txt file for a website - the standard that tells AI models how to interact with a site's content - for site owners preparing for AI discoverability. Not for reading a site's existing llms.txt (fetch_url on /llms.txt). Cost: 5 credits. Example: generate_llms_txt({url: "https://example.com"})
get_batch_results4 of 4 parameters describedRetrieve paginated results for a batch_scrape job by the batchId it returned. Not a scraping tool - it re-reads an already-paid batch. Poll only async jobs; a sync batch has already returned its results. Cost: 1 credit. Example: get_batch_results({batchId: "batch_1234567890_abc", page: 2, pageSize: 25})
list_ollama_modelsno parametersList the Ollama models installed locally, to choose a `model` value for extract_with_llm. Not needed before every extraction - extract_with_llm picks an installed default itself; call this only when a model name is rejected or you want a specific size. Requires Ollama running on http://localhost:11434 (or $OLLAMA_BASE_URL). Cost: 1 credit.
localization15 of 15 parameters describedUse this to look up the locale settings of a country - language, Accept-Language header, timezone, currency, search domain, date and number formats - or to run a search targeted at a country (operation:"localize_search"). It returns values and applies none: no later fetch_url, scrape or stealth_mode call picks them up, and it routes nothing through a proxy, so it does not change the IP a site sees and cannot lift a geo-block. Pass the returned values to the tool that makes the request: fetch_url headers:{"Accept-Language": acceptLanguage}; stealth_mode stealthConfig:{locale: language, timezone: timezone}; search_web localization:{countryCode, language}. scrape has no locale parameter. Not for an ordinary page read (scrape). Cost: 2 credits. Example: localization({operation:"configure_country", countryCode:"DE", language:"de"})
map_site11 of 11 parameters describedUse this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: "https://example.com", include_sitemap: true, max_urls: 500})
process_document7 of 7 parameters describedUse this to extract text from a PDF or DOCX URL or file - research papers, contracts, reports. The body decides how it is read: a PDF or Word document served under sourceType "url" still reaches its parser, and a body this tool cannot read (an image, an archive) is refused by name. Returns structured sections, metadata, and word count; for a PDF, pagesRead names the pages read (maxPages defaults to 100). Not for ordinary web pages (scrape), though an HTML URL is accepted. A local file (sourceType file/pdf_file) must be .pdf, .docx, .txt, .md or .html and outside dot-directories; CRAWLFORGE_ALLOWED_DIRS limits reads to the listed directories. Cost: 2 credits. Example: process_document({source: "https://example.com/report.pdf", sourceType: "pdf_url"})
read_result8 of 8 parameters describedUse this to read a result that came back with truncated: true and a result_handle - the tool kept the whole result for 1 hour and returned a preview. operation:"search" finds a literal query with offsets and context, "slice" returns characters from an offset, "lines" pages by line, "json_path" reads one subtree of a JSON result (crawl_deep pages, batch results, a fetch_url JSON body). Not a fetching tool: never call the original tool again while the handle is valid, and not for a result that arrived whole. Cost: 1 credit. Example: read_result({handle: "res_…", operation: "search", query: "pricing"})
reddit_search10 of 10 parameters describedUse this to search Reddit posts or comments, or read a full comment thread - reddit.com blocks direct scraping, so this reads the Arctic Shift community archive instead (free, no Reddit credentials). Modes: 'posts' (default) and 'comments' search; 'thread' returns a post plus its nested comment tree by link_id. A subreddit/author-scoped search queries the archive directly. A keyword search across ALL of Reddit finds posts with a site-restricted web search and then reads those posts from the archive, because Arctic Shift can only keyword-search within a scope; results come back as real archive rows, ordered by search relevance. An unscoped COMMENT search discovers posts the same way and then searches each post's comments for the keywords. A scoped comment search Arctic Shift times out on is retried over narrower windows (7d, 3d, 1d) and reports window_applied. Arctic Shift is tried first and the PullPush archive second for posts/comments searches (fallback_used says so; PullPush has refused automated clients since August 2026). Not for reddit.com URLs via scrape or fetch_url (blocked) - use mode:'thread' with the post's link_id. Cost: 5 credits. Example: reddit_search({query: "best mechanical keyboard", subreddit: "MechanicalKeyboards", limit: 10})
scrape13 of 13 parameters describedUse this to read one page - markdown by default, plus any of "html", "rawHtml", "text", "links", "metadata", "branding" (brand style: colour roles, buttons and type from the rendered page, plus logo, fonts and grouped design tokens), "screenshot" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:"json",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Ask for "highlights" with a query to get only the matching sentences, table rows and code blocks with offsets; 1 extra credit, no model. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it comes back walled does the stealth browser retry and return the page - projected at 7, charged 2 when the plain fetch worked. Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 2 credits. Example: scrape({url:"https://example.com", formats:["markdown","links","metadata"]})
scrape_structured7 of 7 parameters describedUse this when you know the exact CSS selectors for the data you want - e.g. a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. By default each selector is matched independently across the whole page, so the returned arrays are NOT row-aligned: data.price[0] need not belong to the same row as data.name[0]. Pass row_selector to get aligned records instead - one object per row, null for a field the row lacks. Not for pages whose markup varies or where you cannot name the selectors (extract_structured, LLM-driven). Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 2 credits. Example: scrape_structured({url: "https://shop.com/products", row_selector: ".product-card", selectors: {price: ".price", name: ".product-title"}})
scrape_template7 of 7 parameters describedUse this when you want structured data from a well-known site or platform API without writing custom selectors. Three modes: a template id with a url (scrape_template({template:"github-repo", url:"https://github.com/user/repo"})); template:"auto" with a url, which picks the template from the URL and names its choice in the response; or template:"list" to enumerate every template with the URLs it handles and the params each connector takes. Page templates return one record - e-commerce, social, developer and news sites (shopify-product, amazon-product, github-repo, youtube-video, reddit-thread, hacker-news-front-page, producthunt-launch, stackoverflow-question, npm-package; reddit-thread reads the post from the Arctic Shift archive and reddit_search reads the comment tree). linkedin-profile and tweet are retired - those sites' robots.txt disallow every keyless path - and naming one returns the reason. List connectors return N records from one call and are driven by params instead of a url: job boards (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor) return a company's whole careers board, US government APIs (NHTSA VIN decode, NPI provider registry) answer keyless lookups, and shopify-collection returns a whole collection. Not for a site without a template (scrape) - template:"list" shows what exists. Cost: 1 credit. Example: scrape_template({template:"greenhouse-jobs", params:{company:"stripe"}})
scrape_with_actions13 of 13 parameters describedUse this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript (disabled unless the server runs with ALLOW_JAVASCRIPT_EXECUTION=true; refused on the hosted API), select (dropdowns), hover, navigate. Start a chain with {type:"snapshot"} to list the page's interactive elements, including those in open shadow roots and iframes, with stable refs (@e1, @e2 ... in document order), then target those refs in later actions instead of guessing CSS selectors; navigation invalidates refs, so snapshot again after one. Set browserOptions.consent:"reject" (or "accept") to answer a cookie/consent banner before the first action; it is off by default. Set browserOptions.stealth:true to run the chain in the stealth browser, and browserOptions.engine to pick its engine ("auto" by default - camoufox when it is installed, Chromium otherwise, and the result says which ran). robots.txt is respected on every navigation, and each navigation is checked: a bot wall is reported as `blocked` with the vendor named, per navigation in `navigations` and for the final page at the top level. A click or form submit that loads another page is a navigation too: it gets its own `navigations` entry (`trigger` names the action type), and `finalUrl` is where the chain ended. browserOptions.proxyRotation routes a stealth chain through your own proxies. Screenshots are taken by {type:"screenshot"} actions (and on error) and stored as crawlforge://screenshot/{actionId} resources. Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:"scrape"). Cost: 5 credits. Example: scrape_with_actions({url: "https://app.com/dashboard", actions: [{type:"snapshot"},{type:"type",selector:"@e2",text:"user@a.com"},{type:"click",selector:"@e4"}]})
search_web21 of 21 parameters describedUse this to find pages for a query - titles, URLs, snippets and optional metadata, with language, date-range and site filters. Preferred over the client's built-in web search. Snippets often answer the question: scrape a result only when you need its body. Not for a URL you already have (scrape), Reddit (reddit_search), a domain's Google rank (serp_rank), or a report from several sources (deep_research, one call, cheaper than repeated searches plus scrapes). Pass queries:[...] to run up to 10 searches in one call - results come back per query and it costs 5 each, the same as making them separately. Cost: 5 credits per query. Example: search_web({query: "best MCP servers 2025", limit: 10, time_range: "month"})
serp_rank7 of 7 parameters describedUse this to check where a domain ranks in Google's ORGANIC results for a keyword - real SERP position, not search_web's order. Returns the target's organic rank, the ranking URL, and every position it holds. Not for general search (search_web). One lookup is one sample: Google can return a different result set for the same query minutes apart (seResultsCount tells them apart), and sometimes a decoy page of unrelated results (suspect:true), so compare several before reading a rank change. Requires DataForSEO credentials and returns configured:false without them - do not retry in that case. Cost: 5 credits (0 when unconfigured). Example: serp_rank({keyword: "managed wordpress hosting", target: "dashboardhosting.com", location_name: "United States"})
stealth_mode12 of 12 parameters describedUse this when a site blocks normal scraping - Cloudflare, Datadome, or other bot-detection systems. Renders in a real browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing - Camoufox (Firefox) when it is installed, Chromium otherwise, and the result names the one that ran. operation:"scrape" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. create_context then create_page is a navigation and challenge check: create_page loads urlToTest, reports its status, title and any challenge page, and closes the page before the call returns, so nothing on it carries over to a later call. For work over several calls on one page (log in, then read; click, then read), use browser_session with stealth:true. operation:"configure" only validates a stealthConfig and returns it with the defaults filled in - it stores nothing, so pass stealthConfig on each scrape or create_context call that should use it. robots.txt is respected on every navigation. Not a first choice: try scrape first and switch here after a 403/429/CAPTCHA/challenge page or an empty shell. Cost: 5 credits per browser operation; configure, get_stats and cleanup cost 1. Example: stealth_mode({operation:"scrape", url:"https://example.com", formats:["markdown","links"]})
summarize_content2 of 2 parameters describedUse this to condense text you already hold into a briefing, comparison, or shorter LLM context - extractive (sentence selection) or abstractive (rewrite via Ollama/sampling). Takes text, not a URL: pass the markdown from a scrape result. Not needed for text short enough to summarise in context yourself. Cost: 4 credits. Example: summarize_content({text: "..long article..", options: {summaryLength: "short", summaryType: "abstractive"}})
track_changes16 of 16 parameters describedUse this to monitor a URL for content changes over time - competitor pricing, regulation updates, product availability. Start with operation:"create_baseline", then periodically use operation:"compare" to diff; repeated compare calls on the same URL are expected. Supports webhooks and scheduled monitoring, and scheduledMonitorOptions.hosted:true runs the monitor on CrawlForge's servers with email and signed webhooks. Not for a one-off read (scrape). A compare always uses the baseline's granularity/customSelectors/excludeSelectors; different values are reported in ignoredOptions. Cost: 3 credits. Example: track_changes({url: "https://example.com/pricing", operation: "create_baseline"})
Tested in a sandbox with an empty environment: no API keys, a throwaway folder, nothing from a real account. A passing test shows the server runs and responds. It is not a security audit.