MCP-RQUEST: Bypass Anti-Bot, Convert PDF/HTML

What is MCP-RQUEST: Bypass Anti-Bot, Convert PDF/HTML?

MCP-RQUEST is an advanced HTTP request server engineered to empower large language models (LLMs) like Claude with human-like web interaction capabilities. Built on the rquest framework, it emulates real browser traffic using precise TLS/JA3/JA4 fingerprints and HTTP/2 headers, enabling seamless navigation of anti-bot protected sites. The toolchain also includes intelligent content conversion, transforming PDF and HTML files into structured Markdown for optimized LLM processing. This combination of anti-bot evasion and document parsing makes it ideal for automated web tasks requiring both stealth and semantic clarity.

How to Use MCP-RQUEST: Bypass Anti-Bot, Convert PDF/HTML?

Deployment starts with installing via uv/pip followed by configuration in your LLM environment:

• Installation: Use uvx mcp-rquest or pip install mcp-rquest
• Claude Configuration: Add the server definition to your mcpServers settings with proper execution commands
• Request Execution: Leverage REST endpoints like /api/http_get or use conversion tools like get_stored_response_with_markdown to process documents

Full instructions include optional development setup using virtual environments and marker library optimizations.

Key Features of MCP-RQUEST: Bypass Anti-Bot, Convert PDF/HTML?

This toolchain combines enterprise-grade capabilities:

• Bot Evasion: Supports Chrome, Firefox, Edge and OkHttp browser profiles with cryptographic fingerprint accuracy
• Format Agnosticism: Auto-detects PDF/HTML content and converts to Markdown using the Marker library for semantic preservation
• Enterprise Scalability: Handles large responses with token tracking and secure temporary storage
• Authentication Flexibility: Implements bearer tokens, basic auth, and custom header configurations
• Protocol Mastery: Implements all standard HTTP methods including TRACE and OPTIONS

Use Cases of MCP-RQUEST: Bypass Anti-Bot, Convert PDF/HTML?

Common application scenarios include:

• Content Extraction: Safely scrape e-commerce product pages protected by bot detection systems
• Document Processing: Convert PDF reports and HTML articles into machine-readable Markdown for NLP analysis
• API Testing: Test REST endpoints with full HTTP method coverage under realistic browser conditions
• Model Training: Generate synthetic training data by rendering dynamic web content through emulated browsers
• Error Recovery: Automatically restart stalled PDF conversion processes using built-in model management tools

FAQ from MCP-RQUEST: Bypass Anti-Bot, Convert PDF/HTML?

• Q: Does browser fingerprinting really bypass modern anti-bots?
  A: Yes, uses cryptographic TLS fingerprints and HTTP/2 behaviors indistinguishable from real browsers
• Q: How does PDF conversion maintain formatting?
  A: Leverages the Marker library's advanced layout analysis to preserve headings/tables
• Q: Can I customize user-agent strings?
  A: Fully configurable through header parameters, with pre-defined browser templates
• Q: What response size limits exist?
  A: Token counting handles large responses; stored securely in system temp directories
• Q: Does it work with cloud-based LLMs?
  A: Designed for local server setups communicating via MCP protocol standards

mcp-rquest

A Model Context Protocol (MCP) server that provides advanced HTTP request capabilities for Claude and other LLMs. Built on rquest, this server enables realistic browser emulation with accurate TLS/JA3/JA4 fingerprints, allowing models to interact with websites more naturally and bypass common anti-bot measures. It also supports converting PDF and HTML documents to Markdown for easier processing by LLMs.

Features

• Complete HTTP Methods : Support for GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, and TRACE
• Browser Fingerprinting : Accurate TLS, JA3/JA4, and HTTP/2 browser fingerprints
• Content Handling :
  • Automatic handling of large responses with token counting
  • HTML to Markdown conversion for better LLM processing
  • PDF to Markdown conversion using the Marker library
  • Secure storage of responses in system temporary directories
• Authentication Support : Basic, Bearer, and custom authentication methods
• Request Customization :
  • Headers, cookies, redirects
  • Form data, JSON payloads, multipart/form-data
  • Query parameters
• SSL Security : Uses BoringSSL for secure connections with realistic browser fingerprints

Available Tools

• HTTP Request Tools :

  • http_get - Perform GET requests with optional parameters
  • http_post - Submit data via POST requests
  • http_put - Update resources with PUT requests
  • http_delete - Remove resources with DELETE requests
  • http_patch - Partially update resources
  • http_head - Retrieve only headers from a resource
  • http_options - Retrieve options for a resource
  • http_trace - Diagnostic request tracing

• Response Handling Tools :

  • get_stored_response - Retrieve stored large responses, optionally by line range
  • get_stored_response_with_markdown - Convert HTML or PDF responses to Markdown format for better LLM processing
  • get_model_state - Get the current state of the PDF models loading process
  • restart_model_loading - Restart the PDF models loading process if it failed or got stuck

PDF Support

mcp-rquest now supports PDF to Markdown conversion, allowing you to download PDF files and convert them to Markdown format that's easy for LLMs to process:

1. Automatic PDF Detection : PDF files are automatically detected based on content type
2. Seamless Conversion : The same get_stored_response_with_markdown tool works for both HTML and PDF files
3. High-Quality Conversion : Uses the Marker library for accurate PDF to Markdown transformation
4. Optimized Performance : Models are pre-downloaded during package installation to avoid delays during request processing

Installation

Using uv (recommended)

When using uv no specific installation is needed. We will use uvx to directly run mcp-rquest.

Using pip

Alternatively you can install mcp-rquest via pip:

pip install mcp-rquest

After installation, you can run it as a script using:

python -m mcp_rquest

Configuration

Configure for Claude.app

Add to your Claude settings:

Using uvx:

{
  "mcpServers": {
    "http-rquest": {
      "command": "uvx",
      "args": ["mcp-rquest"]
    }
  }
}

Using pip:

{
  "mcpServers": {
    "http-rquest": {
      "command": "python",
      "args": ["-m", "mcp_rquest"]
    }
  }
}

Using pipx:

{
  "mcpServers": {
    "http-rquest": {
      "command": "pipx",
      "args": ["run", "mcp-rquest"]
    }
  }
}

Browser Emulation

mcp-rquest leverages rquest's powerful browser emulation capabilities to provide realistic browser fingerprints, which helps bypass bot detection and access content normally available only to standard browsers. Supported browser fingerprints include:

• Chrome (multiple versions)
• Firefox
• Safari (including iOS and iPad versions)
• Edge
• OkHttp

This ensures that requests sent through mcp-rquest appear as legitimate browser traffic rather than bot requests.

Development

Setting up a Development Environment

1. Clone the repository

2. Create a virtual environment using uv:

   uv venv

3. Activate the virtual environment:

   Unix/macOS

source .venv/bin/activate
# Windows
.venv\Scripts\activate

4. Install development dependencies:

   uv pip install -e ".[dev]"

Acknowledgements

• This project is built on top of rquest, which provides the advanced HTTP client with browser fingerprinting capabilities.
• rquest is based on a fork of reqwest.