Skip to content
English - United Kingdom
Sign in
Contact us
  • There are no suggestions because the search field is empty.

The SF VPS and Cloudflare Tunnel Stack

The SF VPS stack that lets the whole team run professional crawls from Claude.

What is it?

A Hostinger Linux VPS running Screaming Frog headlessly, exposed to Ops Hub via a Cloudflare Tunnel. This gives the whole BTT team programmatic access to full SF crawls without anyone needing to own the SF licence, open the desktop app, or manage exports manually.

Architecture

The call chain from an AM's conversation to a finished crawl:

  1. AM asks Claude: Run an SF audit on getevenly.com.
  2. Claude calls the sf_run_crawl MCP tool at /api/sf-mcp (Ops Hub, Vercel).
  3. lib/sf-vps.js attaches two auth headers (Cloudflare service token + Bearer token) and POSTs to sf-api.bristechtonic.co.uk.
  4. Cloudflare Tunnel receives the request, validates both auth layers, and forwards it to http://localhost:4280 on the Hostinger VPS.
  5. The Node API service (tools/sf-vps-service/) spawns screamingfrogseospider --headless and writes results to /var/sf-runs/{job_id}/.
  6. Claude polls sf_get_status until the job completes, then calls sf_get_results_link to retrieve the CSV bundle.
  7. Every crawl is logged to the sf_audits Supabase table.

Security layers

The VPS has no public-facing ports. Three security factors stand between the internet and the box:

  1. Cloudflare Tunnel (outbound-only): the VPS initiates the connection to Cloudflare. No inbound ports are open.
  2. Cloudflare Access: two policies. ssh.bristechtonic.co.uk requires Google Workspace SSO (@bristechtonic.co.uk only). sf-api.bristechtonic.co.uk accepts the ops-hub-sf-mcp service token (machine-to-machine).
  3. Bearer token: even if the Cloudflare layer were bypassed, the Node service requires Authorization: Bearer $SF_VPS_BEARER_TOKEN on every request.

VPS specification

  • Hostinger KVM 2: 2 vCPU, 8 GB RAM, 100 GB SSD, London region
  • OS: Ubuntu 24.04 LTS
  • Java: OpenJDK 21 (SF requirement)
  • Node.js: v20 LTS
  • SF JVM heap: 6 GB (set to -Xmx6g)

Per-client configuration profiles

Large or quirky client sites can use a custom .seospiderconfig file:

  1. Configure the crawl in the SF GUI on your Mac and save as .seospiderconfig.
  2. Upload via the Ops Hub client detail page (SEO tab > SF Configuration card).
  3. The file is stored in Supabase Storage (canonical) and mirrored to /opt/sf-configs/ on the VPS.
  4. The clients.sf_default_config_name column records which profile applies to that client.
  5. When sf_run_crawl is called with a client_id, the config is auto-applied.

The sf_run_crawl tool also accepts an explicit config_name parameter to override the default for a single run.

Ops Hub integration points

  • /api/sf-mcp -- the MCP endpoint the team calls
  • lib/sf-vps.js -- the HTTP client (attaches auth, logs to sf_audits)
  • sf_audits table -- every crawl history, archived after 90 days via daily cron
  • sf_configs table -- per-client config file registry
  • Vercel env vars: SF_VPS_URL, SF_VPS_BEARER_TOKEN, CF_ACCESS_CLIENT_ID, CF_ACCESS_CLIENT_SECRET

Ongoing operations

  • SSH to the box via: ssh sf-vps (Cloudflare Access prompts Google login on first use)
  • View service logs: tail -f /var/log/sf-api.log
  • Restart after config change: sudo systemctl restart sf-api
  • Rotate bearer token: update /opt/sf-vps-service/.env on the VPS and SF_VPS_BEARER_TOKEN in Vercel simultaneously
  • Disk cleanup: the service auto-removes /var/sf-runs/{job_id}/ 24 hours after upload. Config files are never auto-deleted.