Skip to content
English - United Kingdom
Sign in
Contact us
  • There are no suggestions because the search field is empty.

Headless Screaming Frog on the Hostinger Ubuntu VPS

Stand up the BTT Linux VPS that powers team-wide SF crawls through Claude.

Who this is for

Chris (primary) or anyone helping stand up or maintain the Linux Screaming Frog VPS. Time needed: approximately 90 minutes end-to-end. After setup, the team runs SF crawls from inside Claude without touching the VPS directly.

Prerequisites

  • A freshly-provisioned Hostinger Linux KVM 2 VPS (Ubuntu 24.04 LTS)
  • A Cloudflare account with bristechtonic.co.uk already on Cloudflare nameservers
  • Admin access to BrisTechTonic Google Workspace
  • Your Screaming Frog licence.txt file

What this gets you

  1. Team Claude usage: AMs ask Claude to run an SF audit -- Claude calls sf_run_crawl -- the VPS runs the crawl headlessly -- results come back into the conversation.
  2. Direct SSH: ssh sf-vps from your Mac.
  3. Locked down: No public SSH port. Three security layers: Cloudflare Access (Google Workspace SSO), Cloudflare Tunnel (outbound-only), SSH key authentication.

Architecture overview

AM asks Claude
  Claude calls sf_run_crawl (ops.bristechtonic.co.uk/api/sf-mcp)
  lib/sf-vps.js attaches auth headers and POSTs
  sf-api.bristechtonic.co.uk (Cloudflare Tunnel)
  Node service on Hostinger VPS (localhost:4280)
  screamingfrogseospider --headless
  results in /var/sf-runs/{job_id}/

Stage 1 - Provision the VPS

Hostinger > VPS Hosting > KVM 2 (2 vCPU / 8 GB / 100 GB) > Ubuntu 24.04 LTS > London region. Enable the malware scanner, skip Docker manager. Set a strong root password and note the public IPv4.

Stage 2 - Harden the box (5 minutes)

SSH in as root (ssh root@IP), then run the hardening block: create a chris user with your SSH public key, disable root login and password auth, install fail2ban, ufw, curl, wget, openjdk-21-jre-headless, enable UFW with OpenSSH allowed, enable unattended security upgrades. Test from a new terminal -- key-only connection should succeed.

Stage 3 - Install Screaming Frog headless (10 minutes)

  1. Download and install the latest .deb from screamingfrog.co.uk/seo-spider/
  2. Copy your licence.txt to ~/.ScreamingFrogSEOSpider/licence.txt via scp from your Mac.
  3. Set JVM heap: echo "-Xmx6g" > ~/.ScreamingFrogSEOSpider/.spider-memory
  4. Smoke test: screamingfrogseospider --headless --crawl https://bristechtonic.co.uk --output-folder ~/sf-test

Stage 4 - Cloudflare Tunnel

Wire two services through one tunnel: ssh.bristechtonic.co.uk (SSH) and sf-api.bristechtonic.co.uk (SF API).

  1. Install cloudflared on the VPS via the Cloudflare apt repo.
  2. Run cloudflared tunnel login and authorise in browser.
  3. Create the tunnel: cloudflared tunnel create btt-sf-vps
  4. Create /etc/cloudflared/config.yml pointing the two hostnames at ssh://localhost:22 and http://localhost:4280.
  5. Route DNS for both hostnames.
  6. Install as systemd service and enable.
  7. In Cloudflare Zero Trust, add Google Workspace as identity provider. Create an Access Application for ssh.bristechtonic.co.uk (SSO, @bristechtonic.co.uk only) and one for sf-api.bristechtonic.co.uk (service token for Ops Hub).
  8. Add SSH config on your Mac with ProxyCommand cloudflared access ssh. Install cloudflared via Homebrew.
  9. Once SSH-over-Cloudflare works, close the public SSH port: sudo ufw delete allow OpenSSH

Stage 5 - Install the Node SF API service (15 minutes)

Clone tools/sf-vps-service/ to /opt/sf-vps-service via sparse checkout from the btt-ops-hub repo. Run npm install. Create /opt/sf-vps-service/.env with SF_VPS_BEARER_TOKEN (generate with openssl rand -hex 32), SF_BIN, SF_RUNS_DIR, and PORT=4280. Wire as a systemd service (sf-api.service). Smoke test: curl https://sf-api.bristechtonic.co.uk/healthz with the service-token headers should return {ok:true}.

Stage 6 - Wire Ops Hub

Set four env vars in Vercel for btt-ops-hub: SF_VPS_URL, SF_VPS_BEARER_TOKEN, CF_ACCESS_CLIENT_ID, CF_ACCESS_CLIENT_SECRET. Redeploy. Test by asking Claude to run an SF audit.

Stage 6.5 - Per-client SF config profiles

For large or quirky client sites, upload a saved .seospiderconfig via the Ops Hub client detail page (SEO tab > SF Configuration card). The file is stored in Supabase Storage and mirrored to /opt/sf-configs/ on the VPS. The clients.sf_default_config_name column records which profile applies. The sf_run_crawl tool also accepts an explicit config_name parameter for per-run overrides.

Stage 7 - Ongoing operations

  • View logs: tail -f /var/log/sf-api.log
  • Restart service: sudo systemctl restart sf-api
  • Restart tunnel: sudo systemctl restart cloudflared
  • Update SF: download newer .deb and sudo dpkg -i
  • Rotate bearer token: update .env on the VPS AND SF_VPS_BEARER_TOKEN in Vercel simultaneously

Common issues