SF Config Builder

The SF Config Builder is a powerful tool designed for inspecting, comparing, and programmatically modifying Screaming Frog SEO Spider .seospiderconfig files. Utilizing Screaming Frog's own classes ensures that the configurations remain compatible across different versions of the software.

Key Features

This tool provides several commands to enhance configuration management:

Usage Examples

Inspecting a Config

To view a configuration in a readable JSON format, the following command can be executed:

./scripts/run.sh --inspect --config /path/to/your.seospiderconfig

Comparing Configs

To compare two configuration files and see what has changed, the command is:

./scripts/run.sh --diff --config-a default.seospiderconfig --config-b modified.seospiderconfig

Building a New Config

Creating or altering a config can be accomplished with:

./scripts/run.sh --build \
  --template default.seospiderconfig \
  --output my-crawl.seospiderconfig \
  --patches '{"mCrawlConfig.mMaxUrls": 10000, "mCrawlConfig.mRobotsTxtMode": "IGNORE"}'

Testing Selectors

To ensure your selectors are functioning correctly against a live URL, use:

./scripts/run.sh --test-extraction \
  --url "https://example.com/product" \
  --selector "//span[@class='price']" \
  --selector-type XPATH \
  --extract-mode TEXT

Editable Fields

The tool allows modification of several fields, including:

• Crawl settings such as mCrawlConfig.mMaxUrls, mCrawlConfig.mMaxDepth, etc.
• User agent settings like mUserAgentConfig.mUserAgent.
• Exclude patterns including mExcludeManager.mExcludePatterns.

Note: Field availability depends on the installed version of Screaming Frog. The tool ensures validation by returning appropriate errors if a field is not present.

Error Handling

Any errors encountered will return a JSON response to stdout, providing a clear indication of the issue:

{
  "success": false,
  "error": "Error message here",
  "errorType": "VALIDATION_ERROR | IO_ERROR | PARSE_ERROR | UNKNOWN",
  "details": {}
}

The tool also provides exit codes for quick reference on success and failure states.

For more information or to raise questions, reach out via Twitter to @skippermissions.