Settings Refactoring: Upgrading and reorganizing your project settings

28th June 2017Laura Scott

Botify is introducing a new crawl configuration option and reorganizing your project settings with settings refactoring. Starting tomorrow, you will see a new organization of your project settings that will allow you to easily configure a standard crawl or utilize advanced crawling options.

Also expect a new advanced crawling option, follow rules, that will be available to all users in the coming week. Follow rules allow users to configure their crawl to include AMP, Hreflang, and canonicals among other link types to give you even more options to customize each crawl.

Settings Refactoring

With settings refactoring, Botify is replacing the “Scope”, “Crawler” and “Sitemaps” tabs with “Main” and “Advanced Settings” in your project settings.

All mandatory crawl settings will be included in the “Main” tab. This will include settings like start URLs, allowed domains, Sitemaps and crawl parameters. The settings in the “main” tab will help you identify the proper scope for your standard crawls.

Advanced Settings

For the technically-savvy SEO, Botify is introducing advanced crawl settings within project settings. This section includes familiar crawl options like compare with previous analysis, respect no-follow rules, ignore canonical tags, and evaluate content quality.

You’ll also find robots.txt and rules selection options, custom HTTP headers, parameter options for URLs, and Advanced URL rewriting within the advanced settings tab.

New Advanced Crawl Settings

With settings refactoring, Botify is also introducing a new advanced crawl setting, follow rules, in the coming week. Follow rules allow users to configure a project to follow:

• AMP links
• No-follow links
• Canonical links
• Hreflang links
• Alternate media links

Botify then crawls the pages discovered through these links. This new crawl capability gives SEOs the ability to perform targeted crawls of specific content types.

Get Started

Get acquainted with Botify’s new project settings. Visit the advanced settings page next week to see what crawl insights you can find with new crawl configuration follow rules.