How to Leverage In-Market & Out-of-Market SEO Tests to Prepare for Site Migrations

Most SEOs run tests — it’s how they operate. Test a change on a small portion of the site, measure the results, and if they’re positive, roll them out at scale. That methodology is less common when it comes to pre-launch migration testing though. 

Typically, in preparation for a migration, SEOs will crawl a staging site to check it against known best practices, making sure search engine bots can access what they need to access, and that all on-page elements are squared away. But what if we took the way we run tests on our live sites and applied it to site migrations? 

That’s exactly what Michael Brown, Senior SEO Manager at Jellyfish, proposes that we do.

I sat down to talk with him about what he refers to as “in-market” and “out-of-market” testing, and how leveraging both methods can help us better determine if a migration will go as planned (or better).

What are in-market & out-of-market SEO tests, and why are they important?

“The most common thing for SEOs to do before a site migration is set benchmarks or baseline reports, which would be used to measure impact later on,” said Michael. 

“We attempt to forecast the impact a site migration will have on performance by leveraging experiences from prior migrations. The only problem with that is there are typically too many things happening all at once in a migration to truly know how impactful your changes were.”

Conducting both in-market and out-of-market tests is a great solution to that problem, but those terms aren’t universally understood in the context of SEO, so what exactly do they mean?

According to Michael,

• In-market tests: Field testing changes on pages that are currently open to indexing and capable of ranking — tests on your live site. 
• Out-of-market tests: Running tests on staging environments that are not currently open to indexing and ranking.

Why refer to them this way?

“SEO is the practice of trying to earn real estate on the SERP — it’s like its own market, and like a market, it changes all the time. That means even though I may have run this type of test six months ago, I would still repeat the test because the market is constantly fluctuating.” 

Michael also added that “markets differ depending on the industry or vertical you’re performing the site migration on. The results of a controlled migration on an outdoor gear site differ from that of a health insurance site. This is another reason why I think of site migrations at the market-level.” 

With that in mind, let’s learn a bit more about what each testing method entails. 

Out-of-market SEO testing: Standard for pre-launch migration QA 

Out-of-market tests are the most common, according to Michael.

“Running tests on staging environments prior to launching them is traditional best practice for pre-launch testing and QA-ing to ensure your migration goes to plan.”

💡 Related resource: The Website Migration Checklist: How To Safeguard Your SEO While Updating Your Site provides a list of checks to follow before migrating your site. This is the out-of-market testing Michael is referring to.

But the limitation with this style of testing is that we’re left relying on SEO best practices and anecdotal information from past migrations that may no longer be relevant due to market changes. It also doesn’t typically help you make traffic and engagement projections. 

“Out-of-market testing isn’t as subjective as in-market testing and mostly technical,” he added, explaining why that method is so common amongst SEOs. 

This type of testing can help you prevent glaring technical errors that would likely harm your performance after launch, but in order to make more accurate predictions of post-launch performance, Michael thinks we should add in-market SEO testing to the mix. 

What does it look like to perform in-market testing for site migrations?

In-market testing is common, according to Michael, but not when it comes to preparing for site migrations. 

“SEOs are already testing at the task level, but even if you’ve done something at the task level before and are confident in it, it’s still a good idea to test before you apply something at scale, such as in a migration. Then, incorporate your learnings into your QA checklist for the migration.”

To learn more about what that could look like, I asked him for some examples.

“Say I’m comfortable with internal linking. I know that’s something that’ll change during the migration, so what kind of internal linking test could I roll out related to the migration? In other words, Find the areas you’re confident in and focus on doing an in-market pre-launch test you can learn from before you roll out the full migration. That would help you understand the impact of that specific activity but also helps you set expectations for your client or boss, like ‘Based on this test, I predict an uplift in traffic or revenue of X.’”

In-market testing is a bit more tricky than out-of-market testing, but according to Michael, “helps an SEO become a better strategist and can assist in predicting a more accurate growth forecast.”

How do we measure a site migration?

“In general,” Michael told me, “we forecast the value of a migration by looking at estimated audience growth and site engagement.”

However, if you’re running both in-market and out-of-market SEO testing, you’ll need to have different measurements for each:

• Out-of-market tests are measured by looking at how a site validates a change and if it meets the requirements of everything we know about crawlers.
• In-market tests are measured by changes in things like rankings, traffic, and engagement.

This makes in-market measurement a little more involved since “that means we have to keep things controlled so we understand exactly what action influenced the SERP change.” 

Michael added, “There are a ton of variables involved with in-market tests, so the fewer things you have going on, the less chance of disruption of the test and the better the outcome.”

Because anything you touch on the website could impact results, you’ll want to try and make one change at a time in order to get the most accurate gauge of how that change impacted KPIs like ranking and traffic. 

“Ideally, you’ll want to run your in-market tests independently, but you can also run concurrent tests as long as they don’t conflict.”

How do we prove it?

Once you know how to measure the impact of both in-market and out-of-market tests, it’s time to run some tests.

Michael gave me examples of priorities by SEO channel and how you could test for them.

Technical improvements

• Question: Are we improving a site’s ability to communicate with crawlers and bots?
• Most common testing approach: Validating redirects and page rendering 
• Example: We inventory all internal URLs to make sure that our response codes are positive and that our content can be rendered by a bot. 

On-page improvements

• Question: Is the information we are promoting improving our acquisition of audience and site engagement?
• Most common testing approach: Information architecture changes and content consolidation
• Example: We push sampled IA and content changes to the existing market (live) so that we can anticipate what kind of shifts we would see in SERPs when rolled out in scale

Off-page improvements

• Question: How much authority will I lose?
• Most common testing approach: Link decay through redirects or changes to a pages status code
• Example: We inventory backlinks and isolate a few authoritative links to test via redirects

To sum things up, the idea of pre-launch testing is to help validate that the changes you’re making to the site will turn out the way you anticipated. 

“There is a sea of unknown variables that influence our results as a channel,” Michael said. “The more we can do to reduce the number of unknowns, the closer we get to positively influencing results. It’ll also help SEOs become better strategists along the way.”