Check Destination URLs in Bulk Using Screaming Frog

YouTube Preview Image

If you ever find that Google Analytics tracking is showing up not set, or that your clicks to visits ratio is higher than 30% between Adwords and Analytics, it could mean that you have a destination URL redirect problem.  Web teams, without thinking, frequently go in and redirect pages where paid search traffic is being sent to without telling the paid search team.  Tracking gets stripped off, attribution of conversion does not get reported and it's very annoying.

The hard part is figuring out which URLs redirect and which don't.  If you have a few URLs in your paid search account, this is easy.  When you have a lot of URLs to check, this becomes the most boring and monotonous task.

In Excel, you could use a formula like =hyperlink(A1) and click the URLs one-by-one.

If you need to check many destination URLs on both the keyword and ad-level, use Screaming Frog.

I've used Screaming Frog software for over a year now and it's been a huge time-saver.  The tool provides you with a lot of SEO data regarding your site in a nice table-friendly format that can be exported to Excel.  The paid version actually has the ability to check for specific text on a page.  Think about a website with hundreds or thousands of pages and you need to know the "Stock Status" but you have a lousy team that is not technically savvy enough to run such a report from their database.  You can use this tool to crawl pages you send paid search traffic to and it will check if the product is 'in stock' or 'out of stock'.  Pretty awesome!

I primarily use Screaming Frog for checking paid search destination URLs using the List mode feature.

Here are the steps:

  1. In Adwords Editor, download most recent changes.
  2. Export your account as .CSV
  3. Delete all columns except 'Destination URL' (this column contains keyword and ad-level destination URLs)
  4. In Excel, on the Data tab, use the Remove Duplicates feature
  5. Copy your list of URLs and save them in a .TXT file
  6. In Screaming Frog, under 'Mode' in the top menu, select 'List' to put Screaming Frog into list crawling mode
  7. Select your file, hit Start
  8. When the crawler has finished, go to the 'Response Codes' tab and click 'Export'
  9. Open your 'Response Codes' exported list and sort by response code.

Learn this!  This is paid search 101 for managing large paid search accounts.

Build PPC Campaigns Faster With

PPC Campaign Generator

Build PPC campaigns in 5 simple steps!
Simple enough for beginners, powerful enough for advanced users
Training videos and support
FREE stand-alone version of my Keyword Grouper software with purchase (Limited-time!)

Tags: ,

2 Responses to “Check Destination URLs in Bulk Using Screaming Frog”

  1. Tracy February 19, 2013 at 5:31 am #

    Thanks for these great tips on using screaming frog. Really useful for large ecommerce, recruitment, real estate - and loads of others I'm sure.

    It would be useful for your article to specify if there is a limit on the URL list size (paid or free version).

    • Jarad Collier February 19, 2013 at 3:14 pm #

      There is a 500 URI crawl limit. I think that means if you want to crawl more than 500 URLs in one shot, you'd have to buy a license. I mentioned this in the video but not in the article. Thanks for the feedback.

Leave a Reply