When it comes to web crawlers there are 3 main solutions when you get right down to it: Xenu; Screaming Frog; and IIS SEO Toolkit. For our purposes here today, I will assume that you agree with me (and not this Moz article comparing Xenu and ScreamingFrog) that Xenu just doesn’t make the grade in comparison to the other two. If you feel different, please let me know why I’m terribly and pathetically wrong, but in my opinion Screaming Frog and IIS simply offer much more than Xenu. So in my mind the toss up between crawlers comes between to Screaming Frog and IIS.

The biggest difference right off is that if you’re already running Windows Server, the IIS SEO Toolkit is free and Screaming Frog is not (although it is a very reasonable annual subscription. With a little creativity, you can also crawl larger sites in their entirety if your willing to set up crawls for all subdirectories, be patient, and marry a whole lot of reports into one). My mom always encouraged my two brothers and me to be satisfied with the free option as opposed to the alternative (pickup ball versus rec and water at every restaurant), or free not being an option, the less expensive (I don’t know how long it took us to experience a name brand soda). But my mom was never an SEO.

So regarding expense the Frog starts with a delicious leg up.

Let’s take a look at the functionality.

For the following items, we’re assuming that the crawler checks for all possible problems. For example, issues related to the title tag could be: missing; too long; too short; duplicate; multiple. It would be near impossible to list every piece of data each tool offers. Below focuses on many of the core aspects an SEO is likely to find valuable but is really a sample of what each tool is capable of.

Passes are generally accurate passes, although it’s important to note that subtle differences still exist. As an example, IIS reports pages where the branding occurs at the beginning a title tag or description as opposed to the end, and while this information can be found by viewing all page titles on Screaming Frog, it’s not readily available as an automated or stand alone violation.

	Screaming Frog	IIS
Title Tag
Meta Description
H1
H2
ALT Text
Inlinks
Outlinks
Recommendations
Subdirectory Level
Capitalization in URL
Canonical Tag
Unnecessary Redirects
Excessive JS/CSS
Broken Links
Server Status
Multiple Tag Formats
Google Analytics Integration
Google Search Console Integration
Slow Pages/Page Elements
Directive (ex.: noindex)
Valid Markup
Indentical Tags (ex.: title and description)
Check/Scan Customization

So each crawler clearly has its advantages. One of the coolest functionalities of these tools is the ability to customize what is being checked for. Screaming offers custom reporting where you can define text to check for via 10 filters and in addition, a very handy extraction setting. IIS also allows custom checks and while the possibilities are exhaustive, you can’t define whatever you want.

Both are fast, both are thorough.

What ultimately sways me is the UX (user experience). I find IIS to be much more accessible. While much of the same data exists in both, I’m able to navigate better and get different perspectives on whatever issue I’m interested in at that moment. Creating custom reports is also much more seamless.

ScreamingFrog

IIS SEO Toolkit

Conclusion

When all is said and done, I use both crawlers for any site audit or investigation I’m working on and I recommend you do the same. But if I had to pick one and be stranded on a deserted island with internet access, I’d have to go with IIS.

Sorry Screaming Frog – I still love you…

…Just not exclusively…

…Or as much…

….But I’d still like to be friends.