Website Auditor Tools: Compare the 10 Greatest Hits

SEO

Website Auditor Tools: Compare the 10 Greatest Hits

Ever since I realized the importance of optimizing every element on every page on the Web, website auditor tools have been in demand. These tools “crawl” your website one page at a time by following links on each page and/or by following a sitemap. Data collected during the crawl is then delivered in some form of a report, and it is at this point where the tools differ.

Some website auditor tools are focused on page load speeds and user experience optimization while others may be more interested in how deep some pages are located within your site. I’m not really interested in these.

It should also be noted that I am by definition a lazy person. The idea of performing keyboard gymnastics with multiple spreadsheets open does not appeal to me in the least. Instead, I prefer ways to gather as much relevant data as possible and have it turned into understandable and useful information from which I can spot trends, be alerted to potential problems and take action.

For the purpose of this review, I was interested in seeing results useful for search engine optimization (SEO). In short, I am looking for information about on-site issues with page titles and meta content, anything I may have goofed-up that prevents search engines from finding site pages and how my page content is helping or hurting my effort.

Even with those specific goals in mind, results from the tools I looked at varied widely in the level of detail and information included in their crawl reports. Expert tools provide gobs of information down to the smallest levels of detail.

Tools better suited for non-experts may provide less detail but include guidance to potential problems, further explanations and even actionable recommendations. And, of course, some sit squarely in the middle providing both a ton of detail while pointing out potential problems and providing recommendations.

Included in this review are both desktop and Web-based tools. Some are only months old and one has its roots in the 1990s. No bias here.

Desktop programs

  • Screaming Frog SEO Spider v2.3
  • Beam Us Up v1.0.0.0
  • Xenu’s Link Sleuth v1.3

Web-based

  • Google (Webmaster Tools & Analytics)
  • Raven Tools Site Auditor
  • WooRank
  • DeepCrawl
  • Moz Pro SEO Web Crawler
  • Alexa’s Site Auditor
  • Site Condor

For the most part, all of the above are geared to accomplish my goal of finding what’s helping and not helping my websites’ SEO while catering to my lazy person preferences.

Should you pick a desktop program over a web-based site auditor tool? That depends on your needs. The desktop tools reviewed here all provide detailed information, but you need to know what it is you’re looking for in order to find it.

For years, SEO experts have been living inside spreadsheets, compiling and massaging tons of data. So for them, this is familiar territory and the desktop programs tend to cater to that familiarity. Desktop programs are also nimble, unencumbered by delays of hours or days before providing crawl results or imposing maximum page count limitations (this is limited more by your computer’s memory size in most cases). But, they do require installing on a computer and maintenance when updates are released.

Web-based tools are often preferred because they don’t require you to install or update software (which also makes them OS independent) and can be accessed from anywhere you have Internet access. But, they also need to be compensated for the bandwidth, at least used during the crawls.

Below are my assessments of 10 website auditor tools worth your consideration. The “Cost” shown represents the cost of the service level I utilized for my use of the tool. Most of these companies offer higher levels of functionality, support and crawling capacity than I needed for this review.

Google Webmaster Tools

  • Advantages – Google bot is always crawling making nearly immediate results always available.
  • Disadvantages – No direct control over when data will be updated.
  • Cost – Free
  • Platform – Web-based
  • Recommended for – Experts

Website Auditor Tool - Google

You may be wondering why (or even how) Google came to be listed among these more specialized tools. When asking around for suggestions and finding out about what website auditor tools others were using, Google kept coming up. This was a surprise to me at first, but after considering all the enhancements and upgrades made to Webmaster Tools and Analytics over the past couple years, it is understandable. And getting your information from “the horse’s mouth” isn’t necessarily a bad thing.

If you haven’t looked beyond your crawl errors and site messages in Google Webmaster Tools (GWT) in a while, you should take the time to drill down through the new menus to see what’s available.

Within Google Webmaster Tools (GWT) you now can locate (menu paths provided in parentheses):

  • Meta Tag Problems (Search Appearance > HTML Improvements)
  • Non-Indexable Content (Search Appearance > HTML Improvements)
  • Microformat Usage (Search Appearance > Structured Data)
  • Backlinks (Search Traffic > Links to Your Site)
  • Internal Links by Count (Search Traffic > Internal Links)
  • Pages Indexed (Google Index > Index Status)
  • Significant Keywords (Google Index > Content Keywords)
  • Crawl Errors (Crawl > Crawl Errors)
  • Pages Crawled (Crawl > Crawl Stats)
  • Blocked Pages (Crawl > Blocked URLs)
  • Page Load Time (Other Resources > Page Insights)

That’s a pretty substantial depository of information that is already being collected without much effort on your part and for free.

Raven Tools Site Auditor

  • Advantages – Comprehensive detailed results are combined with easy to understand problem identification and actionable recommendations; can compare crawl results against each other
  • Disadvantages – Can take a while to get crawl results
  • Cost – $99/month
  • Platform – Web-based
  • Recommended for – Both novices and experts

(Honesty declaration: Raven Tools has been my choice for SEO research and monitoring resources for the past couple years. There are likely to be additional insights not included in coverage of other tools in this review.)

Website Auditor Tool - Raven

Raven Tools provides a collection of tools and services for search and site optimization, which goes beyond their site auditor. But this doesn’t mean the site auditor capabilities are diminished. I found quite the opposite. And, while the “beta” label is still firmly in place, the site auditor certainly doesn’t feel or perform like it.

Crawls can be scheduled for weekly and monthly intervals or you can start them manually whenever you want. The number of pages crawled can be capped at 50 to 1000 per crawl; there is a 10000 page per day limit on all crawls.

A crawl summary page is provided and clickable tabs, in order of importance, shows any potential problems detected in each category. Categories are Visibility, Meta, Content, Links, Images and Semantics (i.e. Hx tags). Also provided is a page speed analysis and crawl comparisons. Within each tab, a list of evaluated items is provided identifying potential problems.

Each potential problem area gets its own summary showing a count of flagged pages, a visual indicator of importance, a general description of what you should be concerned with on the flagged pages and a “Learn more” link that will take you to a page in the Google Webmaster Tools help library specific to the topic.

Clicking on the title of the potential problem’s description (e.g. “n internal links are missing anchor or ALT text”) takes you to the detailed page-by-page listing of the site’s flagged pages. The list can be exported to a CSV file or PDF report.

Raven Tools is an example of a site auditor that provides the details experts need as well as does a lot of handholding with descriptions and recommendations for novices (i.e. soon-to-be-experts).

WooRank

  • Advantages – No delay in seeing results; simple to use and easy-to-read results
  • Disadvantages – Some assessments are made based on just one page of the site
  • Cost – $49/month
  • Platform – Web-based
  • Recommended for – Novices

Website Auditor Tool - Woorank

WooRank is one of the easiest site auditor tools available. Enter your site URL and hit enter. Immediately, you will see the dials start spinning and gauges start moving as an overall site score is being calculated.

Unlike other site auditor tools, WooRank skips past all those nasty lists of flagged pages and jumps right to recommendations of what it sees as needing fixing. While this may leave experts uneasy about trusting these findings without having some detailed evidence to drill into, novices may find this comforting and less confusing.

A summary is provided at the top of the results page that shows a score for three areas of concern. They are Optimize, Convert and Promote. Further down the page are “Top Priorities” and a link to the “Marketing Plan.” This is a detailed task list with some items possibly already checked off as completed and others awaiting attention based on what was detected during the site crawl.

Many of the SEO-related measures include the phrase “for this Web page” in their description meaning that only one page was crawled to come up with this finding. This means crucial items, such as broken links, are not being detected on pages beyond the front page.

In the “Marketing Plan” segment, WooRank recommends using other tools like Google Webmaster Tools to be aware of such crucial SEO details. While able to provide quick site and server information, take the SEO recommendations with a grain of salt. You are going to need to use another tool from this list to identify exactly which pages need correcting.

DeepCrawl

  • Advantages – Fully configurable for many different crawl settings.
  • Disadvantages – May take a few crawls to become adept at using the correct settings.
  • Cost – $80/month for 5 crawls and 100K pages crawled per month
  • Platform – Web-based
  • Recommended for – Experts

Website Auditor Tool - DeepCrawl

DeepCrawl is an exciting new entrant to the website audit space. It has a very flexible crawler. For instance, you are able to select a crawl type, which will strictly follow a sitemap, a list of backlink sources, a list of site pages and links found starting from a given page or some combination of these. You also are able to link your Google Analytics account to your DeepCrawl account so that it can find pages which may not be in your sitemap or are linked from other pages on your site but which are still receiving traffic.

For this review I used a predefined “Universal” crawl that incorporated my sitemap, Google Analytics data (to find any indexed orphan pages) and links found on all pages crawled to determine what pages would be crawled.

I could have included or excluded pages based on a myriad of factors including respecting follow/nofollow links, specific URL constructs (both negative exclusion and positive inclusion options). I could also have created several rewrite rules to remove or add elements to select URLs.

The tool, rather than set the correct maximum length tolerance for my titles and meta descriptions for me, allowed me to adjust these to fit my preferences. Similar customization was available for many issues such as maximum links per page, minimum content, maximum URL length, maximum load time and more. This is very useful in accommodating recent changes, such as Google changing its SERP layouts or when testing against other listing locations’ display constraints.

I got a lot more “hits” for standard measures than I did with other site auditors in this review. For most, I could reconfigure the crawler better for future crawls. Knowing to set it to “ignore” the tag and category pages of my WordPress site in the future and URLs containing YYYY/MM/DD URL constructs (something I moved away from but only sometime after launching the site) would go a long way toward dialing back many “false positives.”

I’m still not sure why it flagged every page as “Missing Title” but maybe there was something I goofed in the crawl settings to account for that.

Screaming Frog Site Crawler

  • Advantages – Intuitive interface. Lots of crawl and tolerance settings.
  • Disadvantages – Assumes you know what you are looking for when assessing the results.
  • Cost – $165/year
  • Platform – Desktop application
  • Recommended for – Experts

Website Auditor Tool - Screaming Frog

This website auditor tool has been a favorite among experts for a while. Since it is a desktop program, you are able to run crawls however and whenever you want and see results as quick as the pages can be crawled.

Crawl settings offer full control over what pages are included or excluded from those crawled. There are also user-configurable tolerance values for maximum title and description character counts and pixel lengths. Since you are not limited to any set number of crawls per month, you can practice with the wide array of crawl settings and become adept rather quickly.

The results are divided up across several tabs. They are Internal, External, Response Codes, URI, Page Titles, Meta Description, Meta Keywords, H1, H2, Images, Directives and Custom. A pull down menu is available in each tab that shows the different segments of pages flagged making it easy to quickly scan results in the absence of a summary view (although there is a crawl overview available in XLSX or CSV format).

Clicking on a page brings up that page’s details in a lower window. Tabs in that window provide details about URL info, In Links, Out Links, Image Info and SERP Snippet. That last tab provides a mockup of how that page will appear in SERPs and also provides character and pixel lengths.

Beam Us Up

  • Advantages – No installation required. No limits on pages crawled.
  • Disadvantages – Requires installing Java on your computer.
  • Cost – Free
  • Platform – Desktop application
  • Recommended for – Experts (may be good for novices in the future)

Website Auditor Tool - Beamusup

This is a new contender in the desktop application crawler space. Because it’s a program you run on your computer, you can run crawls whenever you want. A recent product update enables an unlimited numbers of pages can be crawled.

Since this is a Java app, there’s nothing to install (besides Java). Download their app to your computer and double-click it to run. That’s all there is to running it on your computer.

There are only a few custom crawl parameters to set and, once you’ve entered what URL to begin crawling from, simply click “Start” and it’s off and running.

As with the other desktop applications reviewed here, there’s a primary display window at the top which shows a list of pages in detail and another window beneath it with a single page’s details. The pages you see in the top window are controlled by clicking in the right-hand sidebar window that lists all the different types of potential problems detected during the crawl.

Below that sidebar window is another that appears to have the intention of providing more detailed explanation of the error type. Unfortunately, it currently only shows a broken link to “See more details.” But remember, this is a newly launched tool that continues to get updates so, over time, this feature is likely to get built.

For the purpose of this review, it is difficult to know the complete list of issues Beam Us Up will identify since I could only see those that were detected during a crawl.
This tool should be considered “one to watch” as they’re on the right track to provide novices with a desktop option. But, I hope they can move away from the Java requirement. I installed Java only to be able to review this tool but will be uninstalling it right away for security reasons. Me paranoid? Could be.

Moz’s Site Auditor

  • Advantages – Comes with lots of other tools and resources
  • Disadvantages – Limited to weekly crawls
  • Cost – $99
  • Platform – Web-based
  • Recommended for – Both novice and expert

Website Auditor Tool - Moz

Moz (previously known as SEO Moz) has been a deservedly respected go-to resource for tools and education for all things SEO for a long time.
Moz offers numerous tools and resources, so finding their “SEO Web Crawler” took some digging, as it’s not really one of their featured tools. Look for it in the sidebar under “More Tools & Resources” on any of their other research tools pages.

Once on the tool’s page, starting a crawl is simple enough. Simply provide a URL, hit “Crawl this URL” and their crawler will head out and follow links found as it crawls up to 3,000 pages. When it is done crawling, you will get an email with a link to the CSV file generated during the crawl.

But wait. Come to find out, via an email a couple days later, the crawl stats are provided online in the site’s campaign dashboard under “Search > Crawl Diagnostics.” That’s information that would have been useful earlier.

The crawl report is separated into tabs labeled Issues Overview, High Priority, Medium Priority, Low Priority and Site Info. For each tab, a graph of crawl history for that topic is displayed (hopefully showing fewer detected issues over time) followed by a detailed listing of pages relevant to the tab and problem selected.

So while identifying potential problems, Moz stops short of actually giving advice about resolving problems. Instead, they direct their users to their member community forums where the question has likely been asked and answered already and, if not, a user can ask for the community to respond.

Site Condor

  • Advantages – Able to set tolerances for different problem alerts
  • Disadvantages – Difficult to pull actionable information
  • Cost – Free for 2 crawls/2K pages per month
  • Platform – Web-based
  • Recommended for – Experts

Website Auditor Tool - Site Condor

The Site Condor website auditor tool has been available for only a few months. Even so, it provides a polished and complete site crawl report. There’s an API available so you can incorporate their site crawler into your agency or company in-house research dashboard.

Each crawl is considered a “job” on Site Condor. Enter your site’s URL; set the advanced options to your preferred UserAgent, throttling, etc., and your crawl will be scheduled. Upon completion, you’ll receive an email notice including a link to view your reports.

When reviewing the crawl report, you will find one page for each type of potential problem with the option to download a CSV of all pages flagged. There are a lot of issues being monitored hence a number of pages to a crawl report. But Site Condor goes one step further, providing graphical representations for the site’s hierarchy/internal link structure and for the site’s keyword usage as aligned with URL structure.

Unfortunately, my small site wasn’t able to create enough common associations to make these graphics readable. They have the potential to shed light on larger more unwieldy sites. This is a tool to keep an eye on.

Xenu’s Link Sleuth

  • Advantages – Nimble desktop application
  • Disadvantages – Doesn’t provide as much post-crawl information as other options do
  • Cost – Free
  • Platform – Desktop application
  • Recommended for – Experts

Xenu’s Link Sleuth is the “old man” within this collection of website audit tools given its origin back before the turn of the century, if my research is accurate. Its primary use is to check for broken links, but I uncovered a few additional SEO research capabilities when scratching a bit beneath the surface.

The program’s user interface is an unapologetically 1990’s style with little user guidance provided. But if you know what to expect, it isn’t too hard to look for the expected crawl preferences and to start a new crawl. Once the crawl has completed, you will be asked if you want to create a report (or you could have told it via the crawl preferences to send you an email instead).

Opting for the report brings up a page (the source is a temporary file stored on your computer) in your browser. It’s here you realize the primary purpose is to identify broken links as the report categories are Broken Links – ordered by link, Broken Links – ordered by page, List of Redirected URLs, URLs you can submit to search engines, Sitemap of HTML pages, Broken Pages – local links, Orphan Files and Statistics for Managers. Each section is presented as a list of URLs each accompanied by related information. It’s not pretty, but it is complete.

The crawl data is also presented within the program window in table form. Clicking the available column headings enables sorting by that column. This makes it easy to scan for things like duplicate or missing titles or image alt-tags. Right clicking on a page enables viewing of a page’s properties, which includes the list of internal link sources.

Alexa Site Auditor

  • Advantages – Provides a complete roster of audit alerts with descriptions and recommendations
  • Disadvantages – Expensive considering how few crawls are allowed
  • Cost – $49/mo.
  • Platform – Web-based
  • Recommended for – Novices

Alexa has undergone a recent facelift. If it’s been a while since you’ve paid them a visit, then you should. Their site design has been brightened and modernized and the “An Amazon.com company” bug next to the Alexa logo is hard to miss. According to the Alexa website, “On April 10, Alexa rolled out a completely redesigned site.  In addition to the new look, there are new web analytics services for digital marketers and content publishers, as well as expanded services for site owners.”

Once you have added your website to your Alexa dashboard, it will be queued up to be crawled. When the crawl has finished you will receive an email to let you know. Returning to your dashboard, you will be able to view the pages of information for that crawl. Each page is dedicated to a different potential problem and contains a brief description of the problem and why it’s a problem. Recommendations about how to resolve the problem are also provided along with a detailed listing of the pages containing the potential problem as well as the ability to download the page’s list as a CSV file.

Alexa’s reporting is one of the more complete among the tools reviewed here. The report layout is intuitive to understand, problems are identified in laymen terms, as are recommendations. These all make this tool worth considering if you are not yet an SEO expert.

If you want to learn how to conduct an SEO audit like a pro, read more from Raven’s Jon Henshaw.

Related Posts Plugin for WordPress, Blogger...

Filed under:
SEO

Joe Beaulaurier is an SEO expert and founder of Whatcom Marketing. Whatcom Marketing specializes in web site development, social media and search engine optimization (SEO) for different industries nationwide.

More about Joe Beaulaurier | @JoeBeaulaurier

Tell us what you think

  • http://GodOfSEO.Co/ Charles Floate

    I really don’t get why Bing don’t collab with the creators of that toolkit.. They’d have such a better search engine than there current Scroogle campaign has given them.

  • http://www.jeremyriveraseo.com/ Jeremy Rivera

    Glad to see that Beam us up made the list, glad to be a part of that growing tool. Yes, we’re currently working on fleshing out those areas you pointed out that need improvement.

    Also realy glad to see that Raven isn’t afraid to have their other tools reviewed on their blog, too many companies are afraid to expose other brands that have similar tools. However, in this case it’s a win because people such as me are happy to share a good quality review like this.

  • http://WhatcomMarketing.com/ Joe @ Whatcom Marketing

    Good ideas all around, @disqus_HseZwwpbwz:disqus. I toyed with the idea of extracting a best feature from each to identify how to best use each tool in conjunction with each other. But as a “lazy” SEO, that idea only lasted for about 30 seconds. ;)

  • Chris Green

    IIS SEO Toolkit is the only one I’d have said was missing too – although I’ve been a solid fan of Screaming Frog for a while now.

    Although, if you’re able to use more than one tool set in conjunction and know what each is best for, I think you can get the best results. For really serious crawls I use Screaming Frog, Link Sleuth and GWT as they all sneak ahead in different areas – although, I have been tempted by the Raven tool set for a while now.

  • http://WhatcomMarketing.com/ Joe @ Whatcom Marketing

    That is a good recommendation. Kicking myself now for not including it. When I first used it I was surprised by the number of layers and ways to view the data. And it’s free too, right? Good catch @rjonesx:disqus!

  • rjonesx

    Let me put in an upvote for Microsoft IIS SEO Toolkit – it is a great audit tool.