A message from our CEO: Raven to drop rankings, other scraped data on Jan. 2

rankings reporting

UPDATE: Using Raven’s SEO Tools, you can now report on rankings using authorized data from Google Webmaster Tools and Bing Webmaster Tools.

You make business decisions every day. So do I.

It’s the most difficult decisions that define a business. Abandoning your comfort zone for the sake of growth and stability. Going with your gut even though you know it will cost you money in the short term.

This month, I’ve had to make the most difficult business decision about Raven since we created the software.

It means letting go of a tool we’ve offered for seven years.
It means removing data that our customers rely on us to provide.

Above all, it means preparing for a future that my Raven co-founders and I strongly believe is coming to this industry.

Raven has decided to remove unauthorized Google data from our platform. As of 5 p.m. PST (GMT-8) on Jan. 2, 2013, we will no longer offer Google rankings and are eliminating our SERP Tracker tool in its entirety. Also, we will remove all data provided to us by SEMRush. You have until then to include any of that data in your reports.

According to the Google AdWords API team, these actions by this deadline will bring our software into compliance with the AdWords API Terms of Service; as a result, Raven will continue to offer AdWords tools with data directly from the AdWords API well into the future.

Raven’s co-founders — myself, Jon Henshaw and Scott Holdren — know this change will create hardships for some of our customers. We wish we had more time to make this transition, but that was not in our control. Raven worked hard to make sure you could run reports for all of your data through the end of the year.

We have gathered step-by-step instructions, FAQs and additional information you’ll need to know about this transition into one place. That’s also where you’ll find a public forum for reactions, comments and conversation.

Why we’ve made this decision

Raven integrates more than 15 sources of data for a variety of online marketing methods, from SEO and PPC to social and content.

For many years, Raven has operated with data we received directly from Google and data about Google scraped by third parties. It’s a common model for software companies in the Internet marketing industry. It’s partly stable, partly risky—but more or less comfortable.

That’s not an option going forward. Software companies and their customers will have to choose if they want authorized Google data or scraped Google data.

After much reflection, built on years of experience, Raven has come to the conclusion that the future of scraped Google data is one of diminishing returns. As a private, profitable company planning a long-term future, it was time for us to choose stability over risk for ourselves and our customers.

We want to be a platform for SEO, social media, PPC, content and related analytics. And we want to be a platform with the most reliable, accurate data that we can provide so that our customers can report to their clients with confidence. That’s why I made this choice.

Google AdWords prompted immediate action

On Nov. 2, 2012, Raven submitted its application for Google AdWords’ annual audit. If you pass the audit, you can continue to access the AdWords API. If you fail, you lose access. It’s an audit we’ve passed two years running, and we didn’t anticipate any problems.

On Nov. 5, the AdWords team emailed to tell us Raven had failed the audit and our access to the AdWords API would be revoked in 7 days. We replied immediately to ask for details and more time to make any necessary changes.

On Nov. 8, the AdWords team replied and said we had 7 additional days before our access was revoked to present a plan about how we would comply with the AdWords Terms of Service. They were specific:

“In order to be in compliance with the AdWords API Terms and Conditions please cease offering any product or product feature that collects scraped data or uses scraped data acquired from another source (this includes the functionality offered under website and keyword research in your tool).”

We received that email on a Friday evening.

It was clear what it would take to pass the audit. We had to choose: authorized AdWords data or scraped Google data of any kind. Over that weekend, my partners and I started to think through options.

On Monday, Nov. 12, we emailed the AdWords team to say we wanted to remove Google scraped data. We asked for four months—until March—to execute a plan of action, and we asked to keep our AdWords API access during that time. We didn’t know when we would hear back or if they would approve our plan.

On Tuesday, Nov. 13, working under the assumption they would revoke access within days, we turned off Raven’s AdWords research, management and reporting tools and contacted our customers.

Then, for the next two weeks, we explored scenarios while waiting to hear about our plan. Could we keep scraped Google data and live without AdWords API access? Possibly. If we removed scraped Google data from the SERP Tracker, would that tool in its current form have any value? Probably not. Could we allow customers to import Google rankings to the SERP Tracker and still be in compliance with the terms of service? No.

On Monday, Nov. 26—right after the U.S. Thanksgiving holiday weekend—the AdWords team replied. They approved our plan, stipulating only a timing change. We couldn’t have four months to be in compliance, but we could have 30 days. After another email exchange, the AdWords team granted us an extension to Jan. 3, 2013, to execute our plan and pass the AdWords API access audit.

On Wednesday, Nov. 28, we turned our AdWords research, management and reporting tools back on in Raven.

We appreciate the additional time that the AdWords team allowed us so that we could communicate with our customers. That way, our customers could compile reports through the end of the year, and they could do it with all of their data.

Why scraped data isn’t worth it

In the case of scraped Google ranking data, we were paying a lot of money to get these things in return:

  • Unreliable results. It’s impossible to be 100% accurate with scraped Google rankings because geography, personalization, server timing and types of results influence SERPs for each searcher.
  • Inflated expectations. Most customers expected 100% accuracy regardless of geography, personalization, server timing and types of results. Any variance from what they saw when they (or their clients) performed a search caused confusion. Adding rank editing did not alleviate the problem as much as we hoped. What’s more, recently we’ve had more requests than ever for daily, universal and local results.
  • High risk. Using scraped Google data violates Google’s terms of service. It doesn’t matter whether we provided scraped data for rankings, research or really boring Friday night entertainment.
  • An unpredictable future. This metric, though deemed essential by some SEOs, would only continue to diminish in relevance as campaign performance metrics replaced it in importance.

As an experienced businessman, I’d rather provide tools and data that prove ROI, conversions and goals, not simple rankings.

What we predict about the industry

Raven wasn’t singled out during this AdWords audit. We know that two of our third-party data providers have lost AdWords API access at some point in the past two years. Google seems to be universally enforcing its terms of service regarding scraped Google data, which it has every right to do as a business.

If this enforcement continues—and I believe it will—then every software company will have to make the same choice as Raven: either you comply with Google’s requests or you don’t. Every SEO tool provider will have to choose between a very risky future (unauthorized, scraped data) or a less risky future (authorized API data).

Regardless, it will be impossible for any software company to provide scraped Google ranking data and authorized AdWords API data.

And what happens when Google realizes that it’s not just its editorial content that’s being scraped, its money-making advertising data is being scraped, then sold as authorized data, too?

Raven’s plans

We love what we do. We want to provide the kinds of efficient, essential tools for Internet marketers that prompt them to write, “If you’re thinking about getting Raven, do it. It’s one of the best decisions that I have ever made. It has been part of our business tripling in the past year alone.”

The convergence of content, SEO, social, advertising, email and other marketing methods into single agency offerings is already happening and will accelerate. What’s more, traditional marketers—ad agencies, public relations firms and the like—are in the game now. We’re building our platform with that future in mind.

One more reminder: You can find step-by-step instructions, FAQs and additional information about this transition right here. There’s also a public forum for reactions, comments and conversation.

— Patrick E. Keeble, Co-Founder, President and CEO