Diagnosing the Diagnostics: Background

This is the first part of a series about Google's Safebrowsing Diagnostics page.

Google's Safebrowsing Diagnostics pages are a valuable source of information for webmasters. Unfortunately the information can sometimes be difficult to understand. Hopefully the next few posts will help clarify - I've been struggling with them for the past couple of months since they're fairly complex. Bear with me and jump in with any questions you may have.

Before we get to the diagnostics pages, we'll need to understand how malware works. Malware distributors have created sophisticated infrastructures. Typically, they use three types of servers and domains:

• Distribution Servers: Malware distributors usually have a set of servers that they control and use as a base of operations. The servers host malware tools and exploits and are usually on a network where they are unlikely to be shut down. They often don't have domain names associated with them, only IP addresses.
• Compromised Domain: Malware authors compromise legitimate domains and insert malicious scripts. The owner of the website usually doesn't realize this has happened. Most of the entries in Google's Safebrowsing malware list are these sorts of servers that have unintentionally become dangerous.
• Intermediary Domains: Between the distribution servers and the compromised domains there are often one or more intermediary domains. Malware authors establish domains to obfuscate their malicious code. For instance, they may use a common domain name with a slight misspelling, like gooqleanalytics (notice the Q), hoping that webmasters won't spot the typo.

The structure of Google's Safebrowsing Diagnostics pages mirrors these types of servers and domains. Most flagged domains will only be one type of server eg. compromised domains usually aren't also distribution servers.

With that background, you may already be able to understand the diagnostics pages a bit better. In future posts we'll dive into the details of each section of the diagnostics page.

Diagnosing the Diagnostics series:

• Background
• Domain Summary
• More to come...

Suspicious and Really Suspicious

Google's Safebrowsing Diagnostic page lists "the last time suspicious content was found on this site". But what does "suspicious" mean?

Google's automated malware scanners have been highly accurate with an astonishingly low false-positive rate. Part of that success has been because their definition of "suspicious" actually means "has nasty malware". If the scanners aren't really sure that a site has malware, they won't add it to the malware list. And that's the definition of "suspicious" ("has nasty malware") that Google's Safebrowsing Diagnostic page uses - content bad enough to get a site added to the malware list.

When the scanners do a review of a site to check if it should be removed from the malware list, they use a more stringent definition of "suspicious". If there's any suspicious activity at all then the site will not be removed from the malware list. Often sites have been infected with malware in multiple ways and the scanners need to be sure that it has been thoroughly cleaned up.

Those different definitions of "suspicious" may cause confusion when looking at Google's Safebrowsing Diagnostic page for a site that has been reviewed. The review may have found "suspicious" content that was not "suspicious" enough to have added the site to the malware list - but it is "suspicious" enough to prevent it being removed from the list. Google's Safebrowsing Diagnostic page won't list the date of that review scan.

If you're looking for the status of a malware review, log into Google's Webmaster Tools - the same place you reqested the malware review. It will show whether the review succeeded and will list urls that were still found to be "suspicious".

Cross-site Warnings

Several browsers use data from Google's malware list to protect users. Firefox 3, Chrome and Safari all check sites that users are visiting against Google's list and warn users if they are about to visit a dangerous site. There are some small differences in implementation across browsers that can cause confusion.

All three browsers check the address of the top-level page a user is navigating to. That protects most users in most cases. But, a web page can include content from another web page and if the included content is malicious then users may be exposed. Chrome (and Safari*) check every request against Google's malware list. This means those browsers will protect users even if malicious content from a flagged page is embedded on a non-flagged page.

Although that approach provides better protection for users, it may be confusing for webmasters if content on their site comes from another site. Some users (those with Chrome or Safari) will get warnings even though the webmaster's site is not blacklisted. Because the webmaster's site isn't blacklisted, they won't be able to request a malware review via Google's Webmaster Tools. Fortunately, this situation usually doesn't exist for very long. Google's scanners have already identified the embedded content as malicious but they haven't yet flagged the webmaster's site that includes the dangerous content. As they continue to crawl the internet, the scanners will quickly flag the webmaster's site.

If you're a webmaster in this situation, you'll need to examine all the content you're including from other websites. Look carefully at the warning page that browsers display since it usually includes the name of the domain that caused the problem.

* I can't say for certain exactly how Safari behaves because I haven't seen the code. But based on observation, Safari seems to have adopted the approach of checking every request.

Updated: FF3.5 checks every request against the blacklist and helps better protect users. FF3.5, Chrome and Safari all behave the same now.

Kick Them All Out!

On the Google Webmaster Central blog, Ben asked why sites with malware aren't removed from search results.

Users often use Google to search for sites they already know about. Urls are hard to remember but users can remember a couple of search terms and use those instead - like "barak obama" or "john mccain". If a user typed either of those queries, they'd expect the candidate's official site to rank near the top. But malware can strike any website: big or small; Democrat or Republican; corporation, government or private citizen. If Google removed the candidate's website from search results, there would be a lot of confused users. (1)

By leaving the website in the search results and placing a warning on it, users can find what they are looking for but know that visiting the website may be dangerous.

(1) I am not suggesting that either candidate's website has ever had malware. It's just an example.

Update: Yes, yes, I misspelled "Barack". Fortunately, Google still finds the right site: http://www.google.com/search?q=barak+obama

Information

Webmasters often ask why Google can't give more information about malware on websites. A couple of factors come into play.

Google's automated scanners only see the end result of the infection. They have no way of knowing how or exactly when the malicious content was added to a website. There are many ways it could have happened, from sql injection to stolen passwords. The automated scanners just know the bad content is there now.

To some extent, the success of Google's automated scanners relies on the secrecy of the methods used. The team has published malware papers (like Ghost in the Browser [pdf]) that include overviews of the system. But we're uncomfortable publishing any more detail because malware authors can read that information too. We've observed malware trying to hide from our automated systems and a large part of my job is tweaking the system to adapt to new types of malware. Staying on top of what malware is doing isn't easy and providing exact descriptions of the scanners and what they detect would make it impossible.

Earlier this year Google released the Safe Browsing diagnostic page which list general information about what Google's automated scanners found on a website. Some webmasters have found that information helpful in cleaning up their site. Hopefully we can find other ways to share information with webmasters without compromising the success of the scanning system. If you've got specific ideas, please let me know!