Yahoo! & Chinese Censorship

There is much applause in the blogosphere for today's Washington Post Op-Ed condemning Yahoo! for its collaboration with Chinese authorities in the conviction of Chinese journalist Shi Tao.
(See my previous blog posts on the issue here and here.) 

Yahoo! co-founder Jerry Yang said the company was merely following Chinese law - it had no choice. But as human rights groups have been pointing out, Yahoo! has been going above and beyond the strict legal requirements for some time. In 2002 it signed the Internet Society of China's Public Pledge on Self-Discipline for the Chinese Internet Industry.

This "self-discipline" is reflected in the way Yahoo! filters (censors) its search results. The ONI China Filtering Report describes the process in detail. To illustrate the process, I've taken screenshots to show what happens when you try to search for terms like "Tiananmen massacre" on the Chinese language versions of Yahoo! and Yahoo! News.

Here are the steps I took:

1) Sitting at home in Cambridge, MA, using my Comcast internet connection, I accessed Chinese Yahoo! News and Chinese Yahoo!, then typed in the Chinese words for "Tiananmen Massacre": 天安门屠杀. (Click image to enlarge)

Here's the result:

A big fat error message. Same thing happened when I did the same search on the Chinse Yahoo! News.

2) Then I configured my browser to use a Chinese proxy server, so that my searches would be going through an internet service provider based in China. (Chinese ISP's configure their routers to block politically sensitive web pages and filter politically sensitive search results.)

Same error message, plus the Chinese ISP barred all further requests from my computer's IP address. Punishment for trying to access blocked material.

3) By way of comparison, I repeated the same steps for Chinese Google and Google News searches, both with and without a Chinese proxy.

Here's the result of a Google News Chinese search through my normal Comcast connection:

A search on the term "Tiananmen Massacre" returns a long list of results.

In contrast, when I do the same search on a Chinese ISP, I get this:

The error message tells me that no results can be found, etc......

What do these screenshots illustrate? They show that Yahoo! actively filters politically sensitive terms from within its own service. Even if a Chinese user finds a way to access Yahoo from outside China or via a foreign proxy server, they will still get filtered search results on politically sensitive terms. By contrast, Google does not actively filter. The filtering of Google search results is done only by the Chinese ISP... although Google helps hide this process from Chinese users by choosing not to show results that the Chinese user would be unable to access.

As a result, Chinese users of Yahoo! have no way of discovering that the search results they seek actually do exist -- even if they are tech-savvy enough to use a proxy server. This is the same experience that Chinese users have when using home-grown Chinese search engines like Baidu. Google users have a better chance of finding forbidden material if they know how to use proxies, but those who don't (i.e., most Chinese internet users) also have no way of knowing what they're missing.

Like their Chinese competitors, these American companies are not being transparent or honest with Chinese users about the fact that search results on their services are being censored in compliance with Chinese legal and extra-legal requirements.  A bit of transparency and honesty, at very least, would be a step in the right direction.

Comments

[Kyle Hasselbacher]

I like this experiment, but I don't think it shows what you think it does. If you go to google.com from a Chinese IP address, you get Google in Chinese. (I use a proxy that causes my requests to come from all over the world, and I routinely see Google in a language I don't understand.) Google can show you a different front page based on your IP address; I think they can show you different search results based on your IP address.

I think the difference that you've found is that Yahoo! thinks that a request for search results in the Chinese language implies that the searcher is in China regardless of the IP address. Google relies on the IP address to tell whether to block something regardless of what language the user has chosen. What happens if you search on English language Google from an IP address in China? Maybe I'll try this later today.

[zhwj]

Interesting. My results are slightly different from yours.

Accessing cn.yahoo.com from Beijing and searching for that phrase gives me a Yahoo page that reports no results found, and recommends retooling my search terms. So it seems to be doing the same thing as your Google example. It's just picking up that exact the phrase, though; reverse the two words and you'll return a set of results that are essentially "healthy" but include several unviewable on the mainland (the suggested alternative search terms are similar - they look really threatening ("bloodbath" and such) but the engine breaks the phrases into constituent words and returns innocuous reslts).

Also, cn.yahoo.com is not only Chinese-language but actually hosted in China (traceroute goes through bjtelecom), so it's still a matter of location rather than language. tw.yahoo.com (accessible via proxy) gives a full set of results, but hk.yahoo.com throws an error at that search term, proxy or not.

I'm not particularly clear on what you mean by "actively filtering" - if Google shows fewer results results for a search than it actually has indexed, isn't that as much "actively filtering" as if it simply tossed up a "page cannot be displayed" message, as you suggest Yahoo does in your first example, when it encountered sensitive terms/results (I believe that it's the ISP doing the filtering in that example, though)?

[More info: searching for the term via Google gives the same results directly as via proxy; only if I search directly the page gets sent a "connection reset by peer" as soon as it loads. I get blocked with no proxy when I attempt the news search.]

--Joel

[Nart]

In many locations China's gateway filtering is bi-directional. In the case of Yahoo!, the search server (cn.yahoo.com successfully redirects to cn.websearch.yahoo.com) is located in China. Your request to the search server is being filtering by China's "firewall" -- not by Yahoo!. It is likely that a portion of the URL is filtered by China. (For example, try to access http://www.baidu.com/nothingtoseehere/ you will be redirected to http://www.baidu.com/search/error.html a standard error page. Now try appending the offending URL path from Yahoo! to Baidu -- http://www.baidu.com/search/web_cn?ei=gb&stype=&p=%CC%EC%B0%B2%C3%C5%CD%C0%C9%B1&scch=on

It should generate the same error page.)

Same goes for the proxy. The requested URL is sent from your browser to the proxy in plain text through the "firewall", given the description of access from Beijing in the coments above, that is the spot where its filtered.

Proxy servers in China are good for determining if content is acessible, but will return inconclusive results when checking to see if content is blocked. There are two many points of filtering and failure between you and the proxy and the proxy and the requested content.

I think that what's been shown above is China's "firewall" interfering with requests for Yahoo!. (Yahoo does filter in another way though, see below.)

However, your Google example is actually much better. If no results are being returned, but the connection is not blocked then that is Google filtering the results. Google has admitted to removing results from urls blocked in China and the search for the term above yeilds epochtimes and peacehall bothe of which are blocked by China and removed from Google's Chinese New site for user in China (by geolocation).

I've shown something similar with Yahoo! here:http://ice.citizenlab.org/?p=67

The other thing to watch is that the keyword in URL filtering by China is not 100% accurate, sometimes requests will go through.

Also, the inability to access the proxy is a by product of China's filtering emchanism, explained here (http://ice.citizenlab.org/?p=113) -- the same behaviour can occur bi-directionally in most cases.

[Uncover China]

But not much can be done about it really. If they go against the law then problems will occur. Best to follow their instructions for now..

[Don Mitchel]

yahoo will not let me post anything just says internal error, but then I see where other people have posted. this is an example of what I can not seem to post on yahoo:

Jobs and immigration would not be such a hot button
issue, had Senator Boxer support ballast water legislation in 2008. How can anyone know who to vote for, when politicians will not talk to specifics (current bills etc.)? The news media will only ask superficial questions. Americans hopefully have learned that voting for someone because they say the word "change", should raise red flags and mean- to be careful what you wish for. Restoration programs are great, but by themselves without proper prevention, will only continue to be, a repair to the damage still being allowed to occur and a continued economic drain. NY has spent enormous amounts of money, (that could have been stimulus money) on ballast water protection. The Federal government could not act to protect all states equally after a house vote for ballast laws 395-7, because our country is now dependent on foreign manufactured products to survive which are primarily delivered by foreign ships. Candidates for governor in NY, should be ask if the money the state has spent to protect their waters without infrastructure for enforcement will be wasted, by disregarding NY regulations, should the Coast Guard initiate any plan, weak or strong in the presidential election year 2012. Are they going to be granting permits or passes to ships while this plan to pressure the Federal government to act responsibly, plays out? NY laws are important to uphold as they affect all Great Lakes states, with international implications as well, and NY regulations may pressure this administration to finally act responsibly, before the presidential election year 2012. At best immediate action would be 2 1/2 years after historic legislation passed the largest elected legislative voice of the people 395-7 for the real change we needed in 2008. This legislation was killed by one member of the presidents party, while the three top members of this administration were Senators, leaving the states in competition with each other as they negotiate on an international level with foreign economic interest. Our countries policy of promoting economic globalization should stop this pass destroying our environment, and take our economic security back, by creating a level playing field for American manufacturing. We already have enough store clerks, as store clerks are now the largest career for Americans to aspire. This means the American dream for many American will only be a store clerks standard of living. Unfortunately this scenario has been created by the last two decades of political dynasty's interlacement of our free market economy to the point of dependence with the controlled economy of a communist country.