RConversation: Global Voices Online launches new custom search

My colleagues Ethan Zuckerman and Boris Anthony have put together a new "Global Voices Web" search using the new Google Co-op platform which enables you to create your own search engine. Check it out. You can find it in the yellow search bar near the top of the page right under the infamous "tag cloud" (which will likely change in the upcoming redesign).

In addition to being able to search all of the Global Voices site itself or all of Google, now you can also search "the Global Voices Web." Right now, that includes about 4,800 blogs that our Regional Editors follow each day, and from which they select their "Daily Links". In other words, when you use this search function, you are searching all the blogs that we regularly link to or which our editors have found worth following.

GV doesn't cover North America and Western Europe because we believe the views of people from those parts of the world get disproportionately more attention on the world wide web and in the global media than people from all other regions. GV is meant to be a small effort towards counter-balancing that imbalance. Thus the search includes few blogs from N.America or W.Europe except for blogs by members of various diasporas currently living in the West. The point is to have a search that covers the same footprint of online citizen media that GV covers.

This new "Global Voices Web" search was constructed by Ethan and Boris using Google's Co-op search, with a bit of help from some people at Google who responded to their requests for changes. When Ethan and Boris started putting it together, Ethan blogged about the lack of results on some terms in a post titled "What Google Coop Search Doesn't Do Well." The folks at Google eventually read his post and fixed the problem. As Google engineer Vrishali Wagle wrote on the Google Custom Search blog:

After a week of some serious engineering, we believe we've made searching on Ethan's Custom Search Engine -- and all CSEs, for that matter -- much, much better. In particular, Ethan's search for Ghana, which originally returned only three results, now retrieves a much healthier number. All around, you should see much better performance in the quality of the search results.

The reason is fairly complex. Custom search engines are based on approximation algorithms that aim to search over the entire contents of the sites you specify. As with all approximations, there is always room for improvement. We're constantly working on our algorithms, and your search engines will continue to get better. If you see any anomalies or problems, please let us know. We want to hear from you about what is and is not working.

Ethan is now much happier:

Google Coop Search is now a reasonable solution to my search problems, removing a particularly thorny problem from my “to do” list.

But it’s also good news for Google as a whole, and anyone using their tools. It’s a really good sign when engineers are watching reactions to their products closely, monitoring the blogosphere, and proactively solving problems that users are identifying. What impresses me even more is that I didn’t file a bug report about my experiences with custom search, or attempt to twist the arm of anyone I know who works at Google - this was the team’s response to my observations about the tool.

Thanks for solving my problem, folks. I’m impressed, not just by the re-tuned approximation algorithms, but by the way you guys are doing business.

It's pretty exciting to be working on a project that has enough influence that we can impact Google's products - to the benefit of everybody using them, I hope.

Note that our search is still very very "beta." Because it was constructed by importing the feeds from editors' aggregators, we had to weed out a bunch of non-blog and off-topic feeds (news sites, U.S. tech blogs, and things like that). If you try it out please let us know if there are any non-blog or off-topic sites we've failed to weed out or if there appear to be glaring omissions. I'm sure it is far from perfect at this stage which is why we need as many people as possible to test it out and let us know what's wrong.

As Ethan explains in his blog post about this new search function: "a future version will include all the sites we link to on GV, which should expand the collection quite a bit. And an even further off version will integrate with the giant aggregator we’ll be offering on the site next year, which will let you look at new posts from all the countries we cover, as well as offering suggestions for feeds we should be watching - the blogs covered by that aggregator will be the same blogs tracked by the search engine."

Please help us make it better. There is a lot to tweak, no doubt. In playing with it just now, one thing I want to improve is to return the results in some kind of chronological order so that the most recent blog posts show up at the top. Right now you have to resort to hacks like putting in "2006" along with the rest of your search terms if you're not interested in posts from 2005 and 2004.