A while back I created Trend Sweet Trend and added a search phrase tracker to it. It tracks the most popular search phrases people type into Yahoo! Search thanks to Yahoo! Buzz which updates each weekday. The search phrases are on the left and on the right is a grid showing what rank each term was at in the past. American Idol for instance is in 1st place and has been for the last two days that Yahoo Buzz was reporting. Before that it was 2nd and before that 12th. Green means it’s rank has risen or held steady after rising. Red means it fell. Purple shows you that it’s the first time it’s ever been on the list. Ever being not a very long time(I started tracking in late November 2005).

I am always amazed that most of what people search for are celebrities. Go here if you want to see the live page and click on links.

Since I’ve been following this I’ve noticed that entertainment is really what bonds us all together. We don’t all do the same work or really spend much time worrying about the same news. But we do watch the same hit TV shows and follow the most famous celebrities. This page is really just there to try and make sense of what’s most popular and was an interesting extension for me of the first Trend Sweet Trend page. If you like it or want to see something changed about it, let me know.

The other day I began to think about search engines and their innately commercial nature. I thought, why is it that the Internet, which exists as a large network of networks, relies on commercial organizations to index the web and serve search results? Shouldn’t the Internet index itself and provide search as a basic utility or protocol? With the success of the open source development model there must be great potential to create a community-owned-and-operated search engine.

What would be the benefits of such a system? It would ideally collect only aggregate and anonymous(hash IP addresses with random session keys) logs that would be used for the sole purpose of improving search. The servers providing search would be donated via a distributed effort similar to Seti@Home except limited more to local ISPs with the server resources to provide the search results and maintain replicated indices.

After a bit of thought on this concept, I did some research. Turns out there are a couple of projects over at the Apache Software Foundation by the names of Lucene and Nutch. These are free software projects with the goal of developing world class indexing and search application software. I’m not sure if they’re trying to build a distributed web search infrastructure but their project is certainly important to the idea I’ve been thinking about. One thing I learned from their FAQ that I did not know is that indexing the web is not the most bandwidth intensive task a company like Google or Yahoo has to deal with…it’s actually serving the search results. Makes sense when you think of how many times people look at a specific site versus how often it would change.

One other thought before I call an end to this post. Free software is generally good at recreating or making small improvements to commodity proprietary software where features have been stable for some time. Since search engines are being rapidly developed in the commercial sector, it is difficult for free software to supplant leaders like Google and Yahoo at their own game. It may be necessary to start the search engine I’m talking about by looking to index sites that would provide little or no ad-revenue. Think scientific research and academia.

I’m writing my own search engine. I know it sounds crazy but it’s just the sort of project that fuels my fire. At this point, I’m keeping the scope of the search to urls that have appeared on my Trend Sweet Trend(retired in 2008) page currently numbering (>7000). I’ve written 3 perl scripts so far and setup 5 tables on my newly LAMP’d web server.

One table stores the urls and the last time they were indexed. Another is a queue of urls to be indexed. A third stores the parsed text from the indexing operation. The other two aren’t storing anything yet but will be for the cache of each page and to keep track of links between sites once I unleash a crawling feature.

As for the three scripts, one grabs the urls from my Trend Sweet Trend database and puts them in my search engine’s crawling table. The second checks the last time the sites in the url table were indexed and adds urls that need indexing to the queue. Finally the third script uses curl to visit each page, parse out text using HTML::Parser and stores the counts of each term in my search terms table. It’s quite primitive, indexes a lot of garbage text, and can’t do any booleans but besides that it’s great. Watch out Google. Anybody got a spare datacenter with 100000 commodity boxes?