A while back I created Trend Sweet Trend and added a search phrase tracker to it. It tracks the most popular search phrases people type into Yahoo! Search thanks to Yahoo! Buzz which updates each weekday. The search phrases are on the left and on the right is a grid showing what rank each term was at in the past. American Idol for instance is in 1st place and has been for the last two days that Yahoo Buzz was reporting. Before that it was 2nd and before that 12th. Green means it’s rank has risen or held steady after rising. Red means it fell. Purple shows you that it’s the first time it’s ever been on the list. Ever being not a very long time(I started tracking in late November 2005).

I am always amazed that most of what people search for are celebrities. Go here if you want to see the live page and click on links.

Since I’ve been following this I’ve noticed that entertainment is really what bonds us all together. We don’t all do the same work or really spend much time worrying about the same news. But we do watch the same hit TV shows and follow the most famous celebrities. This page is really just there to try and make sense of what’s most popular and was an interesting extension for me of the first Trend Sweet Trend page. If you like it or want to see something changed about it, let me know.

I’m writing my own search engine. I know it sounds crazy but it’s just the sort of project that fuels my fire. At this point, I’m keeping the scope of the search to urls that have appeared on my Trend Sweet Trend(retired in 2008) page currently numbering (>7000). I’ve written 3 perl scripts so far and setup 5 tables on my newly LAMP’d web server.

One table stores the urls and the last time they were indexed. Another is a queue of urls to be indexed. A third stores the parsed text from the indexing operation. The other two aren’t storing anything yet but will be for the cache of each page and to keep track of links between sites once I unleash a crawling feature.

As for the three scripts, one grabs the urls from my Trend Sweet Trend database and puts them in my search engine’s crawling table. The second checks the last time the sites in the url table were indexed and adds urls that need indexing to the queue. Finally the third script uses curl to visit each page, parse out text using HTML::Parser and stores the counts of each term in my search terms table. It’s quite primitive, indexes a lot of garbage text, and can’t do any booleans but besides that it’s great. Watch out Google. Anybody got a spare datacenter with 100000 commodity boxes?

Trend Sweet Trend suffered a glitch over the past several days. To begin with my power went out over xmas so updates stopped occurring until I could get back and turn on my Fedora server.

Then TST began to show only one link rather than the 25 or 35 it usually does. This was caused by a deficiency in my livemarks curl script that couldn’t handle multiple matches per line. It’s all fixed now but it gave me pause.

If you’re thinking of using “if(/pattern/) ” to match something in a feed and do something with it, you might as well replace that bit with “while(/pattern/g) ” so you’ll match pattern as many times as it might occur in a line.

With web services, you’ve got to think ahead and program for the odd change that might happen upstream. You’ve got to think what if there’s more than one match per line in this regex script or if the odd formatting change should occur.

Using my patented(not really) trend technology, I’ve created another trends page(retired in 2008). This one is based on Yahoo! Buzz, their almost daily listing of the most popular search terms for the day.

Whereas Trend Sweet Trend tends to show lots of developer and blog links, the search phrases page shows a lot more celebrities and holidays. It might be useful if you want to get your site or blog read by the masses.

Now if I can just start listing the popular search terms for a few days from now…

Over the past day we’ve seen some weirdness over at del.icio.us. Last night I saw this error:

System error

raw error

Then on clicking raw error it showed this stuff here:

Can't call method "prepare" on an undefined value at /www/del.icio.us/comp/user/get line 13.

Trace begun at /usr/share/perl5/HTML/Mason/Exceptions.pm line 131HTML::Mason::Exceptions::rethrow_exception('Can\'t call method "prepare" on an undefined value at /www/del.icio.us/comp/user/get line 13.^J') called at /www/del.icio.us/comp/user/get line 13HTML::Mason::Commands::__ANON__('dbh', undef, 'user_name', 'dsterry', 'auth_user', 'dsterry') called at /usr/share/perl5/HTML/Mason/Component.pm line 134HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x1d3f4f0)', 'dbh', undef, 'user_name', 'user', 'auth_user', 'user') called at /usr/share/perl5/HTML/Mason/Request.pm line 1069eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1068HTML::Mason::Request::comp(undef, undef, 'dbh', undef, 'user_name', 'user', 'auth_user', 'user') called at /www/del.icio.us/site/dhandler line 194HTML::Mason::Commands::__ANON__('dbh', undef, 'auth_user', 'user', 'items', 100, 'tagview', 'cloud', 'tagsort', 'alpha', 'bundleview', 'hide', 'cookies', 'HASH(0x1b52740)', 'browser', 'ff') called at /usr/share/perl5/HTML/Mason/Component.pm line 134HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x1b14410)', 'dbh', undef, 'auth_user', 'user', 'items', 100, 'tagview', 'cloud', 'tagsort', 'alpha', 'bundleview', 'hide', 'cookies', 'HASH(0x1b52740)', 'browser', 'ff') called at /usr/share/perl5/HTML/Mason/Request.pm line 1074eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1068HTML::Mason::Request::comp(undef, undef, 'dbh', undef, 'auth_user', 'user', 'items', 100, 'tagview', 'cloud', 'tagsort', 'alpha', 'bundleview', 'hide', 'cookies', 'HASH(0x1b52740)', 'browser', 'ff') called at /usr/share/perl5/HTML/Mason/Request.pm line 760HTML::Mason::Request::call_next('HTML::Mason::Request::ApacheHandler=HASH(0x1ad11c0)', 'dbh', undef, 'auth_user', 'user', 'items', 100, 'tagview', 'cloud', 'tagsort', 'alpha', 'bundleview', 'hide', 'cookies', 'HASH(0x1b52740)', 'browser', 'ff') called at /www/del.icio.us/site/autohandler line 110HTML::Mason::Commands::__ANON__ at /usr/share/perl5/HTML/Mason/Component.pm line 134HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x1b52bd0)') called at /usr/share/perl5/HTML/Mason/Request.pm line 1069eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1068HTML::Mason::Request::comp(undef, undef, undef) called at /usr/share/perl5/HTML/Mason/Request.pm line 338eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 338eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 297HTML::Mason::Request::exec('HTML::Mason::Request::ApacheHandler=HASH(0x1ad11c0)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 134eval {...} at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 134HTML::Mason::Request::ApacheHandler::exec('HTML::Mason::Request::ApacheHandler=HASH(0x1ad11c0)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 793HTML::Mason::ApacheHandler::handle_request('HTML::Mason::ApacheHandler=HASH(0x1acc1c0)', 'Apache=SCALAR(0x1087150)') called at (eval 43) line 8HTML::Mason::ApacheHandler::handler('
HTML::Mason::ApacheHandler', 'Apache=SCALAR(0x1087150)') called at /dev/null line 0eval {...} at /dev/null line 0

then Wednesday from around 4:30 pm to Thursday 1:00 am PST del.icio.us was down showing the following message:

del.icio.us is down for maintenance. we’ll be back in one hour.

I just checked again, I saw my user page and then I saw the first error again. It kind of looks like the database is having issues. Unfortunately all this meant that Alex’s Livemarks and my Trends(retired in 2008) were both having issues today too. Hope they get all this worked out soon…

I’ve taken Trend Sweet Trend(retired in 2008) further. Now, you can see screenshots of each link via css rollover. The page takes a little longer to load but I think it looks cool and is at least a little bit stickier.

To do this, I created a script that submits urls to Browsershots and later checks to see if they’re done. Then I store the images on my server and use css to display them next to each link. The lightweight version of TST is still available here.

I want to thank Johann Rocholl at Browsershots for his help in getting this to work and for making such a cool web service.

I’ve just created my first RSS feed for trend sweet trend(retired in 2008). To do it, I scraped my own page and built the file. You’ve got options: add it to your feed reader, use a site like feedburner to add it to your page, or subscribe with Firefox so it’s always close at hand.

Creating an RSS feed isn’t too hard but there were a couple of stumbling blocks. The first one I ran into was that firefox refused to recognize my php generated document as rss.

This may have been because I had some extra whitespace outside of the code, but I thought it was because of the webserver not sending the output of php suffixed scripts as type application/rss+xml. That’s why I’m now generating the page using a remote perl script.

The other problem I ran into was encoding of funny characters like double greater than symbols. Why people want to use these in their page titles, I don’t know but I solved that using a CDATA section for each title.

More trends and feeds to come…

Livemarks became slow yesterday for an hour or so. That meant I had to update my database by trying to connect and download links manually. I didn’t think of the possibility that my script might not connect but it’s important to plan ahead when you’re using a web service.

You’ve got to think, what happens if it’s not available for an hour, day, or week. What does your site look like then? Turns out my trend sweet trend(retired in 2008) displayed its header only…no links, no trends, nothing.

Today, I fixed it by displaying a message that livemarks was down and showing how many hours the info is delayed. I’ll have to work on the scraping script since running it manually seemed to work with a 4-5 minute time for curl to get the page.

The del.icio.us crop that is. I’ve taken the data I’m gathering for trend sweet trend and made a best-of page. This page(retired in 2008) shows only the sites that have made it to #1 on livemarks for about an hour or more.

I expect the list will grow by about a link a day so if you check the page every week or so you should get some pretty good old links. The goal of all this is either to make a list of the most useful web tools ever or to help people waste time as efficiently as humanly possible.

For a coder the pinnacle of frustration is testing…finding and fixing the bugs you want to squash. You can take one approach which is to find a bug, fix it, and pray that no other bugs appeared during the process. This unfortunately doesn’t work when you’re dealing with an application of any but the most trivial complexity.

You’ve got to have a plan: a repeatable process for testing your code to make sure nothing broke. I just created a test plan for a site I’m working on and it came to about 35 actions to test. This is for a site that has about 8 real pages and some database code.

Testing a web site is a unique challenge in that it’s hard to write a script to drive all your browsers. What’s needed is a standard browser control API. With broad support for such a standard you could write a test harness that drives IE, Firefox, Netscape and Opera all at the same time.

You could put that 21″ lcd to use tiling the four browsers while setting the zoom in each to simulate highres surfers. While I’m imagining this, why not have all the tests be recorded as macros? Wow, wouldn’t life be perfect if I had that?!?! I doubt it but there is some hope.

After a bit of searching I found this Mozilla ActiveX control project which is a step in the right direction. (tangent: Why don’t we have a free software analog to activex?) If I get a chance to check out their project, maybe I can automate some of my website testing.

Until then, I’ll be repeating my 35 steps by hand and writing test scripts whenever the opportunity arises.