M4 – My top secret learning project :-)

Yahoo! is now going through a very tough time and is being ignored for mostly. There is hardly any attention being paid to Yahoo! products. I recently tried to play with Yahoo! Application Platform and YQL. One can develop web based and standalone applications using YAP SDK and YQL. These apps are more or less similar to Facebook Apps in architecture. I used YQL and quickly created a small site that fetches Sensex data and displays it. Please visit http://www.indianbizdirectory.com/.

The domain IndianBizDirectory.com was lying idle for more than 4-5 years. I ran a small example fetch weather data for Beverly using YQL console. The example is so simple and self explaining. I was encouraged and decided to use YQL in some real life stuff. I used CodeIgniter to set up a project. Codeigniter is my framework of choice for rapidly developing prototypes. I searched for some feeds and web pages for data and as usual Yahoo! and Google were there to help me. Apart from this I hooked up BSEIndia.com, NDTV.com, Bing.com and MoneyControl.com for data. Initially I decided to run real time calls to these data sources. I setup a very basic logger with CI that mailed me every time a call to any data source or feed failed. I asked my friends to start hitting my site. They hammered it for two weeks. It was slow. And occasionally the feed from BSEIndia.com was failing. I studied logs an realised that the speed was slow due to multiple calls, there were at least 4 to 5 calls to different data sources, were making the page slow. I decided to cache the data. This increased the speed. So

  • on every page load, CI checks if cache is available and loads it, if not then it updates cache for that particular data source/feed.
  • there is a background process that caches data for Sensex during trading hours. The cache is refreshed every 15 minutes.
  • there are chances that a feed may fail during the background process, in that case the process leaves the cache in its original state. The cache is updated by CI.

I found there that BSE feed failure log entries were reduced though the data was not real time. After fetching data I used Xpath to transform data into HTML. Result, I could develop a standalone self sustaining website that displays useful Sensex, Forex and Commodity data. One can see Sensex top looser and gainers, clicking on a scrip name shows further detail about that scrip. On top of it, I added Bing search to fetch and display news related to that particular scrip. All in all, it was a great experiment and learning.

To monetize this effort, I added Google Ads on all pages. I wanted to add more ads but don’t have experience with other platforms.

After this, I am planning to dirty my hands with Python, PyGTK specifically to solve a problem I usually face. 🙂