Spark streaming reduceByKeyAndWindow unstable application

I wanted to run a job which runs 24×7 and which reports if certain keywords occur more than a N times in the stream. Spark streaming looked a ideal candidate for this task. Spark has a reduceByKeyAndWindow function which was exactly what I was looking for.

Spark streaming: Fixing all executors not getting jobs

I was working on a feature recently which needed a streaming job that runs 24×7 and processing 100 million rows per day. The spark web ui is a wonderful tool to look at how things are running internally. While debugging I noticed that the streaming jobs were getting allocated to only one machine. Spark has a set priority to dispatch jobs to the executors based on proximity (on the same host, in the same pool etc) and if they complete the job within a fixed interval then all the jobs are sent to the same executor.

Sending mails from munin using custom command

We use Sendgrid for email delivery on our cloud. Using custom commands in munin can be tricky. After struggling for some time I wrote a small python script to send emails from munin using sendgrid.

Motorcycle review: Bajaj Avenger 2013

I bought Avenger in July 2013. Specification can be read at http://www.bajajauto.com/avenger/. I have toured extensively in Karnataka and Leh, so I think this review will definitely help you make up your mind.

150 Kg Leg Press

Yesterday I did 150 kg leg press.