Evanescent Thoughts

Evanescent Thoughts


leave a comment »

What I would remember 2011 for :

  1. Finally hitting the gym after 22 years of ¬†‘relaxing’. Lost 60 pounds over 4 months. Think I can consider that an achievement ? ūüėČ
  2. Interning at Facebook over the summer, working on Freight and finally building something considerable from the scratch. Getting an offer finally after a lot of excitement.
  3. Shaking hands with Mark Zuckerberg over a party at his house. This guy is very different from what was portrayed in the movie.
  4. Living with cousins over the summer and watching Appu (my new born nephew) grow exponentially.
  5. Biking around 1100 miles in California over 12 weeks.
  6. Surprised parents by showing up one fine day at their door step when they were thinking I would come for a video chat. (for anyone else, MUST DO once in your lifetime)
  7. Hiking 13 miles in the Grand Canyon (Havasupai Trail) to find the most gorgeous turquoise-blue waterfall I have ever seen.
  8. Finally, working on the Linux Kernel :).
  9. The Door country trip; How to possibly screw up all the plans and yet have fun.
  10. Buying a Kindle and finally reading some of those books I must have read ages ago .

Written by Sathya Narayanan

January 30, 2012 at 5:55 pm

Facebook Internship

leave a comment »

Statuatory warning : Bear with me if this post is long or boring. Its been a long time since I blogged.

This summer, I interned at Facebook, Palo Alto. It also happened to be my very first experience outside academia, writing code that actually goes to production. I got the offer about a month after I started my graduate studies at Univ of Wisconsin, Madison. I had just moved into the US (late august, 2010)and there was a career fair around that time which I missed attending. A couple of my friends got internships from Qualcomm, but I decided to take some time to settle and then start looking for internships (after all, I had just moved in). Around mid september, facebook had come to college for a giving a talk about memcache. It happened to be that the person who was giving the talk  was a alumni of NIT Trichy and UW Madison. They asked us to forward our resumes if interested and I did. Within few weeks, the recruiting team was on campus again and they had a couple of interviews before offering me an internship. I was so excited that my first interview experience landed me with an offer and to top it off, it was FACEBOOK.

The internship was for 12 weeks, a very competitive offer that I¬†didn’t¬†consider sitting for any more companies. Besides google started their process very late (not that I would have been interested in taking it if I had been¬†offered). ¬†Started my internship on May 23rd. I worked for the Application Operations team under Scott and Alex, ¬†with another fellow intern, Dustin (Alex’s intern). We worked on building a service called Freight ¬†and later another thin layer of abstraction on top of it called PubSub (a brief report about it here) . It was an amazing experience to build something from scratch, have daily scrums, work agile and learn a shit load of new things, and the most important of all, to live and work in the the reality. Here goes a brief description of what happened in the next 12 weeks.

Week 1: Orientation, get your mac books and iPhones ( yes .. interns get brand new iPhone 4), get your dev environment setup, meet with your manager, discuss about the internship. We had a full team meeting discussing about Freight and what problems it should address. I should say Week 1 had my head in a whirlwind. A lot of things were pretty new for me. Though I was a little bit scared, I was happy that I would in the worst case, learn a LOT for sure.

Week 2: Started playing around with rsync and it was wiki¬†time. Did a ¬†_very_ dirty hack to rsync by mid second week to make it work for our end goal. It was so dirty that I dare not speak about it. But it works brilliantly! ¬†Spent most of ¬†the time saying hello to git. I had never worked much with any version control systems and I should say I was confused for half the time. But in the end, everything made sense. Dustin could relate to git since he had experience with other version control systems. So he was my 911 for git trouble. Also some dude stole my bike parked in the San Jose caltrain station ūüėź

Week 3: Dustin was making fast progress with the first draft of Conductor ready and I was understanding the facebook infrastructure and network topology. I had the basic monitoring framework ready. Integrated with the conductor.

Week 4: More scrums, more lines of code. Trains, henchman, shunter etc .. Dustin had a major change in the architecture of conductor and the shunter was born. I was working on anti-spikes. Started working on torrents.

Week 5: More features added to freight and I had the basic throttling framework ready with anti-spikes. Started helping out Dustin with some conductor stuff. Voila! Merge conflicts and git said it was doing a three-way !

Week 6: We sort of had a basic freight working and running ( I remember we did a fist pump). A lot of work was done in process management and clean shutdown by Dustin. Blocking / Non-Blocking servers explored. Had fun. I was also working meanwhile on the first draft of PubSub. Couple of experiments, trying to figure out how to do a cheap check-sum of huge files. Took the weekend off to go to this amazing 127 hour like trek in grand canyon.  (pic below.. also I thought it would be cool to drop a beautiful pic to wake up readers who have reached so far)

Week 7: Dustin started work on PubSub, now that we thought Freight was stable and a we had this long debate on phabricator about one of Dustin’s diffs. It was so long that both our managers never bothered reading it and just asked for summaries in the next scrum ūüėõ

Week 8: Back to Freight, we started putting it to test with some small jobs, figured out a lot of changes need to be done. I discovered that transmission ( our torrent s/w ) had issues running stuff at the scale of our bandwidth. Ran a couple of tests and figured out what the problem was, source dived and fixed it ! It felt great.. was my first significant patch to a open source software. Co-incidentally, happened to be the one I had used countless number of times back in my undegrad for downloading what nots ūüėõ

Week 9: Ran first of Cory’s ¬†jobs. Discovered that more hacks need to be done for transmission ( the ones that cant be pushed upstream). Dustin spent some time working on stuff like peer-caching that was screwed up with our setting in transmission.

Week 10: More jobs for freight. We got couple of dedicated servers to host the service.. ¬†started work on making a front end( not fun at all, Dustin pitched in). Meanwhile, the conductor started freezing occasionally. Culprit being thrift’s python server implementation and python’s GIL causing performance bottlenecks. Mild architecture epiphany.. re design couple of things. ¬†Dustin was working on some bugs in the python thrift server implementation¬†were causing us problems .

Week 11: At this point of time, we had a mashed up version of freight running on the servers. It was a mash-up of hot fixes and the master was out of date for a while. Techincally, this was the last week we were supposed to push code. Making new changes to codebase in the last week was not encouraged.  Persistence was also achieved. Thank God !

Week 12: Last week .. team dinner at SF, final fixes to freight, running big jobs,  still trying to fix a lot of performance issues.. The last day, I realized another major performance bottleneck caused by transmission.. but hardly had time to source dive and think about fixing it . Meanwhile, got offered a full time position..  party time !! Also had to  take a flight to India the next day, showing up without any notice and surprise parents.

In all, it was exactly how I wished my internship would be. Dustin was a great guy to work with. Learnt tons of stuff from him. Though he was an undergrad, he had lots of experience working on stuff. At the end, we hated that we wrote most of it in python. Some small parts of it was written in c++ by  me, so I felt slightly better. One valuable lesson learnt was that, never write thread critical stuff in python .. stick to C/C++.

We touched a great deal of stuff in the course of internship .. the whole experience was overwhelming and Alex and Scott helped us a great deal in figuring out stuff.  We used to have daily scrums to discuss about the current progress and what has to be done next.  I had absolutely no time for anything else.  Rather, should say I didnt find anything else interesting in the course of the internship.

To cheer us up, facebook had a lot of intern events surfing, kayaking, go-karting, scaenger hunts, hiking. We also had a bbq party at Zuck’s house.

Throughout my internship, I never once felt like I was going to work. Its tough to call facebook working environment to be company-like. People go around the office in rip-sticks, have hackathons, work at crazy times on crazy things.. you could see hacks lying around everywhere, both in the code-base and the walls of the so called company. Every one works there because they feel so passionate about what they are doing. In my first week, I felt like every one around me was working 10 times faster than me. Took me a while to get used to it. We had amazing talks given by employees on programming. There was a series of c++ talks that just blew my mind off. I never thought about a language from such a perspective and I ended up knowing that the guy who gave the talk, sits few desks away from me and has a wiki page on him¬†! It was a fun working environment.. interns had their own weekly hackathons on thursdays.. pizzas would arrive sometime past midnight and we would gate crash into zuck’s acquarium ( his room.. more of ¬†a room with glass walls, that looks like an¬†aquarium¬†from the outside when its occupied).. no one had cubicles.. you could just walk to any guy and start a conversation with him.

Food .. need I say ?? Cafe X \m/ .. It was so hard on  me that so much was offered and I had to restrict  myself from grabbing everything and piling it on my plates. I used to bike to work everyday (16 miles up and down) and that seemed to take care of all the desserts consumed everyday.  We had a different cuisine for every meal and friday evenings we had happy hour in the bball court after the Q & A with Zuck in the cafeteria.

I am sure I missed mentioning a ton of other fun stuff that we did .. the timeline might be slightly skewed by +/- 1 week .. My patience to sit and write a blog post has taken a dip, I should say.

Building something from scratch, hacking well known opensource software, pushing code everyday, scrums, whiteboard discussions, learning tons of stuff .. I miss all that now, sitting at home. Cant wait for the next summer and I apologize for the sloppy blog post!

Written by Sathya Narayanan

September 2, 2011 at 2:06 am

Flash Player 10.2 Workaround to Copy Video Files

with 12 comments

You might wanna check ¬†this to get a context of what this post is about. With the previous versions of flash, I could get a copy of the videos being played in my browser by taking it from the /tmp directory. But with the new flash player, it deletes the file from /tmp after creating it. Why ?? I seriously have no ¬†<insert blasphemy > idea !!. Generally when your browser plays a video using flash player, a process would be fetching the video from the internet, storing it in /tmp directory with a hashed value and the browser tab would be playing it. But now, we need to get around a lot of shit, just to copy your¬†favorite¬†videos. You may ask me why am I so bothered to do this. Answer is that, I generally stream the videos when I have an ultra fast net connection(which doesn’t happen to be my home), store them in my comp and then later watch it on vlc. You might also use some youtube-dl or other addon…. But those generally work only with youtube. Say you had a video which was directly uploaded on facebook or other website, I am not sure if you can copy that. Enough of trying to convince ¬†you, that this is actually something interesting !! Lets get to the problem.

First, we may assume that the video was never saved on the disk. But who would be insane enough to ¬†put videos that size to MBs in your main memory ?? So it has to be on the disk somewhere. When the video is being deleted from the /tmp directory, it doesn’t mean that its actually deleted. Lets take a small de-tour to the OS ¬†basics. When two or more processes access the same file, they all get a file descriptor ¬†and when one process deletes the file, it doesn’t actually get deleted unless all the other process close their own access through their file-descriptor. And here, we have a process which is buffering the video into the disk and later deletes the file from /tmp and another process ( your browser tab) which is still ¬†playing the file. And voila !! We can still recover the video. Now this is no black magic !! You can see countless articles online about how to recover deleted files which are still open by some other process ( I think I have an a post on ¬†that somewhere in my blog too). But what complicates the things ??

  1. Browsers nowadays run each tab as a separate process. So identifying how to access the file descriptors of the particular tab playing the video is the first concern.
  2. I generally stream  multiple videos. So  I sort of need to write a script that finds all deleted video files from /tmp and copy them back.

How to do this ??

  1. Get the PID of all the tabs of your browser, in my case chromium (Google chrome’s momma).
  2. List the open files of all the PIDs we got from above  and see if any of them have a link to a deleted file, specifically in /tmp
  3. Process the output of lsof which also gives the file descriptor.

Here is a run through

sathya@Phoenix:~$ pgrep chromium

Now list the open files of all these process by appending them into a comma separated and grep them for deleted files with /tmp/Fl*. Assume for now that $PIDS has all the PIDs of the tabs with video open.

sathya@Phoenix:~$ lsof -np $PIDS | grep deleted | grep /tmp/Fl*
chromium- 4506 sathya   25r      REG        8,5 33356235   998384 /tmp/FlashXXK9gZKa (deleted)
chromium- 4506 sathya   32r      REG        8,5 10544205   998386 /tmp/FlashXXO0GYZw (deleted)

Now the column 2 gives the PID of the tab and the column 4 gives the file descriptor. I have opened both the videos from the same tab and  hence the PIDs are same. Now , the 4th column gives the file descriptor followed by the permission (which in this case is r (read)).  Now how to get to this fd ?? The /proc exposes the RAM which has all info about the processes to the user and we can get to the file using the file descriptor from there.

sathya@Phoenix:~$ cp /proc/4506/fd/25 ~/Desktop/GotItMF.flv

There !! We have the video file !!  Now all we need to do is, put all this together in a script, which I did.

for i in `pgrep chromium`
PIDS=`echo ${PIDS:0:${#PIDS}-1}`
export IFS=""
for i in `lsof -np $PIDS | grep deleted | grep /tmp/Fl*`
     PID=`echo $i | cut -d " " -f 2`
     FD=`echo $i | cut -d " " -f 6`
     FD=`echo ${FD:0:${#FD}-1}`
     cp /proc/$PID/fd/$FD ~/Desktop/$PID_$FD.flvdone
export IFS=" "

If you are too scared about meddling with the IFS, use the following which does some clever awk scripting.

for i in `pgrep chromium`
PID=`echo ${PID:0:${#PID}-1}`
lsof -np $PID | grep deleted | grep /tmp/Fl* | awk '{gsub(/[a-z]*/, "" ,$4)} { system("cp /proc/" $2 "/fd/" $4 " ~/Desktop/" $2 "_" $4 ".flv") } {print "Copied "$2"_"$4".flv"}'

Thanks to Hari for making me revisit bash strings and Jai for the AWKgasm !!

UPDATE : As Rik pointed out in comments, lsof with -n is much faster since a lot of time is wasted in host name lookup (which we obviously dont care abt).. -n makes it not convert the addresses to hostname. Makes it 10X faster !!

UPDATE : I recently switched back to firefox and to make it work on firefox, change the “pgrep ¬†chromium” to “pgrep -f libflashplayer.so”

Written by Sathya Narayanan

February 15, 2011 at 10:23 pm

The Indian Scenario

with 11 comments

The following is a random premature thought of what in my opinion is going wrong in India, which I am sure a lot of people think about.¬† Well I am¬† not gonna put on the Raja Ram Mohan Roy costume and lecture¬† about changing the society and¬† stuff ,¬† but I do¬† feel¬† that filling up a few bytes on the internet about this isn’t¬† a very bad idea either.

So, Whats wrong with India ??? I mean we are a body of 1.2 billion people constituting 20% of the world population. Which means out of every 10 guys on this planet, 2 of them are from India and¬† 2 are from China and the rest 193 nations of the world constitute the remaining 6. Yet we just have nine Noble¬†Laureates, and three of them are of foreign Origin. When we are 1/5 th of world population,¬†¬†no matter how screwed up you were with¬† your 10th grade probability calculations, its not a tough job to figure out that Indians must be in top of everything.. not just population. But why is it that, when u take any field, its tough to find an Indian doing better at it ?? I am not talking just about Academia or Research for which every Indian’s instant response is the two worded phenomenon – “Brain Drain”.¬† Lets talk about sports.. How tough is to find a runner who can at least fetch a bronze medal in Olympics. Why is it that when we have Hockey as our national game, a majority of Indian kids don’t even know the names of the players(not like I know any of them). But every one does know that Sachin Tendulkar did start playing cricket at the age of 16 and has blah blah runs in his pocket and runs a restaurant in Mumbai. If we are so obsessed with cricket, why don’t we bloody change our national sport to Cricket !!!. Whats stopping us from doing that. Oh wait !! we are a democratic nation run by a bunch of corrupt politicians who roam around like zombies in wheel chair.

Population :

Population IS THE biggest feature of India. As a strength, it gives us diversity. You can go all around the US and  you will pretty much get bored of the same burger and pizza and the food varieties and culture it throws at you. Now back in India, you walk a few kilometers out of your district, you might actually end up in a totally different culture.. A different sect of people who talk a different language,  who listen to a different music, eat different food, have a whole array of local cinemas and cine-stars.

I sometimes get curious as to how India ended up with such a big population. I mean how ?? May be the so called Gods had such a plan you know, to stuff maximum number of people into this country and¬† to top it , they had to make the whole country¬† HOT and TROPICAL that when Mr Software-Engineer returns from work in the evening, he has to deal with both the HUGE traffic and import supernatural powers to tackle the hot weather outside. At the end of the day, all that is on top of anyone’s mind is¬† how to survive in this place. At times we even manage to have entertainment amidst all this and that’s why people all over the world respect Indians. They can almost survive any bullshit thrown at them. They are cheap labor who don’t mind sweating it out because compared to what they were doing back at home, anything that you throw at them sounds like luxury.

If India has a 1 billion population, it also has¬† an equal¬† number of ¬†divine supernatural entities aka God. At times, you even wonder if they count the ¬†number of Gods into the population stats. And Sachin Tendulkar does contribute under both the human and ¬†divine category to the population stats.¬† How are you even supposed to survive in¬† a country with so much population??.¬† On one side, having a diverse population is a positive thing. But what actually happens in India is a different story.¬† Lets do a simple math. Say X guys were supposed to live under a km-square of area and it takes Y units of money to provide all necessary amenities to that area. Doesn’t it make sense that with the X very high in India, it would actually be easier to develop infrastructure and facilities ?? No .. that doesn’t happen because, One – we don’t pay our taxes properly. We just love painting our money black. And two – we cause more damage to infrastructure than the amount we contribute to build that.¬† And the moment we step into a foreign nation, we don’t spit,¬† we don’t urinate in public. We suddenly get a public sense of responsibility. We enforce that because every one else around you does so. In India, if you don’t cause any public nuisance, you get a nice portrait of yourself¬†¬† in the “Hall of Clowns who follow rules”.¬† And nobody wants to change. We just continue living our lives putting up with all of the crap happening around us and contribute to it.

The Indian Rat Race at Schools  :

Lets get back to where all the talented people in India go to ??. The first thing that you notice is, every body in India wants a shortcut to success. By success we mean making more black money. How many of us actually think about living the life on our own ??. Half the Indian kids grow up listening to their parents. The parents mold¬† their kids into who they wanted to be when they were once in their childhood. Its¬† a sort of a pipeline where your dad wants you¬† live his dream and you want your kids to live your dream. You just throw your dreams onto your kids.¬† “Here kiddo, take all this books and digest them and become a doctor., Because, I once wanted to be a doctor and I didn’t have¬† facilities back in my childhood to achieve that.¬† And this will also make you rich and powerful in the society”.¬† I agree on the fact that parents have the responsibility to teach their kids of how to¬† become successful. But going a step further and choosing how they do it and in what field they do it is an over kill. And hence every kid gets lectured to be an engineer or a doctor. No one advices their kids or even talks to them about other areas like sports, literature, science, history,¬† economics, politics, art etc. Those fields are for kids who missed writing their 12th std board exams or those who forgot to bring a pencil to exam hall and lost out on ¬†marks because they didn’t manage to underline the important points in their answers.

I feel like kids of this age are stereotyped into doing things. Each mom compares her kid with the other kids in the neighborhood and lectures him/her about how they need to take them as role models and score better marks in the next exam. And thus the rat race of engineers¬† begins with a big fat salary cheque on the other end. So most¬† of¬† the kids never get to¬† explore the opportunities in the other fields. A lot of talent actually comes ¬†out in this Rat Race. Thats why when you go abroad and see a graduate student struggling with calculus, you just¬†realize¬†how great the Indian Schooling was. But most of these kids who could do calculus by just hand waving, don’t actually end up as successful engineers compared to what ¬†US produces as undergrads and graduates. Thats just because once the Rat Race is over, we all take a well deserved rest during our undergraduate period. In the contrary, students abroad have a steep learning curve during Undergrad and graduate courses. Which just doesnt happen in India. Thats why even though we have a very successful schooling system, we have a very bad undergraduate experience ( IITs in my opinion can be exempt). There are several perspectives to this but lets rest on this.

Why Brain Drain??

So we only manufacture engineers in our country and what happens next ??¬† All these bright engineers run away to better places in the world because, those¬† not-so-talented kids who chose¬† the other important fields that constitute a society, do a bad job in their field and make the place a hell to live. To become a developed nation, India doesn’t need top notch computer scientists, but it needs talented people in the other fields who make India a better place to live. You need a good politician, a good economist a good city planner, ¬†so on. But no one wants to choose these because they are poorly paid or have no clear shot at success in Indian terms and scenarios. So why not give these guys who do the less interesting but important jobs a better future. Lets lure the talented people into these fields.¬† One of my¬† batch mates¬† who graduated with an Instrumentation¬† & Control Engg degree, decided that Politics was the field he wants to be after he joined a company for work.¬† He even went on to enroll for some correspondence courses in politics as he works. Now if some one had put a thought in him when he was a kid that a politician was some one whom the society respects and needs,¬† he wouldn’t have wasted 4 years in NIT Trichy attending crappy engineering classes.¬† We would have had a¬† good politician who could have run the CWG without any corruption or not cause $36 Billion ¬†loss to Indian government over the spectrum allocation. What India needs is not eradicating brain drain and stopping the talent from going outside the country. But¬† what India¬† needs is to make sure that¬† a fair share of the young talent pool goes on to contribute and develop the¬† other¬† fields that contribute¬† to the society. And of course yeah better politicians.

PS : All of the above are  random thoughts feel free to smirk them off if you are not in the mood to listen.


Written by Sathya Narayanan

November 18, 2010 at 12:55 am

One month into the Grad Life !!

with 8 comments

Its been more than a month into the Graduate Life at Univ of Wisconsin Madison for me. The first thing I felt missing from my Indian life is Time.  Back in India, at times, I was pretty much jobless that I would stay up late in the night and discuss some arbit crap with friends on the garnet lawn or watch some TV Series. I actually ended up watching 18 TV series as a part of my Undergraduate Life :P. But once course work started, it was a round robin between Assignments, Cooking, doing groceries and getting used to America !!.

I am not sure if this applies to all other universities in US, but Wisc-Mad needs you to only enroll for 6 credits per semester if you are a TA. Hence my calendar initially was only filled on Tue and Thurs for a couple of hours. And once they let in TA allotments, it all turned out to be messy. I finally managed to squeeze all TA and classwork commitments to just Tuesday, Wednesday, Thursday. ¬†So the long weekend starts on the Friday and lasts till my alarm rings on the next Tuesday morning at 8.00am :P. ¬†Needless to say, there are assignments to be done or some or the other thing that comes up. ¬†Life here is awesome. You have people who are very sweet and polite to you.¬†Doesn’t¬†mean ¬†they actually are in real life. Its just the thing that they have a sweet outer-cover. So you walk into the street full of people, you ll end up with a lots of face gestures wishing you ¬†a good day. In particular, in one of the invited talks, a guy from west coast mentioned that people in Madison are super sweet.

Fall is beautiful. Just a pleasant thing to watch. I was usually this person in India who rarely manages to step outside the 5m invisible ring around a laptop. But out here, the best thing I like to do is go and sit in front of the lake. Madison has two beautiful lakes and since my university and Madison city are a part and parcel of each other, ¬†I am in close proximity to these awesome buildings and places of Madison. The state capitol which looks like a replica of the White House is few minutes walk from my apartment and department. And hence each and every day I go to dept, I feel like I am actually going to work. Doesn’t give you this feeling of you walking inside a campus where every one you see is a student or a person with academic ties.

Couple of funny things I encountered here :

  1. Wanna meet a prof ??, No problem. Just ¬†walk in … Wanna have your hair cut ?? ¬†get an appointment beforehand and be there on time. And beyond that, most of the saloons have women cutting your hair and they charge you anywhere between 10 to 20 dollars for a simple haircut. Taxes are also to be paid btw :P. And I had to pay 16 dollars (and ideally supposed to have followed the golden-20%-tipping rule) for just ending up in an¬†Identity¬†crisis at the hands of a woman in a span of just few minutes. The lady who cut my hair was actually perplexed when she heard that hair cut at India cost me like 75 cents.
  2. Cycling is a lot of fun. And you have a separate bike-lane here everywhere. The whole city is actually very scenic to cycle.
  3. Ever since I came to US, I have always carried only 5 dollars in my wallet as Cash. And to this date, I still¬†haven’t¬†used it at all.
  4. To all those who sit in India and act like this crazy calculator that converts ¬†their friends/spouses/kid’s salary in dollars to INR, you¬†wouldn’t¬†be willing to even buy a small candy if you keep doing that over here before spending.
  5. Temperature plays a see-saw most of the time here. It might happen at times, that you see a clear and sunny sky out there and wanna go for a bike ride. You might still end up biking at 8 degree Celsius. Happened to me. So trust me :P.
  6. You gotta figure out clever ways to get your TV episode if you don’t wanna end up missing it or watching it in a tiny 400×300 pixel in your browser.
  7. People take your word for granted here. If you say you dunno who is responsible for this big chunk of data downloaded from your phone eating up your data plan, they just push you up to a bigger data plan free of cost :P.
  8. Beyond seven, anything is not funny at all. So I stop here ūüėõ

Well there actually are many interesting things to put up here, but all that comes to my mind right now is this Lattice framework for compiler optimization which I cant get a hang of.

Written by Sathya Narayanan

October 4, 2010 at 10:06 am

Finding Invisible Friends and Dynamically Updating Your Status in Google Chat using XMPP Scripts #linux #python #ubuntu

with 24 comments

Read on, only if you Wanna do any of the following :

  1. Update your  google Chat status with info dynamically or fetch score from a website and display it every minute in your status msg.
  2. Find invisible friends in your chat list.
  3. Wanna change your status message as soon as some one signs on ?? aka make your own Buddy Pounce.
  4. Wish all your contacts by running a simple prog on important days. Like say, I wrote a script that wished ¬†my ¬†friends “Happy Friendship Day” when they appeared online ūüôā .
  5. Get a step closer to writing  your own back end for an IM client.
  6. Wanna remind someone about something over chat at a particular time or period when you might not  be online.
  7. And lots more that your imagination can limit.

Warning : I dont promise that everything in this is right. There might be ¬†hell lot of mistakes. Feel free to point them out in the comments ūüôā

Being someone who does a lot of IM and chatting, I was more curious about how a web chat works. How is it that you are instantly notified of your friend’s online presence and know when he is about to type something on your chat window. A little more digging and reading a couple of sites on XMPP ¬†helped me figure out what I have been wanting to do a long time back in College. What basically you need for having a chat with your friend miles away is an XMPP Server (eXtensible Messaging and Presence Protocol)¬†and an authenticated connection established to it. And of course an IM client (like google talk or pidgin or empathy) unless you wanna write your own command line scripts using some Python API for XMPP.

Firstly, I m not a big fan of gtalk, just cos it doesnt exist for Linux platform and who the hell uses an IM client just for google. An IM client is something that maintains all your chat accounts across various domains that u have a authentication on. An XMPP server listens to your xmpp requests on port 5222 and 5223. To give an idea about how the whole thing happens, imagine an XMPP server with which Alice and Bob have established their authentication. Unlike VOIP, chatting is done ¬†with a man in the middle which happens to be the XMPP server. When Alice wants to send a IM(Instant Message) to Bob, Alice sends it to the server and the server sends it to ¬†Bob. This sending and receiving is done in the form of XML packets. An XML stream runs between the Server ¬†and Alice. A similar stream runs with Server and Bob. Whatever Alice puts on the XML stream is received by the server and the server reads the XML packet and looks at to whom the message is intended for and the server puts an XML packet to Bob’s XML stream and hence Bob receives the message Alice sent :). All this happens each time ¬†u type something on ur IM client or change your status. Now this is ¬†all about messaging. What about the status notifications and stuff ?? Thats where the presence part of XMPP comes to play. Basically a stanza in the XMPP stream between the server and the client can be etiher a Roster/Message/IQ/Presence. A Roster is used to notify the client about who all are online and registered with the server. A Message as I said above is a stanza that is used to communicate from one client to another through the server. A IQ is Info/Query where in u get specific info about a user. And finally a Presence is used to notify your type of availability to the other users (like busy , dnd ¬†, away and put a status). This is the view of the XMPP ¬†like how say you might see earth flying from an airplane . Each of these have their own specifications and finer details. For more, if you are interested, google it up .

When google opened a subdomain aka talk.google.com, users (the geeky ones) were able to authenticate and start chatting even before google officially announced that it has actually started an IM service :). ¬†This happened back in 2004 when google first started the talk.google.com subdomain. Since then it kept on adding features to it chat client and led to group chat and stuff. But as I see, gtalk is not even close to an IM client. Its specific to google and has very limited¬†capabilities when compared to pidgin or empathy. I am not talking about just using multiple accounts but about stuff that you can do with just your google accout like say control the presence notifications, add a¬†¬†buddy pounce n stuff :). Well I can throw in more info with all my current enthu ¬†about xmpp but I ll stop here and break straight to the part of writing cool scripts to do stuff with your google Chat that you cant do with your gtalk ūüôā

Stuff that you might need : Python interpretor, xmpppy library for python. If you are in Linux , install python-xmpp and python-dnspython packages and you should be clear to go. If you are on windows, Keep breaking your head ūüėÄ

The first script I am gonna show here is how to send a message to a friend. This is how I started it. You might wanna cross check with the documentation here.

import xmpp
import getpass

passwd = getpass.getpass("Enter Password: ")

client= xmpp.Client('gmail.com',debug=[])
message = xmpp.Message(to,msg)

A quick look should tell you that Lines ¬†1 and 2 import the necessary modules. The getpass module helps you to enter the password like the way you would enter it on a computer. Otherwise, you would have to put it somewhere and expose it (even better ways to do this are welcome in comments). I am declaring the variables in Lines 4,5,6,7. Line 9 , I create a xmpp Client with the parameters. When u specify the debug=[] option, the debug messages dont get thrown at you in the terminal. You connect to the server on port 5223 in Line 10 and authenticate in ¬†11. Line 12 sends a Roster request and Initial Presence notification. You construct a message packet in the Line 12 and set its attributes in Line 13 and send it to server in Line 15 ūüôā .

The next one I worked on was a script to dynamically update my status message with something like a clock or a cricket score. I am gonna leave to US on Aug 19th and thought how cool would it be to put a countdown on my status that updates how much time is left every minute(First I set it up every second, but it gets irritating to chat on the other end unless you have a very patient friend ūüėõ ).

import xmpp
import time
import datetime
import getpass

passwd = getpass.getpass("Enter Password: ")

client= xmpp.Client('gmail.com')

	diff = datetime.datetime(2010, 8, 19,4,40,00) - datetime.datetime.today()
	minutes=str(diff.seconds/60 - (diff.seconds/60/60 * 60))
	seconds= str(-seconds1+(diff.seconds))
	timeleft=days+" Days, "+hrs+" Hours, "+minutes+" Minutes more !!"
	pres=xmpp.Presence(priority=5, show='chat',status= timeleft )

Lines ¬†upto 12 are self explanatory . I then start a while loop and calculate the difference in time from the current time to the time when I ll be boarding my flight. The diff gives back a structured variable with diff in days and seconds. The seconds difference is done mod 86400 (ie no of seconds in a day). So make appropriate calculations to count the days, minutes and hours and seconds left in lines 15-20 and I construct a XML Presence stanza by calling the xmpp.Presence() and send it on to the XMPP server. After this, I pause the loop for 60 seconds in Line 24 to send the next update packet after a minute. You could do something like put up Live cricket scores ¬†and stuff here to keep your friends informed or put a horizontally scrolling status msg or whatever that comes to your mind !! ūüôā .

The next script that I have is a script to find invisible friends in gtalk ūüôā . ¬†This I find pretty much useful. I used to do the other way around (that is to make myself invisible from pidgin by sending an XMPP req) ¬†cos Pidgin’s Inivisible status doesnt work well. I found the info here when I was googling it back in my coll. Basically I put the following XML snippet in my Pidgin’s XMPP console to make myself go invisible.

<presence type="unavailable">

So I try to do the reverse of this from the Roster I get from the server and that should pretty much give the list of people who are invisible on gTalk :). Which I do in the following python code.

import xmpp
import getpass

user = "sathya.phoenix"
passwd = getpass.getpass("Enter Password: ")
server = "gmail.com"


def check(connec, event):
   if event.getType() == 'unavailable':
     print event.getFrom()

client.RegisterHandler('presence', check)

while client.Process(1):

Lines 1-10 again do the same authenticating and stuff while in Line 11 , instead of just sending the initPresence , we request for a Roster which gives us the status info of all my friends who have registered with the XMPP server. Now here is the magic. Even though you are invisible , your client registers with the XMPP server and sets the status as unavailable which the other Clients detect that and put you under the offline category(those who havent registered). ¬†So all we got to do is to check for which of the registered users in the Roster are having a status “unavailable”. Pretty simple thing to break someone’s privacy huh ūüėČ . ¬†So we have a handler to handle the presence notifications in the Roster and display the gmail ids that have status as “unavailable” and have registered with the server. ¬†Now only disadvantage is that, if your friend is invi , he sure ends up on this list, but not vice versa. There are a couple more to check for that as well :).

So you could write handlers that can do a lot of stuff for you. Like you can send wishes to all online contacts by calling the client.send() in Line 15 and remove the if statement on Line 14. Or when some buddy of yours comes online , you can change your status message to soemthing else . Guess you can figure all that out by looking at the scripts above and combining them and making suitable modifications. Its all exciting !

PS : Any more cool ideas to do, put them on comments. Will try out ūüôā Also check this out if you wanna insert code snippets in your blog the easy way ūüôā

Written by Sathya Narayanan

August 3, 2010 at 11:25 am

The Hyderabad Weekend

with one comment

The last time when I visited my cousins at Hyderabad was sometime around my First year. So I planned a small trip with mom over the weekend and it was one of the most exciting weekends for me after coll ended :). Not to forget the adventurous trip on the way back :P. Things that I did :

  1. Did a lot of shopping with cousins. Thanks to Kalapana and Kavitha akka, now 80% of my shopping work is over. We were right on time for the Sales at Lifestyle at In Orbit. I felt like I really did shop something for the first time in my life  (I never did any shopping till I was out of College :P).
  2. Met Srinivas, Abhinav and Mrinal at CCD near Hitec City. Abhinav has become like a hindi film star, mrinal is on race with Ankit to gain weight and Srinivas is all the more fair and handsome ūüėõ . Had a nice time with them though had to run back home quickly as one of ¬†my cousins had to catch a flight to Chennai that evening.
  3. The new airport Ring road is Awesome ! 8 lanes and u can easily go on 140 kmph ūüôā
  4. Went for a Rock and Jazz concert at Mariut Convention Center. This was my first concert if u dont count those we had in Coll during Festember and all. Was pure fun and was a pleasure meeting some of my cousin’s close friends :).
  5. Karthik picked me up on the way back at 12 in the night and went to his home :). NR and co were already there and Naga sir arrived early in the morning ūüėõ .. Tirth too came all the way at 1 in the night ¬†and we had a nice chit chat till around 3 in the morning :). Not to forget the Zaveri call and the Hyderabadi briyani at some 2 in the night ūüôā
  6. Spent the whole of next day at cousins place. Beautiful house .. .given that my cousin is an Interior Designer herself :).  I was always scared of dogs as pets. But I found  Deva aka the Lazy Labrador very very friendly. He nibbled me a couple of times and that was it. Such a cute one :).
  7. Went for lunch with family at ¬†Mainland China. Went to the same one at Banglore few weeks back with Suren and co and guess what I am back :P. This one in Hyd had a better Chineese¬†ambiance¬†and the food was better too ūüôā
  8. Got a Leather jacket on the way back and visited a Harley Davidson Show room. Some day , I shall own one ūüôā
  9. Went on a Ride on the New Royal Enfield Classic 500 that Ramesh uncle got . It was so stud !! I always felt that bullets were more appealing to me than bikes(even the racing ones.. earlier on the same day , balu let me drive his R15 ).
  10. Got back on a train and reached Chitoor only to find that I am supposed to cross Andhra border and reach my home ¬†through some weird way cos it was a State wide Bandh for Chandrababu Naidu’s Arrest. And this area seemed to be a TDP stronghold and they blocked all roads that led to city outskirts and TN border. Not even Cars were let to passby. Only emergency vehicles and 2 wheelers were allowed to pass. Had to take 2 autos to cross 2 such barricades ( the auto guys were also afraid to go cos they were asked strictly not to take on public transport). Cost me some 350 bucks to cross some 12 kms. On the last barricade at the check post at TN border, nothing worked and the auto guy also gave up. We walked all the way across the border with those heavy shopping bags. Luckily some nice guy gave me and my mom lift on a Hero Honda ( yes !! ¬†we had no other option left ). So Triples it was. Me and my mom clinging on to that¬†metallic¬†fixtures on the back so that we dont fall off :P. Some how reached home after some 5 tensed hours ūüôā

A great weekend in all … A memorable one too . Thanks to my cousins ūüôā

Written by Sathya Narayanan

July 20, 2010 at 9:50 am

Posted in general, Its my life, Trips