Evanescent Thoughts

Evanescent Thoughts

Facebook Internship

leave a comment »


Statuatory warning : Bear with me if this post is long or boring. Its been a long time since I blogged.

This summer, I interned at Facebook, Palo Alto. It also happened to be my very first experience outside academia, writing code that actually goes to production. I got the offer about a month after I started my graduate studies at Univ of Wisconsin, Madison. I had just moved into the US (late august, 2010)and there was a career fair around that time which I missed attending. A couple of my friends got internships from Qualcomm, but I decided to take some time to settle and then start looking for internships (after all, I had just moved in). Around mid september, facebook had come to college for a giving a talk about memcache. It happened to be that the person who was giving the talk  was a alumni of NIT Trichy and UW Madison. They asked us to forward our resumes if interested and I did. Within few weeks, the recruiting team was on campus again and they had a couple of interviews before offering me an internship. I was so excited that my first interview experience landed me with an offer and to top it off, it was FACEBOOK.

The internship was for 12 weeks, a very competitive offer that I didn’t consider sitting for any more companies. Besides google started their process very late (not that I would have been interested in taking it if I had been offered).  Started my internship on May 23rd. I worked for the Application Operations team under Scott and Alex,  with another fellow intern, Dustin (Alex’s intern). We worked on building a service called Freight  and later another thin layer of abstraction on top of it called PubSub (a brief report about it here) . It was an amazing experience to build something from scratch, have daily scrums, work agile and learn a shit load of new things, and the most important of all, to live and work in the the reality. Here goes a brief description of what happened in the next 12 weeks.

Week 1: Orientation, get your mac books and iPhones ( yes .. interns get brand new iPhone 4), get your dev environment setup, meet with your manager, discuss about the internship. We had a full team meeting discussing about Freight and what problems it should address. I should say Week 1 had my head in a whirlwind. A lot of things were pretty new for me. Though I was a little bit scared, I was happy that I would in the worst case, learn a LOT for sure.

Week 2: Started playing around with rsync and it was wiki time. Did a  _very_ dirty hack to rsync by mid second week to make it work for our end goal. It was so dirty that I dare not speak about it. But it works brilliantly!  Spent most of  the time saying hello to git. I had never worked much with any version control systems and I should say I was confused for half the time. But in the end, everything made sense. Dustin could relate to git since he had experience with other version control systems. So he was my 911 for git trouble. Also some dude stole my bike parked in the San Jose caltrain station 😐

Week 3: Dustin was making fast progress with the first draft of Conductor ready and I was understanding the facebook infrastructure and network topology. I had the basic monitoring framework ready. Integrated with the conductor.

Week 4: More scrums, more lines of code. Trains, henchman, shunter etc .. Dustin had a major change in the architecture of conductor and the shunter was born. I was working on anti-spikes. Started working on torrents.

Week 5: More features added to freight and I had the basic throttling framework ready with anti-spikes. Started helping out Dustin with some conductor stuff. Voila! Merge conflicts and git said it was doing a three-way !

Week 6: We sort of had a basic freight working and running ( I remember we did a fist pump). A lot of work was done in process management and clean shutdown by Dustin. Blocking / Non-Blocking servers explored. Had fun. I was also working meanwhile on the first draft of PubSub. Couple of experiments, trying to figure out how to do a cheap check-sum of huge files. Took the weekend off to go to this amazing 127 hour like trek in grand canyon.  (pic below.. also I thought it would be cool to drop a beautiful pic to wake up readers who have reached so far)

Week 7: Dustin started work on PubSub, now that we thought Freight was stable and a we had this long debate on phabricator about one of Dustin’s diffs. It was so long that both our managers never bothered reading it and just asked for summaries in the next scrum 😛

Week 8: Back to Freight, we started putting it to test with some small jobs, figured out a lot of changes need to be done. I discovered that transmission ( our torrent s/w ) had issues running stuff at the scale of our bandwidth. Ran a couple of tests and figured out what the problem was, source dived and fixed it ! It felt great.. was my first significant patch to a open source software. Co-incidentally, happened to be the one I had used countless number of times back in my undegrad for downloading what nots 😛

Week 9: Ran first of Cory’s  jobs. Discovered that more hacks need to be done for transmission ( the ones that cant be pushed upstream). Dustin spent some time working on stuff like peer-caching that was screwed up with our setting in transmission.

Week 10: More jobs for freight. We got couple of dedicated servers to host the service..  started work on making a front end( not fun at all, Dustin pitched in). Meanwhile, the conductor started freezing occasionally. Culprit being thrift’s python server implementation and python’s GIL causing performance bottlenecks. Mild architecture epiphany.. re design couple of things.  Dustin was working on some bugs in the python thrift server implementation were causing us problems .

Week 11: At this point of time, we had a mashed up version of freight running on the servers. It was a mash-up of hot fixes and the master was out of date for a while. Techincally, this was the last week we were supposed to push code. Making new changes to codebase in the last week was not encouraged.  Persistence was also achieved. Thank God !

Week 12: Last week .. team dinner at SF, final fixes to freight, running big jobs,  still trying to fix a lot of performance issues.. The last day, I realized another major performance bottleneck caused by transmission.. but hardly had time to source dive and think about fixing it . Meanwhile, got offered a full time position..  party time !! Also had to  take a flight to India the next day, showing up without any notice and surprise parents.

In all, it was exactly how I wished my internship would be. Dustin was a great guy to work with. Learnt tons of stuff from him. Though he was an undergrad, he had lots of experience working on stuff. At the end, we hated that we wrote most of it in python. Some small parts of it was written in c++ by  me, so I felt slightly better. One valuable lesson learnt was that, never write thread critical stuff in python .. stick to C/C++.

We touched a great deal of stuff in the course of internship .. the whole experience was overwhelming and Alex and Scott helped us a great deal in figuring out stuff.  We used to have daily scrums to discuss about the current progress and what has to be done next.  I had absolutely no time for anything else.  Rather, should say I didnt find anything else interesting in the course of the internship.

To cheer us up, facebook had a lot of intern events surfing, kayaking, go-karting, scaenger hunts, hiking. We also had a bbq party at Zuck’s house.

Throughout my internship, I never once felt like I was going to work. Its tough to call facebook working environment to be company-like. People go around the office in rip-sticks, have hackathons, work at crazy times on crazy things.. you could see hacks lying around everywhere, both in the code-base and the walls of the so called company. Every one works there because they feel so passionate about what they are doing. In my first week, I felt like every one around me was working 10 times faster than me. Took me a while to get used to it. We had amazing talks given by employees on programming. There was a series of c++ talks that just blew my mind off. I never thought about a language from such a perspective and I ended up knowing that the guy who gave the talk, sits few desks away from me and has a wiki page on him ! It was a fun working environment.. interns had their own weekly hackathons on thursdays.. pizzas would arrive sometime past midnight and we would gate crash into zuck’s acquarium ( his room.. more of  a room with glass walls, that looks like an aquarium from the outside when its occupied).. no one had cubicles.. you could just walk to any guy and start a conversation with him.

Food .. need I say ?? Cafe X \m/ .. It was so hard on  me that so much was offered and I had to restrict  myself from grabbing everything and piling it on my plates. I used to bike to work everyday (16 miles up and down) and that seemed to take care of all the desserts consumed everyday.  We had a different cuisine for every meal and friday evenings we had happy hour in the bball court after the Q & A with Zuck in the cafeteria.

I am sure I missed mentioning a ton of other fun stuff that we did .. the timeline might be slightly skewed by +/- 1 week .. My patience to sit and write a blog post has taken a dip, I should say.

Building something from scratch, hacking well known opensource software, pushing code everyday, scrums, whiteboard discussions, learning tons of stuff .. I miss all that now, sitting at home. Cant wait for the next summer and I apologize for the sloppy blog post!

Advertisements

Written by Sathya Narayanan

September 2, 2011 at 2:06 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: