Intelligent Tutors, Robotics and Fun: RISS’18 @ CMU

I’ve (Shiven Mian, B.Tech. Class of 2019) had the privilege of working with the Robotics Institute at Carnegie Mellon University (the world’s largest Robotics facility) for the past 9 months, and spent Summer 2018 (June – August) at CMU’s sprawling campus as a FICCI Research Scholar in the Robotics Institute Summer Scholars (RISS) program (where I was one of 35 undergrads selected around the world). I’m working with Dr. Jack Mostow (Research Professor Emeritus, CMU) on the RoboTutor project (formerly Project LISTEN), which is one of the five $1M Global Finalists in the $15M Global Learning XPRIZE Challenge, funded by Elon Musk. The project is an Android-based Intelligent Tutoring System that enables 7-10 year old children, with no access to schools in remote regions, to learn basic literacy and Math without adult assistance. RoboTutor is currently being beta-tested in Tanzanian villages, and its design as well as the Challenge’s aim could potentially enable millions of deprived children in the future get a basic quality education in their formative years.

If you’d told me 10 months ago I would be writing all this now, I’d have laughed. Seriously. Three months on, the experience still seems surreal today – so this blog would also be a bit long.

I. Applying

Having dabbled with various different fields in the past two and a half years, I really wasn’t sure where to spend my summer experience this year. Though very broadly I had done projects involving Data Science and HCI, I didn’t have one particular area of interest and was mostly versatile. What I did always want though, was that my work should involve (or analyse) interaction with people, and hopefully impact their behavior and lives positively in some way, and I’ve abided by it in most of the work I’ve done during this time. So when I came across RoboTutor and XPRIZE (through a friend on Facebook), I checked if my interests were a match and spoke to a few people in the team, and I realised this is exactly the high-impact social good project I’ve always wanted to be a part of.

Things went uphill quickly from there. I directly emailed Dr. Mostow for applying to RoboTutor, and a couple of interactions later, I started working remotely in Feb 2018, till May. My remote work involved re-designing and developing RoboTutor’s login system for children that utilised faces, streamlining the way we collected logs from the beta test sites, and some server-side work. I also heard about and applied to RISS in the meantime on his suggestion. It’s a fairly standard application – like most others, it requires only an SoP, CV, transcripts and LoRs, along with mentioning what your interests are among the areas RI does research in. Luckily I was one of the 35 selected worldwide in April. To make things sweeter, I also received funding from FICCI for pursuing RISS with RoboTutor. It was pretty flattering for me to be contributing to a project of such massive real-world impact, but being selected in one of the best undergraduate research programs at a place like CMU and being funded for it, however, was well beyond what I had expected. Little did I know that this wasn’t just a research internship – it ended up showing me fields I had never known existed, and impacted my future plans and interests more than anything else I had experienced.

(Cohon University Center, CMU. Glad it wasn’t Winter)

II. Work @ RISS

Since I’d already been working with the team for 4 months, so once I got to CMU in June, I didn’t have much to do to know all the main people in the RoboTutor team, or the massive system architecture RoboTutor had. However, I did work on a different area of RoboTutor for RISS. Me and another fellow RISS scholar, Mononito Goswami, worked on a new analytical tool for aiding RoboTutor’s design improvement process, which we dubbed SPOT. SPOT uses student-interaction data logged by RoboTutor, screen capture videos and AI to automatically detect ‘hot spots’ (major design issues) in the app which are most detrimental to student engagement and learning. The tool then provides designers & developers of RoboTutor with video evidence and causal characteristics of those hot spots – helping them with deriving actionable intelligence to improve RoboTutor accordingly (an example of data-driven iterative design). After getting feedback from Dr. Mostow, the rest of team, and a few other people (especially Ken Koedinger) from CMU and the week-long LearnLab Summer School which we attended, we managed to co-author a working paper on SPOT (for CMU’s Working Papers Journal (WPJ)). We also managed to submit our paper to a top-tier conference. Though this was a great summer outcome, seeing videos of children in our beta sites happily interacting with the things we designed and improved was very satisfying. We knew were contributing to a better learning experience for them, and in future hopefully millions of children. (Update: Accepted to AAAI’19 as a Student Paper 🙂 )

(Part of the RoboTutor Team @ LearnLab. Clockwise: Dr. Mostow, Amogh, me, Corrie, Rebecca, Mononito, Qiwei, Nirmal. Gates-Hillman Centre, CMU)

III. Beyond Tutors

RISS for me wasn’t just a foray into the world of Intelligent Tutoring Systems or Educational Data Mining, it was far more holistic. I am fortunate and privileged to have met some of the brightest peers, and the small cohort size led to us really bonding together as a group. We learned and helped each other in our projects, organised study groups, peer-reviewed each other’s WPJ papers, toured labs, attended RSS’18 (and heard Sergey Levine talk about his work), and even shared lab space (props to Viraj, whose lab space I shared frequently). Being new to many aspects of Robotics, I particularly enjoyed touring my peers’ labs at RI, especially the FRC and the MSL Lab – some of the work being done in the labs was just amazing and futuristic, the kind of stuff I previously saw in those short this-is-the-future videos on YouTube. Lunch talks by RI faculty were hosted every Wednesday as well. Personally, the very first lunch talk (hosted by Red Whittaker) was one of my summer’s highlights because of the impact it left on me (he had spoken about his journey in Robotics and lessons from life), and it was one of the most motivational and insightful talks I had ever heard.


(Left: The Search Based Planning Laboratory (SBPL); Right: one of the ballbots at the MSL Lab – the first ballbots ever built)

But the most fun we had as a cohort was in the Robotics workshops. There were three workshops: on ROS (organised by NREC), drones (by DJI) and humanoids (by UBTech). All the workshops were 2-3 days long, and involved full-day tutorials and a one day team competition in the end. It was really exciting to work with the drones and the humanoids, get acquainted with their development environments, and the limited time for the competitions enabled our teams to stay up at nights dabbling with them. It was an amazing experience, and some of us did walk out with some cool stuff to say the least.

(PS: we own them now 🙂 )

RISS also gave us a lot of opportunities to share our work. Apart from the poster session at the end of the program, the lab tours and the WPJ, there were multiple events organised over the summer to share our work with others. I was fortunate to have been one of the panel speakers at the AI4ALL Summer School at CMU, where I interacted and shared my work with RoboTutor with some of the brightest high school STEM students in the US. All in all, seeing the work done in the labs, attending the talks and the workshops, and moreover giving back and sharing what I learned, all massively enhanced my interest in Robotics. It also enhanced my perception that the Robotics Institute (and CMU) is one of the most inclusive research institutes in the world, and more importantly, a place without limits. It truly is.

(AI4ALL Panel Discussion. “How did you get started with AI?”)

IV. The Fun

While it may seem all the various events we had literally every week took up all the time, the 10 weeks at RISS were nothing without the fun. Apart from Pittsburgh (where I was based), I was lucky to get time to explore most of the East Coast – I visited Philadelphia, Washington DC and New York City (which was particularly awesome because I visited on the July 4 week).


(Left: Times Square; Right: Fireworks Show over the Hudson River. New York, July 4th 2018)

But the fun I had with the RISS cohort is what I cherish the most. Whether it be the late night parties at each other’s places, checking out different cuisines in Oakland and Shadyside regularly, exploring Downtown, witnessing the Pirates play, going for Arctic Monkeys (!), playing squash or table tennis or ultimate frisbee (and sometimes just FIFA) everyday after work, or simply hanging out near the coffee machines (which are frankly amazing at RI) – we always found a way to stay together as a close group. We all still remain in close touch, and on a personal level, one instance which exemplifies how amazing our bond is (or would turn out to be), was on my birthday when, beyond all my expectations, the entire cohort surprised me with a cake in the middle of one of the talks (and this was barely 2 weeks into RISS). It couldn’t have been any better for me.


(Clockwise: AM, one of those random dinners, me with aforementioned cake, Pirates vs Mets at PNC Park)

V. Final week

All good things must unfortunately come to an end, and though I lived the best 10 weeks of my life, time just flew by. The last few days of RISS were both exciting and poignant. The poster presentation sessions went pretty well, where CMU was kind enough to get prominent researchers from both RI and outside of CMU to see our work, and since it was during a time most incoming Graduate Students were moving in, the poster session had an amazing crowd and I made a lot of connections.

(Me, Mononito, Dr. Mostow – RISS Poster Session, Newell-Simon Hall)

The final day had the RISS closing ceremony, which was pretty much a celebration of the summer – our time to say our goodbyes to our advisors and the RISS organisers. The day ended with all of us in the cohort having an amazing farewell party overnight.


(Left: receiving my RISS graduation certificate from Dr. John Dolan and Ms. Rachel Burcin, RISS Program Organisers. Right: me with Dr. Mostow)

VI. Acknowledgements

There are few experiences in my career that I could call ‘life-changing’ (it’s also sometimes loosely used), but RISS is comfortably one of them – not just because of the work, the mentorship and the peers, but because it solidified my interests in the fields I liked, exposed me to others I had never imagined existed and helped me be more versatile. I came in as a novice to the field of Robotics and Intelligent Tutors, having never been in a Summer Research Program (or even to the US!). I left more proficient in these fields but more importantly with endless connections, a bag full of memories (and a full SD card), and some invaluable life lessons. I consider myself very lucky and fortunate to have lived this experience.

I’m truly grateful to Dr. Mostow for believing in me through these 9 months, enabling me to live what I lived this Summer at RI and contribute to such an amazing project, but most importantly for the life lessons and anecdotes he shared with me so that I improve as a human and work to my fullest. Whatever happens next for me in my career, these are lessons I will always keep with me. I’m grateful to Mononito for being an amazing colleague, without whose work our summer would never have been as good. I’m also really grateful to Dr.  PK and Dr. Vinayak Naik for recommending me for my RISS application, Indira for helping me on it, and FICCI for sponsoring my stay. And finally, I’m extremely thankful to Precog for providing me an unparalleled environment and exposure to get involved in research at IIITD to begin with and motivating me to aim higher – I couldn’t have been part of a better group here.

(RISS Cohort 2018, Closing Ceremony – University Center, CMU)

Lastly, here’s my 2¢ if you like: Try to work on socially relevant projects if possible, it’s extremely satisfying. Be open to other fields especially during undergrad, you never know when you’ll find something interesting. And more importantly, apply to CMU and RISS. However highly you think of this place, your expectations will be exceeded.

(PS: As always, for any doubts / advice / other help you may require or just a chat, you’re most welcome to contact me through emailLinkedin or Telegram. I’d love to talk!)

class undergradAtPrecog(IIITD):

def  __init(self)__:

How I got to know about Precog is quite dramatic (call it fate? hahaha). It had been a really long day (Intro to Engineering Design project demo, IIRC) and I was walking past GB Pant towards the metro station. To give a rough idea about how far back I’m talking about: Phase II construction of the college was yet to begin, attendance was never recorded in any lecture, and Ubers were allowed to enter college premises.

As I was walking, absolutely dreading the hour long ride back home, a car stopped by and offered me a ride till the metro station. And I begin an extremely foolish (and lazy) kid, got into the car without giving it a second thought. However, the rest of the story doesn’t pan out like a Bollywood kidnapping thriller. To the contrary, I met two of the most fantastic people I know, Megha and Rohan! It was during this ride down to Hauz Khas metro (yep, they saved a lot of the metro anguish that day), that I got to know about Precog, PK, and most importantly that I don’t need to have taken any advanced courses in order to join Precog (many professors ask you to take their course before joining in).

Unfortunately, I wasn’t able to apply straight away (mostly since I was spending any time I got those days writing code for FLINT). However, I did realise that there is a lot more to computer science than Byld and Foobar (which I was completely obsessed with during that period). So as soon as I wrapped up my GSoC project, I shot PK an email asking whether they’re taking any students.

My application process was very straightforward. After sending in my CV and SoP, I was called directly for a brief chat with The (now Dr.) Dewan and Anu. Both of them have mentored and assisted me for extended periods with my projects, and I’ve learnt much more than just how to write research abstracts from them (including PTK’s art of cracking the best puns ever, and Anu’s art of subtle and effective sarcasm). Of course, I did destroy multiple servers in the process, but it was always forgiven within a week with restored access 🙂 After the first round, PTK and Anu recommended me to PK, who took the final screening.

All in all, it was a really smooth and professional process, much more to the point than the majority of the job interviews I’ve been taking recently. What still surprises me, is the fact that I’ve never come across another lab member who wasn’t given a task during the interview process. Guess it became mandatory after I clogged up a couple of servers hahaha?

My initial time at the lab seemed to be somewhat jinxed I’d say. The first two projects I worked on between Jan 2016 and April 2016 got scrapped in succession due to some technical issues. Both involved some analytics on data curated by an external agency. At this point it felt like a “local minima”, and I started wondering whether I was doing something wrong. When I meekly pointed it out, thankfully PK intervened and explained how it was not at all an issue on my end, and I should be expressing my thoughts more frequently rather than overthinking. That’s one thing I’ve always kept in mind, and tried to improve upon. Yes, folks, communication is the key.!

The summer of sixteen nights

During Summer ’16, PK gave me the opportunity to go spend some time at IIIT Hyderabad. They were hosting IASNLP over two weeks (16 nights :P) and had invited some people from Precog. He explained that we would like to do some work with code switched social network text, and gave me a free hand to Indira, Mallika and I to work on whatever we like.

IASNLP proved to be an extremely important stepping stone for me, as it was my first introduction to computational linguistics and machine learning (something I am working on till date!) Upon returning, we finalised our target to be a set of low level tools (LIDF, POS tagger, NER, Sentiment) for English – Hindi code switched text. Although I didn’t realise it back then, the problem was not as trivial as I thought. In fact, I ended up working on it for almost a complete year (taking it up as my BTech Project eventually), and finally published it at ACL this year (link and link).


Lab members and interns, summer 0f 2016. Missing a lot of people here

Why Precog?

Being a part of Precog is much larger than just completing your thesis/ project. When you’re a member of Precog, you’re a member of Precog. Weekly group updates keep you up to date with what’s going around in the lab. Regular paper reading sessions keep you updated with the cutting edge in areas other people are working in (I’d admit I wasn’t able to take full advantage of this, something I deeply regret). If I had to cherry-pick five top reasons I’m gonna miss being here

  • I got access to whatever resources I required to work. Anshuman, Divam and I constantly used to fight over GPU servers. PK noticed and ordered a separate one for each project 😀
  • Members are extremely helpful. If you ever need to get a code or paper draft review, all you need to do is shoot an email on the mailing list. And it is not just work. There were countless times I’ve pestered Anu/PTK/Srishti/NS and even PK to review my SoP for various applications!
  • There’s no need to be diplomatic. When it comes to work, people expect you to speak freely what’s on your mind. Even if it hurts sentiments.
  • Just like the IIITD culture, there is no concept of senior / junior / “call me sir/ma’am” amongst members. I think we inherit this really well from our college. Anu kept a special jar where you had to put in 10 bucks every time you called her ma’am :’)
  • We don’t just work together, we party together too, and a lot! Apart from regular PK treats at Barbeque Nation and sometimes even his apartment (how many of you can boast of having had dinner in the faculty residence :P), often we head out to GK after a tiring week (PTK always seemed to be fixated with The Beer Cafe for some reason).

You never walk alone. Winter 2017 semester, I took a break from college. On the eve of Precog’s birthday, members called me up via Skype for the celebrations 😀

To say Precog has played an instrumental role in shaping who I am today would be a massive understatement. I’ve spent more than half my undergrad life associated this lab, its members and its wonderful wonderful machines (JARVIS, FRIDAY and DEXTER). I have failed to produce results multiple times, missed multiple paper deadlines and annoyed a lot of members. However each time, I’ve learnt something really important, which allowed me to succeed in the end. And oh boy, when I finally got a paper accept (two in a row in fact, ACL SRW and SocialNLP @ ACL!), it was a beautiful closure.

PK and I after my (almost) convocation in August this year

What I’m grateful and indebted for, is the fact that I was allowed to make mistakes and learn from them. An essential piece of PK “gyan” (knowledge, for the non-Hindi speakers) which I’ve valued increasingly over the years, is that “working hard is good, working smart is better”. Choosing your battles is as equally important as working hard to win your battles. To succeed, I think it’s essential to surround yourself with people that believe in you. My journey with Precog was certainly not easy, it had its fair share of ups and down. However, it comprised of everything that was required for me to get out of my comfort zone and succeed.

Has the battle for General Elections 2019 shifted online?

Social Media, today, is playing a very important role in the politics of almost every nation. In India, there has been a boom in social media political campaigns, which was more marked in the 2014 General Elections.  All major political parties used social media to promote their manifestos and interact in the public domain in a one-to-one manner. They analysed different views of the people, and worked on them. The Home Minister, in a seminar after the 2014 general elections, also said that “Through social media, the government is outlining its plan, its vision”. After assuming power, the NDA government has been using Twitter, Facebook and blogs to outline its plans, vision, showcase the progress, and other updates.

One of the major reason for this was to involve and cater to the interest of the youth of the country, that are active on most social media websites. Owing to this, political parties have invested huge amounts of funds into social media campaigning, hence also increasing their reach.

Formerly our work for the 2014 General Elections involved heavy analysis of Twitter data. The work was supplemented by developing a portal that captured the intricacies of the political campaign leading up to the election as a mirror of the social media.

In order to kick off the next sprint of analysis for the upcoming general elections of 2019, we decided to revisit the data gathered in 2014. We had over 21 million posts by roughly a million handles collected in the last sprint of analysis that ran in 2014. An inspection of the handles lead us to find that currently only 31.64% of the handles are still active (defining activity as the handle having posted at least one tweet in the year 2018). Thereafter, the deleted and suspended handles constitute upto 15.65% and 19.81% of the total users respectively.

We are building a  portal to analyze the 2019 data and help see through the data that is getting generated on social media. Stay tuned as we unravel further analysis for the forthcoming election cycle. Below are some images from the portal landing page. We hope to do more analysis of the data that we are collecting from 2019 Elections.

If you have any questions for us to answer, please drop an email at pk[at]iiitd[dot]ac[dot]in we will be happy to answer it for you and credit the same for you.



A Summer to Remember

I spent my summer 2018 at Carnegie Mellon University (CMU) as a Robotics Institute Summer Scholar (RISS). It is one of the most sought-after internship programs out there, partly as it is hosted by CMU and partly as it allows you to immerse yourself into research in the field of Computer Science. I had spent my past two years at Precog working under PK on several developmental and research projects. I was hesitant at first while applying for the program as I was not sure if my experience was enough to have a decent shot at selection. As the application deadline approached I thought to at least give it a try and was able to submit it just in time. The next few months were filled with eager anticipation for the result and it was on 14th March that I received an interview call from my to-be mentor at CMU. After my interview, I was informed regarding the confirmation of my selection in the program at the end of March after two long months.

The summer began with the orientation of ~35 scholars from around the world. I was working in Intelligent Coordination and Logistics Lab (ICLL) led by Dr. Stephen Smith along with Dr. Isaac Isukapati. I worked on two projects during my time there. The first project was regarding developing a traffic simulation which incorporates a bus dwell time model built using Bayesian hierarchical inference. The second project involved testing and developing an IOS app which would help visually impaired people to cross the intersections using minimal gestures. Both of the projects had SURTRAC at the heart of the problem statement. SURTRAC is a system developed by ICLL to make intersections smarter by seeing the oncoming traffic using the DSRC technology to change the phases of traffic lights. I worked on these projects with Aidan – another member of the cohort and one of the best colleague you could ask for.

With Dr. Stephen Smith and Aidan during the poster session
With Dr. Stephen, Dr. Isaac, and Aidan during the closing ceremony

RISS was an enriching and a holistic experience. I met with some wonderful people during the time and made some lasting friendships. This program not only focuses on encouraging students to heavily engage in research but also allows you to build yourself in an all-round manner. During the summer I got the opportunity to attend the RSS conference at CMU. It was at this conference that I got to interact with world-class researchers and even attended a talk by Dr. Sergey Levine.  We as a cohort also had the opportunity to visit the office of the mayor and interact with a councilperson regarding our internship. This platform provides several opportunities like the UBTech and DJI workshops where we worked with drones and a humanoid robot, which to my surprise were given to us at the end to take back home. It felt like I was learning something new in every second that I spent there. Initially, I used to think of research in a more technical way with a narrow-minded focus on results rather than what the results we’re trying to convey but after regular philosophical discussions with my mentor, I got to know that it is much more than that.

Humanoid robot One with everyone!
During one of our late night walks! L-to-R: Mononito, Akari, Kyuto, Aidan
At the mayor’s office. L-to-R: Dr. John Dolan (Program Director), Me, Mononito :), Nick, Ziqi, Ms.                             Rachel Burcin (Program Co-program Director), Hameed, Yike

I owe most of my success to Precog which has been an intrinsic part of my career for the past two years. They are a bunch of people who genuinely care for you and are willing to help if you face any issues regardless of its triviality. They are highly welcoming for new students and make them feel like a part of a family. The weekly group discussions allowed me to ensure regular progress and brainstorm ideas to fix any issues that I would face in my project. I highly recommend everyone to be a part of Precog and experience the warmth of this family.