One night in 1996 – The Story behind ESPN CricInfo

Anytime anywhere in the world when fans need to know what’s going on in a cricket match of any note, they turn to ESPNcricinfo. Ball-by-ball coverage of every international match involving the Full Member nations has distinguished the site, and its ball-by-ball service is unmatched in any sport for its depth and reach.

Like many other inventions, ball-by-ball updates was the result of a crisis. I intend to give you a first-person account of what happened one fateful night, 17 years ago, when Cricinfo started covering games ball by ball.

The Promise

Cricinfo started as a volunteer-run organisation in 1993. It was managed largely out of virtual offices, on IRC (internet relay chat) chatrooms called #ci and #cricket. You could think of IRC as text-only “hangouts” where like-minded people logged in and chatted using a program. I was a graduate student at the University of Massachusetts pursuing my PhD during that period, and part of the volunteer management of Cricinfo.

Fast forward to the World Cup in 1996. Through the efforts of a large number of people, Cricinfo reached an agreement with the 1996 World Cup organising committee, PILCOM, to jointly provide live coverage of the tournament.

The organising committee agreed to make available via file transfers the live scorecards they would create and update. We would take them, massage them into our format, and publish them on the site.

In 1995, along with Travis Basevi, I completed loading the entire Test and ODI archive of scorecards onto the Cricinfo database. Since I already had the scripts that created and formatted our scorecards, I said I would handle the task of porting the World Cup scorecards into our format. Easy.

All of us were excited when a big press release went out announcing the partnership and the availability of live coverage on the internet. Everybody awaited the event with tremendous anticipation.

The Disaster

In the run-up to the big event, we kept reminding PILCOM to send us some sample cards so I could tweak my scripts and be ready. Silence.

I was getting increasingly nervous.

At the time, Cricinfo used to provide an approximation of live coverage, in the form of manually edited scorecards, covering about one of every five international games or so. A volunteer watching the match on television would update the scores in a text file, which would, once every ten overs or so, be uploaded to the website. Clunky, yes, but much better than no coverage. The way PILCOM were going, we were beginning to wonder about worst-case scenarios, if they failed to come through.

On day one of the World Cup, there was no sign of a scorecard from the organising committee. About 17 overs into the first game, I received credentials for an FTP site from where to download the card. When I hurriedly logged in, using my broadband connection at university, there was no card. Five overs and much frantic checking later, one appeared.

When I got a good look at it, my jaw dropped and I banged my head on my table. Not only had a lone card arrived when the game was past 20 overs, it was only updated to over No. 12, so it it was already way out of date. It was the ugliest-looking card you could imagine, riddled with formatting errors – a code-writer’s nightmare.

In three places on one card, a player would be variously referred to as “Graeme Hick”, “G.A.HICK” and “HICK”. The same dismissal could be rendered as “ct” or “CAUGHT.” There was no way I could write a program that could read, interpret and translate this card into the Cricinfo format, even if we received it ball by ball from PILCOM, which we wouldn’t anyway.

Getting the format right was not a matter of just aesthetics, it was critical for the stats generation program Travis and I had prepared to provide continuous statistics during the World Cup. (Travis has since taken those same stats generation program to new heights with Statsguru).

I informed our team that depending on the PILCOM cards was a non-starter. In the meantime, the traffic caused by hits to the hand-edited card on our server in Oregon on the opening day caused it to melt (and I mean literally: the network card burned). The world was awaiting live coverage. We didn’t have it.

#cricket then mobilised its resources and jumped into action. At all points during a game, someone or the other somewhere in the world would watch and edit the scorecard directly on our server. Volunteers pitched in from all over the world. Shifts were set up.

That was how our first few days of our World Cup went by. We were providing live coverage as promised, but our plans behind the scenes had been completely upturned. Nobody on the outside realised or cared how we were doing it, and the popularity of the site was rising with every game.

The Victory

Anybody who has scored even a club-level game knows it is hard work and requires enormous concentration. Doing it off television – editing a text file on a slow connection and loading it onto an overworked server during TV commercial breaks – was proving impossible.

Not least, it became difficult to line up people to provide coverage. One night, one of our volunteers, Duane Pettett in New Zealand, had to edit three cards simultaneously because there was no one else around.

I told the folks on #ci (the Cricinfo management irc channel) that the situation was not sustainable, and that I needed to go and find another solution.

The place to start was Dougie. Or rather dougie, a cricket scoring program written by a South African graduate student, Jaques de Villiers, about four years before the World Cup.

Dougie was brilliant because it automated the creation and updating of scorecards. A scorer on dougie simply needed to enter what happened on a particular ball, using commands like “.” or “2.” (for a dot ball, and two runs scored). The program would automatically update the batsman’s score, team score and bowling analysis.

Other users could connect to dougie to follow a game live. While it was brilliant, it didn’t account for the millions of users expecting to follow games on Cricinfo; and it required those users to download, compile and configure the Unix-based dougie. Great for nerds but not for the rest of the world just about getting familiar using Netscape on Windows 3.1.

The web was great and easy for consuming content but it was hard for us to create scorecards – which Dougie, on the other hand, was fantastic at. I decided to connect dougie and the web to provide the missing parts for both. Innovation is not always about creating complex things, but rather extracting simplicity from complexity.

I had about ten hours to do it, though, before the next day’s games began. While we had discussed using dougie for live scoring within Cricinfo on and off, it had never gone beyond a random conversation. Things were going to change that night.

Ten hours to go

I started with getting hold of the source code for dougie. After I got a sense of what it was doing, I tried to compile and run the code myself. It promptly crashed (many months later I realised that the copy I had was old and de Villiers had since fixed many of these issues). My machine was running Linux, which is the most popular variant of Unix today but was a young operating system in those days, in a constant state of flux. Imagine a world where your iPhone or Android device gets an operating system upgrade every week, and you’re trying to run an app that was built to run on a version 20 versions prior to the one you have.

Dougie was a distributed system and I spent time tracking its networking inconsistencies – again by reading the code and going through documentation on Linux and various newsgroups devoted to these issues. (Remember, Google was still a few years away.)

I managed to fix the problems and the program started running. The networking seemed fairly stable in whatever testing I could manage in a couple of hours.

Now that was done, the cricket nerd in me took over – and I started testing the program to check if it handled all the possibilities that a game could throw up.

It handled a large number beautifully. A few interesting ones it didn’t, though. Stumped off a wide? No way to score that. Getting run out on a no-ball while taking the third run when the first one has been declared short by the umpire? You could score that but there was a careful and complex sequence of commands to manage it. Running two byes on a no-ball – did the score and bowling analyses update correctly?

Brilliant though it was, dougie still needed some fixing, and I was running out of time. It was about 11pm in Amherst, Massachusetts, where I was trying to put together our scoring infrastructure. Adrenaline (and generous doses of caffeine) kept me going but I knew I could not finish the task alone on time.

We were providing live coverage as promised, but our plans behind the scenes had been completely upturned.

Help from Holland

I contacted Louis van Dompselaar, a Dutchman and an avid #cricketer, who was an expert programmer and crucially, a few time zones ahead of me.

I requested him to write a C function that would take the dougie output and print it out in the Cricinfo format, which he was familiar with. My plan was to finish fixing the scoring-logic portion of dougie, get a couple hours of sleep and then in the morning resume coding and testing the part of connecting dougie with our web infrastructure.

I finished modifying the scoring logic to take care of the issues I had discovered, and went off to sleep. When I woke in a few hours, Louis had, as I had trusted he would, delivered. I had the function sitting in my inbox and I quickly integrated it in dougie.

We weren’t through yet. The next step was making the ball-by-ball scorecard appear live on our website.

There was no precedent or comparable system that I knew of at that time. Since our master web server would get overloaded during games, with hits from users, it was not practical or feasible to run dougie on it to do the actual scoring.

I devised an architecture where our master server would run a dougie slave, and the master dougie would run elsewhere, on Unix machines our volunteer base had access to. The slave dougie would receive ball-by-ball updates and generate a fresh card every ball and write it out to a file. I wrote scripts that would constantly monitor all such files (there were many simultaneous games we covered), and whenever a change was discovered, it would take that file, pad it with some html and copy it over to the designated spot on our web database.

Half an hour to go for the world’s first global live blog

I simulated a game by running both the master and slave dougie and the whole system seemed to work. Dougie still occasionally crashed but by and large it was stable.

On the #ci channel, I announced to the other guys that I had a working scoring system based on dougie that we could start using immediately. All our wonderful, eager assembled volunteer scorers were given access to the program. Using dougie we covered the games ball by ball that day. If my memories serves me correctly, it was March 1: Australia were playing Zimbabwe.

Since that World Cup in 1996, Cricinfo has not missed live coverage of a single international game between cricket’s Full Member nations. The infrastructure we created that night remained the basis for the site’s live coverage for the next ten years.

That night transformed Cricinfo from being a reproducer of content to probably the earliest known real-time content producer on the web.

We had volunteers all over the USA, Australia, New Zealand, UK and elsewhere watching a game being played on the Indian subcontinent, sending micro updates out to our server, which were instantly relayed to millions of users around the world. You can say we were tweeting ten years before Twitter was born!

A big leap for cricket-kind

In Amherst, Massachusetts, where I was, we had access to the cricket coverage via a large satellite dish antenna in the UMass-Amherst Foreign Language Learning Center. We begged and pleaded and bought television coverage of the World Cup from a provider.

I ended up scoring the final of that tournament. That’s right – the first live ball-by-ball coverage on the internet of the final of cricket’s biggest event came from a small college town in the US in the dead of the night, with about two feet of snow piled up outside, and it was followed all over the world. That was, and is the continuing magic of the internet.

Within a year, Cricinfo’s popularity went through the roof, showing an eight-fold increase over the previous year (and this was when barely anyone in India had internet access). At that time I didn’t realise how big a part that one night would play in creating cricket history of its own kind. Even today when I see those live scorecards refresh, a part of me smiles inside and goes back to that night.

Sri Lanka won the World Cup that year, but it was also a big victory for Cricinfo, our army of cricket fans and volunteers, and for sports coverage on the internet.

[Editor’s Note: This is a guest post by Vishal Misra is a professor in the computer science department of Columbia University and is also the founder of the company Infinio. His contributions to ESPNcricinfo, besides early technical work, were also sometimes poetic.]

Note: We at Inc42 take our ethics very seriously. More information about it can be found here.