A public transport model built on open data

Europeans may like to laugh at the UK for leaving the EU, but at least trains and buses leave on time there, especially in the capital. London's transport authority has transformed its public network into one that actually works. And it's done that with real-time, open data.

It started with ticketing data - things like the Oyster Card, a simple "tap on, tap off" payment system that has given Transport for London (TfL) masses of data on customer behavior. And they have gone from that to a trial in late 2016 that used WiFi in underground station to monitor the way people move within station, how they make connections, and how that can lead to - or avoid - overcrowding.

"We know there are times on our network in certain locations that are crowded and if we can explain that - at those busy times and busy locations - to our customers, then that's hugely helpful," says Lauren Sager Weinstein, TfL's Chief Data Officer.

Read more: Tunnel vision: Inside London's new subterranean railway

"Some customers will take a different route or change their timing," says Sager Weinstein. "And it also helps us because if the system is a little less crowded it works better for everybody."

Maybe it's because I'm a Londoner…

Now, before we continue our journey, I should declare a bias. I really like public transport in London. But it's not because I'm a Londoner, as the old Cockney song goes, "that I love London so." It's simply because the network works. 

I left London in 1992 at a time when very little worked. And I moved to Cologne, where pretty much every public service worked like a dream. All these years later I find the tables have turned. Like a nightmare. Every second tram in my adopted city is bust, pasted with stickers, reading "Tür defekt" (defective door).

The system is ramshackle at best. You can never quite be sure if your train will turn up. And if it does, it certainly won't leave on time. Are these petty gripes? Perhaps. But not when you consider the growth in the local population and the increasing numbers of people commuting to and from Cologne.

In London, the system has almost always been overloaded, or close to it, and the population never stops growing, but Londoners have found a way to deal with it through, I hesitate to admit, technology.

Better living through data?

The other thing I need to declare is that I'm no fan of our contemporary belief that life can only get better the more data we have at our disposal. If you allow, that's hogwash - it's no universal law.

But when it comes to public transport data in London, it seems TfL have made good use of the public's property to improve the public's daily lives. (And I hope I never have to eat those words.)

That said, and as with every use of private data, there are concerns and caveats aplenty, even when the data is "anonymized."

"We now know from [Edward] Snowden that data becomes the government property by all sorts of mechanisms, so it's sort of a question of what Transport for London does," says Bob Miles, an associate dean for Study Abroad at the University of North Carolina at Chapel Hill. He just happened to be one of the random people I asked on the Tube. "Security," he says, "is always in doubt because of the State."

TfL's Sager Weinstein, however, describes the authority as a "guardian of the public's data." When they were planning the WiFi trial, which ran at the end of 2016, she says it was "privacy by design."

"We took the principle of data minimization," she says. "So you only take the data that you need to take. You don't collect more than you need for [your stated] reason. And we were very transparent about it, so customers could opt-out if they didn't want to participate."

Of pings and crowds

So what exactly was this WiFi trial all about? Because it wasn't just to let travelers access their social media feeds underground. 

Well the Oyster card gives TfL good data on how people enter and exit the network.

But TfL wants better data on how people move about the network when they are in it, and how that affects crowding on trains. 

"London's network is very intricate, particularly in the center of London where there are multiple routes that go through the center," says Sager Weinstein.

Just think you can take 18 different routes travel go between two major stations: King's Cross St. Pancras in the north and Waterloo in the south.

Using ticketing data - from the Oyster card and contactless payments - TfL can see how many entries there are at King's Cross and how many exits at Waterloo, and they can construct individual journeys with that data.

But the data is still rather crude - there is no detail about the actual routes people take and what demands that places on the network and at what times.

So that's where WiFi comes in.

"Through the WiFi data connection pilot, we would see a device, depersonalize the information by scrambling the device ID - and you scramble it consistently - so you see it at King's Cross and then you see it along the way," says Sager Weinstein.

The system picked up devices whenever they came within range of one of the 1,070 WiFi connection points. When a device has WiFi enabled, it's constantly sending out so-called probing requests, or pings. And as those pings are tied to particular devices through a Media Access Control (MAC) address, TfL was able to construct a far more precise picture of the way people moved about the city.

TfL collected an encrypted, depersonalised MAC address, with dates and times.

Bluetooth on the buses

But this is just one small slab of TfL's use of data. Another major pillar of TfL's approach is open data. TfL allows third parties to make use of some of its data to "extend the reach."

A recent study by Deloitte, a financial consultancy, suggests TfL's open data policy was generating annual economic benefits and savings of up to £130 million (145 million euros) for travelers and the authority itself.

They say it also supports about 500 jobs. And there are a range of small companies hoping to profit from TfL's open data, like Urban Things, a developer behind an app called Bus Checker.

UrbanThings is launching a new service called Ticketless, which uses Bluetooth beacons, rather than barcodes or near-field communication technology for mobile ticketing. The beacons will track when people get on and off buses, as well as make the sale of tickets feel seamless. 

"You'll be able to quite accurately estimate how many people are on a bus," says Tej Palladino Peters at UrbanThings. "Knowing not only when your bus is coming but also how full it's going to be, and whether to wait for the next one … at rush hour in London, that can be a big issue." 

Peters makes one concession - he says "assuming Bluetooth stays as popular as it is" - but if anything the technology has seen a rude resurgence since the release of Bluetooth Low Energy.

Can London lend a hand?

Not all of London's recent public transport success is down to "leveraging technology."

So while a lot is expected of passenger data, less and less seems to be expected of the passengers themselves. On most lines, passengers are no longer required to even press a button to open doors, and that's a good thing. All the buttons have been removed. It takes control out of the passengers' hands, but ensures a more efficient service - whether that was the intention or not.

And to think that the last time I worked in London in the late 1990s, the then newly extended Jubilee line regularly struggled to get trains to line up with glass platform barriers. An office worker behind me one day said "everywhere else in the world they get this technology to work, but not in London." Well, London's lifted its game.

Meanwhile in Cologne, passengers assume so much control over train doors that they are constantly ripping them open, or blocking the automated system in some other way, that it is my - purely empirical, subjective view - that this is why the doors are always broken and the trains run late. If only Germany would relieve its citizens of a little autonomy, public transport might also be a service that actually works. 

Read more: If you want German trains, go to Switzerland

TfL's Lauren Sager Weinstein would be open to a collaboration.

"We already work closely with a number of agencies across the world. It's fundamental for us to share. We don't compete with each other," she says. "So there's a huge opportunity to work with different agencies and share our know-how."

And, for instance, they have done that with New York City to help it introduce contactless payment technology. 

But some, like Andrew MacDougall of CityMapper, say the Germans are not really the sharing kind.

"In Germany, government agencies tend to use their data to power their own 'official' transit apps, releasing only a poorer subset of that data to third parties," wrote MacDougall in an email. "Sometimes it's the transport company themselves that isn't open with data. For example, Deutsche Bahn doesn't share their data widely, preferring to share it with an exclusive partner, in this case, Google."

Data protection laws tend to be stricter in Germany than in many other countries. But, despite MacDougall's somewhat piqued tone, you get the point.