How do we measure reliability?

Introduction

At Transit Committee (TC) on the 10th of April, reliability was the big issue of the day. Reliability statistics, reliability breakdowns, reliability plans, the discussion was centered around reliability even if the solutions were not found that day. 

In light of this, and in light of my own criticism of OC Transpo's reporting of reliability, I wanted to ask the question: how do we measure reliability? And how should we? 

KPIs

Currently, OC Transpo reports three statistics for general service "reliability" at erratic intervals at Transit Committee. (Para Transpo is reported separately, but I do not have the background to comment on the validity of their approach. I suspect it is also poor.) 

The first metric is "service delivery." This measures cancellations - or rather, lack thereof. The service delivered is the percentage of service that did not get cancelled. 

This metric becomes more blurry when you realize that OC Transpo is in the habit of pulling trips from the schedule at random. These are "temporarily cancelled trips," TCTs, and inquiries came up with unclear responses on whether these are included in service delivery metrics. 

The second metric is regularity. This metric refers to the headway between buses, measuring bunching. OC Transpo's metric is 40% of variability on a headway is okay, ie. six minutes variation on a fifteen minute headway or four minutes variation on a ten minute headway. 

Essentially, the gap between buses on the midday 88 can be twenty one minutes, and it will register as "regular." 

The third and final metric is punctuality, which is measured on routes with a headway higher than 15 minutes. Buses are considered late if they arrive more than five minutes late; buses are considered early if they arrive more than one minute early. 

As of the Apr 2024-Mar 2025, 10% of less frequent buses were early while 16% were late. 

This sounds pretty good. "98% busses aren't cancelled, and 75% of buses are on-time!" 

Of course, the media headlines the service delivery metrics and the general public starts to believe OC Transpo has an on-time performance of 98%. Which certainly isn't true in any event. 

The Dangers of Averages

Find any statistician, or in fact, anyone who has ever taken a statistics course ever and they will tell you a few things about averages, especially an average over a wide scale.  

First, a "97.6% service delivered" average can hide variation between routes. Indeed, OC Transpo themselves present the worst routes, with the worst of them dropping below 90% of trips delivered. Remember, that's 10% cancelled, not including delays, knock-on effects, bunching, and so on. 

It may be the case that the least busy routes (which we didn't actually know until December last year) may have a high cancellation rate while the 88 has none. Or it may be that the 6 and 88, the busiest routes, are constantly cancelled, which seems to be the case based on data inconsistently presented to TC. 

And frankly, the cancellation rate of an Orleans bus route is irrelevant to a transit user in Westboro, and vice versa. 

When it comes to reliability, it is also the case that cancellations vary over time of day. This is data that is less easily reported, although some kind of monthly average or yearly average for X route at Y hour would be appreciated - this is the kind of work that could be automated, if we so desired. 

I think one would find that cancellations are highest at the peak periods, when transit demand is also at its peak. There are few late night cancellations, but their patronage is also relatively few. 

It is not only cancellation rates that are averaged, but the data for on-time performance as well. 

This is even more egregious than the averaging of cancellation data, because delays compound on delays, especially in an interlined network such as ours. 

OC Transpo reports its on-time performance as a "headline statistic." The statistics, described above, are reported as a single number with no further breakdowns. 

As with cancellations, this can present a skewed view of the network. 

Delays and bunching are more likely to occur during peak periods, I have found - this is also when the most people ride transit. (midday ridership is higher as an absolute number, but it is also spread over six hours, the combined AM + PM length). 

It also doesn't distinguish between, say, high-ridership route 19 or low-ridership route 54; neither does the regularity statistic. Nor does it distinguish between a midday 88 trip, which may carry over a hundred riders, to the 87 trip with sixteen riders on the trip - and the knock-on effects of these delays. On capacity-limited routes like the 6 or 88, one cancellation or delay may lead to pass-ups (the bus doesn't stop because there is no room), which is a functional delay for the rider. OC Transpo does not report pass-by data. 

The data also does not cover time of day or weekdays/weekends. 

Wilson Lo, at the TC meeting on the 10th of April, asked for more granular data, but he only asked for certain days of the week; what we need is a systematic approach. 

Proper Data Keeping

I have just spent much digital ink talking about the practice of averaging reliability data. 

Does OC Transpo collect this data internally? I don't think so. They maintained an active online portal for route-by-route information until 2019, but in response to a query from Capital Current, they said: 

Capital Current first asked the City of Ottawa for the performance of all routes within OC Transpo’s network. Processing that amount of data was not feasible, said a City of Ottawa media relations official. The spokesperson said the city only had the resources to provide data on 10 routes, a process that ultimately took more than a month.

That was the sound of my head smashing against the wall. 

Either OC Transpo is lying, which is unlikely but not impossible (it was speculated that they did not keep ridership records based on their responses to queries) or they really do not keep this data around. Neither option is acceptable. 

What do we need? We need detailed, public data that is available to any member of the public - you and I, the Ottawa Citizen, or the Prime Minister - can access at any time via the internet. 

This data should include per-route breakdowns, preferably by time of day. 

One good example is Edmonton's Route Report Card. With a page for each route, it includes average ridership, on-time performance (early/late), service hours, stops, headway, and so on (ours would probably include a "% cancelled"). 

Even if this data is not publicly shared, OC Transpo must collect this information. How else would they know which bus routes were problem routes, and which ones they could leave alone? 

I might forgive this data not being collected if OC Transpo had a reputation for punctuality and on-time performance. However, with their chronic unreliability and tattered reputation, how could this information be ignored? 

Ultimately though, even all of this is a symptom rather than a cause. 

Systematic Problems at OC Transpo

What all of this - the averages, the data collection, the headline number - underlies is a lack of real commitment to improving reliability. 

There's no direction from Transit Committee and no direction from OC Transpo, and the system continues to drift, rudderless, while riders wait and the tax bill goes up. 

Statistical reporting must be done with care and needs to be detailed enough to be useful without becoming noise. Right now, OC Transpo is only capturing noise - headline numbers are not useful without detailed breakdowns, which we do not get. 

To improve service, we must know what the problems are. To know what the problems are, we must have a robust set of data to base our decisions on. That is something that we do not have, seemingly internally or externally. 

The lack of care given to reporting this information implies a lack of care given to improving reliability. For example, Renée Amilcar has admitted that New Ways to Bus is based on 2023 runtimes. Why? I can pull this data. Why can't OC Transpo? 

Thoughts

Reliability is a many-headed monster. There's always more problems to attack and more ways to improve. Reporting on reliability seems like a small step. But as any expert in any field can tell you, statistics can tell a story, or it can tell a lie. It comes to the framing of the reporting. 

"Reliability" comprises many factors and cannot be explained with a single headline number. It's useful at a very high level, but often bears little resemblance to the service transit riders face on the ground. It's a framing device for the discussions we have on improving service - it will lead us where the numbers tell us to go, but bad reporting will lead us in a bad direction. 

Improving any facet of transit service, including reliability, starts with the admission that there is a serious problem, then gathering information on how it can be improved. Without serious, detailed information on reliability, all three types, we won't achieve the improvements that we want to our transit system. 

That's the real shame in how we measure reliability. 

Until next time.