Carol Ketcherside

December 30, 2008 (Version 2)

Yonel Grant

San Francisco

18 Month Review – Service
Efficiency and Effectiveness
Study Phase II

Bryan Jungwirth

This technical memorandum constitutes Booz Allen Hamilton’s 18-month evaluation of
the Service Efficiency and Effectiveness Study (SEES) framework implemented in early
2007.
1. INTRODUCTION
Brief Background
•
Booz Allen Hamilton consultant team was hired in 2006.
•
The Phase I SEES Report was finalized in March 2007.
•
The SEES framework was adopted by the Valley Metro/RPTA Board in
March 2007.
•
SEES performance targets were adopted by the Valley Metro/RPTA Board in
June 2007.
•
RPTA issued the FY2006-07 Transit Performance Report in January of 2008.
•
18 month evaluation (Phase II) was scheduled for November-December 2008.
Purpose of 18 Month Review
The purpose of the 18 month review is to provide an independent evaluation of the
SEES framework implementation, and associated performance measures and processes.
The consultant scope is also to provide recommendations for revising or amending
specific components of the route and system performance criteria.
Approach
During the course of several days in November 2008, Booz Allen interviewed three
RPTA Planning staff and two staff from City of Phoenix Public Transit Department.
Booz Allen also conducted telephone interviews with a half dozen RPTA member
agencies to solicit feedback on their experiences with the Transit Performance Report
(TPR) data reporting and process. Booz Allen also reviewed the draft TPR report and
supporting data sheets, as well as early data submittals from the City of Phoenix.

Carol Ketcherside
December 30, 2008
Page 2

2. FINDINGS
Overall, the implementation of the SEES framework is going well (B+). There have
been, and continue to be, some challenges related to communications, technology, data
quality, and staffing, but this is to be expected when implementing a new framework
such as this for a transit system that is not truly regionalized.
A major success came early when the Valley Metro/RPTA and Valley Metro Rail
Boards adopted the recommended performance framework. The two policy boards
actively supported RPTA’s transit performance reporting goals.
Findings and observations, which are discussed in this section, are grouped as follows:
•
Framework adoption
•
Target setting and adoption
•
Communications with reporting agencies
•
Data reporting into regional framework
•
Web tool implementation
•
Transit Performance Report (TPR)
•
Contract negotiations regarding performance reporting
•
Route maturation guidelines
•
Proposition 400 routes versus other routes
•
Reporting transit performance to policy bodies.
Framework Adoption
The framework was successfully adopted in March 2007 by the Valley Metro/RPTA
Board and subsequently by the VMR Board (for the rail portion of the SEES).
Target Setting and Adoption
The SEES framework, route, mode and system targets were presented and discussed at
TMC, VMOCC, Valley Metro/RPTA Board and VMR Board meetings. Initial Board
approval of performance targets occurred in March and April 2007. Following
recommendations from rail staff, the Total Rail Boardings target for the first year of
operations was reduced from 10,665,000 to 7,827,000. The revised targets were
approved by the RPTA and VMR Boards in June 2007.
In November-December 2007, RPTA Planning staff prepared the FY2007 Transit
Performance Report using updated performance targets. Financial targets (e.g.,
operating cost per boarding, operating cost per revenue mile) were increased by the
percentage increase in the CPI-U for transportation items. Effectiveness targets were
not changed. The targeted annual increase in boardings remained unchanged at 3%.
The fixed route farebox recovery ratio target was also held at 25% (the RPTA Board

Carol Ketcherside
December 30, 2008
Page 3

adopted the 25% farebox recovery ratio target in September 2007 when the regional fare
policy recommendations were approved).
RPTA staff commented to Booz Allen Hamilton that the 3% target for annual increases
in paratransit ridership is problematic given the growth in operating costs. The source
of ridership target was the Financial Plan assumptions that were in place in 2007. RPTA
staff desires consistency with the Paratransit Study.
Staff from one of the reporting agencies commented that targets should be revised on a
regular basis, to challenge the operating agencies to continuously improve.
For the November-December draft FY2008 TPR, RPTA staff have updated the targets
using the same assumptions: CPI-U increases for financial targets; other targets
unchanged. In December 2008, RPTA staff uploaded a series of documents to the
Valley Metro website, including the draft TPR, supporting modal summaries, and a
master table of targets.
Communications with Reporting Agencies
Communications to reporting agencies on TPR planning and reporting occurs mainly
through the monthly VMOCC process. In addition, RPTA sent notification to member
agencies by e-mail in September 2008 about the October deadline for annual transit
performance reporting. The notification included a spreadsheet template for reporting
agencies to populate as well as a complete list of performance indicator definitions.
The primary challenge for communications with member agencies is the number of
agencies involved in the process and the turnover (real and procedural) of member
agency staff involved in this process. Two examples are provided:
•

The Technical Working Group (TWG) established for the SEES was made up
primarily of data practitioners from each agency, who would have the most
familiarity with the data and data reporting that would ultimately be
required. Not counting RPTA staff, the TWG had 12 reporting agency
members. Of these 12, only one person holds the same role now (18 months
later). In at least one case, the responsible staff member is no longer with the
agency and it appears that the replacement staff person may not have been
trained on the reporting requirements.

•

Even with several of the larger reporting agencies (e.g., Phoenix, Scottsdale,
Tempe), there is turnover in terms of attendance at VMOCC meetings. As a
result, the same person does not attend VMOCC meetings each month. Both
in-person and telephone interviews with member agencies revealed a
significant information gap in the history of the SEES initiative, the rationale

Carol Ketcherside
December 30, 2008
Page 4

for the regional reporting requirement vis a vis Proposition 400 and the
upcoming audit, the need to report performance by route, and the need to
flow down reporting requirements to contractors.
A secondary challenge which results from turnover in reporting agency staff is the
technical ability to accurately report all of the transit data required for the TPR. For
instance, during the review Booz Allen found that Boarding data initially submitted by
a new transit analyst at the City of Phoenix was 17.5 percent too low. RPTA staff
immediately suspected an error, based on comparisons to prior year results. The error
was confirmed and subsequently corrected. In cases where there is turnover among
staff responsible for collecting and reporting data, it is critical to provide for training of
replacement staff – and if possible, cross-training of other agency staff.
Data Reporting into Regional Framework
Member agencies interviewed for the 18 month review reported that the transition from
the old Performance Management Analysis System (PMAS) to the regional
Transportation Performance Report (TPR) process was seamless. It is fairly difficult for
RPTA Planning staff to spend considerable time auditing data provided by member
agencies, and in most cases they do not have access to the source data to be able to audit
it. RPTA Planning staff do tally the information provided and compare it to expected
levels and prior year results. Questionable numbers typically are subjected to a second
opinion and/or are sent back to the reporting agency for correction or explanation. In
the current cycle, this occurred with Phoenix ridership data as previously described.
Another agency that revised data in response to a question from RPTA Planning staff,
suggested that it would be helpful to receive a copy of the revised report before it is
forwarded to the VMOCC.
The agencies interviewed reported that they had not used the Operating Cost template
distributed during the 2006 SEES. This template provides a method to calculate total
operating costs for member agencies, including allocated administrative and overhead
costs. However, on careful questioning, the agencies stressed that the reported
operating costs include both direct and allocated costs, including contractor costs as
well as administrative overhead for the respective cities.
RPTA Planning staff noted that some performance categories needed for the TPR are
not available due to large gaps in the data available from contractors. Examples include
safety/security, and the maintenance indicator. One agency reports roadcalls in a
manner that is consistent with the TPR definition, but manual data manipulation is
required to pull those instances from the more broadly defined maintenance service call
data that the agency tracks. On-time performance data appears to be fully reported in
the detailed tables, but not in the graphics section of the TPR. In at least one case, on-

Carol Ketcherside
December 30, 2008
Page 5

time performance data is not available for neighborhood shuttles because there is no
reporting system in place to capture the data.
Although several agencies said that they are reporting performance data at the route
level, RPTA is experiencing some push back from some member agencies for detailed
route level performance data.
Web Tool Implementation
In the SEES concept, RPTA would work with member agencies to develop a final
reporting format, complete the transition from PMAS (the old performance
measurement reporting system), develop a reporting platform, and work with
individual members to facilitate the process. The hope was that these activities could
occur between April and December of 2007.
Ultimately, this schedule proved to be unrealistic because RPTA engaged a contractor
to develop the web-driven, automated data reporting tool. Development took longer
than expected but now includes a detailed System Manual and User Manual. As of
December 2008 the contractor, Systems Technology Group, Inc. (STG), is still working
through minor bugs. Other RPTA Planning priorities also interfered with the schedule.
RPTA decided therefore to pursue two parallel tracks:
1) Develop the FY2008 TPR manually, sending spreadsheet templates to all
reporting agencies and compiling the report based on the feedback
2) Continue development of the automated Performance Planning System with
STG, with gradual implementation during calendar year 2009. The intent for the
web-based reporting system is to begin quarterly performance reporting in
addition to the annual reporting.
RPTA has formed a working group to provide feedback on the STG web tool. As soon
as the beta version is ready, members will be able to comment on the input screens.
Phoenix has already made a request to enter data by batch file.
Transit Performance Report
The TPR has replaced the old PMAS. The transition occurred in the fall of 2007. The
Valley Metro/RPTA Board was pleased with the FY2007 report presented in January
2008.
Booz Allen reviewed the draft FY2008 Transit Performance Report (prepared in
December 2008) and provided feedback to RPTA. The report has an appealing layout

Carol Ketcherside
December 30, 2008
Page 6

and features clear performance updates for system-wide bus, paratransit and vanpool.
However, Booz Allen also identified several gaps:
•

The presentation of performance indicators does not include all the indicators
in the SEES framework. For example, the fixed route system-wide pages lack
data on safety/security performance, on-time performance, and mechanical
performance.

•

The draft TPR also does not contain any route-level data. The SEES
framework advocated a comprehensive set of route level indicators as shown
below.
Exhibit ES-2
Fixed Route, Route Level Measures
Cost Efficiency/Effectiveness
Farebox Recovery Ratio
Operating Cost per Boarding
Subsidy (Net Opg Cost) per Boarding
Cost per Revenue Mile
Service Effectiveness
Total Boardings
Boardings Avg. Weekday, Sat., Sun.
Boardings per Revenue Mile
Boardings per Revenue Hours (Express Bus)
On-time Performance
Miles between Mechanical Failures

Source: SEES Report, March 2007

It would be cumbersome to provide this much information in the TPR for all
routes. However, a summary of the best performing and worst performing
routes (e.g., the highest and lowest quartile boardings per revenue mile and
subsidy per boarding), and a link to the complete route level data should be
additions to the TPR.
Contract Negotiations Regarding Performance Reporting
Since the SEES adoption, there do not appear to have been any changes in contractual
requirements for reporting route level data and operating costs by the three primary
fixed route bus contractors in the Valley (Veolia, First Transit, MV). In most cases, the
contractors derive route level performance from larger samples of data.
As one example, representatives of the City of Phoenix said they were not made aware
that their contractors had to report cost per boarding or miles between mechanical
failures on a route by route basis. Now that they understood the rationale for the
regional reporting requirement, they could begin collecting this data for July 2008 (i.e.,

Carol Ketcherside
December 30, 2008
Page 7

first quarter of FY2009) so that the FY2009 TPR could include route level performance
results.
Route Maturation Guidelines
By December 2008, route maturation guidelines have not specifically been discussed in
TMC or VMOCC meetings since the SEES. These are expected to be included in the
Short Range Transit Program (SRTP). Generally, the current understanding is that there
is a two- year maturation period. However, since neither the TPR nor other processes
currently lend themselves to a quarterly comparison of route level performance data,
discussion of this topic at the regional level is difficult.
Another agency noted that the TPR does not make it clear whether data for “immature”
routes (i.e., those that have been operating for less than two years) are excluded from
the performance indicators. That agency went on to make the case that data from
immature routes should be excluded from the systemwide data because it is likely to
have a negative impact on systemwide performance. Performance of these routes
should be reported only at the route level.
The Transit Life Cycle Program (TLCP) provides guidance for Proposition 400 Route
implementation and funding, above the original legislation. Given the state of the
Arizona and national economies, regional discussions are under way to consider
phased route implementations (i.e., SRTP).
Proposition 400 Routes Versus Other Routes
For the 18-month review, the consultant team reviewed annual transit performance data
files and confirmed that the transit performance data is available at the route level,
including all of the Proposition 400 routes. Each route receiving Proposition 400
funding was identified, however it was not immediately possible for the consultant to
know what percentage of the route was funded through Proposition 400 funding.
Reporting Transit Performance to Policy Bodies
The annual TPR is proceeding on schedule, with presentations to the TMC and Board
planned for January 2009.
Currently, there is no quarterly regional transit performance reporting. It is expected
that regional reporting can begin in the third or fourth quarter of FY09 (i.e., by June
2009), as the web-based data implementation tool is rolled out.

Carol Ketcherside
December 30, 2008
Page 8

3. RECOMMENDATIONS
Recommendations for the SEES implementation are provided below and organized by
the following topics:
•
Target setting and adoption
•
Communications with reporting agencies
•
Web tool implementation
•
Transit Performance Report
•
Contract negotiations regarding performance reporting
•
Proposition 400 routes versus other routes
•
Reporting to policy bodies.
Target Setting and Adoption
First, Valley Metro/RPTA is to be commended on the establishment and adoption of
performance targets. As the reporting process matures, so too will the process for
updating the targets.
The appropriate Board(s) should always be involved in changes to performance targets.
Performance targets can be divided into four groups with respect to how and when
they should be updated (from less frequent to most frequent)
Group

Example Measure

1

Farebox Recovery Ratio

2

Service effectiveness measures
Miles between Mechanical Failures
Security Incidents per X Boardings
On Time Performance
Customer Satisfaction
Etc.
Demand measures
Annual Increase in Total Boardings
Annual Increase in
Saturday/Sunday Boardings
Standard cost efficiency measures:
Operating Cost per Boarding
Subsidy per Boarding
Operating Cost per Revenue Mile
Etc.

3

4

Source of
Performance Target
Regional fare policy
recommendation
Actual performance

Rationale for
Update
Regional policy;
Regional fare policy
recommendation
Regional policy

Update Frequency
Regular pulse check, but unlikely to
change unless regional policy
changes.
Regular pulse check, but unlikely to
change unless regional policy
changes.

Financial Plan
assumptions

Should be consistent
with assumptions in
RPTA financial plan

Yearly pulse check; change
according to Financial Plan
assumptions

Actual baseline data
from FY05-06

For the FY07 and
FY08 TPRs, the
target was escalated
based on national
CPI-U for
transportation items
Future targets
should be based on
budgeted costs and
service levels

Need to update every year.

RPTA staff requested advice on the rationale for updating cost efficiency measures such
as operating cost per boarding. Simply inflating historical operating cost data using
national inflation trends can result in increasing the divergence between a theoretical

Carol Ketcherside
December 30, 2008
Page 9

goal and what is observed (or attainable) locally. A better approach would be to
develop the target annually as a part of the budgeting process rather than externally
through the CPI-U. The Finance Department would take the lead in updating the
targets based on current financial projections, ridership projections and interest rates.
Annual transit budgets are developed to provide specific service levels, with costs built
up at the line item level to reflect the direct (e.g., labor, benefits, fuels, parts) and
indirect (or overhead) costs of providing a given level of service. Therefore, the budget
effectively establishes the target for annual efficiency measures such as operating cost
per boarding or operating cost per revenue mile. Transit budgets also identify revenues
by source, and may therefore also be used to define targets for revenue measures, such
as subsidy per boarding. It is therefore recommended that RPTA use budgeted costs
and revenues per unit of service delivered or consumed to calculate the annual targets
for those indicators. For instance, the paratransit ridership target of 3% annual increase
could be dropped given the disproportionate cost of paratransit trips.
In addition to being relatively simple to calculate, an advantage of this approach is that
it makes it easy to explain variances in terms of budget line item costs. It is also likely to
be more relevant to local experience than the rate of change in a national index that may
or may not reflect local conditions.
RPTA should also periodically evaluate effectiveness performance targets (i.e., not
dependent on cost). For instance, the 3% target for annual increases in paratransit
ridership, although a baseline assumption in the Financial Plan, should be dropped as it
given the disproportionate costs involved.
It is recommended that RPTA take advantage of the regular TMC and VMOCC
meetings during the October cycle to discuss potential updates to the SEES targets (by
exception only), with the objective of presenting recommended updates for Board
consideration during the January cycle.
Communications With Reporting Agencies
The main strategy to address the turnover among member agency staff involved in
reporting transit performance data is regular communication and solid documentation.
RPTA already has a working group of data practitioners responsible for TPR updates
and Proposition 400 reporting. The working group listing needs to be kept current and
should include back-ups (alternates) for participation when the primary member is not
present. RPTA should also keep a running tally of participation.
RPTA staff should develop a standard package of documentation to share with current
and future working group members. This documentation could include:

Carol Ketcherside
December 30, 2008
Page 10

•
•
•
•
•
•
•

Working group list (primary member and alternate)
Proposition 400 legislation/summary
SEES Report
Operating cost template
Performance measure definitions (longer version)
Latest TPR report
Data implementation tool User Manual.

Web Tool Implementation
RPTA should more proactively advertise the data tool implementation with its working
group. Early participation by the data input staff will contribute to tool acceptance, and
will likely prevent costly fixes to the software ahead of the beta version release. The
City of Phoenix request for batch data entry is a good example of both the former and
the latter.
Another advantage of more proactive participation in the data tool implementation
project is increased member agency awareness of upcoming quarterly reporting, and
the ability to sort performance by route including by Proposition 400 routes.
To facilitate understanding and implementation of the tool, it is recommended that
RPTA offer several training sessions for the working group members (ideally both
primary and secondary members should be trained). RPTA could use this opportunity
to reiterate the importance of total operating cost reporting, and to provide additional
training on the operating cost template.
Transit Performance Report
Booz Allen has worked with RPTA staff to make improvements to the draft FY2008 TPR
report. As indicated in the Findings section above, two major opportunities for
improvement include the inclusion of the missing performance indicators (e.g., on-time,
maintenance, safety and security) as well as the route analysis summary.
Individual route performance is currently not available in the TPR, but definitely
should be included in the future. The following table illustrates three approaches RPTA
peer agencies use to evaluate and present individual route performance.
System
Houston METRO

Evaluation Measure
 Subsidy per Boarding

Evaluation Approach
Routes ranked from best to worst
Routes sorted by quartile
Focus on 4th quartile for improvement
Routes in lower 4th quartile subject to
elimination

Carol Ketcherside
December 30, 2008
Page 11

Denver RTD

 Subsidy per Boarding
 Boarding per Revenue Hour

Routes ranked from best to worst
Focus on least 10% for either measure
Focus on least 25% for both measures
Routes subject to modification or
elimination

Dallas DART

 Subsidy per Boarding
 Passengers per Trip
 Passengers per Revenue Mile

Routes ranked from best to worst
Routes with Route Performance Index
(RPI) less than 0.6 require corrective
action; with 0.6-0.9 examined for
corrective action

Source: Booz Allen Hamilton, Technical Advisory Committee Presentation #7, SEES, November 2006

The complete listing of the route level performance indicators is included in the SEES
report and listed above (Exhibit ES-2). Booz Allen does not recommend that all
measures be included in the TPR report, but they should all be available upon request.
It is expected that this data will be readily available once the data implementation tool
is operational.
Specific recommendations for what route level analysis to include in the TPR did not
exist up to this point. To cover the route cost component, it is recommended that
subsidy per boarding be tracked, at a minimum. For service effectiveness, it is
recommended that RPTA track passenger boardings per revenue mile. A
representative sample of the best and worst performers would be shown in the TPR.
From the base data, it is recommended that RPTA rank the routes from best to worst for
each indicator, then sort them by quartile (top 25%, next 25%, etc). Performance for the
routes can be shown by quartile in a few simple tables as illustrated below.
Route-Level Performance Summary, FY08
Route-Level Subsidy Per Boarding
Route-Level Subsidy Per Boarding
Best performers (Top 25%)
Worst performers (Bottom 25%)
Route A: $a/Boarding
Route B: $b/Boarding
…

Route Z: $z/Boarding
Route Y: $y/Boarding
…

Passenger Boardings per Revenue Mile
Best performers (Top 25%)

Passenger Boardings per Revenue Mile
Worst performers (Bottom 25%)

Route A (not the same A as above): a/mi
Route B: b/mi
…

Route Z: z/mi
Route Y: y/mi
…

It is worth noting that in monitoring regionally funded service programmed under the
TLCP, the draft SRTP (FY08/09 to FY13/14) recommends using both passenger

Carol Ketcherside
December 30, 2008
Page 12

boardings per mile (target: 2.1) and the regional local bus passenger loading (standard:
125%). The passenger loading measure tracks whether for the same trip (in a period of
6 months within the year) there were three or more consecutively sequenced trips that
exceeded the regional local bus passenger load of 125% (i.e., 25% standees).
Monitoring route performance is key to identifying areas of improvement. This is
desirable at all times, but particularly important in a recessionary economy. Possible
actions for poor performing routes could include targeted marketing, headway changes,
days of service changes, elimination of non productive route segment(s), or route
elimination, among others.
Contract Negotiations Regarding Performance Reporting
Progress needs to be made with respect to requiring operators to meet the regional
reporting requirement contractually. “Problematic” areas include safety and security,
mechanical, and on time data. Reporting agencies need to flow down the reporting
requirements to their contractors for both the system and route level performance.
RPTA should monitor and check compliance with the requirements. Copies of the SEES
indicator table should be helpful for communicating these needs.
Proposition 400 Routes Versus Other Routes
During the review, the question arose as to whether it would be appropriate for the TPR
to provide specific reporting for Proposition 400 routes. The TPR was designed as a
reporting tool to manage the system on a regional basis. It is expected that the webbased data implementation tool will have the functionality to drill down to any
indicator by route or groups of routes (e.g., for all Proposition 400 routes as a system).
For this reason it is not recommended that RPTA transform the TPR into a Proposition
400 system report. It is recommended that RPTA ensure that a similar report can be
produced through the automated system either by quarter or for all of FY2009.
Reporting to Policy Bodies
At the moment, the only regional transit performance data reporting occurs on an
annual basis, in January for the prior fiscal year. During the SEES, the recommendation
was that regional transit performance reporting should occur quarterly. The consultant
team recommends maintaining the quarterly reporting cycle to the extent possible, and
that the Operations Department should become involved in the process.

Summary Recommendations – 18 Month SEES Pulse Check
CATEGORY
Target Setting and Adoption

Communications with
Reporting Agencies

Web Tool Implementation

RECOMMENDATION

ACTION OWNER

TIMEFRAME

Discuss potential updates to SEES targets with TMC and VMOCC

RPTA Planning Staff

October each year

Consider change in base escalation for cost efficiency measures
Present annual TPR; present recommended target change (s) to
RPTA and VMR Boards

RPTA Planning Staff

January each year

RPTA Planning Staff

January each year

Develop up to date working group listing
Conduct monthly Data Tool/Transit performance meeting

RPTA Planning Staff
RPTA Planning Staff, reporting agencies

February 2008
Monthly in 2008

Emphasize to working group importance of total cost accounting

RPTA Planning Staff

February 2008

Keep running record of participation
Develop standard package of documentation for working group
members

RPTA Planning Staff

February 2008

RPTA Planning Staff

February 2008

RPTA Planning Staff, STG

March 2009

RPTA Planning Staff, STG

March 2009

Finish roll out of Beta version
Test Special Report for Proposition 400 individual route and System
performance for all of Prop 400 routes together
Work with individual members to maximize input; monthly working
meetings
Finish roll out
Plan and Execute Training Session for member agencies

RPTA Planning Staff, reporting agencies
member staff
RPTA Planning Staff
RPTA Planning Staff

March 2009
May 2009
May 2009

Add snapshot of best route level performance: recommend subsidy
per boarding and boardings per revenue mile for best and worst
RPTA Planning Staff
performing quartile routes

February 2009

Add safety and security, maintenance and on-time performance
indicator performance data. Indicate where/why not available

RPTA Planning Staff

February 2009

Contract Negotiations

Flow down reporting requirement to Contractors
(system wide and route level)

Tempe, Phoenix, Glendale
RPTA Operations

January-March 2009

Reporting

Begin monthly data entry with new system

Transit Performance Report

Begin quarterly, etc briefing to Audiences - Bus/DAR
Begin quarterly, etc briefing to Audiences - Rail

RPTA Planning Staff, reporting agencies
member staff
RPTA Operations Mgr
VMR Operations Mgr

May 2009
June 2009 and quarterly
June 2009 and quarterly