Application case: Tracking Citywide Performance

Tracking Citywide Performance

Massive systems produce massive amounts of metrics. The challenge is to determine how to slice the data in ways that benefit users rather than confuse them, especially when the users have a wide variety of experiences with information technologies. This was one of the major issues facing New York City when it decided to develop its Citywide Performance Reporting (CPR) Online System, an interactive dashboard providing agencies and citizens with "user-friendly access to the most critical performance indicators for every city agency, with monthly updates and automatic evaluation of trends within specified program areas."

Development of CPR

CPR is part of NYCStat (nyc.gov/html/ops/nycstat/html/home/home.shtml). NYCStat is New York City's one-stop-shop for all essential data, reports, and statistics related to city services. NYCStat provides access to a wide variety of performance-related information, including citywide and agency-specific information, 311-related data, and interactive mapping features for selected performance data and quality-of-life indicators. 

CPR was launched in February 2008. Initial plans for CPR were developed by the Mayor;s Office of Operations in concert with the Department of Information Technology and Communications (DoITT) in mid-2005 (NYCStat, 2009). The project had three components:

• Performance management application. A backend computer system providing a single point of access for agencies to input data.
• Analytics tool/dashboard. The frontend system to provide standardized reporting format with drill-down capabilities, performance summaries, and trend graphics.
• Data definition. Review and identification of the topics measures and critical indicators to be included in the CPR system for 44 mayorial agencies.

Development of these components continued through July 2007. At this time, the system was opened to the 44 agencies and mayor's office for review. At the end of the summer, Mayor Bloomberg directed that the system be made available to the public through the city's NYC.gov Web site as soon as possible. Subsequent work on the system focused on making the dashboard as easy and flexible as possible for public use. The system was finally opened on February 14, 2008, as part of the release of the Preliminary Fiscal 2008 Mayor's Management Report.

Refining Metrics

CPR is an offshoot of the Mayor's Management Report (MMR), an annual evaluation of the city's agencies that measures 1000 performance indicators, including everything from grade school test scores to the time it takes to fix a pothole. Rather than simply providing an online version of the MMR, the CPR required the agencies to take a fresh look at the services they were delivering and how best to measure the outcomes of those services. Initially, this resulted in thousands of measures, many of which did not exist at the time. Eventually, the list of initial measures was reduced to 525 critical measures. The majority of these measures represent final outcomes of services that directly impact the city's residents. This is why the decision was made to open up the system to the public.

In addition to refining the measures, other key decisions had to be made. Because data was available by month, year-to-date, and full year, choices had to be made about how best slice and dice the data and make comparisons across the variious periods. Decisions also had to be made on how to measure and present targets--designating the desired direction of each measure and tresholds for good and bad performance. Finally, because the total number of measures and agencies was so large and none of the public was likely to be familiar with each of the individual agencies, the presentation and navigation needed to be simplified. The developers decided to categorize the measures into eight citywide "themes" that captured the way that the city goverment serves the people who work and live in New York City. The themes included: city-wide administration, community services, economic development and business affairs, infrastucture, education, legal affairs, public safety, and social services.

Impact of CPR

From the viewpoint of mayor's office, CPR improves performance management in three ways--accountability, transperency, and accessibility. It does this by providing the following capabilities (NYCStat, 2009):

• Tracks performance for the most important "outcome" measures that directly reflect how the lives of the public are affected by city goverment.
• Measures performance by comparing current data to prior data for the year before, thereby holding agencies accountable for year-over-year improvements.
• Highlights agency performance through graphs and color coding, making positive or negative performance trends immediately obvious.
• Drill-down capability, allowing users to review comparative trends over a 5-year period.
• Aggregates important measures into citywide themes, which cut across agency silos and disciplines to reveal the overall picture about city government performance.
• Updates each measure monthly, quarterly, or annually so that the most recent data is always available.
• Offers the ability to download data in a variety of formats for more detailed review and analysis.
• Provides detailed information about each measure, including an explanation of what the measure means, its reporting frequency, and other useful details.

In March 2009,the CPR system was recognized by the Ash Institute for Democratic Governance and Innovation at the JFK School of Government at Harvard University as one of the Top 50 Government Innovations for the year (New York Non-Profit Press, 2009)

Lesson Learned

According to Sarlin, "It is interesting to see how government agencies at all levels are implementing best practices to be more run like a business." In the case of New York City's Mayor's Office of Operations you might expect this, because Mayor Bloomberg comes from the world of business, more specifically, the world of real-time financial information (see bloomberg.com). However, other government agencies have announced online dashboards even though they have virtually no connection with business. For instance, the new U.S. CIO Vivek Kundra recently announced the U.S. Federal IT Dashboard, which provides public access to data reported to the Office of Management and Budget (OMB), including general information on over 7000 federal IT investments and detailed data for nearly 800 of these investments classified as "major". Mr. Kundra has spent his entire career in government positions.

These efforts suggest that business may be able to learn quite a bit from these initiatives. 

These government dashboard initiatives demonstrate the following (Buytendijk, 2008):

• Transparency makes a difference. These initiatives provide widespread, public access to large collections of performance data. Many businesses could learn from the depth and breadth of the information these agencies are sharing.
• Importance of collaboration. Many BI projects suffer from silo approach, with different departments doing their own dashboards or scorecards. Each department may realize a short-term ROI. However, the combination will be suboptimal and is not likely to lead to an overall ROI. Both the CPR dashboard and the federal IT dashboard show that it is possible to create an organization-wide initiative spanning multiple domains.
• Continuous improvement. CPR is based on trends, not targets. This means that the performance indicators are aimed at continuous improvement instead of static goals. It sets a good example for how an operational or tactical dashboard should look.

--o0o--

Question for the case:

1. If you were a mayor of a city in Indonesia, what would you do after you get the lesson learnt from the case above?