Performance Measurements for Transportation Libraries

Slide 1
Performance Measurements for Transportation Libraries

• December 13, 2007
• Session: “Performance Measurements for Transportation Libraries”
• Hank Zaletel, MTKN/Iowa DOT Library/CTRE
• Maggie Sacco, Transportation Library Connectivity Pooled Fund Study
• www.mtkn.org
• www.libraryconnectivity.org

Slide 2
Performance Measurements for Transportation Libraries

• Hank Zaletel
  MTKN/IOWA DOT/CTRI
• Maggie Sacco
  Transportation Library Connectivity Pooled Fund Study

Slide 3
Definitions of Performance Measures

• Ways to objectively measure the degree of success a program has had in achieving its stated objectives, goals, and planned program activities. ...
  www.ojp.usdoj.gov/BJA/evaluation/glossary/glossary_p.htm
• Indicators of transportation system outcomes with regard to such things as average speed, reliability of travel, and accident rates.
  plan4operations.dot.gov/Glossary.asp
• The actions that can be objectively observed and measured to determine if a task performer has performed the task to the prescribed standard.
  www.neiu.edu/~dbehrlic/hrd408/glossary.htm
• Assessment methods based on student activities or products, as opposed to tests or surveys, to evaluate students' knowledge, skill and development.
  www.unk.edu/academicaffairs/assessment/Resources/index.php
• Qualitative measures or indicators of progress toward specified outcomes or benchmarks.
  https://competitivesourcing.navy.mil/reference_documents/defs.cfm

Notes:
A Google search produced the following 10 definitions of Performance Measures. They are all pretty similar, but when you’re approaching this as a group you need to make sure everyone agrees with the stated objectives.

Slide 4
Definitions of Performance Measures, cont'd.

• The indicators used to measure the performance of policies, programs and processes.
  www.fiu.edu/~pie/sec8appglossary.htm
• Performance measures can be either outcome or output measures. Program performance should be monitored and assessed.
  ocsevalinitiative.com/resources/glossary.asp
• Specific indicators used to evaluate how well a person, organization, or a system is operating.
  www.crfonline.org/orc/glossary/p.html
• Qualitative or quantitative indicators for assessing the achievement of outcomes.
  www.wtcc.sa.gov.au/webdata/resources/files/sp_glossary.doc
• 1. Explicit statements that define how success is to be measured, and identify the criteria, or benchmarks, to be used in judging success. 2. Discrete and predetermined information collected in order to assess performance against targets.
  www.treasury.tas.gov.au/domino/dtf/dtf.nsf/03b2a1d6613ba1894a2566bf001d6c3f/3fd01dff12836ef64a2566dd000cd01b

Slide 5
What To Measure & Why: The Big Picture

• Demonstrate Value of Library Services
• Performance Measures
• Quantitative: Keep good statistics
• Report Findings
• Qualitative: ROI, Success stories, Customer feedback
• Strategic Planning

Notes:
What to Measure and Why
Performance measurement allows librarians to gather data and draw conclusions about the overall success or impact of library services. Tracking library services provides data for senior management (CEOs, decision-makers, managers, etc.) with a clearer picture of the daily functioning of the library, its impact on the agency and the value of providing library services (also the value of retaining highly trained information professionals to provide these services).

Librarians should approach performance measurement as though they are reporting to people who rarely use the library and don’t understand that it is a vital unit within their agency, as this is frequently the case. To present a clear picture of the library as a busy unit supporting the agency’s mission statement and the professionals whose work they do understand, librarians have to keep good statistics of the work we do. That is the quantitative component of the performance measurement task. There is a qualitative component, too, and using customer feedback or “success stories” has a real impact on senior managers. Library services don’t lend themselves to scientific measurements, but we can make assumptions based on solid facts. We use what we know factually and from experience to extrapolate certain conclusions.

Slide 6
What To Measure

Quantitative Data : Keep Good Statistics

• ILL
• Reference
• Literature Searches
• Circulation
• Cataloging
• Outreach Initiatives
• Website activity

Notes:
Many ILSs will track materials checked out, new items added to the database and ILL/document delivery activity. Librarians can create reports that are presented clearly, often with charts and graphs from these ILS modules. Many libraries track reference, literature searches and ILL statistics in Excel, which is available almost universally in any Microsoft Office environment. The pooled fund study has created a statistical tracking tool to record transactions quickly and easily. (See Appendix H.) There is no need to purchase additional software to accomplish accurate and well-presented statistical reports. Excel also has a fairly versatile charting and graphing function. It is a good idea to pick out some data sets and represent them graphically for both impact and visual interest.

Display Tracking Tools:
Hank: MTKN
Maggie: Pooled Fund
Email Hank / Maggie for copies of these documents.

Slide 7
What To Measure

Networks:

Identify the inputs / outputs in common or establish baseline metrics for measurement.

• ILL
• Reference
• Literature Searches
• Circulation
• Cataloging
• Outreach Initiatives
• Website activity

Notes:
Networks need to agree upon which metrics they are going to focus on for an apples-to-apples comparison and to identify trends (benefits) resulting from working collaboratively.

Slide 8
What To Measure

Qualitative Data: What do the statistics tell us?

• Return On Investment (ROI) calculations
  Value of library services? How much $$$ was saved by having trained staff provide these services?
• Success stories
  How did the library’s services help the organization fulfill its mission?
• Customer feedback
  Customer testimonials speak volumes.

Slide 9
What To Measure

Qualitative Data: Return On Investment (ROI)

• What is the value of library services?
• How much $$$ was saved by having trained staff provide these services?

Sources:
ROI for Libraries Remains High, Roger Strouse, http://www.outsellinc.com/store/insights/3538.
Mn/DOT Library Accomplishments, Jerry Baldwin,
http://www.dot.state.mn.us/library/mndot_library_benefits.html.

Notes:
Qualitative data is also important when demonstrating the ROI of library services to senior managers. Several studies claim that libraries return at least four to six times the investment

Outsell 2007 Research

Important Details: Libraries looking to justify their existence with quantitative return-on-investment (ROI) numbers need look no further than their own end-users’ feedback. In Outsell research from Spring of 2007, users reported substantial, quantifiable benefits from involving the library in their information gathering. Library ROI was highest for the corporate and government sectors, but also positive for healthcare and education.

Benefits of library use included:
Time Saved: Government sector users topped this metric, reporting that each interaction with the library saved them 12.2 hours on average.
Dollars Saved: Government users report $2,575 in savings per use.
Decision Support: Two-thirds (65%) of users across sectors used library-provided information to support decisions, actions, and strategies. In this regard, government end-users were the most reliant on library information (72%)

Implications: Given the time that end-users spend on information tasks (now around 11.6 hours per week) and the increasing costs of content, libraries can use the data above to make a powerful argument for their own value. In light of Outsell’s newest ROI data, it seems penny-wise and pound-foolish for enterprises to eliminate library budgets in a time of increased information challenges. Outsell recommends that libraries, publishers, and information providers alike collect and publish ROI data gathered from their own users as one method of illustrating their contribution to the bottom line.

Display Jerry Baldwin’s ROI, Toolkit version
Jerry Baldwin (Mn/DOT) has developed an ROI formula that many people have adapted for their own use that demonstrates reduced costs and added-value ratios.

Slide 10
What To Measure

Qualitative Data: Success Stories

“An internal customer at WisDOT was about to solicit a Request for Proposal on a $50,000 project to determine the demerit point/administrative license withdrawal system used by other states. The library not only had the NHTSA related study on the shelf but was able to deliver it to the general counsel’s Office within 15 minutes of receiving the request.”

Notes:
The pooled fund study has been collecting Success Stories as Qualitative Data. Our libraries have submitted a number of high-impact accounts of how the librarians help customers avoid duplication of research, provide quick turn-around on requests for vital information, save customers’ time, enhance productivity and save their departments’ $$$.

Source:
Example provided by John Cherney, Head Librarian, Wisconsin DOT.

Slide 11
What To Measure

Qualitative Data: Customer Feedback

“The KDOT Library has been an indispensable resource that I have used often over the past few years. Without it, I would have spent hours looking or articles, visiting local libraries, making phone calls and waiting on faxes or mail. This has meant not only better informed decisions by staff engineers, but an actual savings to the taxpayers of the state of Kansas.”

Source:
Excerpt from letter of thanks from a bridge hydraulics engineer at Kansas DOT. Example provided by Marie Manthe, Librarian, Kansas DOT.

Notes:
The pooled fund librarians feel that customer feedback speaks volumes to management.

Slide 12
Why?

Reporting: Communicating Value To Management

Value of the library to the organization

Value of investing time and resources in a consortium or network

• Statistics
• Expenditures
• ROI
• Success Stories
• Customer Feedback

Source:
Transportation Librarians Toolkit (forthcoming)

Notes:
Statistics, expenditures, ROI and success stories should all be part of the reports to management. Reporting informs the decision-makers of the role of the library within the organization and allows library staff members to step back and review their progress. A good rule of thumb for reports is to include the mission statement up front, to inform the reader of the library’s purpose and follow with statistics and narrative illustrating how the mission was fulfilled and what factors may have interfered with the library’s objectives. The report should also include a summary of customer populations (percentage of civil engineering staff, planning, research), which parts of the collection get the most use, most frequently used services and the results of a customer survey.

Show WSDOT Library Use Charts

Slide 13
Why?

Strategic Planning: The Library in Context

• Performance Measurement
• Demonstrate Value
• Align the Library’s Vision with the Department’s Mission

Source:
Transportation Librarians Toolkit (forthcoming)

Notes:
Strategic planning for libraries

Performance measures and reporting provide context for effective library strategic planning. It also provides a framework for ensuring that the library's vision and strategies are aligned to support the mission of the departments they serve.

Slide 14
THANK YOU!

• Hank Zaletel
  MTKN/Iowa DOT Library/CTRE
  Hank.Zaletel@dot.iowa.gov
• Maggie Sacco
  Transportation Library Connectivity Pooled Fund Study
  msacco@ctcandassociates.com
• www.mtkn.org
• www.libraryconnectivity.org