The Metrics of Knowledge Management

PART F –THE METRICS OF KNOWLEDGE MANAGEMENT

INTRODUCTION

How do we measure the performance of knowledge management? The answer is with great difficulty! In this section, we will examine four different methods of measuring performance:

Return on Investment
Balanced Scorecard
Surveys
Fuzzy Logic

Due to the complexity of knowledge management, those that have attempted to measure performance have tried and tested systems which are both flexible and robust, quantitative and qualitative, soft and hard.

IBM have been pioneers in the measurement of KM. The Knowledge and Organizational Performance Forum (KOPF), a multi-client consortium run by the IBM Corporation from 1999 through 2003, focused on advancing how organizations derive value from investments in knowledge, learning and human capital.

Initially chartered as the Institute for Knowledge Management (IKM) and then known as the Institute for Knowledge-Based Organizations (IKO), the program conducted pragmatic, field-based research on topics such as social capital, social network analysis, communities of practice, the role of knowledge in strategic alliances, and collaborative environments.

More than 60 highly respected organizations from a diverse set of industries and the public sector participated during the consortium's five-year history.

RETURN ON INVESTMENT (RoI)

Measuring performance by RoI is usually the preferred option of private sector organizations as an attempt to appraise the value of KM to profit margins.

It can be quite successful measuring how levels of investment support overall business strategy, as such, it attempts to monitor the resources deployed in ICTs and people.

However, the question still remains: How do you measure the RoI in knowledge itself?

Below are two case of how RoI has been successful in monitoring KM performance.

ERNST & YOUNG

Proposition: Faster revenue growth lower costs.

Approach: CoPs, KM managers, content management.

Results: Ten-fold increase in revenue with only five-fold increase in employees.

KRISPY KREME DOUGHNUTS

Proposition: Provide enterprise portal used by employees, franchisees, and store managers.

Approach: Buying enterprise portal software cheaper than building it.

Results: Faster sharing of financial, ordering and supply chain information with store managers. Reduced ordering errors. A vehicle for communicating marketing standards.

In the public sector, particularly amongst governmental and quasi-governmental organizations, Prince2 (http://www.prince2.com/whatisp2.html#intro) methodology is now the standard for measuring performance. This template-based methodology is supported by the use of Microsoft Project and Visio to process map and monitor performance.

BALANCED SCORECARD

A new approach to strategic management was developed in the early 1990's by Drs. Robert Kaplan (Harvard Business School) and David Norton. They named this system the 'balanced scorecard'. Recognizing some of the weaknesses and vagueness of previous management approaches, the balanced scorecard approach provides a clear prescription as to what companies should measure in order to 'balance' the financial perspective.

The balanced scorecard is a management system (not only a measurement system) that enables organizations to clarify their vision and strategy and translate them into action. It provides feedback around both the internal business processes and external outcomes in order to continuously improve strategic performance and results. When fully deployed, the balanced scorecard transforms strategic planning from an academic exercise into the nerve center of an enterprise.

Kaplan and Norton describe the innovation of the balanced scorecard as follows:

"The balanced scorecard retains traditional financial measures. But financial measures tell the story of past events, an adequate story for industrial age companies for which investments in long-term capabilities and customer relationships were not critical for success. These financial measures are inadequate, however, for guiding and evaluating the journey that information age companies must make to create future value through investment in customers, suppliers, employees, processes, technology, and innovation."

The balanced scorecard suggests that we view the organization from four perspectives, and to develop metrics, collect data and analyze it relative to each of these perspectives:

1. The Learning and growth perspective

This perspective includes employee training and corporate cultural attitudes related to both individual and corporate self-improvement. In a knowledge-worker organization, people -- the only repository of knowledge -- are the main resource. In the current climate of rapid technological change, it is becoming necessary for knowledge workers to be in a continuous learning mode. Government agencies often find themselves unable to hire new technical workers and at the same time is showing a decline in training of existing employees. This is a leading indicator of 'brain drain' that must be reversed. Metrics can be put into place to guide managers in focusing training funds where they can help the most. In any case, learning and growth constitute the essential foundation for success of any knowledge-worker organization.

Kaplan and Norton emphasize that 'learning' is more than 'training'; it also includes things like mentors and tutors within the organization, as well as that ease of communication among workers that allows them to readily get help on a problem when it is needed. It also includes technological tools; what the Baldrige criteria call "high performance work systems." One of these, the Intranet, will be examined in detail later in this document.

2. The business process perspective

This perspective refers to internal business processes. Metrics based on this perspective allow the managers to know how well their business is running, and whether its products and services conform to customer requirements (the mission). These metrics have to be carefully designed by those who know these processes most intimately; with our unique missions these are not something that can be developed by outside consultants. In addition to the strategic management process, two kinds of business processes may be identified: a) mission-oriented processes, and b) support processes. Mission-oriented processes are the special functions of government offices, and many unique problems are encountered in these processes. The support processes are more repetitive in nature, and hence easier to measure and benchmark using generic metrics.

3. The customer perspective

Recent management philosophy has shown an increasing realization of the importance of customer focus and customer satisfaction in any business. These are leading indicators: if customers are not satisfied, they will eventually find other suppliers that will meet their needs. Poor performance from this perspective is thus a leading indicator of future decline, even though the current financial picture may look good.

In developing metrics for satisfaction, customers should be analyzed in terms of kinds of customers and the kinds of processes for which we are providing a product or service to those customer groups.

4. The financial perspective

SURVEYS

Survey research is one of the most important areas of measurement in applied social research. The broad area of survey research encompasses any measurement procedures that involve asking questions of respondents. A "survey" can be anything from a short paper-and-pencil feedback form to an intensive one-on-one in-depth interview.

Surveys can be divided into two broad categories: the questionnaire and the interview. Questionnaires are usually paper-and-pencil instruments that the respondent completes. Interviews are completed by the interviewer based on the respondent says. Sometimes, it's hard to tell the difference between a questionnaire and an interview. For instance, some people think that questionnaires always ask short closed-ended questions while interviews always ask broad open-ended ones. But you will see questionnaires with open-ended questions (although they do tend to be shorter than in interviews) and there will often be a series of closed-ended questions asked in an interview.

Survey research has changed dramatically in the last ten years. We have automated telephone surveys that use random dialing methods. There are computerized kiosks in public places that allows people to ask for input. A whole new variation of group interview has evolved as focus group methodology. Increasingly, survey research is tightly integrated with the delivery of service. Your hotel room has a survey on the desk. Your waiter presents a short customer satisfaction survey with your check. You get a call for an interview several days after your last call to a computer company for technical assistance. You're asked to complete a short survey when you visit a web site.

Most surveys of knowledge-based initiatives tend to ask a series of questions based upon aspects of knowledge management, such as, innovation or leadership. In these surveys, respondents are usually asked to rate their response on a numeric scale of 1 to 5.

FUZZY LOGIC

“As the complexity of a system increases, it becomes more difficult and eventually impossible to make a precise statement about its behavior, eventually arriving at a point of complexity where the fuzzy logic method born in humans is the only way to get at the problem.”

(Lotfi A. Zadeh, University of California, Berkeley)

A fuzzy perception is an assessment of a physical condition that is not measured with precision, but is assigned an intuitive value. In fact, the fuzzy logic people assert everything in the universe is a little fuzzy, no matter how good your measuring equipment is.

When humans are the basis for an analysis, we must have a way to assign some rational value to intuitive assessments of individual elements of a fuzzy set. We must translate from human fuzziness to numbers that can be used by a computer. We do this by assigning assessment of conditions a value from zero to 1.0. For "how hot the room is" the human might rate it at .2 if the temperature were below freezing, and the human might rate the room at .9, or even 1.0, if it is a hot day in summer with the air conditioner off. You can see these perceptions are fuzzy, just intuitive assessments, not precisely measured facts.

By making fuzzy evaluations, with zero at the bottom of the scale and 1.0 at the top, we have a basis for analysis rules for the fuzzy logic method, and we can accomplish our analysis or control project. The results seem to turn out well for complex systems or systems where human experience is the only base from which to proceed, certainly better than doing nothing at all, which is where we would be if unwilling to proceed with fuzzy rules.

Below is a fuzzy logic test designed to measure the performance of communities of practice.

COMMUNITIES OF PRACTICE: SELF ASSESSMENT TEST

Do people in your organisation share a mental model of its aims and objectives?