Statistics for Improving the Efficiency of Public Administration Daniel Peña Universidad Carlos III Madrid, Spain NTTS 2009 Brussels

Outline 1. Introduction 2. Building a quality index: attributes and weights 3. Methods to determine weights 4. A simple model 5. Extensions of the model 6. Conclusions

1. Introduction Public Adminstration provides services to people (Health, Education, Transportation, Energy...) Analysing the efficiency of these services requires measuring their cost and their Quality. This talk is related to evaluating service quality

In order to evaluate and monitor the quality of a service we need to build an index (scalar measure) to summarize its performance: a quality index. This a similar procedure to summarizing the evolution of the prices in a cost of living index. Understanding the structure and the level of a quality index is needed for decision making about resource allocation and cost/benefit analysis.

We usually know how the budget is split among different projects or areas in an institution. We do not know how different activities contributes to the overall quality of an institution.

Note that for services provided in a free market prices can be considered as a summary of the sevice quality. This is not the case for public services where there is not an open market. For instance: basic research We present a model for building a performance quality index which can be applied to the services provided for Public Administrations.

2. Building a Quality Index It is assumed that the quality Q of a given service depends on some attributes: Q=f(X 1,…, X k ) For instance, for a university X 1 : quality of teaching = f(X 12,...X 1a ) X 2 : quality of research = f(X 21,...X 2b ) X 3 : quality of innovation and transfer = f(X 31,...X 3c )

University Quality X 1 (teaching) X 2 (Research) X 3 (innovation+Transfer) X 11 X 21 X 31 X 1a X 2b X 3c X 111 X 11e X 1a1 X 1af X 311 X 3ch

A linear quality Index (quality index from now on) combines all the different attributes into a single number: Q = w 1 X 1 +…+w k X k, In order to build this index we need: The list of attributes. The weights.

The most important part are obtaining the weights because we can always write a longer than needed list of attributes and give to some of them weights equal to zero.

In most application of quality indexes (as for faculty evaluation) we do not have objective weights. This is a great difference with a standard statistical index number, as Cost of living index, where the weights are objective (for instance the weight of a product in a cost of living index can be its relative contribution to the total cost of a familiar unit)

The procedure I will present can be applied to build a quality index for A service process (student registration in a university or in a hospital) An area of activity (research, teaching,..) An Organization (A universtity or hospital)

3. Methods to determine the weights 1. For consensus among a small group of experts. Advantage: it is simple to apply and can be presented as a political decision Disadvantages: Different groups of experts can give different weights Problems of coherence with many attributes

2. For indirect evaluation of the attributes and the quality from a sample of experts from some population of experts. The weights are obtained by statistical analysis. Thus the data will be the evaluation of the overall quality of the service and also the levels of the attributes which determine the overall quality. For instance, overall university quality and also the quality of teaching, research and innovation

Two ways to carry out this evaluation: a) fix the values of the attributes and ask for a global evaluation (value of Q). Then use experimental design linear models to estimate the weights. This is conjoint analysis, we can use fractional factorials to evaluate many attributes

This procedure, which has been used for evaluating technical services (as telephone service), is not easy to apply when there is not and objective ways to fix the levels of the the attributes.

With three attributes each with two possible levels, eight possible services are defined and evaluated evaluation

b) the second approach is to evaluate in several units both the attributes and the global performance ( o global quality) and use regression to build a model and estimate the weights.

Example: some universities are evaluated by defining the global quality and the quality of the attributes which determine the overall quality The evaluation of some specific attributes is asked, X 1 =?, ….,X k =? (for instance in a 0-10 scale) The evaluation of the global performance Y is asked in the same 0-10 scale.

Then for each Judge we have the explanatory variables, X, the response o global performance, Y, and the regression coefficients will be the weights. Problem: the weights will be different for different judges or referees and we want to estimate the distribution of weights in the population and the average weights to build the index.

4. A simple model

Note: This assumption is strong and can be relaxed as we will explain later

an example

5. Extensions of the model Instead of assuming the weigths follow a normal distribution we can assume that they follow a Dirichlet distribution. In the way: (1) they are positive and add up to one; (2) the variability of each weight depends on its importance;

We can test for missing attributes with the data allowing that the sum of the weights of the attributes considered to be smaller than one. We can allow for mixture distributions for the weights (clusters of customers with different wishes)

We can predict the weights for each respondent and then relate these weights to personal characteristics to undertand the structure of people wishes

Further extensions We can also assume that the attributes are evaluated with error, that is they are also latent variables. Then we have a random coefficient model with errors in variables which is a challenging estimation problem.

We may also assume that we have measurement error in the X and Y and that both are multidimensional, that is instead of a response variable we have a vector of response corresponding to dimensions we do not want to put together

Then we may have a LISREL (linear structural relation model) model in which all the key variables are latent variables and are related to some observed variables, as:

The simple model we have presented have no measurement error in the X, have an scalar responde Y and assumes some joint distribution for the regression parameters.

LISREL models have been applied to Quality indicators usually assuming fixed, instead of random, coefficients in the relationship between Y and X.

An example: ACSI index

6. Conclusions Building linear quality indicators seems to be an increasing important task for quality service evaluation The key part of this task is estimating the weights There are many important statistical problems link to this objective:

Cluster analysis to determine groups of customers with similiar weights Factor and LISREL models and the EM algorithm for estimating these models Multivariate Outlier Analysis for finding measurement errors and robust analysis

Multivariate time series analysis for monitoring these indexes over time Non linear models for capturing the more realistic effects among the attributes and the overall quality

In summary, this is an important area in which statisticians can play a key role in the future.

Thank you for your attention