Evaluating the impact of public interventions: methods and case studies


Research Project supported by the Italian Ministry of Education and Scientific Research, 2005

[Home] [Versione italiana] [Printer-friendly version]

Projects | 1997 | 1999 | 2001 | 2003 | 2005 



The aims of this research programme are discussed with respect to the four broad themes it consists of. In what follows we will not deal with data requirements, as the relevance of appropriate information for the specific case studies is discussed in greater detail elsewhere in this proposal (see the projects of the five RU for details).


The central issue in programme evaluation – as for all observational studies that aim at identifying causal relationships – is how to control for confounding factors. Sticking to the basics, the problem is how to measure the causal effect of any policy – namely, the contribution of the policy to the observed change – when the analyst is only allowed to observe the events taking place in the real world. The problem arises because in an observational study – as opposed to what happens in a controlled experiment, which is often unfeasible in the social sciences – the analyst does not control the process determining the assignment of subjects to the ‘treatment’ and to the ‘control’ groups. The analyst only observes the outcome of the process. Then, by looking at the differences between the two groups s/he seeks to sort out the causal effect of the policy from the differences s/he would have observed even if the policy had not taken place. In a technical wording, the causal effect is defined as the difference between the factual outcome experienced by the subjects in the treatment group and the counterfactual outcome – namely, what subjects in the treatment group would have experienced had they not taken part into the programme. The operational problem then is how to obtain a good approximation of the counterfactual event, which is not observable by definition.

In general terms, the solution to the problem crucially rests on a careful analysis of the selection process. It takes having a thorough understanding of the institutional and economic reasons leading some subjects to undergo the policy while some others do not. As a result, it turns out feasible (i) to list the individual characteristics with respect to which the two groups – treatments and controls – are likely to differ and (ii) to design the collection of information needed to control for such differences. Summing up, the credibility of any estimate of the impact of a public policy ultimately relies on the analyst’s ability to provide an accurate picture of the selection process (and, of course, on the availability of all the relevant information needed to control for the differences between the two groups). Good data combined with the appropriate econometrics is the line to pursue to obtain credible impact evaluations. Within the current project special care will be devoted to the use and – when needed – to the development of non parametric methods, namely methods that avoid imposing strong, often unwarranted, restrictions on the functional form of the models specified to represent the phenomena under analysis. Typically, this methods provide robust inferences.

In addition – and as a complement – to impact evaluation, we shall consider its links to the cost-benefit analysis and to the economics of welfare. Impact evaluation studies provide estimates of the short-run effects of a policy disregarding their general equilibrium effects. In between such partial equilibrium analyses and a proper general equilibrium analysis there is room for empirical studies, which the RU in Turin plans to pay some attention to.

Research units involved: all.


The human capital and labour market policies whose causal effect on suitable outcomes this research group will focus on are the following: (a) vocational training programmes; (b) policies promoting equal opportunities to higher education; (c) flexible labour contracts available in the Italian labour market since the late 1990's; (d) the Mobility Lists; (d) the 1998 Italian reform removing barriers to the entry of new firms in the retail industry.

The evaluation of the causal effect of active labour market policies will take place with reference to the characteristics of the post-programme work history of the individuals. It follows that longitudinal data on the work history of “treatment” and “control” individuals are required. Both administrative and survey data will be exploited.

Research units involved: Firenze, Padova, Piemonte Orientale, Torino.


The policies providing incentives to firms this research group will deal with are the following: (a) programmes providing either financial incentives or tax credit to promote capital and R&D investments; (b) business incentive policies in declining areas.

According to the goal of the programmes, the evaluation will aim at measuring their impact on employment, on firms survival, on the nature and quality of investments as well as on firms efficiency.

The data needs are quite diversified here. On the one hand, when it is either the firm survival or the impact on employment to matter, longitudinal micro-data are required which will be recovered from administrative archives and from “ad hoc” surveys. On the other hand, when it is the amount and quality of investments to matter, data will be exploited from a survey run by Capitalia.

Research units involved: Firenze, Piemonte Orientale, Salerno.


The welfare policies this research group will deal with are the following: (a) the ‘Reddito Minimo di Inserimento’ demonstration as well as other policies - mainly at a local scale - promoting social inclusion; (b) policies to promote fertility, in the background of which we also plan to study the impact of education on fertility.

As for their goals these policies are quite heterogeneous. However, they raise very similar methodological issues once one considers evaluating their impact: to our knowledge, an unsettled issue so far.

Research units involved: Padova, Torino


Info: anna.giraldo@unipd.it