Post hoc, propter hoc? Counterfactuals, placebos, and spillovers in evaluating a local mobility policy

Policy reforms are often unique, in the sense that it is hard to find comparable changes and circumstances that would make it possible to clearly identify their net consequences and thus unambiguously support causality attributions. Without the appropriate counterfactuals, there is no way scholars can avoid the uncertainty of their estimates. However, we should accept the causal complexity that characterizes social science, give up on the idea of a model’s precision, and increase the robustness of our empirical evidence through multiple testing. This is the research strategy that we adopted in evaluating a reform in the mobility policy of the municipality of Milan, in Italy, which cannot easily be compared to other policy changes. Overall, we found evidence of the direct and indirect effects of the policy reform. However, the research design helped us refine some of our original expectations and fine-tune the underlying mechanisms. This project uses the proposed case study to emphasize the methodological importance of evaluating any policy change using redundant and robust empirical evidence – even accepting some degree of indeterminacy –rather than relying on isolated positive findings.