The C-Word... modelling the counterfactual

Decisions can only be defended if the alternative – the counterfactual – is also modelled and the implications of doing less or doing nothing are clearly presented

What would happen if government policy regarding carbon emissions changes, as in the recent DfT Decarbonising Transport plan

What would happen if government policy regarding carbon emissions changes, as in the recent DfT Decarbonising Transport plan

No, it’s not that 4-letter word, nor is it Coronavirus or COVID-19. It’s the word ‘counterfactual’, meaning the condition that would be true if a previous condition had been different, including what a situation might be if nothing was done. 

The public response to the COVID modelling by Imperial College, Oxford University, the British Medical Journal and the Universities of Sydney and Melbourne amongst others, illustrates well that the need for action and the benefits of what are seen by some as too draconian measures, can only be defended and expected to be adhered to if the alternative, the counterfactual, is also modelled and the implications of doing less, or doing nothing, are clearly presented.

That’s why we, as transport planners and modellers, need to be involved more in, and spend sufficient effort on the determination and critical assessment of the counterfactual. In that way, transport models will be as helpful in learning and adjustment of policy and plans, as epidemiological models will be in managing the COVID-19 crisis (and that’s two C-words!)

In transport modelling this is no different. Transport projects or policies have long-term societal impacts, with winners and losers, and generate heated debates that are more useful if informed by models or other data-based analyses.

In forecasting, the counterfactual would be the Business as Usual or Do-Minimum case, against which a policy or intervention is tested. In post-opening evaluation the counterfactual is trickier to create.

The model needs to estimate what the travel demand and traffic conditions would have been if the project had not been built, or the policy not implemented.

In that context, others have mentioned the Prophet’s Dilemma – the success of an intervention to some extent negating the public’s belief that it was necessary. Being able to present the counterfactual is key.

Some have argued that, rather than comparing a transport future against the counterfactual, we should compare it to what people experience today. But as with comparing alternative public health responses to COVID-19 to life before lockdown, I don’t think that’s realistic or insightful when assessing the intervention’s effectiveness, although it is useful to provide context.

In transport modelling, the counterfactual is not about alternative futures. That is done through scenario-testing:

• What would happen if oil prices dropped (as they have done recently)

• What would happen if government policy regarding carbon emissions changes (as the recent report “Decarbonising transport: setting the challenge” starts to imply)

It’s about setting a baseline for comparison. That’s not easy, and I recall Denvil Coombe making the point many years ago that modelling the counterfactual was probably harder than modelling the policy or scheme itself.

Modelling the counterfactual for evaluation purposes (ex-post) is particularly troublesome.

The models used to inform the decision (ex-ante) would have made comparable input assumptions between the do-minimum and do-something cases. The reality in which the delivered policy or infrastructure project ultimately operates is inevitably different from those assumptions (because of a changed economic climate compared to that originally anticipated, or certain developments not having progressed).

This must be captured and reflected in modelling what is effectively an alternative reality today – not that simple!

That’s why we, as transport planners and modellers, need to be involved more in, and spend sufficient effort on the determination and critical assessment of the counterfactual. In that way, transport models will be as helpful in learning and adjustment of policy and plans, as epidemiological models will be in managing the COVID-19 crisis (and that’s two C-words!).

Tom van Vuren is the Chair of Modelling World and Practice Leader Transport Planning Australia for Mott MacDonald