The DfT should clarify the status of modelling guidance

We refer to your article on the Regional and local strategic modelling and appraisal capability report commissioned by the DfT. This suggests that many models are not fit for purpose. Our response is based on a careful reading of the audit report, and not merely on your article.

The published DfT response to the report concludes with the comment: “The Department is keen to emphasise the importance of modelling and analytical capability and is open to recommendations and suggestions on how this may be achieved.” We welcome this and share the same aims.

As the report notes, the assessment of existing models was largely based on “compliance with the Department’s guidance”. While there are certainly different levels of quality and suitability among current models, as there always have been, we believe that this basis of assessment gives a narrow and partial view of the state of modelling. The report does, however, raise serious questions about the apparent mismatch between DfT guidance and practice that needs further investigation and debate.

Guidance is necessary, and the existing guidance is helpful in many ways. We support the DfT’s efforts to develop and promulgate it. There is a need both for recommendations and application of good practice, and there is certainly room for improvement. But some readers may conclude from the article that modellers are not very capable, that modelling is poorly developed and that nothing has been learnt over the years since the review of the multi-modal modelling studies in 2003. This runs contrary to our experience, which is that there has been much valuable work done, fruitful applications of models and significant progress in modelling ability throughout most of the profession.

The report contains many good insights and positive conclusions about the importance of modelling and the need to enhance the environment in which models are developed. Since the review was conducted in 2008, many of the listed models have been or are being improved, and so has the guidance, and we would expect this to continue. On the other hand, the table you showed from the report classifies models by their “suitability for general uses”. One might challenge that as a rather sweeping categorisation, without reference to their original objectives or to their successful applications. If the objective is to challenge the quality of modelling it would be necessary to review the models against the wider context of the purposes for which they were developed and used.

The report states that “consultancies do not commit sufficient resources to training their staff in the use of the Department’s guidance” and your article indicates that “keeping up-to-date with the mountain of guidance is a challenge”. If the guidance were a coherent body of clear information, in which both the guidance and its underlying research had been peer-reviewed and approved by experienced members of the modelling community, failure to use it thoroughly would be culpable indeed. The fact that the review has itself had to interpret the guidance is illustrative of our concerns. It would be constructive, in our view, to focus on: clarifying the status of the guidance (what is mandatory and why, what is merely ‘advice’); agreeing where the guidance most needs development; and improving its accessibility in terms of proper source referencing and supporting documents.

It is important to emphasise that modelling is a process used to provide information to inform decisionmaking. The report states that: “the other significant potential impediment to modelling adequacy is money”. We concur but would emphasise that while the modelling should be undertaken to the standard necessary to support decisions, it is also wasteful of resources to seek greater refinement than necessary. 

The signatories to this letter have supported and will continue to work with the Department to move forward the professional debate and development of good practice.

• John Bates, John Bates Services
• Andrew Daly, RAND Europe
• Hugh Gunn, HGA
• Paul Hanson, AECOM
• Miles Logie, Minnerva
• Hugh Neffendorf, Katalysis
• David Simmonds, DSC
• George Terzis, Jacobs Consultancy
• Tom Van Vuren, Mott MacDonald
• Bryan Whittaker, Scott Wilson
• Ian Williams, WSP