A ten-point plan to improve how transport models are explained

The modelling community has a lot to learn about how best to communicate what the strengths and weaknesses of transport modelling are to non-specialists, and in particular prospective scheme financiers, infrastructure investment consultant Robert Bain will tell the forthcoming LTT-sponsored Modelling World conference. Rik Thomas outlines Bain’s ten-point plan to bring some logic and consistency to modelling reports

During his years with the infrastructure team at credit rating agency Standard & Poor’s and, more recently, running his own technical consultancy for banks and institutional investors, Robert Bain has reviewed over 100 transport modelling reports from around the world – particularly for investor-financed infrastructure projects such as toll roads – and has determined that “the variance in the quality of reporting is staggering, which is particularly unhelpful at a time when international investor confidence in transport demand forecasts is at an all time low”.

“Although many advances have been made in transport modelling since the 1950s, for the purposes of accurate demand prediction today’s models – while essential – remain crude and imperfect,” he adds. “In the past, policy-makers and financial engineers have been seduced by consultants’ marketing brochures and claims of predictive prowess. Too much reliance has been placed on pinpoint forecasting accuracy leading to the development of aggressively structured transactions and financings with restricted flexibility. This has resulted in project distress and, in a number of high-profile cases, particularly in the roads sector, failure.”

Transport consultants writing for a financial audience also often self-define their work as ‘investment grade’, Bain says. “My analysis, however, suggests that this is commonly more of a marketing ploy than a serious attempt to understand and respond to the needs of potential investors. If toll roads specifically (and transportation projects generally) are going to reassert themselves as attractive investment propositions to a broad investor base, project risk and uncertainty needs to be better understood and communicated.”

Improved study reporting has a central role to play in that context, he adds, observing also that those commissioning transport studies need to be more demanding in their terms of reference. “For too long the outputs from transport studies have been dictated by transport consultants – and not their clients,” Bain suggests. “From my reviews – and from numerous discussions with bankers, bondholders, insurers and fund managers – recurring reporting deficiencies have emerged.  Addressing these deficiencies would go a long way to restoring confidence and I think that there are ten simple yet practical ways in which the quality and transparency of transport modelling reports could be improved.”

1. Clear presentation of (and justification for) the assumptions used

All of the modelling assumptions adopted in a transport study should be made explicit, Bain asserts, adding that in practice, however, this is rarely done. The assumptions should be consolidated in a single table for easy review, rather than being scattered across different chapters – and strong empirical evidence (with robust justification) should be provided in support. The implications of adopting alternative yet still plausible assumptions on any resulting forecasts should be highlighted.

2. Clear description of the product  offering

A number of demand forecasting reports fail to describe the transport facility under consideration in simple terms, highlighting the key characteristics of the project – such as future time savings or improved journey time reliability – that would attract users, says Bain. Facilities tend to be discussed in engineering or modelling terms. Readers – particularly from a non-technical audience – need to understand what a transport facility represents to consumers (the product offering); who would use it, why, how, when, for what purpose(s) and so forth. Toll road investors frequently talk about the “traffic story”, pointing out that a simple story with an intuitive appeal is likely to attract a more positive response from credit committees than facilities (and their attributes) that remain difficult to comprehend.

3. Avoid over-emphasis on supply-side issues

Bain’s third ‘top tip’ relates to the fact that many of the reports he has reviewed place considerable emphasis on the ‘supply side’ of transport models, devoting numerous pages to descriptions of highway networks, for example, in the base and future years. “In itself this is not unreasonable,” he explains. “However, rather less attention is often paid to the representations of base and future year demand in the models. Demand forecasting is frequently described as being a blend of science and art. Supply side modelling represents the science. It can be depicted accurately – indeed, with military precision using today’s digital maps – and is generally uncontroversial. However, demand forecasting models are only as strong as their weakest links – and the weakest links inevitably relate to the ‘art’ of demand representation and the treatment of demand growth. To be of most help to readers, more critical attention needs to be focussed on demand side issues, uncertainties and risks in transport study reporting.”

4. Avoid over-emphasis on a validated base year model

A well calibrated and validated base year model – one that reflects the travel environment and its competitive dynamic today – is an important tool for demand forecasters, Bain says. “It is difficult to imagine that the leading consultancies in this field would struggle to produce validated base year models and this is particularly true when, as is frequently the case, they limit themselves to relatively straightforward weekday peak-period modelling,” he notes. “For the reader, however, the construction of a satisfactory base year model is typically not the end of an important process – it is the beginning of one. It is the work that follows – focussed on the future – that is of most importance. More emphasis placed on possible future states of the world (and their travel demand and asset usage implications) in transport modelling reports would help to improve transparency and understanding.”

5. Provide a clear explanation of any link between demand and revenue forecasts

A number of transport modelling reports present forecasts in terms of aggregate vehicle miles (or revenue miles) for future years – or the total number of fare transactions (and total fare revenues). These output metrics clearly build from projections of demand volumes in the transport model – however the demand volumes themselves are often not reported, Bain asserts. “This makes it difficult for the reader to understand how future patterns of demand are expected to evolve (in any detail) and to determine whether – or not – the projections make sense,” he says. “A clear explanation of how future-year demand volumes translate into project revenues would be of considerable help – specifically to potential investors.”

6. Conduct comprehensive and realistic sensitivity testing

Most of the reports reviewed by Bain present the demand (and/or revenue) results from sensitivity tests. “However, frequently, these tests are limited in both scope and scale,” he says. “Sometimes they appear to have been ‘cherry picked’ for their lack of impact on future cash flows!” Investors and other project stakeholders need comprehensive sensitivity testing of the key project variables about which there is uncertainty, he adds, and these sensitivity tests need to be realistic in terms of the alternative parameter values being evaluated. “Simply reducing an input by 10% (with no explanation or justification provided) does little to enhance confidence,” Bain concludes. “Sensitivity testing needs to respond intelligently, on a project-by-project basis, to the specific risks and uncertainties to which project parties may be exposed.”

7. Discuss what the results actually mean

Some transport study reports leave the reader with the distinct impression that forecasts were produced at a very late stage in the study, Bain believes. “They are presented, fait accompli, at the end of the report – perhaps in a table – with little or no explanatory text,” he says. “This is unhelpful. Report readers want to know not only what the results are but what they mean. Is a forecast of 65,000 vehicles/day or 27 million passengers/year in 2015 high or low? Is it unexpected, in line with other transport project performance, or what? Transport modellers would add considerable value to their work if an explanatory commentary and/or discussion followed on from the presentation of any demand projections.”

8. Provide a candid description of future uncertainties and modelling limitations

“In academic literature researchers are required to bring to their readers’ attention any limitations associated with their work and it would be helpful if the authors of transport studies adopted this practice,” says Bain. “Instead of providing comfort to readers, the avoidance of any discussion about modelling limitations – or other sources of uncertainty that could impact on project performance – simply serves to undermine confidence. This is especially true when these limitations and uncertainties become apparent only under later cross-examination.”

9. Provide consistency in terms of risk analysis reporting

One of the challenges facing the reviewers of transport modelling reports is the lack of consistency in terms of reporting content and style, Bain claims, observing that this is most evident when considering project risks and the modeller’s commentary on risk exposure. It would be useful if a common project risk register or template was used by way of a summary, he says. “This would enable those who have to review reports to build up their analytical experience and expertise over time and, importantly, would assist with the project comparisons and benchmarking often used in policy decision-making and financial analysis.”

For toll roads this type of template has already been developed, although it is employed by different traffic consultancies to different degrees. At Standard & Poor’s Bain developed a ‘Traffic Risk Index’, based on years of credit risk analysis specifically focused on toll road risks. Key project risks are scored on a single summary sheet using a simple ten-point scale. Wider use of this template (or adaptations for different modes) would summarise any project risks quickly and would help readers to apply more consistency to their analytical endeavours, he believes.

10. Commission an independent peer review

As a condition of receiving government support, some funding initiatives – such as the TIFIA (Transportation Infrastructure Finance and Innovation Act) programme in the United States – require independent peer reviews to be conducted of transport demand studies and Bain thinks that this is good industry practice as it provides oversight of the original study by technically-conversant professionals, thus enhancing confidence. “Selection of the impartial peer reviewer, however, is critical,” he warns. “Some are less rigorous than others, suggesting a possible reluctance to be critical of parties who – next time around – might be in charge of the peer review process themselves.”

“There is a rich seam of literature from around the world demonstrating the potential exposure of transport demand forecasts to the influences of error and, in some cases, optimism bias,” Bain also cautions. “In Australia, for example, investor confidence in toll road traffic projections is currently at an all time low, encouraging procuring agencies to shy away from the traditional stand-alone user-paid model.  Better reporting with more transparency could help to reverse this regressive policy trend.

Project stakeholders who commission transport studies need to reassert their requirements, Bain concludes. “You don’t need to tell transport consultants how to do their job,” he explains. “You simply need to reinforce your expectations in relation to study outputs. This is no different from the more general public-private partnership philosophy with the spotlight usefully on outcomes rather than inputs. Transport demand forecasting best serves project stakeholders when it is focused on the identification of likely usage trends and is accompanied by incisive reporting providing an explicit and comprehensive discussion of all project risks and uncertainties.”