Input data a key source of error in transport project forecasts, say planners

LTT’s report of Professor Bent Flyvbjerg’s new paper on forecasting error in transport projects generated a huge volume of discussion among the international members of the Transportation planning and traffic engineers and Transport modelling Linkedin discussion groups. Below we publish a selection of the responses, in anonymised form

“I wish it becomes true ‘the day the transportation forecasters are penalised’. The majority of toll road revenues are not meeting the goals set by the consultants.”

“I agree, the transport planners are in plight. If they produce realistic numbers, the client turns around and says, ‘Maybe you would like to revise these?’ We have even worked backwards, often devising methods to arrive at the ‘suitable’ figures. The vision is myopic: immediate gains of [project] approval/construction and … by the time the fallacies are realised, the people concerned have changed completely.”

“There are problems such as the client changing the consultant if they don’t get the answer they want. But that is easy to address if they [clients] have to declare when they change advisors. I’m thinking about the situation in the UK where a pension scheme has to declare when they change the scheme actuary. If they do it too often they have to explain why to a body with the authority to fine them. The actuary is required by law to whistleblow on dodgy practices and changing actuaries to find one who can/will give the answer you want is one such dodgy practice.”

“The thing is traffic forecasting is pretty much a judgement call. What will the economy be like in 20 years? Will people still drive 15,000 miles a year? Will 75% of folks be licensed to drive? What will fuel cost? All of these are indeterminate. Thus the only thing we know is that our answer will be wrong. The question is how wrong?”

“The title [of the story] “punish consultants” is strange. Consultants are there to consult. It is others that make the decisions. Should we punish a consultant who used the International Monetary Fund’s predictions for GDP growth? On the other hand, let us have access to the results of the traffic predictions. Several associations can become the ‘library’ for keeping a record of predictions versus reality per consultant. Then clients can select their consultant based on ‘past’ performance. And as stockbrokers say, past performance does not guarantee future performance.”

“I find Professor Flyvbjerg’s reported remarks to be totally lacking in evidential basis and suspect in their motives. I am heartily glad that I have never practiced in the excessively litigation-prone USA, where it appears that hard-working professionals can have their integrity questioned and livelihood threatened at will by the likes of Professor Flyvbjerg. Free speech is one thing – this sounds more like libel. I am not a modeller but most of the projects I work on have a modelling and/or forecasting element. In every case I can think of, the major problem has been the quality of the input data, which mainly comes from the client. In the UK the over-riding uncertainty is the direction the overall economy is going to take – how does Flyvbjerg propose accounting for that?

“Flyvbjerg is well-known for publishing retrospective studies of forecast accuracy. Not enough of these studies are done and on the whole I think his work is a laudable contribution. However, I agree that a line has somehow been crossed here. My concern is that draconian ‘punishment’ for faulty forecasts will not lead to better planning using better methods but more likely agencies and consultants reverting to capital investment decision-making based upon non-analytic methods such as Delphi, sketch-planning, or worse, just caving in to political force, whereas what may be needed is the opposite: more forecasts using different models from a variety of analysts to get as many perspectives as possible before acting (‘crowd sourcing’ the planning process is one possible solution to the problem Flyvbjerg correctly identifies).”

“Many of the inputs to the estimates of costs and benefits are remote from the transport consultants – based on unit rates, policy assumptions, growth factors and other assumptions provided by others. Hence inaccurate forecasts are often a consequence of inaccuracies in the associated inputs, and/or weaknesses in the analysis methodologies used – rather than due to errors by transport consultants. There is no malice or incompetence in this… On the other hand, misleading forecasts (malice implied) can only be the consequence of unprofessional behaviour – from either the client, the consultant, or both. I’d hazard a guess that the most common source of misleading information is a poor executive summary – reporting the central results but not the context of uncertainty. Once this is passed up the decision-making chain, diluted further at every step, the context of the advice is lost.”

“It seems to me Flyvbjerg has never had to produce traffic forecasts in the ‘real world’. Traffic or ridership forecasting is as much of an art as it is science and is heavily dependent on numerous variables and assumptions – often dictated by the client, or political or funding realities. Do we ‘punish’ stock market analysts, meteorologists, or sports pundits? No. This is a ridiculous article in my opinion.”

“I do not think the subject being debated is ridiculous. Just because we do not punish others, it is ok for everyone to do the ‘wrong’ thing and get going? … This article is just trying to put forward a thought, ‘can we be more realistic in doing a job as a transport forecaster?’ or do we need to punish them? The ‘punishment’ could also include educating the modeller, the politician, the decision-maker and everyone involved. There is nothing wrong in [raising] subjects that are ‘hidden’ for too long for the sake of job survival.”

“I didn’t say the subject was ridiculous, I said the article was. As a professional engineer I am already held to a high ethical and competency standard. My point is that holding transportation professionals responsible for forecasts when much of the process is not under their responsible charge is unfair and out of touch with reality. Politicians never get punished for these issues – even if they are the root cause. Forecasts, especially where they are the basis for large expenditures of public monies, should be peer reviewed without question. But making the process punitive for the professional is not, in my opinion, the way to improve the process.”

“I have long admired Professor Flyvbjerg’s work because, in spite of the overly dramatic headlines that he likes, the issues that he identifies are real. I don’t mind debating who should take the financial risk for the accuracy/quality of forecasts, but bear in mind that the most important effects of any policy are usually those that were unintended. The most obvious unintended consequence of shifting the forecasting risk onto consultants is the probable increase in consultancy costs (‘insurance premium’). There might also be a counter-reaction with consultants trying to distinguish modelling error from bad model inputs.”

“Ultimately, transport forecasts are ‘wrong’ because the underlying assumptions are wrong. For example:

• Transport demand projections are tied to GDP projections, which in themselves are unreliable. Would anyone consider using projections created in 2005 for their projects given what has happened since?

• It is typically assumed that projects such as business parks and housing developments in local plans will be successful and produce the projected demand of a successful project. This is not always the case and will change the demand profile of the transport system for which we are trying to create projections.

So if transport consultants should be penalised for making unreliable transport projections, should we not also be looking to penalise the Government for ‘wrong’ GDP projections or unreliable local plans? We could also pursue developers for not generating the traffic or the business they claim they will make when they proposed their projects.”