This article was jointly written with Ken Tombs and oringinally published by the UK National Computing Centre.
Peer reviews come in many shapes and guises. Most are informal and low in method; a few are high in method and rigorous in application. Those in the commercial sector usually have teeth.
Those in the public sector often don’t.
Reviews tend to focus on the ‘tin and plastic’ yet pass by the cultural and capability issues.
Programme and project management methods are now common, yet sophisticated industrial projects are still delivered using no more than a spreadsheet or process notation tool, commonsense and of course some attitude.
Wherever you may sit in this spectrum, a more formalised peer review is considered a highly effective project assurance method. In fact, according to an official Independent Review, the UK Government’s OGC Gateway peer review programme is undoubtedly the most effective mechanism government has in improving project delivery.
Project managers within the private sector will probably know something about the OGC Gateway framework. It started life at Marconi with Sir Peter Gershon, travelling with him as he formed the Office of Government Commerce.
OGC Gateway was initiated during 2000/2001 to deliver UK government policy on project performance. With a substantial emphasis from government and permanent secretaries, it was quickly recognised as useful by project management.
By 2006 OGC Gateway review teams had completed many hundreds of reviews, with a few completing the then total of five gates or key stages in a project’s lifecycle.
Today, OGC Gateway is gaining a level of acceptance in Australia, New Zealand and the Netherlands. It now has six gates; an additional Gate Zero looks at programmes and the other five gates at individual projects. Overall, the framework for OGCGateway has remained the same for over a decade, proving to be one of its strengths.
The Independent Review mentioned above was commissioned to ask fundamental questions, with hard evidence supporting the analysis – most notably ‘Was OGC Gateway delivering?’. The expression ‘evidence’ here goes beyond just professional opinion and a few surveys, with the review relying on statistical justification where possible.
Accompanying this analysis was an evaluation of other organisational project assurance schemes as comparators, finding the majority of such schemes lacked real rigour. Only two quasi-governmental schemes were considered valuable, from the World Bank and the North American Space Administration (NASA).
In fact, NASA became the lens through which OGC Gateway was viewed, as it had used peer review successfully to turn around catastrophic failures that had cost lives (not just cash and reputation). Ken Costello, then chief engineer at NASA’s Independent Verification and Validation facility, was an evangelist for this approach and it worked.
So what has NASA’s project management approach taught us? The NASA expression ‘Will it fly?’ (literally) focused everyone’s attention on a project’s likely outcome.
Yet while NASA’s approach got to grips with its live projects, it didn’t overcome pre-project prevarication. For example, NASA’s development of the International Space Station cost billions of dollars and involved the re-invention of many wheels over again
Nevertheless NASA placed great emphasis on parametrics for all designs and components, and its ruthless rationalisation to stop re-inventing wheels paid handsome dividends. NASA began to build working and reliable, but not necessarily the absolute best, solutions.
Apollo 13, with non-interchangeable carbon dioxide filters between the moon lander and command module, was a good example of this as shown in the film of the same name. NASA built a bridge between in-project parametrics and pre-project thinking and planning, which generally remains weak in the majority of projects.
Returning to OGC Gateway, the outcome of the Independent Review was clear: OGCGateway was the only game in town to improve major government projects. Yes, there were some fundamental flaws – though not in the framework itself which was remarkably robust. Rather, gaps existed around the various cultural and managerial interactions, plus a lack of consistently applied solid metrics.
For example, a serious shortcoming was found in the then red, amber, green (RAG) status allocated to projects. This proved to be no more than a bartering tool, with a minimal relationship to a review team’s expressed sentiments. Since then the RAG status has thankfully evolved into a more valuable five-step expression of delivery confidence.
Yet the haphazard collection of review team assessments remains, with its inability to combine quantitative and qualitative techniques to create appropriate checks and balances.
With the UK’s Major Projects Authority now taking up the reins, where does OGC Gateway go from here? Is there a value to the private sector as well as the public sector? Or no value at all? That very question is now being considered by the Efficiency and Reform Group (ERG) in the Cabinet Office, and the authors thank ERG for their open assistance given here.
So what lessons are there for private sector organisations, who work very differently to the public sector? OGC Gateway is UK government intellectual property with strict rules about who can and cannot use it. Regardless, there are many people unofficially running ‘key stage reviews’ based on the framework and workbooks are available for download, as are the report templates.
Historically, the OGC justifiably re-inforced that only authorised reviews should be trusted by management. Experience shows that private sector reviews must have the confidence of their top management; otherwise they will lack weight and make little practical difference.
OGC Gateway has, over several thousand reviews, built up a valuable knowledge base of what can go wrong. This could be invaluable to the private sector on progressing from less to more structured peer reviews.
To make it palatable for the public sector to run reviews that fed back potentially career-limiting information, the reviewers could never realistically recommend stopping a project. All reports were kept confidential, with a few people trying legally, and mostly failing, to access those reports and their assessments. This is an area unlikely to impact the private sector directly, unless a company is facing a shareholders’ investigation.
(NASA took a tougher line. For them, the review team could recommend to the chief engineer that a project should stop; it then became his role to further investigate the various pros and cons, including the political dimensions.)
A ‘Sword of Damocles’ is essential to structured peer review methods, for without it the evidence is that organisational politics will tend to find a way to keep it going. And that was the intention of OGCGateway: that it would challenge projects that weren’t working, or could never work, as sometimes officials found it tough to say ‘no’ to ministers and projects took on a life of their own.
Public sector OGC Gateways have matured, with those being reviewed increasingly knowing how to ‘play the game’. So a review team will spot hyper-polished documents, well-briefed interviewees ‘on message’ and so on.
The private sector more typically just gets on and does things without massive paper trails. Bureaucracy or brevity…both bring strengths and weaknesses to the review process and OGC Gateway is robust enough to deal with both.
Ultimately, of course, you cannot administer a project to a successful conclusion. It requires hard and challenging decisions that are frequently unpopular or counter-cultural. It seems an oddity that this still remains outside of peer review consideration, OGC Gateway or otherwise.
The future for peer review has to encompass not just the tin and plastic but also the psychological and organisational drivers that have the greatest impact on success. We’ve cracked project management methodology, done well with peer review so far.
Now we need proper consideration of team character and the cultural context for a project’s solution.
Undoubtedly OGCGateway needs something of an injection of new spirit, energy, reduced scale, better technique and truly independent reviewers. However, it must not be discarded for it has much experience and method to learn from: we must build on experience, not arbitrarily discard it. The temptation is strong to convince ourselves that there is always a better mousetrap to be invented.
OGC Gateway remains a viable, more management-friendly alternative to IV&V as practised mostly in the United States, with a lot of time, intellect and resources already invested.
OGC Gateway’s strengths and weaknesses, as its own parametrics, are well understood. Additionally, its core framework can scale and adapt remarkably well to meet future political and organisational needs.
So has the time come for the next generation of peer review to take shape around OGC Gateway, joining public and private practitioners into a common, comparable and more independent framework?