After introducing the general differences between BIM Capability and BIM Maturity in Episode 11 , I’ll briefly discuss some of the currently available and applicable maturity models[1]. The intention is to understand what other organisations and individuals have already achieved in this space and try to pin-point an appropriate performance-measurement model that can be adopted or modified to assess BIM competencies [2].
This episode is available in other languages. For a list of all translated episodes, pleaser refer to http://www.bimthinkspace.com/translations.html. The original English version continues below:
Why is this important? If BIM implementations by teams and organizations are to achieve the much touted increase in productivity, these implementations need to be measured, compared against some sort of industry benchmarks and – most importantly – independently certified. Without measurement, organizations offering design, construction or operations’ services have no basis on which to improve their processes and deliverables. Without benchmarks and certificates, clients aiming to employ these organizations have no consistent way of understanding their BIM competencies.
It is really a no-brainer that the AECO industry needs a specialized tool to assess BIM implementations in order to identify real BIM abilities from flagrant BIM wash. What requires real thought is [ONE] what BIM ‘metrics’ should this tool measure, [TWO] how to perform these measurements, and [THREE] how to certify measurement results so they can be trusted and depended upon for selecting project partners and/or improving BIM performance. This lengthy post will address a small part of the first question...
The initial step in identifying suitable metrics would be through searching for an existing and suitable performance measurement tool - rather than developing a new one from scratch - and then improve upon it. So, let’s have a quick look at some existing and applicable tools:
Applicable Maturity Models and Tools
A ‘maturity model’ is simply a set of performance improvement levels that can be achieved by an organisation or a project team. There are many maturity models which are relevant to our quest but I’ll only mention a few of them below:
|
1 |
COBIT, Control Objects for Information and related Technology – Information Systems Audit and Control Association (ISACA) and the IT Governance Institute (ITGI) - weblink. |
|
2 |
CMMI, Capability Maturity Model Integration - Software Engineering Institute/ Carnegie Melon - weblink. |
|
3 |
CSCMM, Construction Supply Chain Maturity Model - Vaidyanathan & Howell (2007) [3] |
|
4 |
I-CMM, Interactive Capability Maturity Model developed as part of the National BIM Standard (NBIMS) Version 1 Part 1 - a project of the National Institute for Building Sciences (NIBS), buildingSMARTalliance™ - weblink. |
|
5 |
Indiana University BIM Proficiency Matrix - weblink (MS Excel File) |
|
6 |
Knowledge Retention Maturity Levels - Arif, Egbu, Alom and Khalfan (2009) [4] |
|
7 |
LESAT, Lean Enterprise Self-Assessment Tool - Lean Aerospace Initiative (LAI) at the Massachusetts Institute of Technology (MIT) - weblink |
|
8 |
P3M3, Portfolio, Programme and Project Management Maturity Model – Office of Government Commerce (UK) - weblink |
|
9 |
P-CMM®, People Capability Maturity Model v2 – Software Engineering Institute / Carnegie Melon - weblink |
|
10 |
(PM)², Project Management Process Maturity Model - Kwak & Ibbs (2002) [5] |
|
11 |
SPICE, Standardised Process Improvement for Construction Enterprises - Research Centre for the Built and Human Environment, University of Salford – Hutchinson & Finnemore (1999) [6] |
|
12 |
Supply Chain Management Process Maturity Model and Business Process Orientation (BPO) maturity model - Lockamy III & McCormack (2004) [7] |
|
... |
... |
Table 1. Sample Maturity Models of relevance to BIM
All the above ‘maturity models’ are relevant to the construction industry (there are many others as well) but only two so far have claimed the ability to measure BIM-specific maturity: NBIMS’ I-CMM and Indiana University’s BIM Proficiency Matrix [8]. Since Indiana University’s effort is fairly new and is not yet well documented, I will only review the I-CMM tool below:
A quick focus on the NBIMS maturity effort
Let’s start with the definition: the U.S. National Building Information Model Standard™ (NBIMS) establishes “standard definitions for building information exchanges to support critical business contexts using standard semantics and ontologies...[to be]..implemented in software". NBIM Standard Version 1 – Part 1 proposes a Capability Maturity Model (CMM) for “users to evaluate their business practices along a continuum or spectrum of desired technical level functionality... [and to measure] the degree to which a building information model implements a mature BIM Standard”[9].
There are two versions of NBIMS’ CMM. The first is a static table identifying 11 Areas of Interest (AOI) measured against 10 Levels of increasing maturity (Fig. 1). The second is the Interactive Capability Maturity Model (I-CMM), a multi-tab Microsoft Excel® workbook based on the static table and employing a point score against each AOI.
Fig.1. NBIMS CMM Chart (NBIMS-US v3 Section 5.2 PDF 1.3MB - link updated Oct 15, 2017)
NBIMS’ I-CMM is based on the concept of Minimum BIM; that is, a project needs to achieve a minimum total score of maturity for it to be considered ‘true BIM’. When it was first released, the NBIM Standard, version 1 stated that “one should obtain a minimum score of 20 [points -weighted average] in order to consider true BIM maturity”. It however stressed that the minimum score is not fixed but is “dependent on the date the [the I-CMM tool] is used”. The minimum score can thus change yearly or “as the rhetorical bar is raised and owners demand more from the models being delivered” [10]. In fact, in the newer version of the Excel tool (v1.9), the Minimum BIM score has since been changed to 30 and lately to 40 points
Limitations within the NBIMS I-CMM tool
NBIMS’ maturity model and tool are still in their early days of developmentand may yet change significantly. However, both the model and tool have significant limitations which I will briefly discuss below [11]:
The I-CMM tool has been designed to be used as an “internal tool...[to]...determine the level of maturity of an individual BIM project as measured against a set of weighted criteria agreed to be desirable in a Building Information Model”[12][13]. I-CMM focuses primarily on measuring BIM information management and “should not be used as a benchmark for any other metrics” [14] including those related to architectural, engineering, construction and management. Also, I-CMM is not intended to be used as a “tool to compare BIMs or BIM implementations” [15].
In addition to the above structural limitations, the I-CMM’s scoring system can theoretically generate different results (certificates) for the same BIM project if the tool is employed by different users or at different times. This is highlighted in how NBIMS ‘allows’ those who use the tool to modify AOI weighting according to their specific requirements (see page 79 of NBIMS v1, part 1). This variability in AOI’s weightings coupled with a ‘date-sensitive’ Minimum BIM score limit the tool’s reliability as well as its usability as an industry-wide, market-independent measurement tool.
The need for a comprehensive tool
If one spends enough time and energy to analyse the many strength and shortfalls of available maturity models, s/he soon realizes what is missing: a specialized BIM Capability and Maturity tool that can be used internally by organisation and externally by independent assessors, can measure all key metrics related to BIM, has a consistent scoring system and is equally applicable across markets, disciplines and organisational sizes.
Of course, it is not realistic to expect organizations to independently develop their own measurement tools and impose them on all others. It is also impractical to ask industry players to rely on tools developed by other industries and are unsuitable for measuring BIM. Finally, it is not useful to adopt existing tools that – although developed for BIM but - can neither measure all BIM indicators nor are consistent in their measurement.
So what is the solution to all this? Can a maturity model and a measurement tool be developed to detect ‘BIM wash’, measure ‘BIMness’ and allow trustworthy certification of organizations which invest, develop and maintain their BIM competency?
...
[1] The term ‘model’ in this post denotes ‘knowledge models’ not ‘object-based models’ as typically associated with BIM.
[2] BIM Competencies are the generic abilities that teams and organizations need to acquire as they adopt and improve upon their BIM technologies, processes and policies. These competencies include technical (e.g. exchanging model data) and non-technical abilities (e.g. virtual team management) but all are important to move from Pre-BIM to IPD through a systematic and measurable approach. BIM Competencies are grouped in sets which are employed to establish either Capability or Maturity benchmarks. I’ll discuss BIM Competencies in a future post.
[3] Vaidyanathan, K., & Howell, G. (2007). Construction Supply Chain Maturity Model - Conceptual Framework, International Group For Lean Construction (IGLC-15). Michigan, USA.
[4] Arif, M., Egbu, C., Alom, O., & Khalfan, M. M. A. (2009). Measuring knowledge retention: a case study of a construction consultancy in the UAE. Engineering, Construction and Architectural Management, 16(1), 92-108.
[5] Kwak, Y. H., & Ibbs, W. C. (2002). Project Management Process Maturity (PM)2 Model. ASCE, Journal of Management in Engineering, 18(3), 150-155.
[6] Hutchinson, A., & Finnemore, M. (1999). Standardized process improvement for construction enterprises. Total Quality Management, 10, 576-583.
[7] Lockamy III, A., & McCormack, K. (2004). The development of a supply chain management process maturity model using the concepts of business process orientation. Supply Chain Management: An International Journal, 9(4), 272-278.
[8] Indiana University BIM Proficiency Matrix includes 8 categories measured against 4 maturity/proficiency levels. The matrix appears to focus on the accuracy and richness of the digital model (as an end-product) and has little focus on the process of creating that model. More information is available here (PDF). Thank you to Dr. Umit Isikdag (University of Salford - UK) for bringing this effort to my attention.
[9] NIST. (2007). National Building Information Modeling Standard - Version 1.0 - Part 1: Overview, principles and Methodologies: National Institute of Building Sciences (Page 75).
[10] Same as above – Page 72
[11] I will be reviewing the NBIMS CMM and I-CMM in more detail as part of my upcoming chapter ‘BIM Maturity Matrix’ in the Handbook of Research on Building Information Modeling and Construction Informatics: Concepts and Technologies (http://bit.ly/BIMhandbook).
[12] See AECbytes Viewpoint #33 (December 6, 2007), http://bit.ly/AECbytes1.
[13] Suermann, P. C., Issa, R. R. A., & McCuen, T. L. (2008). Validation of the U.S. National Building Information Modeling Standard Interactive Capability Maturity Model 12th International Conference on Computing In Civil and Building Engineering, October 16-18. Beijing, China.
[14] NIST. (2007). National Building Information Modeling Standard - Version 1.0 - Part 1: Overview, principles and Methodologies: National Institute of Building Sciences (Page 80).
[15] Same as endnote 10.
