Could you tell us a little bit about yourself, your background and what you are working on today.
PATRICK O'TOOLE: I started out in IT back in the late 70's. During that time period I worked as a systems analyst, project manager, and ultimately program manager. I then made a career switch and assumed more of a systems engineering role working on high-speed imaging products that were being released to the financial services marketplace. By the early 90's, we were capturing images of checks at 30 images or 30 checks per second, which was pretty high speed at that time. We were counting on Intel's 486 chips to be released on time so we could actually do the processing: that's how long ago this was. After that, I decided that I really wanted to consult in the area of project management, so I left the company I was with and started up my own consulting company. I found that I could get jobs as a contractual project manager; nevertheless, I wasn't really doing consulting in the area of project management. Then one day I attended a conference at the Software Engineering Institute that focused on risk management. I was fortunate enough to be seated next to the president of a risk management consulting company and we stayed in touch over the Internet. About three years later, he gave me a call and asked if I was interested in working with him. That's when I joined Teraquest, which is Bill Curtis's company. Bill is one of the authors of the CMM for software. I worked with Teraquest for about 4 and a half years, became a lead appraiser and worked with several organizations to help them achieve a higher level of maturity. However, when I noticed that I was spending 95 percent of my time on the road, I decided that with 2 small kids at home I had better make an adjustment. So I gave Teraquest about a year's notice, made a separation plan, and started up my own company in 2000. Since that time, I have essentially been focusing on training, assessing, and consulting primarily in the area of CMMI, essentially same things that I was doing with Teraquest.
Can you talk a little bit about your status as a visiting scientist at the SEI?
PATRICK O'TOOLE: Essentially, the SEI uses part-time employees to help with certain areas of specialization. I teach the intermediate concepts course which is a 5 day upper level CMMI course that can only be offered by the SEI. It's a simple pass fail class that is a prerequisite for becoming an authorized lead appraiser or CMMI instructor. I also teach the intro course on behalf of the SEI. In fact, I just spent two weeks teaching this class in Brazil. Another thing that I do at the SEI is observe first time instructors.
Could you tell us a little bit about the historical background of the CMM and the CMMI – how it started, how it evolved and any changes that are currently taking place today?
PATRICK O'TOOLE: The story, as I've been told it, is that Watts Humphrey went to Phil Crosby's quality college down in Florida, and as he was heading back home to Pittsburgh he thought, “How can we apply something like the Crosby model, which is a 5-scale model, to the wonderful world of software?” So on the proverbial back of an envelope, he sketched out the 5 levels of the Crosby scale and what they would mean in terms of software, and this ultimately evolved into the CMM. The CMM is simply a collection of good practices that organizations have found to be successful. When they were putting the model together, they invited several highquality software development organizations to congregate in Pittsburgh. They all sat around in a conference room and wrote the practices that had made them successful on yellow stickies. After they pasted the walls with these things, they went through an infinity diagram exercise and clumped them into groups. They found a large group of project management practices, a large group of engineering practices and a large group of support type activities. These groups ultimately became the key process series. That resulting set of practices ultimately turned into the CMM for software, which served many companies very well for several years. The CMM was primarily a software model because companies were finding that they were having problems not so much with hardware, but more with software. They were more and more dependent on software and software was increasingly letting them down. The need for software improvement was what ultimately led to the funding of the SEI and the development of the CMM for software. After several years of use, the CMM started getting crusty around the edges. It was a good first attempt at a model, but the SEI decided that they needed to upgrade it and began revising it and developing the CMM for Software. The other thing that was happening was that more and more maturity models were being produced, including a maturity model for system engineering and another one for people. At one point, the SEI claims they had 17 maturity models either published or on the drawing board to be developed. In response to this, the Department of Defense said, “Wait a minute; every time the SEI releases a new model we have to train people on it and we need another appraisal method and another appraisal to go through. Let's just get it down to one model.” That's when the SEI determined that it was time to do a consolidation. They brought together three models: the CMM for software, the System Engineering CMM, and the IPTD that was just about to be released (the integrated product team CMM). They consolidated these three into what is now known as the CMMI, which was published in early 2000. The CMMI went through a couple of trial periods, then ultimately fell into version 1.1, which became the official release for the model. Last August, it went through another fairly significant revision and now we're up to version 1.2.
What is your opinion of the progress that has been made to date in terms of CMMI penetration in large organizations?
PATRICK O'TOOLE: I think one of the things we have found, especially in the Department of Defense area, is that the CMMI is almost a rite of passage. Organizations have to adopt the model, because the Department of Defense gives preferred nation status to any organization that has achieved maturity level 2 or maturity level 3, depending on what kind of software or engineering systems need to be developed. The use of the CMMI has now extended into hardware engineering, system engineering, electrical engineering and mechanical engineering. At this point, any project oriented engineering activity can use the CMMI to help guide its activities. The Department of Defense is still on the leading edge of pushing it, but it's also expanding into other areas. For example, the automotive industry is very adamant that software suppliers and other engineering suppliers achieve a certain level of maturity, or at least get on the path to doing so. General Motors has a requirement for their suppliers to be at maturity level 2 by a certain date and maturity level 3 by another date. Chrysler has done the same thing, but it has taken a slightly different approach. Rather than focusing on maturity levels, which is a staged representation of the model, Chrysler focused on the continuous representation of the model. They take the process areas they believe benefit them as a customer the most, and then they insist that their suppliers achieve certain capability levels in just those process areas. In some respects, I think this is a good trend because it's pushing organizations to do what they should be doing on their own. In other respects, however, I think it's a bad trend because engineering organizations will wind up striving for the number, not necessarily for the improvement. Many organizations approach CMMI-based initiatives thinking, “Let's do anything we can to get to level 3, whether or not we achieve any improvement.”
We hear a lot about Six Sigma, ITIL and of course the CMM. How do you differentiate between these different improvement programs and models and how do organizations know where they should be putting their energies? What, if any, is the interrelationship between all of these things?
PATRICK O'TOOLE: There are a lot of models out there and it can be very confusing to people. If you're doing data center or maintenance type support – a kind of flow activity where it's not really a project oriented activity – then ITIL is probably the proper choice. Six Sigma can be very beneficial, but it suffers from some of the problems that we see in the low maturity organizations. That is, if they are not doing things in a reasonably consistent and repeatable way, Six Sigma can be applied, and can even be beneficial, but the real benefit won't materialize until a greater consistency is achieved. My typical recommendation is to get to CMMI maturity level 2, stabilize your projects, and then start thinking about how you might be able to apply a little more sophistication to take those stable projects to the next level. At this point, I think Six Sigma and CMMI work very nicely hand-in-hand, especially at higher maturity levels. And once you are looking at maturity level 4 and 5, Six Sigma is a perfectly natural fit. The one thing I would suggest to organizations that have multiple quality initiatives going – Six Sigma, CMMI and ITIL – is that they make sure to pick one as the master. I don't care which one it is, but you're going to get into bizarre turf wars over various quality initiatives if you do not designate a primary initiative. For instance, you might say, “We're doing Six Sigma, and underneath that umbrella we've got an ITIL portion that we have applied to the flow activities and a CMMI portion going on for the projectoriented activities, but it's all being done under the umbrella of Six Sigma.” In other words, one intiative is master and the others are subbordinate to it.
For those organizations that have had a reasonably successful CMM program in place, how would you characterize their return on investment?
PATRICK O'TOOLE: That is always a tricky question, because it is kind of a selfselecting group. Who is going to report their data? Well, it is going to be the organizations that had glorious success. Nevertheless, the numbers that they show are compelling. They can show five-to-one payback – for every dollar invested you get a five dollar return – and that is very good. I think the ones that have the data and are reporting it are being honest; nevertheless, they are self-selected and that is something to keep in mind. What I would be more interested in finding out is what occurred within those organizations that have not achieved success. What have been the failure points? Why did they abandon the effort or settle for less? I think part of the issue, and this is addressed a little bit in the CMMI, is that when organizations measure the benefits achieved or the progress that they've made, they measure this through the accomplishment of maturity levels. But maturity levels are nothing more than a recognition that you've achieved a certain level of process maturity. They do not reflect anything about the value that you have derived from that stabilization. Organizations would be well served to say, “In order for us to be successful in the marketplace, we need to be a high quality producer, so the process we're going to build and the CMMI that we're going to use as a guideline will have to be focused on achieving high quality.” Too often organizations say, “Let's get to maturity level 3,” and they don't consider what they are really trying to achieve as a business.
There are some statistics out there that suggest that 80% of IT money is spent on maintaining existing systems and infrastructure, whereas 20% of the total investment goes toward development. To your knowledge, is the SEI addressing maintenance at all? Is there any possibility that we might see a CMM for maintenance in the future?
PATRICK O'TOOLE: Well, the CMMI that was released in August was called CMMI for development, so this presupposes that the SEI is recognizing the need for different flavors, or what they call constellations. The CMMI for development is the first of three that they have planned, the other two being the CMMI for services and the CMMI for acquisitions. To the best of my knowledge, though, they do not have maintenance on the drawing board yet. A lot of organizations treat their maintenance projects. Consequently, you would leverage the CMMI for development, because teh CMMI for development is designed for project-oriented development involving things like minor enhancements. I think that the thinking around maintenance is that if we were to develop higher quality products in our development activities and if we were to support these with better documentation, better designs, and better architectures, then the maintenance effort over the long term would naturally decrease. So rather than trying to fix the systems that we already feel are eating up 80% of our resources, the best we can hope for is to reduce those same mistakes in the future and to release higher quality, more maintainable, more supportable products in the beginning of the lifecycle.
For organizations that are undertaking the CMMI and implementing it, what are the critical success factors?
PATRICK O'TOOLE: I think something that really helps a CMMI implementation be successful is the burning platform: the last release that went out was a major disaster, it blew up in the field, the customers were irate and it resulted in lawsuits. That failure motivates an organization to say, “We can't keep doing things the way we have been doing them. It isn't working.” Barring that, I have actually recommended to clients that they would be well served to have a major disaster, because that is what they need to convince the workforce that things really need to change. They usually don't listen to that kind of advice, however. Other than that, strong management support is crucial. And this goes beyond just sponsorship. To me sponsorship means we put a team together, we fund the team and after a month we get together with the team and listen to what they have to say. My preference, however, is that have the executives actively get involved. I want them to document some of their processes and to ensure that they are following their processes and make sure that they are improving them over time. I want the executive who can stand up and say, “Look, I'm becoming more process disciplined. It appears to be working for me and my team, therefore I think it would be beneficial to the engineers and the project managers as well.” We need executives who can stand up and lead by example.
For organizations that are attempting to work with the CMM and are having problems, could you outline some of the biggest pitfalls and some of the most common mistakes made?
PATRICK O'TOOLE: I think the biggest problem is the level. They may start off well and say, “We really need to improve and we're using the CMM as a guidebook to help us do that.” Then they go at it for six months, maybe twelve months, and because the changes are gradual, the benefits that they're achieving is not really evident unless they step back and assess how far they have come. If they can take that longer term perspective they will see that things are getting better. But at some point the executive will say, “This is going way too slow. I know we're in it for improvement, but darn if we are going to make level 2 six months from now.” And so the pendulum swings from doing it for the right reasons to getting to the level. The level unfortunately becomes the end all and be all, and whatever process improvement group you have in place starts putting their energies into generating templates and forms to show that they have achieved the level, rather than staying focused on gaining the day-to-day value. When I went to college, I didn't get smarter the day I was handed my diploma. I got smarter over the four years that I was at the University, went to the classes and did the homework. Achieving the diploma was just the recognition of the accomplishment; it was not the accomplishment itself.
What advice would you give to a Level 1 organization if they had 9-12 months to demonstrate value through a process improvement effort? What should they be focusing on and how should they undertake this?
PATRICK O'TOOLE: I would take two different perspectives on the work being performed. The first question I would ask is, “Where's the pain?” In other words, what are the problems the organization is facing and how can we use the CMMI as a medical reference book to find practices that we believe will address the pain? After that I would ask, “Where do you need to be successful?” For example, if you're trying to achieve high quality in order to be successful, why not look at the CMMI for practices that will lead to higher quality? Peer reviews are one example of a practice that will lead to higher quality, but they happen to be staged at maturity level 3. But who cares? If peer reviews are going to lead to higher quality, and higher quality is what is going to make you successful, you should get implemented as early as you can in order to get the value as quickly as possible.
Is there anything else that you'd like to talk about, any area that we didn't cover?
PATRICK O'TOOLE: The one final thing I want to comment on is the appraisal process. The SEI has now come out with a family of appraisal methods. There are class A methods, where you're actually going for the gold, but there are also class B and C methods that are less rigourous. We typically use class B and C methods for gap analysis or progress analysis. These kinds of less rigourous appraisal methods can really be used as a health check. We can not only look for compliance to the model, which I think is a rather bizarre term anyway, but we can also determine how well the practices of the model are being adopted by the organization. An organization is trying to become the high quality producer. Are they achieving that? Is the organization achieving that? Let's use the appraisal method not just to look for conformance to the CMMI, but also to explore personnel's ideas on how they can achieve higher quality and win in the marketplace. I think appraisals are an underutilized tool. Most organizations look to them to determine whether or not they have hit their number. But I think there is a whole other use for these things. They're expensive, they're time consuming and they're somewhat disruptive, but we've got to find a way to optimize their value.
Biography of Patrick O'Toole
Pat O'Toole is the Principal Consultant at Process Assessment, Consulting & Training (PACT) where he provides a full range of services to his process improvement clients. Pat is one of the most active CMMI lead appraisers, and has led appraisals spanning all maturity levels, including one of the largest and most complex CMM Level 5 assessment conducted to date. He is an SEI authorized instructor for the “Intro to CMMI” course who has taught this course more than 40 times in 6 countries. Pat is a Visiting Scientist at the SEI, and teaches the “Intermediate Concepts of CMMI” course on their behalf.