VICTOR BASILI: I started out as a professor at the University of Maryland thirty seven years ago but while at the University I also spent time on many outside projects. That included over 25 years working on the Frauenhofer Project at the NASA Software Engineering Laboratory, where I helped build an environment for ground support software systems. This was essentially a knowledge base, one that contained data about what worked and what didn't work. The project was a success and it helped NASA improve their defect rates and costs. Once it was completed, we built the Frauenhofer Center, where I continue to work on applied research. At the Frauenhofer Center, I work with companies to try to improve their processes. In my opinion, the only way that you can do meaningful research is by looking at real projects within real companies.
What is the value of process and measurement? Why is it so important?VICTOR BASILI: Whatever the product, whether it's software or something else, process is important, because it is the only way to optimize what you are doing. And measurement is important because it's the only way you can observe and get feedback about what is really happening.
How would you have characterized the state of process and measurement in the software and IT industry at the onset of your career?
VICTOR BASILI: Over the course of my thirty-seven year career, I would say it's been slow and spotty. When I first started to talk about measurement in the early seventies, people would ask, “Why are you doing that, why are you asking those questions, why would you possibly want to measure software?” The prevailing attitude was that it was an art, not a science. But over the years an increasingly greater number of people became more and more interested in metrics. At the start, there were a lot of questions that needed to be answered. Questions such as: “Can we measure product success, can we measure the product itself, and can we measure the success of various techniques?” From the very beginning I focused my work on trying to understand how to use measurement as a method for answering such questions. Later on people began to do measurement by data or data management by data. But I was first interested in improving process in particular and for specific environments.
There are so many different organizations and so many different consumers of metrics within all of these organizations. In light of this, how are we supposed to determine what to measure and why we should be measuring?
VICTOR BASILI: Actually that's the Goal-Question-Metric paradigm which was developed in the mid-seventies. It's still the most popular way in the world to measure a model ““ any kind of model. The concept behind it is really rather straightforward. GQM essentially states that before you decide what you measure, you need to figure out what you want to know. Therefore, you have to constantly track your goals, and that's how you track the data. This makes it sound simpler than it really is, but that's the basic underlying premise. The Goal-Question-Metric paradigm came out of our work at NASA. We were trying to analyze data from a variety of projects and we started to recognize that we kept losing track of the questions we were trying to ask. Because of that, we had to keep reorganizing, re-focus setting on the goals, and then re-determining the questions we should be asking ourselves and finding the data to answer them. That's how we avoided losing track of what we were doing. Over time we continued to expand and modify that technique across many, many projects.
Beyond GQM itself, aren't there challenges that IT organizations face in gathering data, both from a technical perspective as well as an organizational or cultural perspective? How do you address both of those issues?
VICTOR BASILI: You start simple by just doing things on paper. You have people fill out data on paper forms. You have them carry them around wherever they spend their time. If you have any defects, you have a defect form that asks questions about the defects. This helps you analyze what to do with defects in the system, what's the class of defect, and whether it's a real defect or just a change. At NASA, there was a lot of data that went all the way back to such forms. Regarding the organizational or cultural aspect of this kind of measurement, what we have found over the years is that people consider measurement more of an annoyance than a threat. Once they recognize that what you are doing is not about them personally but about trying to improve their environment, the resistance almost immediately vanishes.
To what extent is a successful measurement initiative contingent upon standard processes? How do these two things hang together?
VICTOR BASILI: I have a slightly contrarian perspective on this. I think process is a variable that needs to be tailored to the specific problem at hand. Although you may have lots of commonalities in your processes, they each have to be a little bit different for various projects. While we were at NASA Goddard, we developed an approach to this called quality improvement. The first step in using this model is to collect data that doesn't necessarily tell you where you are or what you're doing. We called it “characterized” data but I heard you guys call it “visualized” – and I like that term. I like it because that's what you are doing ““ you are visualizing what's going on in your environment. The second step is to set your goals for particular projects and also for the entire organization. Where does this organization want to be and how will this project contribute to the overall knowledge that is within the organization? In the third step you choose your process, one that allows you to achieve your goals relative to your environment, relative to what the business is about, and relative to what you've got in the present moment. The fourth step is to do it. The fifth step is to try to do as much feedback and analysis as you can in real time in order to help manage the project and to help increase learning from that project. Your real goal is to learn from every project you perform at a particular organization. The sixth step is about the analysis you do after the project has been completed. You've done as much analysis as you can in real time, and that's usually very complicated. But now you must do more analysis. With post-project analysis we try to recognize what really happened, what was a success, what wasn't a success, etc. The seventh step is to package all of this knowledge so that it becomes part of your processes, part of your organization, part of the way you think about how you solve all of your problems. And then the next project comes through and you keep going. You build up and save this knowledge in what we call an experience base. The idea at the root of all of this is to build an “experience factory” within an organization. Such an experience base can tell you at any given point how your projects are being developed. But while all of this is happening you are also learning that every project is an experiment. You're testing and observing what is happening, what should have happened, and what will happen. And you're making changes to the way you understand your organization. At the end of all of this you will end up with a lean and optimized set of processes for various classes of problems. Your future projects will be different, but that's OK because you'll have classes of processes that will work for different kinds of projects. You will be able to conduct predictions and optimize everything you've got, including code.
If you were trying to improve an organization from scratch, from Level 1 ““ how would you proceed? What specific steps would you undertake? What would be the critical success factors?
VICTOR BASILI: The last thing I would do is try and be Level II. The first thing I would do is try to understand where the organization is in terms of defects, effort, process, tools ““ everything that they have available. That would be step one, trying to characterize or make visible everything that goes on in an environment so that you have a basis upon which to make judgments, set goals, and know what to do. As for critical success factors, they would really be dependent upon the organization. What do they want to be? What are their goals? Are they mostly growth goals? Are they mostly success goals? Do they want to be the best they can? What is the rate of growth they want? I can suggest a reasonable set of goals for an organization while also distinguishing between growth goals and maintenance goals. Ultimately, however, organizations have to pick their own goals, on the corporate as well as the project level. They need to figure out where they want to be as an organization and how software will play a role in this. They need to set goals for their organization and relate those goals to software. For example, if they really desire to become an organization that provides great customer satisfaction, they would have to start working on a customer satisfaction goal, and that would become part of the goal set for every project they undertake.
You mentioned maintenance goals. From a process and metrics perspective, how would you address the issue of maintenance? Do you see this as an area of opportunity? If so, what questions should people be asking themselves? Are there any specific metrics that you would advise organizations to make use of?
VICTOR BASILI: Maintenance is a really hot area. When collecting maintenance data you start by asking questions such as, “Where do I spend my time, what are the big bottlenecks in the process, which processes are working, what is the relationship between the kinds of defects I am seeing and the kinds of schedule problems I have, what are the pieces of code that seem to be changing the most?” If you've got a lot of changing code, what do you do? Do you stop and try to redesign? At what point is it appropriate to redesign and what should your criteria be for that? It's a complicated issue.
Biography of Vic Basili
Dr. Victor R. Basili is Professor of Computer Science at the University of Maryland. He holds a Ph.D. in Computer Science from the University of Texas and honorary degrees from the Universities of Sannio (Italy) and Kaiserslautern (Germany). He was Executive Director of the Fraunhofer Center – Maryland and a founder and principal of the Software Engineering Laboratory (SEL) at NASA/GSFC. He works on measuring, valuating, and improving the software development process and product via mechanisms for observing and evolving knowledge through empirical research, e.g., the Goal/Question /Metric Approach, The Quality Improvement Paradigm, the Experience Factory. He is a recipient several awards including a NASA Group Achievement Award, a NASA/GSFC Productivity Improvement and Quality Enhancement Award, the 1997 Award for Outstanding Achievement in Mathematics and Computer Science by the Washington Academy of Sciences, the 2000 Outstanding Research Award from ACM SIGSOFT and the 2003 Harlan Mills Award from the IEEE Computer Society. Dr. Basili has authored over 200 papers, served as Editor-in-Chief of several journals (IEEE TSE, Journal of Empirical Software Engineering) and program chair and general chair of several conferences (ICSE). He is an IEEE and ACM Fellow. Our interview between Dr. Victor Basili and Michael Milutis, Executive Director of the IT Metrics and Productivity Institute, took place in October of 2006.