Acquisition Research

At present, we perceive a critical need to catalyze a quantum increase in the quality and quantity of new acquisition knowledge produced through scholarly research. Although research represents only one of several important knowledge sources—others include, for example, professional practice, trial and error, lessons learned—it is arguably the most neglected at present and the most critical for the future, particularly at this time when "out of the box" thinking and radical process redesign are called for.

Need for Acquisition Research

In his classic work, Kuhn (1970) describes the idea of "revolutionary science" as exemplified by the "paradigm shifts" from Newtonian to Einsteinian physics, or from Ptolemaic to Copernican astronomy. Such revolutions can occur, according to Kuhn, as evolving conditions lead to situations for which the extant paradigm simply cannot account. It seems likely to us that acquisition stands on the threshold of such a paradigm shift. Earlier we noted the heritage of US defense acquisition in the post-World War II, Cold War environment. Recent acquisition reform initiatives notwithstanding, many of the laws, policies, regulations, and practices that govern present-day acquisition are products of the Cold War mindset. Given the fundamental changes of the past decade, we question whether that particular paradigm can long endure.

As the US enters the twenty-first century, it confronts a new military environment characterized by expanding mission requirements, declining defense funds, and the absence of a monolithic superpower threat. This environment has drastically changed the face of acquisition. For example, policy initiatives associated with "reengineering" and "downsizing" are blurring the traditional distinctions between the public and private sectors' acquisition roles in several ways. First, new public-private partnerships may be radically reshaping our view of the proper relationship between government and industry. To illustrate, recent plans called for a government weapons facility at Rock Island Arsenal, Illinois to produce items, in effect as a subcontractor, to United Defense Limited Partnership, the industry consortium under contract to develop the Army's new Crusader howitzer system. Second, the nature of government contracting is changing in significant ways. The government now relies far less on detailed and restrictive specifications and standards, choosing instead to communicate only broad objectives with the details left to industry discretion. We even see evidence of a trend away from strict reliance on contracts in the increased use of simplified agreements—so-called "other transactions"—between government and industry. Perhaps most significantly, functions that previously were performed almost exclusively by DoD—logistics support for weapon systems, for example—are now increasingly being "contracted out" to private industry.

Such changes call for new acquisition processes. The nature, scope, and pace of change required to effectively transform these acquisition processes imply that new knowledge will be required. Change of such magnitude and speed is unprecedented within the defense acquisition system; hence leaders cannot simply reuse old ideas and techniques. Rather, these new processes require new knowledge—theoretical knowledge to guide high-level policy and decision-making; applied knowledge to support transition and execution in the new acquisition environment; reliable, generalizable, cumulative knowledge to leverage problem solutions across many defense programs and avoid redundancy or duplication. New acquisition knowledge such as this calls for research, as the researcher's primary motivation is knowledge creation (e.g., discovery research).

Without research of a relatively fundamental, loosely applied nature, it is next to impossible to achieve paradigm shift, and it would be inconceivable that such a shift could occur through incremental changes in acquisition practice alone (i.e., without research). Researchers provide a unique ability to generalize from experiences. They build cumulatively upon the work of others (what Kuhn calls "normal science") and employ rigorous methods to ensure high validity and reliability of their results. Indeed, only research that stretches the boundaries of current knowledge can be used to leverage solutions across entire classes of problems (e.g., through new theory) and to adapt effective solutions induced from one process or program to many others. And academics are trained to design experiments and employ rigorous research methods that isolate effects and minimize the cost of knowledge creation. Such research requires careful planning and preparation and is time-consuming. But it minimizes exposure to failure from trial and error (e.g., as with professional practice, on-the-job training, lessons learned) and maximizes the impact and dependability of results per unit cost. Thus, academic research is efficient as well as effective at knowledge creation. By building on the cumulative work of others, researchers are able to avoid the redundancy, duplication, and waste that plagues many current acquisition reform efforts in practice. Of course, research also feeds education, training, consulting, and ultimately professional practice itself, as new knowledge creation (i.e., research) sits at the top of the knowledge "food chain."

State of Acquisition Research—Quantity

Despite acquisition's critical role and its changing nature in the defense domain, research in acquisition has been sorely neglected. Scholars outside of the DoD community have for the most part simply ignored acquisition as an area of inquiry. To illustrate, a review of titles for the 100-plus panels conducted at the year 2000 annual conference of the American Society for Public Administration revealed no presentations or papers dealing with acquisition. Such inattention may be attributed in part to society's, and hence academe's, historical tendency to draw distinctions between military and civilian matters, and to the separate identity of the military created by its unique role and ethic. These can lead to ignorance—perhaps even a distrust or fear—of military matters among non-government scholars (Jefferies 1977). At the very least, such perceptions indicate to scholars that "defense is different", and thus they inhibit consideration of similarities between the defense and non-defense domains of acquisition.

As for the few civilian scholars who have attended to acquisition, some are highly critical and some take a more balanced perspective. The critics usually focus on the highly politically charged atmosphere of major weapon system acquisition. Their characterizations of acquisition as an often irrational process are evident in their works' titles, such as The Pentagon Paradox (Stevenson 1993), Foregone Conclusions (Lebovic 1996), and Weapons Without a Cause (Farrell 1997). Scholars taking the more balanced approach are represented by Thompson and Jones (1994), Fox (1974, 1988), and Mayer (1991). Other researchers write on topics that, while not "acquisition-specific", are central to acquisition. Aaron Wildavsky's (1969) work in budgeting and policy analysis is but one example.

Within DoD, the potential benefits of acquisition research have long been recognized (Strayer and Lockwood 1975), yet little substantive research has emerged from DoD sources. Past attempts to enhance acquisition research include establishment of the Army Procurement Research Office in 1969, the Procurement Research Coordinating Committee in 1971, the Federal Acquisition Research Symposia in 1972, the Air Force Business Research Management Center in 1973, the Federal Acquisition Institute, and the Naval Center for Acquisition Research in 1977 (Office of Management and Budget 1980). We speculate that the general ineffectiveness of these efforts is attributable to the dominant Cold War acquisition paradigm under which they were all undertaken. That is, because the underlying assumptions regarding threats, missions, budgets, and so on remained stable, new knowledge (as derived by research) received little value and priority.

DoD's institutions for education and training have also contributed little to acquisition research. Dedicated acquisition curricula are relatively new additions to the respective graduate schools of the Navy and the Air Force. In the past, the faculties have focused on the task of preparing students for acquisition jobs, rather than on research. At acquisition training institutions like the Defense Systems Management College, a few programs of research in acquisition are ongoing, such as the biennial Acquisition Research Symposia, the Military Research Fellowship Program, and a program of Research on Acquisition Research (Abellera 1993).

A need for acquisition research in DoD was reflected in the legislation that established the Defense Acquisition University (DAU). The Defense Acquisition Workforce Improvement Act, enacted in 1990, gave DAU the charter to lead the training and education of acquisition professionals, and it also provided for a research function in DAU. Understandably, DAU's focus during its first several years was on training and education, with little attention being given to research. In 1994, DAU began publishing a refereed journal, Acquisition Review Quarterly. To date, the great majority of articles have been practitioner-oriented works, such as tutorials, "lessons learned" papers, and opinion pieces, rather than papers with theoretical rigor.

State of Acquisition Research—Quality

Exacerbating this situation of the general lack of acquisition research is the fact that much of what is currently performed within DoD tends to be very applied in nature and lacks research rigor. This is not to imply that applied research is less valuable than basic or exploratory work, but research is governed by a well-understood maxim: the more applied the work, the more narrow the benefits of its results. By contrast, the more fundamental the work, the wider the coverage of benefits. Further, unless research is conducted with the kind of rigor demanded by top academic journals, the results risk duplication with previous efforts (e.g., if not guided by a thorough literature review), confounding of causal effects (e.g., not being able to assess a particular result to decisions made or actions taken), non-generalizability (e.g., results that apply only to the specific case, process, program or system studied), and other threats to validity (e.g., rival hypotheses, concept invalidity, unreliability) (Campbell and Stanley 1973, Yin 1994).

Research that tends to be very applied in nature and which is conducted with little rigor is classified as "1-1" and "2-2" work using the research framework depicted in Figure 1 (Nissen, Snider, and Lamm 1998). Briefly, on the horizontal axis we have the fundamentalism or "basicness" of the research, which corresponds roughly to the standard research categories used in the DoD—management and support, engineering development, advanced development, exploratory research, and basic research (Fox 1974, 22). As depicted by the five-point scale for this axis, work toward the extreme end of the scale characterizes research of a more fundamental and general nature that seeks to solve broad classes of problems in a domain of investigation.

Figure 1: Acquisition Research Space

As research moves toward the origin along this dimension (i.e., becomes increasingly applied), the associated research takes on a narrower, more specific, shorter-term character. This helps to depict the natural migration of research from the basic and exploratory development of new knowledge toward management and applied work as research in an area matures. This dynamic pattern also highlights the need for systematic introduction of new knowledge and ideas—that derive from more-fundamental investigations—through applied research. Indeed, without such fundamental (e.g., basic, exploratory, developmental) research, a program based solely on applied work will eventually stagnate and regress into a pattern characterized by recirculation of old ideas. In fact, a number of scholars perceive this pathological pattern exists in the acquisition domain today (Williams and Arvis 1985).

Returning to the research space diagrammed in Figure 1, the ordinate is used to depict the methodological rigor associated with research (in any category, basic or applied). This five-point scale is used to classify the increasing use of high-confidence research methods that leave decreasing margin for refutation of the results. For example, work at level 1 (i.e., lowest level of rigor) may involve an "investigator" who is not even objectively detached from the work being studied (e.g., a knowledge worker simply reporting the results of his acquisition work). At level 2, an independent investigator is at least in a position to objectively observe and describe some acquisition phenomenon of interest. At level 3, this independent investigator conducts a thorough literature review in a particular area, in order to avoid duplicating previous results and to focus on the kinds of high-payoff research targets and topics that can only be identified through an understanding of, and appreciation for, previous work in a research area. At level 4, this investigator ensures reliability and generalizability of the results by employing a well-founded research design (e.g., multiple case study, factorial, stratified survey). At level 5, the researcher may even employ experimental (or quasi-experimental) methods—like those stressed in the physical sciences—in order to promote the highest levels of confidence in the results.

Two main points emerge from this diagram. First, the majority of extant research in the acquisition domain would be classified near the origin of this research space, as depicted by the "extant research envelope" in Figure 1. This tends to represent just POK (plain old knowledge) work and specialized consulting, more than what most academics would even consider to constitute "research", and it suffers from high refutability and lack of generalization. Although the contribution of such work is positive, it is minimal in that it tends to address only one specific problem at a time, is often redundant with previous or parallel work and offers results confounded by poor methodology. This arguably represents a sub-optimal allocation of scarce research resources. Second, any acquisition research—whether basic or applied—needs to be scholarly to overcome the refutability and generalization problems from above. These points are used to establish the acquisition target research area depicted above the horizontal, "scholarly research" line in the figure.

As empirical evidence of these claims, we examined some seventy articles published from the Acquisition Research Symposium (Brown 1997). It represents a principal outlet for acquisition research in the US. Using the same two-dimensional research space described above, we present the results from categorizing these papers in Figure 2. Notice the mean (denoted by a small circle icon) falls within the "2-2" quadrant, and the 90 percent confidence ellipsoid (delineated in two dimensions by lines extending outward from the mean) indicates the average acquisition research paper falls within the ("3-3") extant research envelope delineated above. Indeed, only three papers fall outside this envelope, and no paper crosses the "scholarly" threshold at level 4 along the rigor axis. Another point pertains to the modal value (denoted by the largest diamond icon in the chart): it lies squarely at the "1-1" point. In other words, this empirical evidence suggests the characteristic (i.e., modal) acquisition research paper reflects "1-1" research.

Figure 2: Acquistion Research Plot Points

How to Attract Quality Researchers in Quantity?

The preceding discussion indicates the pressing need surrounding the current state of acquisition research: to engage scholars who can perform high quality (i.e., at "4" or "5" levels) research in quantities sufficient to generate knowledge and understanding of acquisition appropriate for its contemporary problems. Satisfying this need means that researchers in leading civilian institutions must be actively sought out and attracted by some means to this work. It also implies that these researchers must be convinced that the defense domain of acquisition is not in fact "different", but rather that it represents a fruitful area for study and for extending their research into exciting and important new directions. To these ends, the DAU has established the External Acquisition Research Program (EARP).