Developmental Research: Studies of instructional Design and Development

Notes from Developmental Research by Rita C. Richey, James D. Klein & Wayne A. Nelson:

Developmental research can be either
1. The study of the process and impact of specific instructional design and development efforts or
2. A situation in which someone is performing instructional design, development or evaluation activities and studying the process at the same time; or
3. The study of the instructional design, development and evaluation process as a whole or of particular process components.

The distinction is made between performing a process and studying that process. Reports of developmental research may take the form of a case study with retrospective analysis, an evaluation report, or even a typical experimental research report.

Development in its most generic sense implies gradual growth, evolution and change. In the field of instructional technology development has a particular somewhat unique connotation: development is “the process of translating the design specifications into physical forms.” (Seels & Richey, 1994)
Thus it refers to the process of producing instructional materials. The 1994 definition of the field attempts to clarify these issues by viewing design as the planning phase in which specifications are constructed, and development as the production phase in which the design specifications are actualized. However, the word development has a broader meaning when used in connection to research. The focus now includes comprehensive evaluation, as well as planning and development of the instructional materials.
Developmental research may address formative, summative and confirmative evaluation. The next step beyond utilization & maintenance would be impact, the follow-up analysis of the effects of an instructional product or program on the organization or the learner. This type of research typically falls within the scope of traditional evaluation research. The term developmental research emerged in the 1960s and 1970s. This is the same time as the field of instructional technology was emerging from the convergence of audiovisual education and instructional psychology. The distinction between doing and studying design and development provide further clarification of developmental research activities. These distinctions can be described in terms of examining the focus, techniques, and tools of developmental research. Developmental research has particular emphases that vary in terms of the extent to which the conclusions are generalizable or contextually specific. The most straightforward developmental research attempts to produce models and principles that guide the design, development, and evaluation processes. As such, doing development and studying development are two different enterprises.
Two types developmental research: Type 1 and Type 2
Type 1: Emphasis: Study of specific product or program design, development, and/or evaluation projects. Product: Lessons learned from developing specific products and analyzing the conditions that facilitate their use. Context-specific conclusions

Developmental research Does Not include: instructional psychology studies, media or delivery system comparison or impact studies, message design and communication studies, policy analysis or formation studies and research on the profession.

A developmental research project may include several distinct stages, each of which involves reporting and analyzing a data set. Merely conducting a comprehensive design and development project does not constitute conducting a developmental research project even using its most narrow Type 1 definition. One must also include the analysis and reporting sage to warrant being classified as developmental research.

Developmental research may also include a number of component parts. Substudies may be conducted to analyze and define the instructional problem, to specify the content, or to determne instrument reliability and validity…to provide a formative evaluation, a summative evaluation or a follow-up evaluation. Thus developmental reports are often quite long.

Type 1: Emphasis: Study of specific product or program design, development, and/or evaluation projects. Product: Lessons learned from developing specific products and analyzing the conditions that facilitate their use. Context-specific conclusions

Type 2: Study of design, development, or evaluation processes, tools or models. Product: New design, development, and evaluation procedures and/or models, and conditions that facilitate their use. Generalized conclusions.

Type 1:
Some Type 1 developmental studies reflect traditional evaluation orientations in which the development process is not addressed, and only the product or program evaluation is described. The results are typically context and product specific, even though the implications for similar situations may be discussed.
Example: O’Quin, Kinsey and eery’s (1987) report on evaluation of a micrcomputer traiing workshop for college personnel.
Type 1 research studies originate with the design and development of an instructional product or program. This is the crux of Type 1 research.
Petry and Edward’s (1984) description of systematic design, development and evaluation of a university applied phonetics course. They described the application of a particular ISD model as well as the use of elaboration theory in content sequencing. It also addresses production of course materials, as well as the results of an evaluation of student performance and attitudes in the revised course. Studies that do not include the entire design, development and evaluation process emphasize a particular phase of ISD such as needs assessment or formative evaluation. Other type 1 developmental research projects foucs on the production aspect of the ISD approach. These studies often concentrate on the development of technology-based instruction. The describe the authoring procedures so specifically that one could replicate the innovative development processes. Type 1 developmental studies demonstrate the range of design and development procedures currently available to practicioners. Commonly they include an evaluation of the products, programs created, including an examination of the changes in learners who had interacted with the newly developed products.

Research Methodologies employed in Type 1 developmental research:
Case studies: The manner in which case study techniques are used varies widely in developmental research…the case study is seen as a way in which one can explore or describe complex situations, which consist of a myriad of critical contextual variables.
If trying to establish causal relationships, rather than simply providing detailed descriptions, you need a quantitative orientation to your developmental case study. Which becomes mixed methods…
Mixed Methods
Example: Plummer, Gillis, Legree and Sanders (1992) develop job aid and evaluate effectiveness of job aid. 3 instructional situations compared…job aid alone, job aid with demonstration and technical manual with demonstration.
Evaluation methods
Learner surveys, achievement tests, performance measures are often used to collect data.
Sullivan, Ice and Niedermeyer (2000) field-tested a k-12 energy education curriculum by implementing attitude surveys and achievement tests.

Conclusions from Type 1
Contextual Specific Conclusions include:
Suggested improvements in the product or program
The conditions that promote successful use of the product or program
The impact of the particular product or program
The conditions that are conducive to efficient design, development, and/or evaluation of the instructional product or program

See this article…
Type 1 developmental research:


Type 2
Type 2: Study of design, development, or evaluation processes, tools or models. Product: New design, development, and evaluation procedures and/or models, and conditions that facilitate their use. Generalized conclusions.
This Type 2 of developmental study is oriented toward a general analysis of design, development, or evaluation processes, addressed either as a whole or in terms of a particular component. Type 2 research may draw its population either from one target project or from a variety of design and development environments. Typically, conclusions from Type 2 developmental research are generalized, even though there are instances of context-specific conclusions in the literature. Driscoll (1991) calls this type of study Model development and technique development research.
Van de Akker calls them “reconstructive studies (1999)
Examples: Tracey (2002) study is an example of a global design orientation. She constructed and validated an instructional systems design model that incorporated Gardner’s notion of multiple intelligences. Taylor and Ellis’ (1991) study evaluated the use of instructional systems design in the Navy. Jonassen’s (1988) case study using needs assessment data in the development of a university program focuses on only one phase of the design/development/evaluation process.
Ultimate objective of Type 2 research is production of knowledge, often in the form of a new or an enhanced design or development model. This research tends to emphasize:


The key difference between type 1 and type 2 studies that focus on a particular aspect of the total process is that goals of type 2 studies tend to be more generalized, striving to enhance the ultimate models employed in these procedures. Type 1 research, on the other hand is more confined to the analysis of a give project. Type 2 research spans the entire range of design and development process components from needs assessment (Crowell, 2000) to evaluation (Phillips, 2000). Also there are type 2 studies that address design processes in a more generic fashion.

Research methodologies in Type 2:
Experimental and Quasi-experimental designs and qualitative designs (some case studies, but not nearly as many as Type 1)
Experimental and Quasi-experimental:

Surveys are often used in Type 2 studies as a means of gathering data from designers in a variety of settings.
Qualitative research methods are often employed in Type 2 research. Sturctured interviews to gather data from instructional designers.

Conclusions in Type 2 research:
Type 2 studies are ultimately directed toward general principles, which are applicable in a wide range of design and development projects. Type 2 conclusions pertain to a technique or model as opposed to a product or program. Issues addressed in these conclusions:

It is not uncommon for a given Type 2 study to generate more than one type of conclusion.

Tessemer et al (1999) described a theoretical and procedural model for conducting a type of needs assessment called needs reassessment…called their model CODE (criticality, opportunity, difficulty, emphasis)
Jones and Richey (2000) indepth examination of use of rapid prototyping methods in two instructional design projects in natural work settings.

Characteristics of Type 2 studies include:

Defining the research problem:
Is the problem common to many designers and developers? Is it one that is currently critical to the profession? Does the problem reflect realistic constraints and conditions typically faced by designers? Does the problem pertain to cutting-edge technologies and processes?
Focusing the Problem:
Give your problem a developmental twist. Focus the research problem on a specific aspect of the design, development or evaluation process, as opposed to focusing on a particular variable that impacts learning or perhaps the impact of a type of media.
Type 1 developmental studies focus upon a given instructional product, program, process or tool They reflect an interest to identify either general development principles or situation specific recommendations. These studies may ultimately validate a particular design or development technique or too.
Type 2 developmental studies focus on a given design, development or evaluation model or process. They may involve constructing and validating unique design models and processes as well as the evaluation of the instruction. If so, will formative summative and confirmative evaluation be addressed? Will the revision and retesting be addressed?
Developmental studies often are structured in phases.
Framing the Problem:
Research questions rather than hypotheses, commonly serve as the organizing framework for developmental studies. Research questions are more appropriate for qualitative research, a common developmental methodology.
Identifying Limitations:
Developmental is often context specific. Limitations and unique conditions may affect the study and the ability to generalize the conclusions of the study.
Review of literature:
Establishes the conceptual foundations of the study. Addresses topics such as
Procedural models that might be appropriate for the task at hand; characteristics of siilar effective instructional products, programs or delivery systems; factors that have impacted the use of the target development processes in other situations; and factors impacting the implementation and management of the target instructional product, program or delivery system in other situations.
In type 2 studies lit reviews may address topics such as

The methodology is often addressed in the lit review. In all developmental studies, The lit review must focus on the foundational theory of the project, even though the link may be indirect.

Research Procedures:
Often it occurs in natural work environments. This enhances the credibility of the research, as well as creates methodological dilemmas for the researcher…whether the research is conducted during the design and development process or retrospectively the best research pertains to the actual projects, rather than simulated or idealized projects. The “real-life” aspects of developmental research may reflect why this type of research takes more time than other types of research. There are often more changes in one’s research plans and procedures as a result of unanticipated events than is typical in other types of research. Detailed procedures and timelines are most important.
Participants. Multiple types of participants: designers, developers, and evaluators; clients; instructors and/or program facilitators; organizations; design and development researchers and theorists; and learners and other types of users.

Research Design:
Often developmental research projects utilize multiple research methodologies and designs, with different designs being used for different phases of the project.

A common situation that is potentially problematic is when the researcher is also a participant in the study, such as when the researcher is also the designer or developer. Care must be taken to ensure objectivity through consistent, systematic data collection techniques and the collection of corroborating data if possible. Often structured logs and diaries completed by several project participants according to a regularly established schedule creates a structure that facilitates the generation of reliable and comparable data. Maintaining the recall data can be problematic. Many studies rely on self-reports of past projects. Others use structured interviews of participants. Using previously prepared documents or data from others involved in the same project facilitates a triangulation process to validate the data collected.

d Typical types of data collected:
…
including documentation of the target populations and the implementation context and measures of learning transfer and the impact of the intervention on the organization.

Data analysis and synthesis are not unlike other research projects..Descriptive data presentations and qualitative data analyses, traditional quantitative analyses.
Reporting developmental data is problematic…massive amounts of data especially with Type 1 studies makes reporting difficult…even for a dissertation appendix.
Some studies use Web sites as data repositories.

Recent innovative developmental research
Constructivist influences are evident in the emphasis on the role of context in design…examining the social and collaborative nature of learning and in the development of new approaches to instruction such as anchored instruction or case-based instruction. Research addressed include areas such as designer decision making, knowledge acquisition tools, and the use of automated development tools.

Trends in Research on Design and designer decision making
The theoretical basis for most studies of design comes from the literature on human problem solving, where Simon (1981) suggests an all encompassing view of design that incorporates nearly any kind of planning activity.
An alternative theoreticaql orientation views design as an experiential, constructive process where an individual designer shapes the problem and solution through cyceles of situated action and reflection (Suchman, 1987). In this sense, design problems are constructed by the designer through a process of “dialogue” with the situation in which the designer engages in metaphorical processes that relate the current design state to the repertoire of objects/solutions known by the designer. Design typically flows through four major stages: naming (where designers identify the main issues in the problem) framing (establishing the parameters of the problem), moving (taking an experimental design action) and reflecting (evaluating and criticizing the move and the frame). Schon (1983, 1985, 1987) has noted that the designers reflect on moves in three ways: by judging the desirablitiy and consequences of the move, by examing the implications of the move in terms of conformity or violation of earlier moves, and by understanding new problems or potentials the move has created. In part, this involves “seeing” the current situation in a new way (Rowland, 1993). As a designer moves through the design process, the situation talks back to the designer and causes a reframing of the problem. Very cyclical nature…design thinking naturally benefits from reflection in action, and designers often maintain sketchbooks and diaries to support reflection (Cheng, 2000, Webster, 2001). Theses and other aspects of reflection assume that a designer possesses a willingness to be thoughtful and reflective is able to understand the context in which assumptions and actions are formed and is willing to explore alternatives and be exposed to interpretive considerations through dialogue with others (Moallem, 1998). Designer is engaged in learning as well as design, because the designer’s personal knowledge structures are altered by the information present in the design environment (McAleese, 1988). From a social view, design is a collaborative activity where conversation, argumentation and persuasion are used to achieve consensus apbout perspectives and actions that might be taken to solve the design problem (Bucciarelli, 2001), Lave & Wenger, 1991; Stumpf & McDonnell, 1999).

See page 1118….for more details

The instructional Design Task Environment: What makes design a special form of problem solving is the nature of design problems. Ill defined problems ….
Design thinking and instructional design…studies of the cognitive processes of designers in domains other than instructional design indicate that the design process is iterative and cyclical with 2 distinct categories of designer behavior: problem structuring and problem solving (Akin, Chen, Dave & Pithavadian, 1986).
Role of knowledge in the design process: The success of the designer’s problem-solving processes is directly related to the designer’s experience and knowledge in the design task environment. Well organized knowledge base for instructional design is crucial to the process. Recent studies suggest that complex case studies grounded in real-world, ill-defined problems are effective in developing the kind of knowledge and expertise necessary to be an effective instructional designer.
Designer decision-Making Studies: Typically Type 2 and has te ultimate goal of understanding the design process and at ties, producig design models that more closely match actual design activity. Population of the studies are naturally designers, not learners…novice or experts
Effort to identify the impact of various design environments is a common secondary objective…more qualitative in nature, although survey methods are not uncommon.
Trends in research on automated instructional design and development: The systematic instructional design and development procedures common to our field have been developed as a means to organize and control what is a very complicated engineering process. Even with systematic methods, the instructional design process can become very time-consuming and costly. Computer based tools have been designed to streamline the design and development of instruction.
Knowledge-based design tools are becoming common in many design professions as researchers strive to acquire and represent in computer systems the kinds of knowledge and reasoning necessary to interpret design problems, control design actions and produce design specification.
Design productivity tools based on expert system technology have been developed to aid instructional designers in making decisions about various aspects of instructional design. These toos function as intelligent “job aids.” Merrill (1987) was an early advocate for the development of authoring systems for computer-based instruction that provided guidance for the user throughout the design process. Question that remains unanswsered is will these systems be used by practicing instructional designers?