A great link for learning about the history of Case studies came be found at the following address.
http://aprillesjourney.blogspot.com/
Thursday, March 13, 2008
Wednesday, March 5, 2008
Case Studies
According to Polit and Hungler (1983), case studies are detailed investigations of individuals, groups, institutions or other social units. The essence of case studies is their central premise to illuminate a decision or set of decisions...specifically why they were made, how they were implemented and with what results. The researcher conducting a case study attempts to analyze the variables relevant to the subject under study. The major difference between a case study and other types of research studies is that in a case study the study focuses on a specific case and not on an entire population of cases. Most case studies focus on understanding the particularities of the case being studied in a natural setting. By studying a case in a bounded system under a normal environment the system can be understood in a natural state (Stake, 1988). Case studies employ a particular focus using a variety of techniques and can be qualitative, quantitative or mixed method. Generally case studies are conducted over a lengthy period of time. Therefore, case studies are a good design for practical problems such as intriguing questions, situations, or puzzling occurrences arising from everyday practice (Merriam, 1998).
Based on the work of Rossman and Rallis (2003), case studies are considered to be an overall strategy rather than a genre of research. Case studies are an in-depth and detailed exploration of a single example that are instances drawn from a class of a similar phenomena. Case studies are descriptive, holistic, hueristic, and inductive. Case studies are also context dependent. Therefore, the results of a case study can not be generalized in the probabilistic sense. In other words, the results of one case study can not be applied directly to another case study. Yet, through reasoning by analogy the lessons learned in one case study can be applied to another case of similar characteristics. (Kennedy, 1979). Case study evaluations are often the research technique of choice where broad, complex, multi-layered questions have to be addressed in complex circumstances, particularly in medical scenarios. The strength of case studies is their reliance on details and their complexity. Case studies use multiple sources to obtain a variety of perspectives. This results in a thick, rich description that allows the reader to interpret and decide the applicability of case learnings to another setting (Rossman & Rallis, 2003).
"Critical case studies are grounded in a critique of existing social structures and patterns. They assume theoretically that oprression and domination characterize the setting and seek to uncover how patterns of action perpetuate the status
quo" (Rossman & Rallis, 2003, p. 106).
Based on the work of Rossman and Rallis (2003), case studies are considered to be an overall strategy rather than a genre of research. Case studies are an in-depth and detailed exploration of a single example that are instances drawn from a class of a similar phenomena. Case studies are descriptive, holistic, hueristic, and inductive. Case studies are also context dependent. Therefore, the results of a case study can not be generalized in the probabilistic sense. In other words, the results of one case study can not be applied directly to another case study. Yet, through reasoning by analogy the lessons learned in one case study can be applied to another case of similar characteristics. (Kennedy, 1979). Case study evaluations are often the research technique of choice where broad, complex, multi-layered questions have to be addressed in complex circumstances, particularly in medical scenarios. The strength of case studies is their reliance on details and their complexity. Case studies use multiple sources to obtain a variety of perspectives. This results in a thick, rich description that allows the reader to interpret and decide the applicability of case learnings to another setting (Rossman & Rallis, 2003).
"Critical case studies are grounded in a critique of existing social structures and patterns. They assume theoretically that oprression and domination characterize the setting and seek to uncover how patterns of action perpetuate the status
quo" (Rossman & Rallis, 2003, p. 106).
Tuesday, March 4, 2008
Developmental Research: Studies of instructional Design and Development
Notes from Developmental Research by Rita C. Richey, James D. Klein & Wayne A. Nelson:
Developmental research can be either
1. The study of the process and impact of specific instructional design and development efforts or
2. A situation in which someone is performing instructional design, development or evaluation activities and studying the process at the same time; or
3. The study of the instructional design, development and evaluation process as a whole or of particular process components.
The distinction is made between performing a process and studying that process. Reports of developmental research may take the form of a case study with retrospective analysis, an evaluation report, or even a typical experimental research report.
Development in its most generic sense implies gradual growth, evolution and change. In the field of instructional technology development has a particular somewhat unique connotation: development is “the process of translating the design specifications into physical forms.” (Seels & Richey, 1994)
Thus it refers to the process of producing instructional materials. The 1994 definition of the field attempts to clarify these issues by viewing design as the planning phase in which specifications are constructed, and development as the production phase in which the design specifications are actualized. However, the word development has a broader meaning when used in connection to research. The focus now includes comprehensive evaluation, as well as planning and development of the instructional materials.
Developmental research may address formative, summative and confirmative evaluation. The next step beyond utilization & maintenance would be impact, the follow-up analysis of the effects of an instructional product or program on the organization or the learner. This type of research typically falls within the scope of traditional evaluation research. The term developmental research emerged in the 1960s and 1970s. This is the same time as the field of instructional technology was emerging from the convergence of audiovisual education and instructional psychology. The distinction between doing and studying design and development provide further clarification of developmental research activities. These distinctions can be described in terms of examining the focus, techniques, and tools of developmental research. Developmental research has particular emphases that vary in terms of the extent to which the conclusions are generalizable or contextually specific. The most straightforward developmental research attempts to produce models and principles that guide the design, development, and evaluation processes. As such, doing development and studying development are two different enterprises.
Two types developmental research: Type 1 and Type 2
Type 1: Emphasis: Study of specific product or program design, development, and/or evaluation projects. Product: Lessons learned from developing specific products and analyzing the conditions that facilitate their use. Context-specific conclusions
Developmental research Does Not include: instructional psychology studies, media or delivery system comparison or impact studies, message design and communication studies, policy analysis or formation studies and research on the profession.
A developmental research project may include several distinct stages, each of which involves reporting and analyzing a data set. Merely conducting a comprehensive design and development project does not constitute conducting a developmental research project even using its most narrow Type 1 definition. One must also include the analysis and reporting sage to warrant being classified as developmental research.
Developmental research may also include a number of component parts. Substudies may be conducted to analyze and define the instructional problem, to specify the content, or to determne instrument reliability and validity…to provide a formative evaluation, a summative evaluation or a follow-up evaluation. Thus developmental reports are often quite long.
Type 1: Emphasis: Study of specific product or program design, development, and/or evaluation projects. Product: Lessons learned from developing specific products and analyzing the conditions that facilitate their use. Context-specific conclusions
Type 2: Study of design, development, or evaluation processes, tools or models. Product: New design, development, and evaluation procedures and/or models, and conditions that facilitate their use. Generalized conclusions.
Type 1:
Some Type 1 developmental studies reflect traditional evaluation orientations in which the development process is not addressed, and only the product or program evaluation is described. The results are typically context and product specific, even though the implications for similar situations may be discussed.
Example: O’Quin, Kinsey and eery’s (1987) report on evaluation of a micrcomputer traiing workshop for college personnel.
Type 1 research studies originate with the design and development of an instructional product or program. This is the crux of Type 1 research.
Petry and Edward’s (1984) description of systematic design, development and evaluation of a university applied phonetics course. They described the application of a particular ISD model as well as the use of elaboration theory in content sequencing. It also addresses production of course materials, as well as the results of an evaluation of student performance and attitudes in the revised course. Studies that do not include the entire design, development and evaluation process emphasize a particular phase of ISD such as needs assessment or formative evaluation. Other type 1 developmental research projects foucs on the production aspect of the ISD approach. These studies often concentrate on the development of technology-based instruction. The describe the authoring procedures so specifically that one could replicate the innovative development processes. Type 1 developmental studies demonstrate the range of design and development procedures currently available to practicioners. Commonly they include an evaluation of the products, programs created, including an examination of the changes in learners who had interacted with the newly developed products.
Research Methodologies employed in Type 1 developmental research:
Case studies: The manner in which case study techniques are used varies widely in developmental research…the case study is seen as a way in which one can explore or describe complex situations, which consist of a myriad of critical contextual variables.
If trying to establish causal relationships, rather than simply providing detailed descriptions, you need a quantitative orientation to your developmental case study. Which becomes mixed methods…
Mixed Methods
Example: Plummer, Gillis, Legree and Sanders (1992) develop job aid and evaluate effectiveness of job aid. 3 instructional situations compared…job aid alone, job aid with demonstration and technical manual with demonstration.
Evaluation methods
Learner surveys, achievement tests, performance measures are often used to collect data.
Sullivan, Ice and Niedermeyer (2000) field-tested a k-12 energy education curriculum by implementing attitude surveys and achievement tests.
Conclusions from Type 1
Contextual Specific Conclusions include:
Suggested improvements in the product or program
The conditions that promote successful use of the product or program
The impact of the particular product or program
The conditions that are conducive to efficient design, development, and/or evaluation of the instructional product or program
See this article…
Type 1 developmental research:
Type 2
Type 2: Study of design, development, or evaluation processes, tools or models. Product: New design, development, and evaluation procedures and/or models, and conditions that facilitate their use. Generalized conclusions.
This Type 2 of developmental study is oriented toward a general analysis of design, development, or evaluation processes, addressed either as a whole or in terms of a particular component. Type 2 research may draw its population either from one target project or from a variety of design and development environments. Typically, conclusions from Type 2 developmental research are generalized, even though there are instances of context-specific conclusions in the literature. Driscoll (1991) calls this type of study Model development and technique development research.
Van de Akker calls them “reconstructive studies (1999)
Examples: Tracey (2002) study is an example of a global design orientation. She constructed and validated an instructional systems design model that incorporated Gardner’s notion of multiple intelligences. Taylor and Ellis’ (1991) study evaluated the use of instructional systems design in the Navy. Jonassen’s (1988) case study using needs assessment data in the development of a university program focuses on only one phase of the design/development/evaluation process.
Ultimate objective of Type 2 research is production of knowledge, often in the form of a new or an enhanced design or development model. This research tends to emphasize:
The key difference between type 1 and type 2 studies that focus on a particular aspect of the total process is that goals of type 2 studies tend to be more generalized, striving to enhance the ultimate models employed in these procedures. Type 1 research, on the other hand is more confined to the analysis of a give project. Type 2 research spans the entire range of design and development process components from needs assessment (Crowell, 2000) to evaluation (Phillips, 2000). Also there are type 2 studies that address design processes in a more generic fashion.
Research methodologies in Type 2:
Experimental and Quasi-experimental designs and qualitative designs (some case studies, but not nearly as many as Type 1)
Experimental and Quasi-experimental:
Surveys are often used in Type 2 studies as a means of gathering data from designers in a variety of settings.
Qualitative research methods are often employed in Type 2 research. Sturctured interviews to gather data from instructional designers.
Conclusions in Type 2 research:
Type 2 studies are ultimately directed toward general principles, which are applicable in a wide range of design and development projects. Type 2 conclusions pertain to a technique or model as opposed to a product or program. Issues addressed in these conclusions:
It is not uncommon for a given Type 2 study to generate more than one type of conclusion.
Tessemer et al (1999) described a theoretical and procedural model for conducting a type of needs assessment called needs reassessment…called their model CODE (criticality, opportunity, difficulty, emphasis)
Jones and Richey (2000) indepth examination of use of rapid prototyping methods in two instructional design projects in natural work settings.
Characteristics of Type 2 studies include:
Defining the research problem:
Is the problem common to many designers and developers? Is it one that is currently critical to the profession? Does the problem reflect realistic constraints and conditions typically faced by designers? Does the problem pertain to cutting-edge technologies and processes?
Focusing the Problem:
Give your problem a developmental twist. Focus the research problem on a specific aspect of the design, development or evaluation process, as opposed to focusing on a particular variable that impacts learning or perhaps the impact of a type of media.
Type 1 developmental studies focus upon a given instructional product, program, process or tool They reflect an interest to identify either general development principles or situation specific recommendations. These studies may ultimately validate a particular design or development technique or too.
Type 2 developmental studies focus on a given design, development or evaluation model or process. They may involve constructing and validating unique design models and processes as well as the evaluation of the instruction. If so, will formative summative and confirmative evaluation be addressed? Will the revision and retesting be addressed?
Developmental studies often are structured in phases.
Framing the Problem:
Research questions rather than hypotheses, commonly serve as the organizing framework for developmental studies. Research questions are more appropriate for qualitative research, a common developmental methodology.
Identifying Limitations:
Developmental is often context specific. Limitations and unique conditions may affect the study and the ability to generalize the conclusions of the study.
Review of literature:
Establishes the conceptual foundations of the study. Addresses topics such as
Procedural models that might be appropriate for the task at hand; characteristics of siilar effective instructional products, programs or delivery systems; factors that have impacted the use of the target development processes in other situations; and factors impacting the implementation and management of the target instructional product, program or delivery system in other situations.
In type 2 studies lit reviews may address topics such as
The methodology is often addressed in the lit review. In all developmental studies, The lit review must focus on the foundational theory of the project, even though the link may be indirect.
Research Procedures:
Often it occurs in natural work environments. This enhances the credibility of the research, as well as creates methodological dilemmas for the researcher…whether the research is conducted during the design and development process or retrospectively the best research pertains to the actual projects, rather than simulated or idealized projects. The “real-life” aspects of developmental research may reflect why this type of research takes more time than other types of research. There are often more changes in one’s research plans and procedures as a result of unanticipated events than is typical in other types of research. Detailed procedures and timelines are most important.
Participants. Multiple types of participants: designers, developers, and evaluators; clients; instructors and/or program facilitators; organizations; design and development researchers and theorists; and learners and other types of users.
Research Design:
Often developmental research projects utilize multiple research methodologies and designs, with different designs being used for different phases of the project.
A common situation that is potentially problematic is when the researcher is also a participant in the study, such as when the researcher is also the designer or developer. Care must be taken to ensure objectivity through consistent, systematic data collection techniques and the collection of corroborating data if possible. Often structured logs and diaries completed by several project participants according to a regularly established schedule creates a structure that facilitates the generation of reliable and comparable data. Maintaining the recall data can be problematic. Many studies rely on self-reports of past projects. Others use structured interviews of participants. Using previously prepared documents or data from others involved in the same project facilitates a triangulation process to validate the data collected.
d Typical types of data collected:
…
including documentation of the target populations and the implementation context and measures of learning transfer and the impact of the intervention on the organization.
Data analysis and synthesis are not unlike other research projects..Descriptive data presentations and qualitative data analyses, traditional quantitative analyses.
Reporting developmental data is problematic…massive amounts of data especially with Type 1 studies makes reporting difficult…even for a dissertation appendix.
Some studies use Web sites as data repositories.
Recent innovative developmental research
Constructivist influences are evident in the emphasis on the role of context in design…examining the social and collaborative nature of learning and in the development of new approaches to instruction such as anchored instruction or case-based instruction. Research addressed include areas such as designer decision making, knowledge acquisition tools, and the use of automated development tools.
Trends in Research on Design and designer decision making
The theoretical basis for most studies of design comes from the literature on human problem solving, where Simon (1981) suggests an all encompassing view of design that incorporates nearly any kind of planning activity.
An alternative theoreticaql orientation views design as an experiential, constructive process where an individual designer shapes the problem and solution through cyceles of situated action and reflection (Suchman, 1987). In this sense, design problems are constructed by the designer through a process of “dialogue” with the situation in which the designer engages in metaphorical processes that relate the current design state to the repertoire of objects/solutions known by the designer. Design typically flows through four major stages: naming (where designers identify the main issues in the problem) framing (establishing the parameters of the problem), moving (taking an experimental design action) and reflecting (evaluating and criticizing the move and the frame). Schon (1983, 1985, 1987) has noted that the designers reflect on moves in three ways: by judging the desirablitiy and consequences of the move, by examing the implications of the move in terms of conformity or violation of earlier moves, and by understanding new problems or potentials the move has created. In part, this involves “seeing” the current situation in a new way (Rowland, 1993). As a designer moves through the design process, the situation talks back to the designer and causes a reframing of the problem. Very cyclical nature…design thinking naturally benefits from reflection in action, and designers often maintain sketchbooks and diaries to support reflection (Cheng, 2000, Webster, 2001). Theses and other aspects of reflection assume that a designer possesses a willingness to be thoughtful and reflective is able to understand the context in which assumptions and actions are formed and is willing to explore alternatives and be exposed to interpretive considerations through dialogue with others (Moallem, 1998). Designer is engaged in learning as well as design, because the designer’s personal knowledge structures are altered by the information present in the design environment (McAleese, 1988). From a social view, design is a collaborative activity where conversation, argumentation and persuasion are used to achieve consensus apbout perspectives and actions that might be taken to solve the design problem (Bucciarelli, 2001), Lave & Wenger, 1991; Stumpf & McDonnell, 1999).
See page 1118….for more details
The instructional Design Task Environment: What makes design a special form of problem solving is the nature of design problems. Ill defined problems ….
Design thinking and instructional design…studies of the cognitive processes of designers in domains other than instructional design indicate that the design process is iterative and cyclical with 2 distinct categories of designer behavior: problem structuring and problem solving (Akin, Chen, Dave & Pithavadian, 1986).
Role of knowledge in the design process: The success of the designer’s problem-solving processes is directly related to the designer’s experience and knowledge in the design task environment. Well organized knowledge base for instructional design is crucial to the process. Recent studies suggest that complex case studies grounded in real-world, ill-defined problems are effective in developing the kind of knowledge and expertise necessary to be an effective instructional designer.
Designer decision-Making Studies: Typically Type 2 and has te ultimate goal of understanding the design process and at ties, producig design models that more closely match actual design activity. Population of the studies are naturally designers, not learners…novice or experts
Effort to identify the impact of various design environments is a common secondary objective…more qualitative in nature, although survey methods are not uncommon.
Trends in research on automated instructional design and development: The systematic instructional design and development procedures common to our field have been developed as a means to organize and control what is a very complicated engineering process. Even with systematic methods, the instructional design process can become very time-consuming and costly. Computer based tools have been designed to streamline the design and development of instruction.
Knowledge-based design tools are becoming common in many design professions as researchers strive to acquire and represent in computer systems the kinds of knowledge and reasoning necessary to interpret design problems, control design actions and produce design specification.
Design productivity tools based on expert system technology have been developed to aid instructional designers in making decisions about various aspects of instructional design. These toos function as intelligent “job aids.” Merrill (1987) was an early advocate for the development of authoring systems for computer-based instruction that provided guidance for the user throughout the design process. Question that remains unanswsered is will these systems be used by practicing instructional designers?
Developmental research can be either
1. The study of the process and impact of specific instructional design and development efforts or
2. A situation in which someone is performing instructional design, development or evaluation activities and studying the process at the same time; or
3. The study of the instructional design, development and evaluation process as a whole or of particular process components.
The distinction is made between performing a process and studying that process. Reports of developmental research may take the form of a case study with retrospective analysis, an evaluation report, or even a typical experimental research report.
Development in its most generic sense implies gradual growth, evolution and change. In the field of instructional technology development has a particular somewhat unique connotation: development is “the process of translating the design specifications into physical forms.” (Seels & Richey, 1994)
Thus it refers to the process of producing instructional materials. The 1994 definition of the field attempts to clarify these issues by viewing design as the planning phase in which specifications are constructed, and development as the production phase in which the design specifications are actualized. However, the word development has a broader meaning when used in connection to research. The focus now includes comprehensive evaluation, as well as planning and development of the instructional materials.
Developmental research may address formative, summative and confirmative evaluation. The next step beyond utilization & maintenance would be impact, the follow-up analysis of the effects of an instructional product or program on the organization or the learner. This type of research typically falls within the scope of traditional evaluation research. The term developmental research emerged in the 1960s and 1970s. This is the same time as the field of instructional technology was emerging from the convergence of audiovisual education and instructional psychology. The distinction between doing and studying design and development provide further clarification of developmental research activities. These distinctions can be described in terms of examining the focus, techniques, and tools of developmental research. Developmental research has particular emphases that vary in terms of the extent to which the conclusions are generalizable or contextually specific. The most straightforward developmental research attempts to produce models and principles that guide the design, development, and evaluation processes. As such, doing development and studying development are two different enterprises.
Two types developmental research: Type 1 and Type 2
Type 1: Emphasis: Study of specific product or program design, development, and/or evaluation projects. Product: Lessons learned from developing specific products and analyzing the conditions that facilitate their use. Context-specific conclusions
Developmental research Does Not include: instructional psychology studies, media or delivery system comparison or impact studies, message design and communication studies, policy analysis or formation studies and research on the profession.
A developmental research project may include several distinct stages, each of which involves reporting and analyzing a data set. Merely conducting a comprehensive design and development project does not constitute conducting a developmental research project even using its most narrow Type 1 definition. One must also include the analysis and reporting sage to warrant being classified as developmental research.
Developmental research may also include a number of component parts. Substudies may be conducted to analyze and define the instructional problem, to specify the content, or to determne instrument reliability and validity…to provide a formative evaluation, a summative evaluation or a follow-up evaluation. Thus developmental reports are often quite long.
Type 1: Emphasis: Study of specific product or program design, development, and/or evaluation projects. Product: Lessons learned from developing specific products and analyzing the conditions that facilitate their use. Context-specific conclusions
Type 2: Study of design, development, or evaluation processes, tools or models. Product: New design, development, and evaluation procedures and/or models, and conditions that facilitate their use. Generalized conclusions.
Type 1:
Some Type 1 developmental studies reflect traditional evaluation orientations in which the development process is not addressed, and only the product or program evaluation is described. The results are typically context and product specific, even though the implications for similar situations may be discussed.
Example: O’Quin, Kinsey and eery’s (1987) report on evaluation of a micrcomputer traiing workshop for college personnel.
Type 1 research studies originate with the design and development of an instructional product or program. This is the crux of Type 1 research.
Petry and Edward’s (1984) description of systematic design, development and evaluation of a university applied phonetics course. They described the application of a particular ISD model as well as the use of elaboration theory in content sequencing. It also addresses production of course materials, as well as the results of an evaluation of student performance and attitudes in the revised course. Studies that do not include the entire design, development and evaluation process emphasize a particular phase of ISD such as needs assessment or formative evaluation. Other type 1 developmental research projects foucs on the production aspect of the ISD approach. These studies often concentrate on the development of technology-based instruction. The describe the authoring procedures so specifically that one could replicate the innovative development processes. Type 1 developmental studies demonstrate the range of design and development procedures currently available to practicioners. Commonly they include an evaluation of the products, programs created, including an examination of the changes in learners who had interacted with the newly developed products.
Research Methodologies employed in Type 1 developmental research:
Case studies: The manner in which case study techniques are used varies widely in developmental research…the case study is seen as a way in which one can explore or describe complex situations, which consist of a myriad of critical contextual variables.
If trying to establish causal relationships, rather than simply providing detailed descriptions, you need a quantitative orientation to your developmental case study. Which becomes mixed methods…
Mixed Methods
Example: Plummer, Gillis, Legree and Sanders (1992) develop job aid and evaluate effectiveness of job aid. 3 instructional situations compared…job aid alone, job aid with demonstration and technical manual with demonstration.
Evaluation methods
Learner surveys, achievement tests, performance measures are often used to collect data.
Sullivan, Ice and Niedermeyer (2000) field-tested a k-12 energy education curriculum by implementing attitude surveys and achievement tests.
Conclusions from Type 1
Contextual Specific Conclusions include:
Suggested improvements in the product or program
The conditions that promote successful use of the product or program
The impact of the particular product or program
The conditions that are conducive to efficient design, development, and/or evaluation of the instructional product or program
See this article…
Type 1 developmental research:
Type 2
Type 2: Study of design, development, or evaluation processes, tools or models. Product: New design, development, and evaluation procedures and/or models, and conditions that facilitate their use. Generalized conclusions.
This Type 2 of developmental study is oriented toward a general analysis of design, development, or evaluation processes, addressed either as a whole or in terms of a particular component. Type 2 research may draw its population either from one target project or from a variety of design and development environments. Typically, conclusions from Type 2 developmental research are generalized, even though there are instances of context-specific conclusions in the literature. Driscoll (1991) calls this type of study Model development and technique development research.
Van de Akker calls them “reconstructive studies (1999)
Examples: Tracey (2002) study is an example of a global design orientation. She constructed and validated an instructional systems design model that incorporated Gardner’s notion of multiple intelligences. Taylor and Ellis’ (1991) study evaluated the use of instructional systems design in the Navy. Jonassen’s (1988) case study using needs assessment data in the development of a university program focuses on only one phase of the design/development/evaluation process.
Ultimate objective of Type 2 research is production of knowledge, often in the form of a new or an enhanced design or development model. This research tends to emphasize:
The key difference between type 1 and type 2 studies that focus on a particular aspect of the total process is that goals of type 2 studies tend to be more generalized, striving to enhance the ultimate models employed in these procedures. Type 1 research, on the other hand is more confined to the analysis of a give project. Type 2 research spans the entire range of design and development process components from needs assessment (Crowell, 2000) to evaluation (Phillips, 2000). Also there are type 2 studies that address design processes in a more generic fashion.
Research methodologies in Type 2:
Experimental and Quasi-experimental designs and qualitative designs (some case studies, but not nearly as many as Type 1)
Experimental and Quasi-experimental:
Surveys are often used in Type 2 studies as a means of gathering data from designers in a variety of settings.
Qualitative research methods are often employed in Type 2 research. Sturctured interviews to gather data from instructional designers.
Conclusions in Type 2 research:
Type 2 studies are ultimately directed toward general principles, which are applicable in a wide range of design and development projects. Type 2 conclusions pertain to a technique or model as opposed to a product or program. Issues addressed in these conclusions:
It is not uncommon for a given Type 2 study to generate more than one type of conclusion.
Tessemer et al (1999) described a theoretical and procedural model for conducting a type of needs assessment called needs reassessment…called their model CODE (criticality, opportunity, difficulty, emphasis)
Jones and Richey (2000) indepth examination of use of rapid prototyping methods in two instructional design projects in natural work settings.
Characteristics of Type 2 studies include:
Defining the research problem:
Is the problem common to many designers and developers? Is it one that is currently critical to the profession? Does the problem reflect realistic constraints and conditions typically faced by designers? Does the problem pertain to cutting-edge technologies and processes?
Focusing the Problem:
Give your problem a developmental twist. Focus the research problem on a specific aspect of the design, development or evaluation process, as opposed to focusing on a particular variable that impacts learning or perhaps the impact of a type of media.
Type 1 developmental studies focus upon a given instructional product, program, process or tool They reflect an interest to identify either general development principles or situation specific recommendations. These studies may ultimately validate a particular design or development technique or too.
Type 2 developmental studies focus on a given design, development or evaluation model or process. They may involve constructing and validating unique design models and processes as well as the evaluation of the instruction. If so, will formative summative and confirmative evaluation be addressed? Will the revision and retesting be addressed?
Developmental studies often are structured in phases.
Framing the Problem:
Research questions rather than hypotheses, commonly serve as the organizing framework for developmental studies. Research questions are more appropriate for qualitative research, a common developmental methodology.
Identifying Limitations:
Developmental is often context specific. Limitations and unique conditions may affect the study and the ability to generalize the conclusions of the study.
Review of literature:
Establishes the conceptual foundations of the study. Addresses topics such as
Procedural models that might be appropriate for the task at hand; characteristics of siilar effective instructional products, programs or delivery systems; factors that have impacted the use of the target development processes in other situations; and factors impacting the implementation and management of the target instructional product, program or delivery system in other situations.
In type 2 studies lit reviews may address topics such as
The methodology is often addressed in the lit review. In all developmental studies, The lit review must focus on the foundational theory of the project, even though the link may be indirect.
Research Procedures:
Often it occurs in natural work environments. This enhances the credibility of the research, as well as creates methodological dilemmas for the researcher…whether the research is conducted during the design and development process or retrospectively the best research pertains to the actual projects, rather than simulated or idealized projects. The “real-life” aspects of developmental research may reflect why this type of research takes more time than other types of research. There are often more changes in one’s research plans and procedures as a result of unanticipated events than is typical in other types of research. Detailed procedures and timelines are most important.
Participants. Multiple types of participants: designers, developers, and evaluators; clients; instructors and/or program facilitators; organizations; design and development researchers and theorists; and learners and other types of users.
Research Design:
Often developmental research projects utilize multiple research methodologies and designs, with different designs being used for different phases of the project.
A common situation that is potentially problematic is when the researcher is also a participant in the study, such as when the researcher is also the designer or developer. Care must be taken to ensure objectivity through consistent, systematic data collection techniques and the collection of corroborating data if possible. Often structured logs and diaries completed by several project participants according to a regularly established schedule creates a structure that facilitates the generation of reliable and comparable data. Maintaining the recall data can be problematic. Many studies rely on self-reports of past projects. Others use structured interviews of participants. Using previously prepared documents or data from others involved in the same project facilitates a triangulation process to validate the data collected.
d Typical types of data collected:
…
including documentation of the target populations and the implementation context and measures of learning transfer and the impact of the intervention on the organization.
Data analysis and synthesis are not unlike other research projects..Descriptive data presentations and qualitative data analyses, traditional quantitative analyses.
Reporting developmental data is problematic…massive amounts of data especially with Type 1 studies makes reporting difficult…even for a dissertation appendix.
Some studies use Web sites as data repositories.
Recent innovative developmental research
Constructivist influences are evident in the emphasis on the role of context in design…examining the social and collaborative nature of learning and in the development of new approaches to instruction such as anchored instruction or case-based instruction. Research addressed include areas such as designer decision making, knowledge acquisition tools, and the use of automated development tools.
Trends in Research on Design and designer decision making
The theoretical basis for most studies of design comes from the literature on human problem solving, where Simon (1981) suggests an all encompassing view of design that incorporates nearly any kind of planning activity.
An alternative theoreticaql orientation views design as an experiential, constructive process where an individual designer shapes the problem and solution through cyceles of situated action and reflection (Suchman, 1987). In this sense, design problems are constructed by the designer through a process of “dialogue” with the situation in which the designer engages in metaphorical processes that relate the current design state to the repertoire of objects/solutions known by the designer. Design typically flows through four major stages: naming (where designers identify the main issues in the problem) framing (establishing the parameters of the problem), moving (taking an experimental design action) and reflecting (evaluating and criticizing the move and the frame). Schon (1983, 1985, 1987) has noted that the designers reflect on moves in three ways: by judging the desirablitiy and consequences of the move, by examing the implications of the move in terms of conformity or violation of earlier moves, and by understanding new problems or potentials the move has created. In part, this involves “seeing” the current situation in a new way (Rowland, 1993). As a designer moves through the design process, the situation talks back to the designer and causes a reframing of the problem. Very cyclical nature…design thinking naturally benefits from reflection in action, and designers often maintain sketchbooks and diaries to support reflection (Cheng, 2000, Webster, 2001). Theses and other aspects of reflection assume that a designer possesses a willingness to be thoughtful and reflective is able to understand the context in which assumptions and actions are formed and is willing to explore alternatives and be exposed to interpretive considerations through dialogue with others (Moallem, 1998). Designer is engaged in learning as well as design, because the designer’s personal knowledge structures are altered by the information present in the design environment (McAleese, 1988). From a social view, design is a collaborative activity where conversation, argumentation and persuasion are used to achieve consensus apbout perspectives and actions that might be taken to solve the design problem (Bucciarelli, 2001), Lave & Wenger, 1991; Stumpf & McDonnell, 1999).
See page 1118….for more details
The instructional Design Task Environment: What makes design a special form of problem solving is the nature of design problems. Ill defined problems ….
Design thinking and instructional design…studies of the cognitive processes of designers in domains other than instructional design indicate that the design process is iterative and cyclical with 2 distinct categories of designer behavior: problem structuring and problem solving (Akin, Chen, Dave & Pithavadian, 1986).
Role of knowledge in the design process: The success of the designer’s problem-solving processes is directly related to the designer’s experience and knowledge in the design task environment. Well organized knowledge base for instructional design is crucial to the process. Recent studies suggest that complex case studies grounded in real-world, ill-defined problems are effective in developing the kind of knowledge and expertise necessary to be an effective instructional designer.
Designer decision-Making Studies: Typically Type 2 and has te ultimate goal of understanding the design process and at ties, producig design models that more closely match actual design activity. Population of the studies are naturally designers, not learners…novice or experts
Effort to identify the impact of various design environments is a common secondary objective…more qualitative in nature, although survey methods are not uncommon.
Trends in research on automated instructional design and development: The systematic instructional design and development procedures common to our field have been developed as a means to organize and control what is a very complicated engineering process. Even with systematic methods, the instructional design process can become very time-consuming and costly. Computer based tools have been designed to streamline the design and development of instruction.
Knowledge-based design tools are becoming common in many design professions as researchers strive to acquire and represent in computer systems the kinds of knowledge and reasoning necessary to interpret design problems, control design actions and produce design specification.
Design productivity tools based on expert system technology have been developed to aid instructional designers in making decisions about various aspects of instructional design. These toos function as intelligent “job aids.” Merrill (1987) was an early advocate for the development of authoring systems for computer-based instruction that provided guidance for the user throughout the design process. Question that remains unanswsered is will these systems be used by practicing instructional designers?
Interviewing in Qualitative Research
“We interview people to find out directly from them what we can’t know from observation.”
Types of interviews
Informal conversation
General interview guide
Standardized open-ended interview
Informal conversation
AKA unstructured interviewing, ethnographic interviewing
Spontaneous generation of questions in the natural flow of interaction
Maximum flexibility
Unstructured does not mean unfocused
Interviewer must “go with the flow”
Data collection varies
Some interviewers do not take notes, but write down what they learned later
Some take notes or even use a tape recorder.
Strengths: Flexibility, Spontaneity, Responsiveness to individual differences and situation changes
Weaknesses: Requires a great amount of time to collect systematic information, May be susceptible to interviewer effects, leading questions, biases, difficult to pull together and analyze data.
Formal Interviews
Outlining a set if issues that are to be explored with each respondent before the interview begins
Advantages:
Interviewer makes best use of limited time
Interviewing is more systematic and comprehensive
Can be flexible in the specificity of important issues and the extent to which it answers your questions
Standardized Open-ended
A set of questions carefully worded and arranged with the intention of taking each respondent though the same sequence and asking the same questions with essentially the same words
Standardized Open-ended
Strengths:
Same questions are asked
Can compensate fore variation in interviewing skills
Can establish priorities for interview
Used to compare answers to the same questions over a period of time
Makes data anaysis easier
Exact instrument is available for inspection
Can help overcome questions of legitimacy and credibility, limiations of the data can be known.
Weaknesses
Does not permit interviewer to pursue topics unanticipated
Reduces the extent individual differences and circumstances can be queried
Combination of approaches
Combining an interview guide with standardized with key questions offers flexibility in probing and determining when it is appropriate to explore certain subjects in greater depth or a new area
May use conversation
What types of questions to ask
Experience and behavior questions
Opinions and value questions
Feeling questions
Knowledge questions
Sensory questions
Background/demographic questions
Time frame questions
Can make questions different by asking about them as in the past or future
Sequencing questions
Start with non-controversial present behaviors, activities, and experiences,
Follow with opinions and feelings
Knowledge and skill questions need a context
Ask about present, then ask about the same in the past, then possibly broach the future.
How to phrase questions:
Open ended questions never yes or no, questions you can get a significant response to
Horns of a dichotomy
Asking singular questions
Theme the wording used in asking questions can make a significant difference in the quality of the responses elicited.
Clarity of questions:
Ask singular questions
Learn special terms used by people in the setting of the interview
Understand the language participants use among themselves in talking about a setting, activities, or other aspects of life
Avoid using labels use the respondents own terms and clarify them in the interview
Theme use language that is understandable and part of the frame of reference of the person being interviewed
Why: Take care when asking why
Presuppose things happen for a reason that those reasons might or might not be shared
Rapport vs. Neutrality
Rapport is a stance about the person being interviewed (respect)
Neutrality is a stance about the content of what the person says.
Illustrative Examples
Letting the respondent know that you are not interested in the sensational just in the genuine experiences
Share examples from all sides to show that you’re heard it all
Must take caution to avoid leading questions
Role Playing and simulations
Provide a context for the questions to help the respondent hone in on relevant responses
Questions ask the respondent to become an observer
Presupposition Questions
Can increase the richness and depth of the responses
Bypasses an initial step of asking if a person has had such an experience
Prefatory statements and announcements
Alert the interviewee to what is about to be asked to direct awareness and focus attention and to give the respondent time to organize thoughts.
Probes and follow-up questions
Used to deepen the response, increase richness and depth and give clues about the level of depth of responses
Detailed oriented
Process feedback during the interview
Interviewer has a responsibility to communicate clearly what information is desired and why it is important and to let the interviewee know how the interview is progressing
Helpful to maintain the flow of the interview
Support and recognition responses
Common mistake is failing to provide reinforcement and feedback
Words of thanks, feedback and praise.
Maintaining control enhancing quality
Don’t want to jeopardize the quality of the interview
Know what you want to find out
Ask focused questions to get relevant answers
Listen attentively
Give appropriate verbal and nonverbal feedback…
The one shot quezstion
Sometimes things don’t work out as planned
This is the question you ask if you are only going to get a few minutes with the interviewee and it’s the most important question.
If I only had one question to ask what should it have been.
Final or closing questions
How to gather data
Recording the data
Explain the use of the tape recorder to the interviewee
Increase accuracy of the data
Permits the interviewer to be more attentive
Part of the interview is observing the respondent
When not possible to record, notes must be thorough and comprehensive.
Taking notes during the interview
Purpose
Helps to formulate new questions
Can stimulate early insights
Facilitates later analysis
Backup to recorder malfunction
Use key phrases and major points
Helps pace the interview
After the interview
Check the tape to make sure it was functioning properly. If not, make extensive notes immediately go over notes to avoid ambiguity or uncertainty
Check to see if there is a need for clarification or follow up
Record details about setting and observations about interview
Reflect on the quality of the interview data
Critical for reflection and elaboration
Special applications and issues
Think-aloud protocol interviewing
Aims to elicit the inner thoguths or cognitive processes that illuminate what’s going on in a person’s head during the performance of a task
Get people
Focus Group interviews
Interviewing a small group of people on a specific topic
6-10 people
similar background
1-2 hours in length
recognizes that many decisions are made in a social context
discussion though direct interaction among participants
additional comments can be made after one hears what another says
consider one’s views In the context of another.
Focus group interviews:
Advantages
Cost effect
Interactions among participants enhance quality of data
Quick assessment of consistency or diversity of views
Disadvantages/limitations
Umber of questions asked is restricted
Available response time is restrained for any particular individual
Facilitating requires considerable group process skills
Those in the minority ay not speak up
Works best with strangers
Controversial issues are poor topics
Confidentiality is hard to assure
Not good at identify subtle differences
Take place outside of natural setting.
Group interviews
Unstructured conversation interviews that are not focused on particular questions
Cross cultural interviewing
Layers of complexity
Language barriers
To be a good interviewer you must like doing it
Not all interviews are interesting and not all go well
Types of interviews
Informal conversation
General interview guide
Standardized open-ended interview
Informal conversation
AKA unstructured interviewing, ethnographic interviewing
Spontaneous generation of questions in the natural flow of interaction
Maximum flexibility
Unstructured does not mean unfocused
Interviewer must “go with the flow”
Data collection varies
Some interviewers do not take notes, but write down what they learned later
Some take notes or even use a tape recorder.
Strengths: Flexibility, Spontaneity, Responsiveness to individual differences and situation changes
Weaknesses: Requires a great amount of time to collect systematic information, May be susceptible to interviewer effects, leading questions, biases, difficult to pull together and analyze data.
Formal Interviews
Outlining a set if issues that are to be explored with each respondent before the interview begins
Advantages:
Interviewer makes best use of limited time
Interviewing is more systematic and comprehensive
Can be flexible in the specificity of important issues and the extent to which it answers your questions
Standardized Open-ended
A set of questions carefully worded and arranged with the intention of taking each respondent though the same sequence and asking the same questions with essentially the same words
Standardized Open-ended
Strengths:
Same questions are asked
Can compensate fore variation in interviewing skills
Can establish priorities for interview
Used to compare answers to the same questions over a period of time
Makes data anaysis easier
Exact instrument is available for inspection
Can help overcome questions of legitimacy and credibility, limiations of the data can be known.
Weaknesses
Does not permit interviewer to pursue topics unanticipated
Reduces the extent individual differences and circumstances can be queried
Combination of approaches
Combining an interview guide with standardized with key questions offers flexibility in probing and determining when it is appropriate to explore certain subjects in greater depth or a new area
May use conversation
What types of questions to ask
Experience and behavior questions
Opinions and value questions
Feeling questions
Knowledge questions
Sensory questions
Background/demographic questions
Time frame questions
Can make questions different by asking about them as in the past or future
Sequencing questions
Start with non-controversial present behaviors, activities, and experiences,
Follow with opinions and feelings
Knowledge and skill questions need a context
Ask about present, then ask about the same in the past, then possibly broach the future.
How to phrase questions:
Open ended questions never yes or no, questions you can get a significant response to
Horns of a dichotomy
Asking singular questions
Theme the wording used in asking questions can make a significant difference in the quality of the responses elicited.
Clarity of questions:
Ask singular questions
Learn special terms used by people in the setting of the interview
Understand the language participants use among themselves in talking about a setting, activities, or other aspects of life
Avoid using labels use the respondents own terms and clarify them in the interview
Theme use language that is understandable and part of the frame of reference of the person being interviewed
Why: Take care when asking why
Presuppose things happen for a reason that those reasons might or might not be shared
Rapport vs. Neutrality
Rapport is a stance about the person being interviewed (respect)
Neutrality is a stance about the content of what the person says.
Illustrative Examples
Letting the respondent know that you are not interested in the sensational just in the genuine experiences
Share examples from all sides to show that you’re heard it all
Must take caution to avoid leading questions
Role Playing and simulations
Provide a context for the questions to help the respondent hone in on relevant responses
Questions ask the respondent to become an observer
Presupposition Questions
Can increase the richness and depth of the responses
Bypasses an initial step of asking if a person has had such an experience
Prefatory statements and announcements
Alert the interviewee to what is about to be asked to direct awareness and focus attention and to give the respondent time to organize thoughts.
Probes and follow-up questions
Used to deepen the response, increase richness and depth and give clues about the level of depth of responses
Detailed oriented
Process feedback during the interview
Interviewer has a responsibility to communicate clearly what information is desired and why it is important and to let the interviewee know how the interview is progressing
Helpful to maintain the flow of the interview
Support and recognition responses
Common mistake is failing to provide reinforcement and feedback
Words of thanks, feedback and praise.
Maintaining control enhancing quality
Don’t want to jeopardize the quality of the interview
Know what you want to find out
Ask focused questions to get relevant answers
Listen attentively
Give appropriate verbal and nonverbal feedback…
The one shot quezstion
Sometimes things don’t work out as planned
This is the question you ask if you are only going to get a few minutes with the interviewee and it’s the most important question.
If I only had one question to ask what should it have been.
Final or closing questions
How to gather data
Recording the data
Explain the use of the tape recorder to the interviewee
Increase accuracy of the data
Permits the interviewer to be more attentive
Part of the interview is observing the respondent
When not possible to record, notes must be thorough and comprehensive.
Taking notes during the interview
Purpose
Helps to formulate new questions
Can stimulate early insights
Facilitates later analysis
Backup to recorder malfunction
Use key phrases and major points
Helps pace the interview
After the interview
Check the tape to make sure it was functioning properly. If not, make extensive notes immediately go over notes to avoid ambiguity or uncertainty
Check to see if there is a need for clarification or follow up
Record details about setting and observations about interview
Reflect on the quality of the interview data
Critical for reflection and elaboration
Special applications and issues
Think-aloud protocol interviewing
Aims to elicit the inner thoguths or cognitive processes that illuminate what’s going on in a person’s head during the performance of a task
Get people
Focus Group interviews
Interviewing a small group of people on a specific topic
6-10 people
similar background
1-2 hours in length
recognizes that many decisions are made in a social context
discussion though direct interaction among participants
additional comments can be made after one hears what another says
consider one’s views In the context of another.
Focus group interviews:
Advantages
Cost effect
Interactions among participants enhance quality of data
Quick assessment of consistency or diversity of views
Disadvantages/limitations
Umber of questions asked is restricted
Available response time is restrained for any particular individual
Facilitating requires considerable group process skills
Those in the minority ay not speak up
Works best with strangers
Controversial issues are poor topics
Confidentiality is hard to assure
Not good at identify subtle differences
Take place outside of natural setting.
Group interviews
Unstructured conversation interviews that are not focused on particular questions
Cross cultural interviewing
Layers of complexity
Language barriers
To be a good interviewer you must like doing it
Not all interviews are interesting and not all go well
Subscribe to:
Posts (Atom)
