《人类监控自动化系统》英文版 Research methods

Research methods Experimental design 16.422 Human Supervisory Control April 2004

Research Methods & Experimental Design 16.422 Human Supervisory Control April 2004

Research methods Qualitative vs. quantitative Understanding the relationship between objectives(research question and variables is critical Information≠Data Information =data analysis a Planning in advance is a must To include how data will be analyzed

Research Methods „ Qualitative vs. quantitative „ Understanding the relationship between objectives (research question) and variables is critical „ Information ≠ Data „ Information=data + analysis „ Planning in advance is a must „ To include how data will be analyzed

Qualitative Research Methods Social cultural phenomenon Case studies Focus groups Observations Usability testing Can be quantitative Interviews Questionnaires

Qualitative Research Methods „ Social & cultural phenomenon „ Case studies „ Focus groups „ Observations „ Usability testing „ Can be quantitative „ Interviews „ Questionnaires

Quantitative Research Methods Natural phenomenon Mathematical modeling Experiments a optimization Game theory a Surveys a Bottom line statistics are a must

Quantitative Research Methods „ Natural phenomenon „ Mathematical modeling „ Experiments „ Optimization „ Game theory „ Surveys „ Bottom line – statistics are a must

Project Assignment Design and conduct an experiment in which you explore some measure of human performance through testing, analyze the results, and discuss the broader implications Design an actual display that uses automation for decision support.. While formal experimental testing is not required, a small group of users should be used to identify problems with the design to include functionality evaluation as well as recommendation for future improvements and systems integration

Project Assignment „ Design and conduct an experiment in which you explore some measure of human performance through testing, analyze the results, and discuss the broader implications. „ Design an actual display that uses automation for decision support… While formal experimental testing is not required, a small group of users should be used to identify problems with the design to include functionality evaluation as well as recommendation for future improvements and systems integration

The Experimental design Process Research Question (Hypothesis) Design Experiment Collect data Analyze data Draw Conclusions

The Experimental Design Process Research Question (Hypothesis) Design Experiment Collect Data Analyze Data Draw Conclusions

Experimental Design Design of Experiments (doe defined a theory concerning the minimum number of experiments necessary to develop an empirica/model of a research question and a methodology for setting up the necessary experiments a parsimony model Human subject VS object experimentation other doe constraints Time Money

Experimental Design „ Design of Experiments (DOE) defined: „ A theory concerning the minimum number of experiments necessary to develop an empirical model of a research question and a methodology for setting up the necessary experiments. „ A parsimony model „ Human subject vs. object experimentation „ Other DOE Constraints „ Time „ Money

Experimental Design Basics TWo kinds of data gathering methodologies observation Cant prove cause effect but can establish associations Hawthorne effect social facilitation Experimental Cause effect Variables of interest- factors vs treatments Independent variable Treatment manipulations of variables of interest Treatment VS control group Dependent variable is what you are measuring

Experimental Design Basics „ Two kinds of data gathering methodologies „ Observation „ Can’t prove cause & effect but can establish associations. „ Hawthorne effect, social facilitation „ Experimental „ Cause & effect „ Variables of interest – factors vs. treatments „ Independent variable „ Treatment – manipulations of variables of interest „ Treatment vs. control group „ Dependent variable is what you are measuring

More basics Confounds Randomization concerns Randomization prevents experimental bias Assignment by experimenter Counterbalancing Statistical assumptions a requirement for statistical tests of significance Why would you use the observation methodology instead of experiments?

More Basics „ Confounds „ Randomization Concerns „ Randomization prevents experimental bias „ Assignment by experimenter „ Counterbalancing „ Statistical assumptions „ A requirement for statistical tests of significance „ Why would you use the observation methodology instead of experiments?

DOE Terminology Replications Independent observations of a single treatment. a Variance The measuring stick that compares different treatments Internal validity The extent to which an experiment accomplishes its goals) Reproducibility Given the appropriate information, the ability of others to replicate the experiment

DOE Terminology „ Replications „ Independent observations of a single treatment. „ Variance „ The measuring stick that compares different treatments. „ Internal validity „ The extent to which an experiment accomplishes its goal(s). „ Reproducibility „ Given the appropriate information, the ability of others to replicate the experiment