RESEARCH DESIGN

 

Spring 2013

 

Class Information

Instructor Information

PSYCH 6002

Helen C. Harton, Ph.D.

Baker 315

Baker 357

W 12-2:50pm

273-2235; harton@uni.edu

 

Office Hours: WF 11-11:50; W 2-3; pretty much any time I’m around

 

Readings:

Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.). Cincinnati, OH: Atomic Dog Publishing.

 

Kuhn, T. S. (1996). The structure of scientific revolutions (3rd ed). Chicago: University of Chicago Press.

 

Articles/readings listed in the class schedule. The vast majority are available online. When the articles are not available online, instructions on how to find them are included after the citation.

 

Course Description and Objectives: Research is at the heart of psychology; psychology’s focus on research and empirical evidence is one of the things that differentiates the field from other similar disciplines. In this course, you’ll learn about the basics of research, explore several “hot topics” or new methods in psychology, and become a better designer, describer, and critical reader of research. The course is designed to give you a breadth of knowledge on research design in psychology and to facilitate your future research career and thesis progress.

 

By the end of this course, you should

1) know the advantages and disadvantages of the commonly used methods in psychology and when to best apply them;

2) know the advantages and disadvantages of several new methodologies in psychology;

3) be able to critique and integrate previous research;

4) be more aware of ethical concerns in research and how to deal with them;

5) be a better scientific writer;

6) be able to design and carry out your own research project; and

7) have made significant progress on your thesis.

 

Course Format: Each week you will have several readings that you should have processed and thought about before class. In class, we will discuss the readings and the issues they bring up. You should contribute thoughtfully to the discussion and build on and gently challenge the comments of other students.

 

Course Requirements:

 

Class discussion (20%). In graduate school especially, you learn not only from books and professors, but from interactions and discussions with peers. Discussing information also helps you to think about it more deeply and learn it more quickly. You are expected to contribute meaningfully to class discussions. While mere attendance is not enough to get a good grade for this component, it is imperative in that you can’t participate if you’re not here. Obviously, it also requires that you read and think about the readings. Both frequency and quality count. You will get graded on discussion each week; you can drop your lowest score.

 

Thought papers (15%). Each week you’ll also be asked to email me a short (1-2 pages) thought paper addressing the readings for the week. These emails should have “thought paper” in the subject line and be sent by midnight Monday nights (just paste it in the message instead of attaching it). The thought paper should address all the readings for the week at some level. Think about how these readings relate to your thesis; bring up questions you have; relate the readings to other things you know. You want to show me that you’ve read, understood, and carefully processed the readings for the week. In addition, some weeks there will be a specific assignment (e.g., find a meta-analysis on a topic). The purpose of these assignments is to get you thinking more deeply about the method and its application to your research and to facilitate in-class discussion. You can drop your lowest thought paper grade.

 

      Discussion and thought papers will be graded on the following scale:

            0 = not there

            2 = attended but didn’t participate, or turned in, but not very relevant (below average)

            3 = comments or questions relevant, but didn’t involve much insight (average)

            4 = comments or questions relevant and insightful (good)

            5 = more than one comment or question showed a significant contribution (excellent)

 

Research proposal (45%). One of the best ways to show that you understand research design is by applying it. You will submit a complete research proposal (7-8 page introduction, complete method and appendices, section on ethics and validity, plan of analysis, discussion of expected results/what it would mean if you didn’t find them, in APA style). This paper doesn’t have to be your thesis proposal, but if you can write about something that does develop into it (or another research project), all the better. There will be assignments due during the semester to provide you with feedback on this paper (e.g., outlines, rough drafts) that will count toward the “assignment” portion of your grade. Obviously, this should not be a paper that you have written or are writing for another class. If there is any question about whether it might be too collaborative or too close to something you’ve already done, talk to me before you get started to see if it is acceptable. The paper should be in good APA style. Those that aren’t, will be returned ungraded and lose at least 1 letter grade (1 if turned in that same day, -1 more for each day after).

 

Proposal presentation (10%). During one of the last class sessions, you will present your proposal to the class (background, method, expected results, what they would mean, etc.). Your presentation, which should include some audio-visual effects (e.g., PowerPoint), should last no more than 12 minutes, followed by a discussion of the proposal by the class (we will play the role of thesis committee). You can (and should) integrate any helpful comments from the class into your proposal before you turn it in. You’ll be graded on presentation style as well as knowledge about the area. Go to https://docs.google.com/a/uni.edu/spreadsheet/ccc?key=0AvGm2QdOV41CdHQwbUYySDVDeW40V0pmMWxKb1BFWmc to sign up for a time. NOTE: You have to do this from your uni google account.

 

Assignments (10%). In addition to the weekly thought papers, you will turn in several assignments directly related to your research proposal (i.e., topic, articles, outlines, rough draft) and provide feedback on a peer’s draft. You may sometimes be asked to read short articles before class that we will critique in class. You will also present on a particular method or “hot topic.” For your topic presentation, you should read at least one article or chapter on the method itself as well as at least two articles that used the method (if I listed an article with the presentation topic in the class schedule, that should be one of the ones you read). In the presentation, you’ll tell us about what the method is, what its advantages and disadvantages are, when it should/shouldn’t be used, how the articles you read used it and whether that was a good use or not, and depending on the topic, either best practices in using the method and/or how (in general) you would analyze data collected by the method. Sometimes there are additional questions you should address that I put on the class schedule. You’re giving a brief overview so that everyone learns the issues involved, but you don’t need to describe it in enough detail that we could actually go out and do it. Presentations should be 10-15 minutes and include your references in your final slide. Send me a copy of your Powerpoints. Go to https://docs.google.com/a/uni.edu/spreadsheet/ccc?key=0AvGm2QdOV41CdHp6WDI2QkltMGl1LWdSTXQ4cEx1THc to sign up for a topic and time. NOTE: You have to do this from your UNI google account.

 

Makeup and Late Paper Policies: Class discussion grades and presentations cannot be made up. Proposals will be accepted up to three days past the due date, but one letter grade will be deducted for each day until they are turned in. Plan ahead and don’t wait until the last minute to finish (or start) the paper, in case something unexpected arises. 

 

Academic Honesty Policy: Cheating and plagiarism of any kind will not be tolerated and will result in a 0 on the assignment in question. This includes using a paper from another class or that you have worked on with another faculty member to fulfill a requirement in this class as well as using quotes from materials without attribution even in short assignments or thought papers. For more information on UNI’s academic honesty policies, see the University Catalog as well as the information in the Department of Psychology Graduate Student Handbook. If you have any questions about what is acceptable, ask.

 

Disability Services: The Americans with Disabilities Act of 1990 (ADA) provides protection from illegal discrimination for qualified individuals with disabilities. Students requesting instructional accommodations due to disabilities must arrange for such accommodation through the Office of Disability Services. The ODS is located at 213 Student Services Center, and the phone number is 273-2676.

 

 

CLASS SCHEDULE

 

Jan. 16—Foundations and theory

T&D, Chapter 1

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. doi:10.1207/s15327957pspr0203_4

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi:10.1177/0956797611417632

 What is the theoretical basis for your thesis? What is good and bad about theories? What examples have you seen of HARKing? Is it really a bad thing? What do you think of the suggestions of Kerr and of Simmons et al.? Are they good? Are they practical? What should we do to avoid misleading reports of research in psychology?

 In class: Panel of 2nd year students on thesis; video on syllabus; then do on-line discussion in elearning. You need to participate in each of the forums at least once (more is better). No thought paper this week.

  

Jan. 23---Philosophy of science

Kuhn book

 What is Kuhn’s view of science? Do you agree? How does science change? How should it change? What is and should be the goal of science? What are examples of “revolutions” in psychology?

 Turn in topic description for your thesis (research proposal for class); put it in my mailbox by Friday at noon.

 In class: Mary Losch will lead discussion. Send your thought papers this week to both of us.

 

 Jan. 30—Ethics

Belmont report

Hertwig, R., & Ortmann, A. (2008). Deception in experiments: Revisiting the arguments in its defense. Ethics & Behavior, 18, 59-92. doi:10.1080/10508420701712990 

Becker-Blease, K. A., & Freyd, J. J. (2006). Research participants telling the truth about their lives: The ethics of asking and not asking about abuse. American Psychologist, 61, 218-226. doi:10.1037/0003-066X.61.3.218  

Barchard, K. A., Williams, J. (2008). Practical advice for conducting ethical online experiments and questionnaires for United States psychologists. Behavior Research Methods, 40, 1111-1128. doi:10.3758/BRM.40.4.1111

 Are there any ethical issues that would need to be addressed with your thesis? How could you deal with them? Is deception okay? When? How should people be debriefed? Are our current techniques adequate? Should “vulnerable populations” receive more protection? How or why? Should we ask about things like abuse?

 Debriefing presentationIs debriefing effective? When or why? Briefly review research on debriefing and give us some “best practice” suggestions.

  

Feb. 6—Sampling

T&D Chapter 2

Henrich, J. Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61-83. doi:10.1017/S0140525X0999152X

Maxwell, S. E., Kelley, K., Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537-563. doi: 10.1146/annurev.psych.59.103006093735

 Is sampling an issue for your thesis topic? What type of sampling will you use? Should you use? What populations have been used in your area? Is that a problem or not and why? If there are WEIRD samples used, how is this likely to interact with your thesis topic area? When are WEIRD samples more or less of a problem? How large should your sample be for your thesis? How will you determine this?

 Find 10 articles for thesis (research proposal), submit in APA style. If they are not in near-perfect APA style, they will be returned ungraded, and you will get a 0 on the assignment. 

 Cross-cultural research presentation—When is cross-cultural research most useful? What particular things do researchers need to keep in mind? What are best practices?

 

 Feb. 13 --Reliability and validity and scaling

T&D Chapters 3 and 5

 Find an article that describe the development of a scale related to your thesis topic (should include multiple studies, assessing reliability and validity). Evaluate the article and scale (and include a brief description of the scale, including the APA style citation). How did they show the various types of reliability and validity? Do you agree? What type of scaling is the measure? What type of scale? Are there similar concepts that still need a scale developed? How would you go about creating a scale? What would you correlate it with to test reliability and validity?

 Outline of intro for proposal

 Method variance presentation--What is the problem of method bias? How can it be decreased? Find at least 2 articles that dealt with this issue either well or poorly.

Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539-569. doi:10.1146/annurev-psych-120710-100452

 

 

Feb. 20—Surveys

T&D Chapter 4

Tourangeau, R. (2004). Survey research and societal change. Annual Review of Psychology, 55, 775-801. doi:10.1146/annurev.psych.55.090902.142040

Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549-576. doi:10.1146/annurev.psych.58.110405.085530

Find examples of “bad” survey questions. Think about any issues that you might need to deal with in terms of how you ask questions on your thesis. What types of issues (e.g., population, sampling) might come up and how would those affect what method you choose? What are the advantages and disadvantages of survey methodology for your topic?

Diary presentation—Include information on the variety of diary methods and how they are analyzed (sometimes called “event-sampling” as well)

Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579-616. doi:10.1146/annurev.psych.54.101601.145030

 

 

Feb. 27-- Experimental design

T&D Chapters 7 and 9

Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182. doi:10.1037/0022-3514.51.6.1173

MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007). Mediation analysis. Annual Review of Psychology, 58, 593-614. doi:10.1146/annurev.psych.58.110405.085542

Which threats to internal validity are most likely to be a problem in your thesis? How can you diminish them? What potential mediators and moderators might be involved in your thesis topic area? How can we differentiate between mediators and moderators? Should you include covariates in your thesis? Why or why not?

Outline of thesis proposal

Placebo presentation--What are best practices in placebo research?

Price, D. D., Finniss, D. G., & Benedetti, F. (2008). A comprehensive review of the placebo effect: Recent advances and current thought. Annual Review of Psychology, 59, 565-590. doi:10.1146/annurev.psych.59.113006.095941

 

 

March 6 -- Quasi-experiments

T&D Chapters 10 and 11

Shadish, W. R., & Cook, T. D. (2009). The Renaissance of field experimentation in evaluating interventions. Annual Review of Psychology, 60, 607-629. doi:10.1146/annurev.psych.60.110707.163544

Is your thesis a quasi-experiment? If so, what kind? What threats to internal validity might be a problem? Are the elements of “good design” present in your thesis? Explain. What are the advantages and disadvantages of field research? How could it be (or has been) used in the area of your thesis topic?

 

 

March 13—Evaluation research and meta-analysis

T&D Chapter 16

Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. doi:10.1146/annurev.psych.52.1.59

Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63, 665-694. doi:10.1348/000711010X502733

What types of evaluation are relevant for your thesis topic? Why? Which is more important:  clinical, practical, or statistical significance? Have there been meta-analyses on your thesis topic? If not, is the area ready for one? What key words would you use? What mediators or moderators would you test? Find and evaluate a meta-analysis that interests you/is related to your topic (include citation and brief description of method/findings). What is good and bad about what they did? Address things like inclusion criteria as well as fixed vs. random effects, etc.

 

March 20—Spring break

 

 

March 27—Quantitative analysis

T&D Chapters 12 and 14

Kashy, D. A., & Kenny, D. A. (2000). The analysis of data from dyads and groups. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology. New York: Cambridge University Press. On reserve in the library (2 hour hold).

How will you analyze your data for your thesis? What tests will be involved? What assumptions do those tests have? Make sure you include enough description of your method that I can understand whether these analyses are appropriate. How could some of the analyses that you were less familiar with be useful in studying your topic? Find a quantitative article related to your thesis (include citation) and answer the quantitative critique questions below.

 

 

April 3--Writing it up, peer review

T&D Chapter 15

Strunk, W., Jr. (1999). The elements of style. New York: Bartleby.com

Bem, D. J. (2002). Writing the empirical journal article. In J. M. Darley, M. P. Zanna, & H. L. Roediger III (Eds.),  (2002). The compleat academic: A career guide. Washington, DC: American Psychological Association.

Sternberg, R. J. (1993). How to win acceptances by psychology journals: 21 tips for better writing. APA Observer.

Bring in rough draft of thesis proposal for peer review and comment. No thought paper this week!

 

 

April 10--Other methods: Implicit measures and physiological measures

Fazio, R. H., & Olson, M. A. (2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297-327. doi:10.1146/annurev.psych.54.101601.145255

De Houwer, J., Teige-Mocigemba, S., Spruyt, A., & Moors, A. (2009). Implicit measures: A normative analysis and review. Psychological Bulletin, 135, 347-368. doi:10.1037/a0014211

Sowden, P., & Barrett, P. (2006). Psychophysiological methods. In G. M. Breakwell, S. Hammond, C. Fife-Schaw, & J. A. Smith (Eds.), Research methods in psychology (3rd ed.) (pp. 146-159). Thousand Oaks, CA: Sage. Available in resource room and possibly on googlebooks.

What are the advantages and disadvantages of implicit measures? What are they really measuring? When should they be used? What do physiological measures tell us and what can they not tell us? How could either of these types of measures be used in research in your thesis area? What are questions or comments do you have about these readings?

fMRI presentation—What is fMRI? What does it tell us? How can it be used in psychology?

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274-290. doi:10.1111/j.1745-6924.2009.01125.x

ERP/EKG presentation—What are ERP and EKGs? What do they tell us? How can they be used in psychology?

Eye tracking presentation—What is eye tracking? How can it be used in psychology?

Duchowski, A. T. (2002). A breadth-first survey of eye tracking applications. Behavior Research Methods, Instruments, and Computers, 34, 455-470. doi:10.3758/BF03195475

Hormone sampling presentation—What hormones have typically been tracked in psychology? How are these collected? What do they tell us? What are best practices in collecting them (e.g., times of day, people for whom the results aren’t accurate)?

 

 

April 17—Other methods: Spatial analyses and internet-based research

Goodchild, M. F., Anselin, L., Appelbaum, R, P., & Herr Harthorn, B. (2000). Toward spatially integrated social science. International Regional Science Review, 23, 139-159.  doi:10.1177/016001760002300201

Skitka, L. J., & Sargis, E. G. (2006). The internet as psychological laboratory. Annual Review of Psychology, 57, 529-555. doi:10.1146/annurev.psych.57.102904.190048

Lewis, K., Kaufman, J., Gonzalez, M., Wimmer, A., & Christakis, N. (2008). Tastes, ties, and time: A new social network dataset using Facebook.com. Social Networks, 30, 330-342. doi:10.1016/j.socnet.2008.07.002

How does space affect your thesis topic, if at all? Are there potential spatial effects either that may be interesting in their own right or that should be controlled for? Research using the internet brings exciting advantages, but also significant disadvantages (e.g., ethical concerns). What do you think about research on the internet—are there certain methods or populations or topics that shouldn’t be studied there? What other questions or comments do you have about these methods?

Computer simulation presentation—How can computer simulations add to our knowledge of psychology and human behavior? When are they best used? What advantages and disadvantages do they offer?

mTurk presentation—What is mTurk? When is it more/less useful?

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s mechanical turk: A new course of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6, 3-5. doi:10.1177/1745691610393980

Buhrmester, M. (2010). Amazon mechanical turk guide for social scientists.

Smart phone presentation—What types of applications can smartphones be used for? Try one out if possible (e.g., iEar). What are its advantages and challenges and how can the challenges be addressed (think ethics as well)?

Raento, J., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for social scientists. Sociological Methods and Research, 37, 426-454. doi:10.11177/0049124108330005

Virtual reality presentation—How has virtual reality been used in psychology? What types of virtual reality have been used? What are the advantages and disadvantages of these approaches? (Jim Blascovich is a good researcher to start with)

 

 

April 24--Qualitative research

T&D Chapter 6 and 8 and 13

Smith, C. P. (2000). Content analysis and narrative analysis. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 313-335). New York: Cambridge University Press. On reserve in the library.

What are the advantages and disadvantages of qualitative research? What new or useful information could qualitative studies provide on your thesis topic? Find and critique an article using qualitative methods that is related to your thesis topic. Use the question for the qualitative critique below.

Semi-structured interviews presentationWhat are the advantages/disadvantages of this method? What are some best practices? How can the data be analyzed?

Bartholomew, K., Henderson, A. J. Z., & Marcia, J. E. (2000). Coded semistructured interviews in social psychological research. . In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 286-312). New York: Cambridge University Press. On reserve in the library.

Two proposals

 

 

MONDAY, April 29—Proposal presentations

Six proposals

 

 

Monday, May 6 (1:00-2:50)—Proposal presentations

Three proposals

 

 

Papers due Thursday, May 9 at 5pm. Turn in rough draft and peer review comments as well.

 

 


Critique of Quantitative Research Article

 

1.  What is the purpose of the article? How does it fit into previous literature? 

 

2.  What is (are) the research question(s)?

 

3.  What is the research design (be specific)?

 

4.  How are the variables operationalized or measured?

 

5.  Are the measures reliable and valid? What evidence do you have of that?

 

6.  Is the sample representative of the population of interest? How much does that matter for this study?

 

7.  Is there sufficient power to obtain significant results?

 

8.  What types of statistical analyses were done?  Do they match the research questions/hypotheses?

 

9.  What are the results?

 

10.  Is the discussion of the results in line with the results?

 

 

 

 

 

Critique of Qualitative Research Article

 

1.  What type of qualitative study is it? (ethnography, case study, focus group, etc)

 

2.  What is the purpose of the study specifically?

 

3.  How does the study build on previous literature?

 

4.  What sources of data over what period of time were used to “hear the participants’ voices”?

 

5.  What does this study tell you that a quantitative one would not be able to?

 

6.  What did we learn from the study? What were the results?