Academic decision making

In 2011, an enthusiastic (masochistic?) Pre-Dr. Walsh embarked on an ambitious quest to pick the brains of psychology researchers the world over to understand what goes on in the minds of expert researchers. After thoroughly researching decision trees and guidelines (more on that monster below), I came up with a questionnaire that I felt provided the kind of insight lacking from the cold hard flow of a flow chart. I compiled a list of over 800 psychology (and allied) departments worldwide, and set about contacting their faculty one at a time… Also, ambushing people at scholarly conferences. Shameless, shameless ambushing.

Four months later and it became abundantly clear that academics don’t really like doing surveys, and I ended up with far too little data to systematically analyse and form into something publishable in a scholarly journal.

I'm in the very slow process of massaging the specific insights from this (alongside practical how-to advice) into a comic book format for early researchers. But, I think some of the general advice should be shared as soon and as widely as possible.

Click here to see the presentation slides for: Walsh, E., & Brinker, J. (2013) Academic Decision Making for statistical analysis, Australian Mathematical Psychology Conference (Adelaide, 10th - 12th of February)

My 2011 survey attracted just over 100 respondents from all areas of psychology, with the largest number being cognitive (16%) developmental (15%), and social (12) psychology researchers. Most (over 90%) had PhDs, with a good mix of seniority; everything from postdoctoral fellows through to full professors. Years of research experience ranged from three to sixty; so there’s also a nice mix of new and veteran scientists. They came from all around the world, though the majority of responses were from Australia, the United States of America, and the United Kingdom. There was a slight gender skew to males (65% male).

While I asked a lot of things in the survey, one key question was: From your personal research experience, if you could give one piece of advice to new researchers, what would that advice be?

Why should a young researcher be interested in these (mostly unfiltered) musings? Because I also asked: “Do you feel you have sufficient time in your everyday working life to educate yourself in order to expand your knowledge of research types, designs, and methods of data analysis?”  and 60% said “no”. That’s… a little scary to me. Aspiring and actual PhD students and early career researchers - now is clearly the time to get into good habits!

Here's the direct, unedited responses to my open-ended plea for advice.

Don't try to reinvent the wheel- where sound, appropriate methodologies exist, use these to approach your research area of interest. In the long run, this will save you some grief at the review stage of the process. Don’t give the reviewers added cause to trash your work by use of novel techniques for no good reason.

Comparison between mathematical models is a type of analysis that is growing. I recommend to spend a considerable amount of time to learn how to create, fit, compare and select mathematical models. Moreover, the statistical free software R would be THE tool in the near future. Spend some time to familiarize with this tool early on.

Beware of recruitment issues, and thoroughly study the population you'll be recruiting from beforehand.  If you have an "in" or two in the system, even better.

Get a good understanding of statistics.

1. The only way to learn is to do, fail, and try again.  2. Make the commitment to doing your best, don't be sloppy, and have the guts and faith in yourself to be your own worst critic.

Consult, consult, consult!  Ensure that you have a robust research question and one that is feasible to answer within the time frame and resources available.   Ensure that the methodology/method is congruent to the question posed.  Consider ethical issues in-depth.

Be sure to choose a population that you will be able to collect a sample from! It is all well and good to choose to do a community sample, for example, but you have to make sure that your potential participants are actually amenable to being sampled from. Try to work with children and families can create a lot of barriers to interacting with participants and you have to be prepared for those challenges from the start.

Make sure that you use the analyses that test your hypotheses.

Do what you like.

Clearly define your research question and ensure the study/survey is able to coherently answer the research question at each stage of the design.

Think carefully about the problem or problems that your research might help to solve.

Study experiment design and statistics together, formally, early and extensively.  This will pay handsome dividends later on.  Don't just assume you can learn on the job, because you will start with one hand tied behind your back.

When doing fieldwork, always be mindful of your hosts! Research in dynamic environments, such as schools, often means that the aims of data collection can be affected by time and opportunity to complete the work. People you need to interview or observe may be subject to many competing pressures, of which your data collection may not be a priority. Adopt a 'go with the flow' attitude, and don't feel pressured to collect data be any means necessary. Your relationship with your hosts is important.

Separate out a portion of the day - EVERY day - to writing.  Even if you do not have anything in particular to write about, getting into the writing habit will pay off.  Although our passion is to do research, and we can spend glorious hours following links between fascinating papers, our productivity is measured in the number of publications we have. If you try the daily writing technique you may find yourself with fully written method sections waiting for you when it's time to write a full paper - you've already written a large chunk of your publication!  If you've spent writing time recording your stream-of-consciousness thoughts on a topic, you may find yourself with new research ideas, perspectives, or even fully fleshed theories (useful for those theoretical papers too!).  And most importantly, when it comes time to write up your paper, the blank page will not seem so daunting to you - you will have faced it daily!

Find a good mentor and work on their projects.

Create relationships with people who have knowledge in areas in which you do not. Don’t be afraid to admit you don't know how to do something.

Understand the question that needs to be answered before deciding which method and statistical analysis to use.  The question should drive those choices, not the reverse.

Work with multiple people during your training (grad school, internship, post-doctoral fellowship, etc), and make sure that they use DIFFERENT methodologies.  So, if your grad advisor was exclusively using surveys, do a post-doc with someone who uses controlled experiments.  etc etc.  You never know what you will want/need to do once you are "on your own", and the best way to learn the pros and cons of a technique is to do it YOURSELF

Think big but focus systematically.

Work on questions that address multiple levels (i.e., smaller projects that can be completed and disseminated in a timely fashion, bigger picture projects with higher risk/reward). So think big and think small. Perhaps more important, though, always subject your questions to critical analysis to guard against bias.

Find something that interests you and study it.

Take as many statistics courses as you can.

Take the time and do it right.

Do a lot of experiments.

The most important thing is to find a question that fascinates you so that you can continue to ask it in new and different ways.  Doing so builds a research program and will facilitate finding something meaningful and replicable.

Create alliances with other researchers so you can benefit from your very own critical reviewers.

Have a mixture of research projects going at any one time, from ones that are very adventurous and fascinating but could quite possibly fail, to ones that might make incremental advances on past research but are very unlikely to fail.

Find a really good mentor, particularly if your graduate advisor was not very good. Find someone who can read your work, discuss project ideas with you, and help you with figuring out problems that arise in research.

KISS [keep it simple, stupid].

Do not become methodologically rigid.  Having a good working knowledge of various types of data collection methods and analytic tools (statistical analyses, qualitative analyses) will mean you are free to ask a wider range of research questions and then know how to pursue the answer to those questions.

One's learning in methods and statistics should always grow. I have never regretted the time and pain spent learning new tools---they have made my research more effective and more fun.

3 things:  1) constantly re-evaluate your program of research, 2) find challenging collaborators, 3) learn advanced statistics!

Ensure that you really enjoy the content area. You will spend a lot of time immersed in it!

1) Understand the rationale of the statistical techniques, rather than just knowing how to use them. This affords you the possibility to choose what is the more appropriate technique in each case.  2) Statistical techniques are tools to investigate psychological phenomena. Do not use a technique because you were trained on it. 3) Think what is the psychological phenomenon you want to investigate, and then choose the technique that best suits your purpose.

1. Go Bayesian   2. Limit your research question to focus on the most essential elements.

Data analysis means modeling your data to understand it; it is quite possible off-the-shelf models (general linear models underlying regression and ANOVA, for example) are not the best ones, and you should develop the skills you need to treat data analysis as a data modeling exercise.

Spend far longer on getting clearly expressed and focused research questions before collecting data.  If research questions are worked out properly, then problems in research design and analysis will become far less frequent and important than you think.

Always discuss a novel paradigm with a statistician BEFORE running the experiment.

Don't be afraid of math; it is a tool for you for use.

Learn about a wide variety of methods; learn complex statistical techniques (e.g., SEM).

Ultimately the participants in your research are the most important consideration, and retention of the sample is a mark of study quality.  Do everything in your power to make participation in the study as straightforward for your sample as possible.  Consider testing in the home or somewhere convenient for them, and provide them an opportunity to easily get to testing sites.  Be mindful of the demands you are placing on participants, and be very available to answer their concerns.

Do what interests you.

Wherever possible, work with the best people you can find, whether they're senior, equivalent, or junior colleagues.

Learn how to ask questions that can be answered in a reasonable amount of time.

1. Find a question that you as genuinely interested in the answer to - will make everything else easier and motivation higher.

Find a topic which you have significant interest in so that it will maintain commitment to it and, thus, allow or the creation of a body of research.

Be strategic. Most people need to achieve certain goals in order to establish themselves in the field and get tenure at their institutions. Choose the work that will set you up for tenure, and then post-tenure you can start the more complex, longitudinal and theoretical studies that you really want to do. Go with what you know, but be aware of the requirements of the journals in which you would like to publish. Stats is changing--stay ahead of the curve if you can.

Take as many methodology courses as you can. Make sure that you work with professional methodologists and insure there is money in budgets for professional methodologists.

Always let the research questions guide the methodology and the statistics....not the other way around.

Let your research question lead you to your method and analysis.

Statistical knowledge is paramount! Attend the appropriate courses for your analyses.

Find something you are interested in, and find a good theory to explain it.  Let theory guide you at every step, from writing the introduction to selecting methods.  Young researchers too often, I think, go looking at some question without having a good explanation for why they are predicting what they are predicting.  Then they can find themselves in a morass.  Theories help organize chaos.

Should use multiple research methods [mixed method] and should use multi-statistical tests [factor and reliability analysis, group differences [ANOVA] and correlations and linear regressions    Do not know how to complete more sophisticated stats [path analysis, etc], see this as a limitation but ....

Research what interests you. I you no longer find a topic interesting switch to a new area.

Research methods should be driven by your question. Ask bigger questions and don't decide what the answer is before you try to find out. i.e. hypothesis testing is of little use in many social science subjects.

 Frankenstien's monster 

Coming at the advice-for-young-researchers issue from another angle, this is a Frankenstien's monster of a decision tree compiled in 2011 from a mixture of book and online sources, intended to cover the entirety of the research process. Click here to download the .pdf.

  frank_monster_WEBSITE_VER_Page_01.jpg frank_monster_WEBSITE_VER_Page_02.jpg

frank_monster_WEBSITE_VER_Page_03.jpg frank_monster_WEBSITE_VER_Page_04.jpg frank_monster_WEBSITE_VER_Page_05.jpg frank_monster_WEBSITE_VER_Page_06.jpg frank_monster_WEBSITE_VER_Page_07.jpg frank_monster_WEBSITE_VER_Page_08.jpg frank_monster_WEBSITE_VER_Page_09.jpg frank_monster_WEBSITE_VER_Page_10.jpg frank_monster_WEBSITE_VER_Page_11.jpg frank_monster_WEBSITE_VER_Page_12.jpg frank_monster_WEBSITE_VER_Page_13.jpg



Citations for decision tree:
Corston, R., & Coleman, A. (2000). A Crash Course in SPSS for Windows. Oxford: Blackwell.
Grosofsky, A. (2008). Statistics Decision Aids.
Haslam, S. A., & McGarty, C. (2003). Research Methods and Statistics in Psychology. London: Sage.
Hopkins, W. G., & Batterham, A. M. (2005). A Decision Tree for Controlled Trials. Health (San
Francisco), 9, 33 –39. Retrieved from
Howell, D. (2008). Fundamental Statistics for the behavioral sciences (6th). Belmont, CA: Wadsworth.
Matsumo, D., & Van de Vijver, F. J. R. (2010). Cross-Cultural Research Methods in Psychology.
Cambridge University Press. Retrieved from\
Mock, T. J. (1972). A Decision Tree Approach to the Methodological Decision Process. Analysis, 47(4),
826–829. Retrieved from
Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate Statistics (Fifth). USA.