Demystifying the Peer Review Process: Advice from Associate Editors

May 1, 2017

Guest post by Ethan R. Van Norman, Georgia State UniversityIn a previous post, the ECF offered guidance on identifying where to submit a manuscript. One piece of advice was to consider the composition of the journal’s editorial board. From that we can infer that the likelihood your paper will be viewed favorably depends in part on the background and theoretical orientation of the individuals that will review your paper. Many of the readers of the ECF serve on those editorial boards. In the frenzy to submit papers and hopefully obtain tenure, it is easy to lose sight of the importance and necessity of the peer review process. In fact, wading through reviewer comments to revise an article (if we are so lucky to be invited to do so) is often viewed as a chore instead of an opportunity to improve our work. Such woes may be self-inflicted (e.g., we did a poor job setting up the rationale for our study or overlooked critical threats to the internal validity of our findings) or, as many of us would rather believe, we are tasked with addressing reviewer comments that appear to be devoid of meaningful content or consist of unfair criticisms without solutions. The purpose of this blog post is to highlight steps that early career scholars can take to ensure that the reviews they provide are of the variety to make the revision process for colleagues as pain free as possible.In November Michelle Demaray discussed serving on an editorial board pre-tenure. In that post Dr. Demaray cited two important characteristics of a reviewer: (1) promptness and editorial board etiquette as well as (2) writing quality reviews. The purpose of this post is to further discuss what constitutes a high-quality review from the perspective of associate editors. This topic is particularly relevant for early career scholars because many universities expect pre-tenure faculty to serve on editorial boards. Indeed, serving on an editorial board is an effective and meaningful way to engage in service for the field. Beyond early career considerations, peer review is a mechanism to ensure scholarly work of sufficient quality and importance is published. High quality reviews can help authors re-conceptualize or refine the initial premises of the paper, which may ultimately lead to a more meaningful or methodologically sound work.To further explore what early career faculty can do to complete high quality reviews, we solicited responses to a series of questions from two associate editors of school psychology journals. Given the somewhat sensitive nature of the questions, we elected to withhold the names of the associate editors. The ECF is grateful for the feedback from both contributors.Advice to ReviewersQuestion 1: What types of comments do you seek from reviewers? That is, which type of comments do you find the most useful when rendering your editorial decision? Is there a general “approach” you recommend reviewers take when conducting a review?AE #1: I find that reviews that not only provide a general perspective on the manuscript, but also provide specific information regarding strengths and questions/concerns to be the most useful. Reviewers who articulate the big ideas from their review (e.g., conceptualization issues, methodological strengths and problems, how findings were reported) up front and then expand by illustrating specific instances that support their perspectives not only help me make sense of their viewpoints, but also can help the authors with any subsequent revisions. In other words, I think a critical discussion of the manuscript typically is more helpful than a list of issues that require the authors and the action editor to construct the reviewer’s message.AE #2: I find reviews most helpful if they (a) comprehensively evaluate each section of the manuscript, (b) provide an overall evaluation of the manuscript, (c) are constructive, and (d) note positive attributes of the manuscript.Question 2: Relatedly, to what degree do you consider reviewer feedback when rendering your editorial decision? Can you elaborate on a situation in which you rendered a decision counter to the prevailing opinion of the reviewers – or the circumstances in which that might occur?AE #1: Reviewer feedback absolutely informs the decisions that I render. I am one person who cannot possibly have a command of all of the issues surrounding a given topic or study. Quality reviews help me identify and think through issues relevant to a manuscript. In many instances, they also can help with articulating the rationale for a decision and issues to be addressed if a revision is invited. I have yet to render a decision for a manuscript that was not consistent with at least one or more of the reviewers; however, when I have rendered different decisions than those recommended by individual reviewers, it typically had to do with a difference in perspective (e.g., the appropriateness of the methods used, the fit of the manuscript for the journal). As much as we often strive to be objective in our field, subjectivity absolutely is a part of the peer review process.AE #2: I strongly consider reviewer feedback as a critical component of the peer review process. An editorial decision is not, however, a “vote count” or tally. In some instances, I or a reviewer may identify a critical flaw not identified by other reviewers.Question 3: Inevitably every author will receive an undesirable editorial decision. However, it seems that subjective experience of the peer review process can be highly influenced by the tone of reviewers. Do you have any advice to reviewers to ensure that their comments are as constructive and as respectful as possible?AE # 1: The peer review process requires a critical review of manuscripts so that scientifically-sound work gets published. However, it can be easy to get on a roll with issues you identify and not remember that other human beings will be on the other side of the decision letter. It helps me to remember that science is a tool invented and used by humans to try and understand their world, which means that no scientific endeavor is perfect. However, all papers I have reviewed or have been the action editor for had strengths in addition to limitations. Additionally, I would encourage reviewers to remember that nobody is correct 100% of the time. It doesn’t hurt to include language such as “my perspective…” or “my thoughts are…” when you don’t agree with a decision that could be defensible.AE #2: I think it is helpful to make comments that are encouraging, specific, and can be addressed. It is particularly useful when reviewers point to a paper or resource authors can review to improve their work.Question 4:  If you had to give 1-2 key tips for new reviewers, what would they be?AE #1: Remember the reviews that you have found the most helpful as an author. I suggest operating from the spirit of pointing out issues and providing suggestions to assist the authors in improving the scholarship represented by the manuscript. It isn’t a reviewer’s job to tell the authors what to do, but providing your perspectives and suggestions can do wonders for authors who ultimately want to see their work published.AE #2: Accept or decline invitations to review quickly, submit reviews on-time, and create comprehensive and constructive reviews.Advice to AuthorsQuestion 1: It seems that receiving a decision of “Revise and Resubmit” is often a sigh of relief for new authors. What advice do you have for authors to increase the likelihood that they will receive the coveted “Tentative Accept” or similar decision without going through multiple rounds of reviewer feedback and revisions?AE #1: I always recommend putting the decision letter down after the initial read and coming back to it once you have had a chance to reflect on how you will respond. Once you come back to it, I strongly recommend that you make sure that you thoroughly and clearly address any direction for revisions provided by the action editor. S/he basically is telling you what s/he expects to see for the paper to be published. I also would write a very clear response letter that articulates exactly how you responded to the requested revisions and provides a strong rationale for any revisions you decided not to make (I would minimize these instances as they will need to be very compelling to convince an editor that his/her initial direction should not have been addressed by the authors). I also would clearly articulate how you responded to any reviewer comments not directly addressed by the editor.AE #2:1. Include testable research questions that emanate clearly from the literature review.2. Make explicit the importance and implications of the study.3. Clearly link all sections together, based on the research questions.4. Include only implications that are supported by the study findings.5. Adhere to the Publication Manual of the American Psychological Association (6th ed.).6. Attend to details.Question 2: The time from submitting a manuscript to a journal and receiving an editorial decision can seem excruciatingly long. Given that different journals set different expectations for rendering a decision, do you have any advice as to when an author should follow up with an associate editor regarding a paper? Further, do you have any advice in going about making that request?AE #1: I definitely would not follow-up before the anticipated decision date. I can’t speak for other action editors, but I would not mind receiving a follow-up email at any point after the communicated timeframe for rendering a decision (2-3 months for my journal). In terms of making the request, I would recommend letting the action editor know that you understand how busy s/he is, but that you were wondering when you might expect to receive a decision. Most of us have been in your shoes as an author and a respectful email may facilitate us moving up on our to-do list rendering a decision on your manuscript.AE #2: Many journals report average time to decision. I recommend checking the journal website and any associated materials that may identify a timeline. I think it is reasonable to check-in with an Associate Editor or Editor two months after the average time to decision. If a timeline is not reported, I think it is reasonable to check-in after about 9-months.ConclusionFor more information on the peer review process in School Psychology, Floyd, Cooley, Arnett, Fagan, Mercer, and Hingle (2011) highlight, among other things, what editors consider to be desirable characteristics of reviewers. In line with the comments offered from the current associate editors, editors in that study viewed reviewers that provided constructive feedback with specific strategies to overcome perceived shortcomings as being the most helpful. To that end, early career scholars can seek out references to identify how to write constructive feedback for reviews (e.g., http://www.apa.org/pubs/authors/review-manuscript-ce-video.aspx) as well as handbooks that detail desiderata for different quantitative (e.g., Hancock & Mueller, 2010) and qualitative (e.g., Trainor & Graue, 2013) methods. In conclusion, it’s important to remember that conducting high quality reviews is important for our field. Yes, turning in another review may be another item on your weekly to do list, but the next time you grumble about the amount of time it’s been since you heard back from a journal or the next time your call a colleague to vent about the tone and content of a review you received– take a moment to look back at the most recent review you completed. Did you complete it on time? Was the tone respectful? Did you offer specific rationale for the criticisms you offered and provide specific recommendations to address those criticisms? Were the critiques fair to begin with? Reviewing others how you’d like to be reviewed may go a long way in ensuring the revision process is a worthwhile endeavor.Do you have different perspectives for how to conduct a high-quality review? Do you use different resources / approaches to review papers? Feel free to share your insights / experiences in the comment section below!ReferencesFloyd, R. G., Cooley, K. M., Arnett, J. E., Fagan, T. K., Mercer, S. H., Hingle, C. (2011). An overview and analysis of journal operations, journal publication patterns, and journal impact in school psychology and related fields. Journal of School Psychology, 49, 617-647.Hancock, G. R., & Mueller, R. O. (Eds.). (2010). The reviewer’s guide to quantitative methods. New York, NY: RoutledgeTrainor, A., A., & Graue, E. (Eds.). (2013). Reviewing qualitative research in the social sciences. New York, NY: Routledge.

Where Should I Send My Manuscript? Journal Options for Researchers in School Psychology

March 22, 2017

A common concern among early career researchers is how to identify appropriate journals for their work. Researchers in school psychology publish in a broad range of journals – from school psychology, to general and special education, educational psychology and other subfields of psychology, public health, prevention science, and other fields depending on the topic of a given manuscript. Productive scholars know it is essential to understand the nuances of potential outlets and their respective audiences when before submitting a manuscript. Even within a relatively small field like school psychology, each journal is unique, so a manuscript should be tailored to the journal to which it is submitted. The flip side is that barring fatal flaws, there is a fitting outlet for most manuscripts.How do you identify potential publishing outlets?

• Check your reference list. If you cite multiple papers from a given journal or subfield, it’s likely your paper will fit in that journal or subfield as well.
• Consider where similar scholars publish their work. If others in your area publish in particular journals, you can infer that you work might also have a home there.
• Search the Journal Citation Reports.Web of Science’s JCR allows you to search for all journals indexed by category and rankable by journal metrics. There’s no category for school psychology, but most school psychology journals are include in the category, Psychology – Educational. Other categories that may be of interest to school psychology researchers are Education & Educational Research; Education – Special; Health Policy & Services; Psychology – Applied; Psychology – Developmental; Psychology – Multidisciplinary; and Social Issues.
• Ask your colleagues, collaborators, and mentors. This can be especially helpful for early career scholars. Many publications have reputations independent of journal metrics. Colleagues in your department, field, or area of study can help you to understand the stature of specific journals or subfields of journals within school psychology or your institution. Colleagues can also be helpful in understanding the idiosyncrasies of the journal scope, expectations, and review process, which may inform your decision about whether or not to submit your work to an outlet.

How do you select the best fit for your manuscript?

• Review journal websites. Most journals have a website where the purpose, scope, author guidelines, editorial board, and other journal-specific information is provided. When preparing a manuscript submission to a journal, it’s important to ensure your manuscript is consistent with the scope and type of manuscripts accepted (e.g., original study, review, brief report, commentary) and follows the author guidelines or submission requirements (e.g., implications for specific audiences).
• Peruse recent issues and online first publications. You can gain insight into the types of work regarded favorably by an editor and editorial board by considering the topical areas and types of papers recently published. These articles can also provide clues about stylistic idiosyncrasies of the journal.
• Read the editor’s commentaries. Most editors serve discrete terms and it’s common for an incoming editor to publish a commentary on their vision for a journal at the beginning of their term. It’s a great resource when considering fit, particularly if there have not been multiple issues published under the new leadership.
• Check out the editorial boards. It’s likely that a submitted manuscript will be reviewed by one or more members of a journal’s editorial board. You can gauge the applicable expertise of board members, and often even suggest specific reviewers. The constituency of the editorial board provides indication of the expertise—and by extension, the scholarship—valued by the editorial team.
• Review journal metrics. There are a variety of metrics used to evaluate journal stature (e.g., impact per publication, source normalized impact per paper, SCImago journal rank, total cites, eigenfactor score, h-index, immediacy index, article influence score). Select journal metrics are included in the Journal Citation Reports and are often reported on journal websites. Many institutions value publications in high status journals and make inferences about the quality and impact of an article based on select metrics of a journal.  Consequently, the journals in which a scholar has published are often taken into consideration in review of researchers during hiring, promotion/tenure process, grant reviews, and selection for honors/awards. If you are in a unit or institution where journal metrics are a focus, it may be important to consider journal metrics when selecting outlets for your work.

It’s not uncommon for early career scholars to wonder what the journals in school psychology are as a starting point. Here’s a list of several peer-reviewed school psychology journals: Journal of School Psychology, School Psychology Review, School Psychology Quarterly, Psychology in the Schools, School Psychology International, Journal of Educational and Psychological Consultation, Journal of Applied School Psychology, School Psychology Forum, Contemporary School Psychology, International Journal of School and Educational Psychology, and The Trainer’s Forum. keep in mind, however, that most school psychology scholars publish in a wide array of journals, so don’t limit yourself to school psychology journals.So, there are a variety of strategies you can use to determine potential outlets for your work. We invite others to share recommendations and reflections on these tips. Have a question? Post below and we’ll provide more information.

Capitalizing on Technology to Facilitate Writing Partnerships

August 3, 2015

By Anisa N. Goforth, University of Montana and Natasha K. Segool, University of HartfordHere’s the challenge: Get two highly motivated research partners and friends to work together on research studies and manuscript writing, while one is working at a young doctoral program that is focused on APA-accreditation in frontier Montana and the other is balancing different priorities in a teaching-focused specialist program across in Connecticut. It’s no small feat, and one with which that we have grappled with for four years. Yet, we have been very successful in developing two large-scale studies and data collection efforts, producing three peer-reviewed articles, two peer-reviewed conference papers, and three peer-reviewed conference posters, with more scholarship underway.The BackstoryWe met in graduate school at Michigan State University’s school psychology program. Currently, Natasha is an assistant professor at the University of Hartford in West Hartford, Connecticut and Anisa is an assistant professor at the University of Montana in Missoula, Montana. Natasha was able to provide some support and advice to Anisa during her academic job hunt and during her transition into her academic position, which followed a year after Natasha’s own career started at the University of Hartford. Given our distance from one another, our friendship has mostly occurred through phone conversations and visits during academic conferences.As early career faculty, we often talked about the challenges of academic life, particularly developing and growing a research agenda amid multiple competing demands for our time. During one of our discussions, we talked about our mutual concerns and questions about the use of social media among our graduate students, some of whom did not appear to have a sense of professional etiquette online.  It was our big “Aha!” moment and we encourage everyone to look out for theirs—it is easy to miss. Prior to this, we had discussed doing research together, but those plans never took shape. We had tried to brainstorm research ideas, often forcing projects in which we had some shared experience. Yet, without passion and genuine interest, these ideas perished tragically in our shared Dropbox with documents that went unedited, and frankly unseen, for months at a time.The key to our success has been capitalizing on our shared questions and curiosity. Once we realized that we were both having similar conversations in our classrooms about professional behavior and etiquette online, we wanted to examine these issues further. However, we were traversing an area of professional psychology that had limited research to turn to related to the ethical and legal ramifications of social networking use. Thus, with little fanfare, we launched into a new phase of our relationship. From friendship alone, we added “collaborative researchers” to our identity. Technology helped with this immensely.The LogisticsOver the past two years, we have designed and conducted two studies related to social media in school psychology. Both projects have led to peer-reviewed journal articles (one article in the Trainer’s Forum, one articleled by our colleague and another MSU alum Andy Pham, and another one currently under review) and conference presentations. All of these projects have been developed and written collaboratively using various technologies. Through these projects, we problem-solved through trial and error to develop working habits that were efficient and effective.Creating accountability.We set up bi-weekly meetings to ensure accountability and keep our projects moving forward. Like many of us who are early career faculty (or mid- or later-career!), competing responsibilities seep in, and consequently, our work often occurred just prior to the upcoming meeting, knowing that our colleague was relying on the other to complete tasks. Our meetings were held using Google Hangouts, which allowed us to use the video call feature from our computers. After our meetings, the project leader sent an email with assigned tasks discussed during the meeting.To organize the meetings, we often used Doodlebecause our schedules were already busy and our two time zones made it difficult to find overlapping work hours. Although it was only a two-hour difference (EST and MST), we had to balance meetings around students, teaching, writing times, and truthfully, our personal lives! Between other responsibilities and our desire to not schedule meetings after 5pm, we were left with a single hour from noon to 1pm on Fridays.  Protecting this time for our meetings was essential, and by prioritizing it, we believe this resulted in our success.Although we did not always have significant progress to report, by meeting face-to-face every two weeks, we held each other accountable. In addition, we were able to consistently discuss our project, our expectations of ourselves and each other, and we were able to problem-solve before major barriers arose and derailed our vision. By seeing each other, we maintained a close working relationship together and communicated effectively.  In retrospect, in the rare circumstances when we canceled our meetings, our emails to one another were indications of being side-tracked or caught up with other competing responsibilities. It was our face-to-face meetings, however, that got us back on track and moving forward.Writing and research materials. In order to collectively work on our project, we primarily used DropBoxto share research documents (e.g., IRB documents, research questionnaires, manuscript drafts, etc.). The key advantages to DropBox were that 1) we were both already using it for our other professional responsibilities, 2) we could easily use it from our work and home computers, 3) it provided notifications to us whenever a document was updated document, and 4) it retained documents in typical formats (e.g., Microsoft Word). At other times, other writing tools have been more helpful. We have used Google Driveand its tools (Docs, Sheets, Slides) while on a Google Hangoutvideo call so that we could work collaboratively on content in real time.Challenges and RecommendationsThere were a number of challenges that we needed to problem-solve, including time zones, accessing files, and troubleshooting technology (e.g., connectivity problems, dropped video call, camera errors). At times, this certainly hampered productivity in addition to causing frustration when we had set aside precious time to work together. In the end, however, we firmly believe that use of video-calling technology and file-sharing technology has facilitated much more productivity and accountability in our work together.  The technology allowed us to meet frequently, feel connected with one another, and have the opportunity to incrementally advance our work. More often than not, we spent our first five to ten minutes checking in with each other about our personal and professional lives before launching into our work, which was important for us to maintain and grow our friendship as well. We see our work together as a win-win!A few other things to consider:

• Some researchers are going to be more adept to technology than others. Expect to provide some on-the-go tech support in addition to developing unique research ideas and empirically rigorous studies.
• Compromise on technology options and choose ones that suit the needs of mostof the researchers. We recommend considering the reliability of the technology, ease of use, and cost effectiveness.
• Be patient when there are technology issues (internet connectivity, video chat drops). Sometimes you just have to shrug your shoulders and laugh.
• Designate one person to be the group leader for the meeting (perhaps the PI or another researcher). As with in-person meetings, virtual meetings are more efficient and effective when someone keeps the meeting on track.
• Designate someone to email everyone after the meeting with a summary and specific tasks, including due dates.
• Build in time to socialize. Expect to spend 5 to 10 minutes chatting and updating about each other’s lives. This is a surprisingly important and rich part of research collaboration!

Resources for Interactive Research CollaborationsVideoconferencing and Communication Tools:Google Hangouts·         Advantages: Allows for video calls with up to 10 people; group conversations are easy and video facilitates understanding and quality interactions; allows for screen sharing to discuss specific issues (i.e., data analysis); integrated with the Google platform, facilitating communication across email, documents, and video.·         Disadvantages: User interface is difficult to navigate and can result in delays making initial connection; requires a Google or Gmail account; requires one time download of Google voice and video free plugin.Skype·         Advantages: Allows for video calls with up to 5 people; screen sharing is free.·         Disadvantages: Requires installing a program onto a computer.GoToMeeting·      Advantages: Allows users to share screen; personal “meeting room” for users; users can connect by computer or phone; can set up a schedule of repeated meetings.·      Disadvantages: Costs $24 to $49 per month; must install a program; voice calls often have feedback.FaceTime·      Advantages: Since many people use iPhones, video calls are easy.·      Disadvantages: Inconvenient for those without an iPhone or Mac; does not yet have a group video chat option.File Sharing Tools:DropBox·      Advantages: Allows users to share documents with a group; documents retain formating (e.g., Word, Excel); backups of files are created and files can be recovered for up to 30 days.·      Disadvantages: Editing must be done one person at a time, otherwise conflicted files are created; any change by a collaborator affects everyone else who has the shared file (e.g., it could be deleted).Google Drive·      Advantages: Allows for real-time simultaneous editing; facilitates team writing and collaboration.·      Disadvantages: Formatting between Google Drive files and mainstream Microsoft documents is problematic.Box·      Advantages: Allows for real-time simultaneous editing.·      Disadvantages: Less commonly used among researchers.

Developing a Research Agenda

November 12, 2014

By Milena A. Keller-Margulis, University of HoustonDespite how much and how often the term “research agenda” is used, there seems to be no agreed-upon, or formalized definition to be found. Many different entities may have a research agenda including entire fields of study, funding agencies, and of course individuals. We propose the following definition of a research agenda; a roadmap or framework that guides inquiry. A research agenda may be both global and specific. Ideally it is used to specify gaps in knowledge in a specific area and serves to guide the direction and development of new projects and research questions. A clear research agenda serves two important purposes. First, it can help you communicate to others what you study and the area in which you have developed (or are developing) expertise.  Second, it serves to guide your decision-making about what projects or specific research questions to pursue. Ultimately, you must be interested in and excited about the topic(s) at the heart of your research agenda.Strategies for Defining Your AgendaIt is never too early or too late to begin to develop your research agenda. It should not be considered static, as the process of developing an agenda is inherently reflective and ongoing. The best way to determine the general topic of your research agenda is to pay attention to the topics that interest you the most. You might discover this through reading the literature but these ideas may also surface through your practical or field-based experiences. These opportunities help to refine what might be a more global topic area into more specific or narrow research questions that have practical significance. Lastly, do not be afraid to test drive some areas of interest by getting involved in research, seeking out opportunities to explore new areas, and talking to other people with similar interests. These activities will help you narrow your focus to the topics and questions that are the most interesting to you.How to Put Your Agenda into ActionOnce you determine your general topic(s) of interest, the actions you take are what solidify your agenda. The two key ways to accomplish this are to (1) conduct research in the areasthat are the focus of your research agenda, and (2) disseminate the products of your research agenda. Generating new research and then disseminating it through writing for publication is the most critical way to further your developing agenda, and of course, contribute to the scientific literature. This can take many forms including writing proposals for conference presentations, writing grant proposals (small or large), and writing for publication. Dissemination can also be informal and involve reaching out to others who are doing similar work in order to identify potential collaborative relationships. All of these actions serve to communicate your agenda to others while at the same time developing and further refining your ideas.  Ultimately, having a research agenda means that you not only have specific topics that you are interested in studying but that you actively engage in research to advance that literature base.Here are some general tips to consider:

• Use your Research Agenda as your Roadmap: One of the most exciting aspects of working in academia and engaging in research is the limitless topics and projects you can pursue. This is also a challenge because you have to make choices as to where you will invest your time. Evaluate each potential research opportunity for the degree to which it is consistent with your agenda.
• Develop an Infrastructure: There are some logistic or infrastructure elements that you might consider in the early stages of developing your agenda. You should consider what you need to facilitate your work. Do you need space or support in the form of research assistants? Materials? Access to certain settings or populations of interest? Build a team of students to help you advance your agenda.
• Get Connected: Another strategy to further develop your research agenda is to collaborate with other scholars who are interested in the same topics. This can be accomplished in an informal way by meeting at conferences (e.g. attend social hours or networking events) or sending a friendly email but there are also structured mentoring/networking opportunities you can utilize. For example, the Society for the Study of School Psychology (SSSP) hosts the School Psychology Research Collaboration Conference every other year. This opportunity brings together junior and senior scholars to support the development of collaborative relationships. Reaching out to other professionals who work in settings that serve populations you want to study is also a great way to develop field-based research partnerships that are mutually beneficial.

Developing and furthering a research agenda takes time and commitment but having a clear area of interest that you find exciting, ensures that it will be a very rewarding endeavor.