Diane Mizrachi, Alicia Salaz, Serap Kurbanoğlu, and Joumana Boustany

Lessons learned from the Academic Reading Format International Study

Developing and coordinating a large international study

Diane Mizrachi is social sciences librarian at the University of California-Los Angeles, email: mizrachi@library.ucla.edu, Alicia Salaz is associate dean for research and academic services at Carnegie Mellon University-Qatar, email: asalaz@cmu.edu, Serap Kurbanoğlu is professor information management at Hacettepe University, Turkey, email: kurbanogluserap@gmail.com, and Joumana Boustany is associate professor in Institute of Engineering Services at the Université Paris-Est Marne-la-Vallée in France, email: jboustany@gmail.com

© 2018 Diane Mizrachi, Alicia Salaz, Serap Kurbanŏglu, and Joumana Boustany

The Academic Reading Format International Study (ARFIS) was born at the 2014 European Conference on Information Literacy (ECIL) where Diane Mizrachi presented her study of UCLA undergraduates’ reading format preferences, either print or electronic, and invited colleagues to examine their own students’ format preferences and behaviors.1

Serap Kubanoğlu, and Joumana Boustany approached Mizrachi with the idea of performing an international comparative study with colleagues at several other institutions. We now have data from more than 18,000 students in 33 countries, and have discussed and published our findings as they evolve in several venues.

At ECIL 2017, six ARFIS researchers presented a panel discussion of our experiences conducting this large study, including some of the challenges and opportunities associated with such a large and distributed collaborative research project. Responses to our presentation were very positive, and we decided to share our insights with a broader audience of LIS colleagues who might be formulating or participating in distributed research networks.

Some members of the Academic Reading Format International Study (ARFIS) research team. Front row (left to right): Serap Kurbanoğlu (Turkey), Polona Vilar (Slovenia); back row: Vlasta Zabukovec (Slovenia), Elena Collina (Italy), Ane Landŏy (Norway), Diane Mizrachi (United States), Joumana Boustany (France), Almuth Gastinger (Norway), Ana Terra (Portugal), and Pan Yantao (China).

In this article, we discuss the process of organizing a large international comparative study, strategies used for communicating across language and cultural differences, foreseen and unforeseen challenges, lessons learned, and our goals for the near future.

11 tips for organizing a study

• Ensure the research questions are translatable across multiple cultures and languages. The first step was to revisit the original research questions used in the single-institution study out of UCLA. We wanted them to reflect our expansion of the original study’s population to include college and university students at all levels, and our emphasis on comparing format preferences and behaviors across an international sample. Our final questions became: What format, print or electronic, do university students prefer for the majority of their academic course materials? Do format preferences vary by country? How does the language of the reading impact format preferences?
• Prepare the instrument for international relevance and interest. Mizrachi’s original questionnaire needed slight revisions for clarity across an international sample population. For example, Grade Point Average (GPA) systems are not universal, so a demographic question about student GPA was dropped. Many students outside the United States are expected to read academic texts in a language that is neither the institutional language of instruction nor native to the student. We added three questions to measure whether the language of the reading impacted their format preference. A demographic question regarding gender was added, and Mizrachi’s original set of three questions seeking to quantify a “long” or “short” reading was condensed to two (defining seven pages or more as long and under seven pages as short). The final questionnaire consisted of 16 Likert-style statements on format preferences, learning engagement practices and language impact, six demographic questions, a question regarding the device(s) students use for electronic readings, and an open question for further comments. All questions included space for optional comments. We thought it important to keep the survey relatively short in order to encourage participation and completion.
• Establish the methodology using clear guidelines, timelines, and expectations. Serap Kubanoğlu and Joumana Boustany had already concluded earlier multinational comparative studies2, 3 and we decided to follow the same basic structure for ARFIS. Each researcher was responsible for the translation of the questionnaire (if relevant), distribution of the online instrument among their sample population, data collection, cleaning, and submission to the ARFIS coordinators. They could survey multiple institutions in their country, if desired.

  The Office of the Human Research Protection Program and Institutional Review Board (IRB) at UCLA reviewed and approved the international research design and instrument, but research partners were responsible for obtaining any locally required permissions. Boustany uploaded the survey onto LimeSurvey, a free and open source survey platform she had used on previous projects, which includes a translation feature. She assigned each country an account and activated and closed the survey at the researcher’s request.

  After completion of data collection the researchers were free to analyze, use, and publish their own country data. We required them to standardize the data sent to ARFIS to ensure uniformity. Only quantitative data has been amalgamated for ARFIS analysis and statistical tests.

  In our first round we gave the initial research partners a timeline of four months to gather, organize, and submit their data to the ARFIS coordinators. This enabled us to complete the analysis and prepare our results in a timely manner. However, we had to be more flexible as more team members joined us. Some requested extra time to receive their institutional permissions, and we had to consider variations among academic calendars. As a result, collection and analysis of the data took longer than planned.

• Establish coding guidelines. Three demographic questions were structured as open-ended response items and required coding standardizations to facilitate easier comparison. These were age, field of study, and whether any visual or other limitation influenced the respondents’ format preference. Limitations were coded as eye strain; wear glasses/contacts; headaches, migraines/neck aches; and other. Age was categorized by five-year periods.

  For analysis purposes we categorized respondents’ majors into the broad areas of Sciences, Social Sciences, and Arts and Humanities according to the Web of Science (WoS) discipline categories. We found this necessary because of the variances among terms for majors, which reflect institutional cultures. For example, psychology at UCLA is a life science but WoS and many other institutions consider it a social science, therefore all psychology majors were categorized as Social Science. Using WoS categories offered a clear solution to this particular dilemma.

  We also analyzed the raw, uncoded versions of these responses in our PLoS ONE paper for more granular analysis of responses by age.

• Recruit trusted research partners. We first sought research partners from among those with whom we had worked previously, scholars already familiar with the procedures and whom we knew to be reliable. This set a strong beginning, and, after sharing preliminary results, more people wished to participate. Partners had to translate the instrument, distribute it, and submit accurate data. The ARFIS survey was translated into more than 20 languages, and we did not have the means to implement external oversight on the process or accuracy of each translation. All our team members came from leading institutions in their countries with previous research experience, numerous publications, and credible scholarly reputations.

  Even still, irregularities will arise. For example, the Chinese team had mistranslated the statement, “I prefer electronic textbooks over print textbooks” to state, “I prefer print textbooks over electronic textbooks.” The discovery of this irregularity and its solution relied on open, transparent communication among team members. We inverted their answers before proceeding with final analysis.

• Establish reliable channels for communication. Google Groups provides email correspondence, dialogs and document repositories, and most researchers already had Google accounts. This worked well in general, however not all research partners could access Google in their home countries. We, therefore, sent our messages directly to them through email, which meant they could not benefit from the dynamic group dialogs.

  Another important communication challenge was time zone differences. Our team today is spread throughout the world on six different continents. Even among our four coordinators, there are time differences of up to 11 hours. Time sensitive communications and scheduling video conferences must take these differences into consideration. Depending on country, not all researchers could access or use Skype for videoconferencing, and we therefore relied mostly on Microsoft’s Zoom for this function.

• Determine when to conclude. Though researchers are still inquiring about joining ARFIS, we decided to close the international comparative study after the final country data was submitted in 2017. We felt that extending it beyond three years would increase the possibility of statistical irregularities and weaken the integrity of our findings. As our results have come in, we have also found remarkable consistency across countries in the findings, and can predict with confidence that additional data will likely be redundant at this stage.
• Decide how and where to disseminate results. Publications containing amalgamated ARFIS data list all contributing team members either as authors or acknowledged elsewhere in the manuscript. The study originator and coordinators are lead authors. Individuals may publish their own country data and results independently.

  Results from several participants were presented at ECIL 20154 and 2016,5 where the coordinators also shared the first round of amalgamated analysis.6 Taking advantage of these gatherings, team members met to discuss future goals and directions, which were then sent to the others for their feedback.

  Primary among our goals was to publicize our results through venues outside of the Library and Information Science disciplines. We felt that professionals in education, cognitive sciences, technology, policy making, administration, and others would be interested in the findings. Mizrachi and Alicia Salaz presented at the 2018 International Technology, Education and Development (INTED) conference in Spain, and we will consider other venues as relevance and budgets allow. The road to publishing in a general science journal took more time and work. After a few rejections related to scope and several revisions, our article discussing the findings from 10,293 participants in 21 countries was published by PLOS ONE in May 2018.7

• Set short-term and long-term goals as relevant. The team agreed upon a paradigm change away from the “either/or” dichotomy of “print versus electronic” to address the subtleties of when and how students use different formats depending on context, reading task, and desired outcome of a particular assignment. Salaz and Mizrachi are developing a model of reading format behaviors based on their findings and theoretical principles, such as cognitive load and the principle of least effort. We hope to conclude our comparative study this year using the quantitative data collected by all research partners. The full dataset, part of which is already openly available through PLoS ONE, can also be used for statistical testing of further questions, including preference and behavior patterns by field of study, gender, age, and visual limitations. Several individual researchers plan on repeating the study locally within the next few years in an effort to track changes over time and as technologies evolve.
• Thoughts on the General Data Protection Regulation (GDPR). In May 2018 (after the conclusion of the ARFIS data gathering process), the European Parliament and Council of the European Union implemented GDPR to ensure data protection and privacy for individuals within its jurisdictions. The general consensus among academics is that GDPR will not affect research practice much, because good academic practices already safeguard participant privacy.8

  However, the change in policy for Europe reminds us of the challenge of overlapping policy environments for institutional review and ethics in research. ARFIS did not collect any personal identifying information and thus would not have been impacted anyway, but it is worth noting that different regulatory authorities maintain different regulations for handling data. While UCLA reviewed and approved the research plan according to institutional and U.S. standards, collaborating researchers were also asked to follow local institutional and/or statutory guidelines for participation in the project. Whenever possible, using a study design that avoids the collection of personally identifiable information is advisable to protect participants and reduce compliance requirements in multiple jurisdictions.

• Some challenges, some surprises. ARFIS has no unified funding source or budget, each researcher must find his/her own funds as relevant. A central budget or grant would be useful to assist any team member with specific needs, and could be applied to sustain our website and social media presence. Volunteers have begun our Facebook site and webpage,9 but we need more dedicated time and expertise to maintain a sustainable and dynamic presence.

  We have experienced some unexpected positive outcomes, as well. The institution of one researcher established an Ethical Commission Approval process for research in the social sciences after seeing the example of IRB standards in ARFIS. Other participants have reported dialogues with their administrators about the results and how to consider the implications in formulating policies. We have enjoyed some press coverage, including a blog feature in the Huffington Post, as well as inquiries of interest from businesses, such as Google and a large European paper company.

  Among the most rewarding experiences and outcomes of the project have been the collaborative spirit and warm camaraderie among ARFIS team members. We have been enriched professionally and personally by working together and sharing our results in scholarly and informal venues, and by broadening our international network of colleagues and friends.

Notes

1. Diane Mizrachi, “Undergraduates’ academic reading format preferences and behaviors,” The Journal of Academic Librarianship 41, no. 3 (2015): 301–11.
2. Laura Saunders, Serap Kurbanoğlu, Mary Wilkins Jordan, Joumana Boustany, Brenda Chawner, Matylda Filas, Ivana Hebrang Grgic, et al., “Culture and competencies: A multi-country examination of reference service competencies,” Libri 63, no. 1 (2013): 33–46.
3. Laura Saunders, Serap Kurbanoğlu, Joumana Boustany, Guleda Dogan, Peter Becker, Eliane Blumer, Sudatta Chowdhury ,et al., “Information behaviors and information literacy skills of LIS students: an international perspective,” Journal of Education for Library and Information Science 56 (2015): S80.
4. Serap Kurbanoğlu, Joumana Boustany, Sonja Špiranec, Esther Grassian, Diane Mizrachi, and Loriene Roy, eds., Information Literacy: Moving Toward Sustainability: Third European Conference, ECIL 2015, Tallinn, Estonia, October 19–22, 2015, Revised Selected Papers, vol. 552, Springer, 2016, pp. 427–64.
5. Serap Kurbanoğlu, Joumana Boustany, Sonja Špiranec, Esther Grassian, Diane Mizrachi, Loriene Roy, and Tolga Çakmak, eds., Information Literacy: Key to an Inclusive Society: 4th European Conference, Ecil 2016, Prague, Czech Republic, October 10-13, 2016, Revised Selected Papers, vol. 676, Springer, 2017, pp. 215–64.
6. Diane Mizrachi, Joumana Boustany, Serap Kurbanoğlu, Güleda Doğan, Tania Todorova, and Polona Vilar, “The Academic Reading Format International Study (ARFIS): Investigating Students Around the World,” in European Conference on Information Literacy, pp. 215-227, Springer, Cham, 2016.
7. Diane Mizrachi, Alicia M. Salaz, Serap Kurbanoğlu, Joumana Boustany, and ARFIS Research Group, “Academic reading format preferences and behaviors among university students worldwide: A comparative survey analysis,” PloS ONE 13, no. 5 (2018): e0197444.
8. See https://www.insight.mrc.ac.uk/2018/04/16/gdpr-research-changes/.
9. See http://arfis.co.