Criminal justice research may generate special ethical questions including the potential for

Forensic Sci Int. 2020; 2: 60–71.

Abstract

Much good quality research by pre-doctoral students and case-work focused practitioners remains unpublished. However, their findings could contribute to the evidence base underpinning science and practice within international justice system contexts. There are two main challenges to making findings accessible: reaching all criminal justice stakeholders, and encouraging collaborative efforts in research addressing ‘real world’ problems.

This article presents the rationale for a new, open access repository. The aim is to share good quality pre-doctoral and practitioner criminal justice research across traditional disciplinary and international borders. Such a repository should be easy to use, well maintained and sustainable. Its reach, value and impact also need to be measurable. We present the major considerations relating to the operation and workflow of such a repository, and outline the potential value, benefits and limitations. Our research suggests that the proposed repository could foster interdisciplinary and collaborative work to benefit global justice systems and societies.

Keywords: Open access, Repository, Justice, Practitioner research, Student research

1. Introduction

Knowledge generation, knowledge exchange and research provision are vital roles of universities across the globe. Students undertaking undergraduate or Master’s degrees in Higher Education Institutions (HEIs) generally conduct a research project relevant to their course title. This usually leads to a draft manuscript (article), thesis or dissertation. Some of this research is published in professional magazines or peer-reviewed journals, presented at conferences and/or provided as reports to practitioners through academia-industry partnerships. However, such outputs represent a relatively small percentage of the total number of student research projects conducted each year [1]. As a result, useful and potentially valuable forensic, and other justice-related, research of interest to researchers and practitioners remains inaccessible [2]. Equally, there is evidence that research carried out by practitioners often goes unpublished and is therefore inaccessible to prosecution and defence [3].

This study was a small-scale piece of commissioned mixed-methods research conducted in the UK. It is a good example of the kind of research that often goes unpublished. Primary data was collected through an online questionnaire and discussions at a stakeholder workshop. Most of the participants were from the UK, with some international representation. This data was analysed in the light of a global review of literature on information sharing in general and repositories in particular. This review encompassed work from Canada, the US, Central America, Europe, the Middle and Far East, Asia, Australasia, and Antarctica. The aim was to establish whether or not an open access criminal justice repository would improve sharing of knowledge and access to research across disciplinary, sector, and international boundaries.

Internationally, thousands of undergraduate and Master’s level research theses in criminal justice topics are undertaken each year in HEIs. Some of these may hold significant value to the professional community, but that may only be realised with systematic collation, assessment and dissemination. This would enable justice professionals and others to establish cross-sector collaborations, generate standardised methods with increased sample sizes, and identify real-world solutions. With increasing emphasis on open access (OA) [4], dissemination should extend to a diverse range of stakeholders [5]. Creating an innovative repository of student and practitioner research offers the potential for a central resource for all professionals working within the justice system. Furthermore, it would offer a new teaching, learning and research tool for academics and researchers in HEIs. The purpose of this article is therefore to assess the rationale for establishing an international repository for justice system-related research with twin aims. First, to enable access to high-quality student and practitioner research findings for all sectors of the justice system. Second, to foster dialogue and productive collaborations between HEIs, industry professionals and their institutions, potentially across the globe.

Some undergraduate, Master’s level and practitioner research will be of lesser quality and/or lack significant new knowledge, and so be unworthy of formal peer-reviewed publication. However, this is not often the case, even for undergraduate researchers. There is a small body of recent published literature exploring factors that influence publication of undergraduate and Master’s level health research. In New Zealand, Ref. [6] assessed 153 undergraduate medical theses published between 1995 and 2014 at the University of Otago. They found that 50 of these led to a publication in an indexed peer-reviewed journal, and in almost all cases the student was first or second author. The key factor they identified as influencing publication was support from supervisors. Ref. [7] studied 162 successful public health Master’s students at two Spanish universities between 2006 and 2010. They identified four factors, one or more of which made publication more likely. These factors were: women students; a first degree in a non-health science subject; completed Master’s research on time; aiming for doctoral study. They also found that the strongest predictor of publication was academic performance. This seems in line with the findings of Ref. [6]; as it is unlikely that supervisors would support publication of students’ work unless it is good quality. While these are only two studies, they offer valuable insights worthy of consideration.

Across international justice systems, many practitioners and professionals cannot access subscription-based journals. In addition, those who wish to publish in OA peer-reviewed journals may not be able to pay for article processing charges. Unfortunately, such barriers are limiting access to novel methods, emerging technological solutions and/or recommendations that propose change within justice system practices. Overcoming such barriers could lead to new ways of working, increasing interdisciplinary solutions, and creating more effective mechanisms to ensure justice is delivered to society.

There is an increasing need for forensic science practitioners and expert witnesses to demonstrate underpinning theories and communicate their expertise to other professionals [[8], [9], [10], [11], [12]]. This is due to the increasing pressures and scrutiny from the criminal justice sector and wider society in the UK and beyond [[11], [13], [14], [15], [16], [17]]. However, with increasing administrative demands, and reducing financial and human resources, there is now even less time for practitioners to undertake research. As a result, there has been a call to develop new collaborations and partnerships between academia and industry [18]. Even though these partnerships are not always formally coordinated, they serve to increase research activity that addresses ‘real-world’ forensic science problems [19]. They also enable research dissemination across both practitioner and academic domains. Therefore, we argue that the strength of these collaborations could be enhanced through the provision of a relevant research repository.

We have also found relevant work at international level. For example, the Organisation for Economic Co-operation and Development (OECD) have developed a set of principles to access data from publicly funded research. This involved consensual work across its 30 member countries. Similarly, the World Data System (WDS), an Interdisciplinary Body of the International Council for Science, established a partnership working group. The remit of this group was to create a Catalogue of Common Requirements for a repository. The OECD principles and the WDS requirements both include three key points [20,21]:

• 1.

  The need for research and data to be both accessible and secure in a sustainable online environment;

• 2.

  The importance of appropriate licensing systems that function within national and international intellectual property regulations;

• 3.

  The need for a method of assuring the quality of research and data stored.

The OECD does not mention repository workflow, but it is one of the later WDS requirements [20]. Conversely, the WDS does not mention the need to promote or market a repository. However, the OECD discusses the need to make stored data visible and to disseminate information about where to find such data [21].

International justice systems span multiple disciplines and jurisdictions. There is an urgent need to establish interdisciplinary approaches that incorporate all relevant sectors, including law enforcement, security agencies, legal professionals and policy-makers. It is imperative that research is accessible to all international professionals who will directly influence policy, improve practice and deliver justice in society. This will help to proactively and effectively reduce national and transnational crime, and ultimately create safer communities.

A recent UK investigation into forensic science and the criminal justice system [14] identified that “Research and development in forensic science is under-resourced and uncoordinated” (p4). The investigation also found significant barriers to research activities in both the HEI and professional environments, and a lack of co-ordinated strategic thinking about research. US forensic science is coming under similar criticism [16]. These challenges suggest that, there is clear value in a repository of research findings which can be made available worldwide. This article demonstrates the rationale for a repository and outlines the potential benefits and limitations of such a resource within the justice community.

2. Methods

This mixed-methods research was overseen by a steering group with five core and two support members. A questionnaire was designed to identify and establish the requirements of the criminal justice community. A day-long stakeholder workshop was then held to present the questionnaire findings and discuss a way forward. Discussions at the workshop were recorded using a combination of touch-typing and handwritten notes. Each dataset was analysed and the analyses were synthesised with each other and with the literature review outlined earlier in this article.

An online questionnaire was created (see Supplementary Material). The questionnaire included demographic questions about gender, location, primary role, and whether the respondent taught forensic science within a HEI. Substantive questions covered:

• •

  the feasibility of the proposed repository,

• •

  how the quality of contributions should be assessed,

• •

  issues of licensing and embargo periods,

• •

  the levels of access to the repository for information-seekers,

• •

  search terms, and

• •

  the likely uses of the repository.

A final unstructured question was also included to elicit any remaining views. Some questions were included for the purposes of comparison with an earlier piece of market research conducted by the Jorum service (Siobhan Burke, personal communication, 2017).

Once the questions and their order were finalised, the questionnaire was formulated using SurveyMonkey. It was decided not to make any answers mandatory to elicit the greatest number of completed questionnaires. Formal ethical approval was not required to conduct the questionnaire as it was part of a commissioned research project investigating a professional topic, so professional ethics were adhered to [22]. No personal or sensitive data was required, and all responses were anonymous.

The questionnaire was publicised using existing channels of communication primarily within the UK forensic science and justice communities, personal emails from steering group members to relevant contacts, and internationally through social media. Participants were recruited from across international forensic science and policing domains using a convenience sampling approach.

A stakeholder workshop was held to gain insights into key themes identified from the questionnaire and discuss the possibility of a student research repository. The workshop brought together 18 experts from academia, the criminal justice system, and the private sector, as well as staff from Jisc. The workshop focused on six key questions related to licensing, assessment of contributions, security of the repository, embargo criteria, registration level, and repository workflow. These questions were identified from queries and gaps arising from questionnaire responses, and from discussions with Jisc to gather information about designing such a repository. Due to the initial scope of the repository, the questionnaire and workshop focused on undergraduate and Master’s theses and dissertations. However, the remit of this repository has since expanded through discussions with members of the UK’s Knowledge Transfer Network (KTN) Forensic Science Special Interest Group. The project was named Research4Justice. This expansion of remit is further discussed in section 4.

3. Results

3.1. Questionnaire findings

There were 100 questionnaire respondents (52 female, 47 male, 1 other). Table 1 shows the geographic locations of the respondents.

Table 1

Geographic locations of questionnaire respondents.

As shown in Table 1, 76 respondents were located in the UK, and 24 in the rest of the world. Table 2 gives the overview of respondents’ professional status.

Table 2

Professional status of questionnaire respondents – overview.

Of the employed academics, there were 17 lecturers, 9 senior lecturers, 2 readers, 2 associate professors, 2 professors, and 1 post-doc/early career researcher (ECR). Of the students, there were 4 doctoral candidates, 4 at Master’s level, and 3 undergraduates. Of the 14 in other categories provided in the questionnaire, there were:

• •

  crime scene investigator (3)

• •

  crime scene manager (3)

• •

  fingerprint examiner (2)

• •

  Government department researcher (2)

• •

  colleague of users of forensic science information (2)

• •

  user of forensic science information (1)

• •

  honorary/ERASMUS professor (1)

No respondents identified themselves as student supervisors, retired academics, police investigators (CID) or colleagues of forensic scientists, which were also categories provided in the questionnaire.

Seventeen respondents gave the answer ‘other – please specify’ and described themselves as:

• •

  Anatomical Pathologist

• •

  Associate Dean

• •

  Director of Forensic Services

• •

  DNA Reporting Officer

• •

  Forensic Document Examiner

• •

  Forensic Imaging Analyst and Forensic Software Provider

• •

  Forensic Pathologist

• •

  Forensics Supervisor

• •

  Knowledge Transfer Manager

• •

  Lab Manager - Forensic Teaching Lab

• •

  Police Leader

• •

  Retired

• •

  Retired Forensic Practitioner

• •

  Senior Forensic Science Technician (University)

• •

  Senior Police Officer

• •

  Technician (2)

Nearly half (n = 46) of the 100 respondents said they studied or taught on an accredited forensic course in the UK, and 41 said they did not. A further two said they did not know, and nine said the question was not applicable.

Because the sample was small, not all respondents answered all questions, and some of the data was qualitative, statistical analysis was restricted to descriptive statistics.

In preparation for the Jorum-based precursor to Research4Justice, an earlier market survey was conducted in 2013 with 48 respondents. Of these, 76% said they would value a repository (n = 34; question answered by 47). In the 2017 questionnaire, over 90% of respondents (n = 82; question answered by 90) said they would value a repository. This suggests that the perceived value of this kind of repository has risen in recent years. In 2017, 61 respondents gave reasons why they would value such a repository. Qualitative content analysis showed that the main reasons given were that a repository would:

• •

  Enable knowledge-sharing and staying up to date (n = 16);

• •

  Increase accessibility of useful and current research (n = 12);

• •

  Be valuable for students (n = 10);

• •

  Facilitate collaboration (n = 8);

• •

  Help to get research into practice and/or policy, thereby leading to operational improvements (n = 8);

• •

  Avoid duplication (n = 7);

• •

  Raise standards (n = 6).

In the current questionnaire, 74 people answered question 14 about what they would use a repository to look for. Most respondents said they would use a repository to:

• •

  Find out whether a topic has already been researched (87%; n = 64);

• •

  Look for potential collaborators (66%; n = 49);

• •

  Support current casework (53%; n = 39);

• •

  Find information for funding applications (32%; n = 24);

• •

  Support defence work (28%; n = 21).

82 people gave their views about ways to assess the quality of deposits. A majority (63%; n = 54) favoured independent assessment of the quality of deposits into the repository. Respondents’ views about how this could be done are shown in Fig. 1 with ‘other’ (n = 6) assessment mechanisms specifying peer-reviewed publication (n = 3). However, this may suggest that respondents have not considered that such a repository would aim to act as a continuous e-publication in its own right. It would not have the traditional format (volume and issue numbers) of a journal or the costs of OA journal publication (discussed in section 4).

Questionnaire findings on assessment of quality of deposits (NB: the order of the legend is in descending order of % response).

75 respondents expressed their views about the best Creative Commons licence to use. The most popular options for type of Creative Commons licence included both the least and the most restrictive licences, as well as the third most restrictive. The majority of respondents opted for a flexible embargo under certain conditions, which should include issues around commercial and/or professional sensitivities. (No statistical information is provided here as this finding is based on an aggregate of more than one set of answers including qualitative information.) The range of responses regarding licensing and embargo conditions indicated that these requirements warranted further discussion in the workshop. The aim was to establish an appropriate compromise for all potential end users (section 3.2).

74 people shared their opinion about the best kind of access to a repository. The majority of respondents (80%; n = 59) wanted either one or two tiers of registration for people to be able to access the repository. Only 15% favoured complete OA, as shown in Fig. 2.

Questionnaire findings on levels of access (NB: the order of the legend is in descending order of % response).

As registration is important for repository workflow and security, it was identified as another topic for further discussion at the workshop.

The final unstructured question, i.e. “Is there anything else we need to consider about this repository that we haven’t already mentioned?“, yielded only 17 responses. Of these, three said ‘no’ and 13 made a variety of comments ranging in length from a single word to a detailed paragraph. These included suggestions about security and metrics, and questions about publication issues and document formats. There were no discernible themes or categories that could be created within this very small dataset.

Six key questions that either remained unanswered, or were deemed more appropriate for discussion at the workshop rather than inclusion in the questionnaire, were:

• 1.

  Which Creative Commons license should we use?

• 2.

  How can we assure security for contributed material and for personal information collected from repository users?

• 3.

  At what level should people be required to register to use the repository?

• 4.

  Who should assess the quality of material intended for the repository?

• 5.

  What should be the conditions for embargoes on material contributed to the repository?

• 6.

  How should the repository be operated and managed?

The stakeholder workshop was designed to help answer these questions and identify workable solutions from the perspective of potential repository users. Section 3.2 summarises the main considerations and outcomes of these discussions.

3.2. Workshop findings

3.2.1. Licensing

There are six Creative Commons licenses with different levels of flexibility. It was decided that Creative Commons 4.0 International (CC4.0) would be the most appropriate license as CC4.0 permits commercial and non-commercial use. The work may be adapted, as long as any changes are noted, and the author must be credited. CC4.0 was thought to offer the best balance of permissiveness, to ensure maximum usability of content, with protection for items in the proposed repository. Concerns were raised about the possibility of plagiarism. However, the routine use of plagiarism detection software, such as Turnitin, in HEIs, should help to guard against that.

3.2.2. Assessment

As shown in Fig. 1, the majority (60.3%) of online questionnaire responses suggested an independent person should be involved in the assessment of student research output quality before submission/deposition. Yet 48.2% of responses indicated that a signed declaration, stating that the output had met an agreed set of submission criteria, would be sufficient. During workshop discussions it was agreed that HEIs should be responsible for the assessment of potential research contributions due to their quality management processes [1]. These processes are the combination of internal and external assessment marking policies typically adopted when assessing student research. It was also suggested that a RAG (red, amber, green) scale/rating, including Black, could support repository users in assessing research quality, such that:

• •

  Black: thesis and/or data not fit for purpose

• •

  Red: overall thesis quality not great, but data/content may be of some use

• •

  Amber: thesis and/or data of average/reasonable quality

• •

  Green: good quality thesis or innovative project with good quality data and interpretation

A request was also made for the repository to automatically watermark documents with their RAG status, on every page, to make plagiarism more difficult. This may be beneficial and achievable in the longer term, but would not be part of the initial framework for the repository. Section 4 discusses assessment proposals in more depth.

3.2.3. Security

One of the 13 open responses to the unstructured questionnaire question raised some valid concerns about the security of the repository contents. As a result, the option of a secure area within the repository for highly confidential material was considered during the workshop. Following discussion, the decision was reached that such an area is likely to create more problems than it solves. For example, it would require a much more complex tiered registration process that may inhibit repository contributions and/or registration of users. If the responsibility of content uploaded to the repository lies with the contributor(s) then the material uploaded is at the discretion of the contributor.

3.2.4. Embargo criteria

It was concluded that self-regulation of embargo periods was the best approach. Any other options, such as external regulation, were too resource-intensive to be viable. There may be various reasons why a contributor might want to embargo their material, such as a plan to develop their research during postgraduate study. A user would be able to specify an end date for the embargo, and could automatically receive an email giving them the option to extend. However, it was considered that the title, keywords, and abstract of any embargoed research should be visible to avoid duplication and enable future collaboration.

3.2.5. Registration level

It was decided that potential contributors would need to register, before they could make their first contribution, by completing a profile page. This enables the collection of data and metrics about users, which assists understanding of where and how information from the repository is used. Title, abstract, author name(s) and institution/organisation should be openly accessible to all information-seekers. Login should require a username and password for all users wanting to access further details regarding a repository submission, including contact details for corresponding author(s). Ideally, a common and existing authentication service should be used to facilitate international login for all users.

3.2.6. Workflow

It was decided that students should prepare their contribution as per their assessment guidelines. Then supervisors should ‘rate’ (discussed in section 3.2.2) and upload it to ensure that the research was relevant to the repository and suitable for OA. The intellectual property (IP) of students’ work may be owned by the university (as encouraged by the UK Government and the European Commission [23]), or it may be owned by the student (See section 4.2 for a fuller discussion of this topic).

Repositories also hold ‘metadata’, or data about contributions, such as the title, author(s) or creator(s), and date of a piece of research or a dataset [24,25]. Metadata quality standards have been established and should be used by repositories [25,26] to ensure that contributions are easier to find [24]. Workshop attendees stated that metadata should be specified by the contributor. It should include, for example, name of author(s) and supervisor, title of research or dataset, date contributed, and details of grants/funders. Persistent digital identifiers such as digital object identifiers (DOIs) were considered important. These could be used for contributions of research, data, and any other relevant materials such as ethics applications. Jisc noted that the repository would create a DOI for each repository upload. Also, the value of persistent personal identifiers for contributors was identified, to help keep track of people moving between institutions. The best-known example is probably the Open Researcher and Contributor ID (ORCiD).

3.2.7. Additional considerations

Further decisions were taken at the workshop regarding specific aspects of workflow, site configuration and management, feedback and reporting, governance, and marketing. It was agreed that the repository should be open to international contributors. It was also agreed that the website should have the capability for conversion to other languages – although again this would be for longer term development. Usage data such as views, downloads and geolocation should be reported so users can measure and evidence the extent of research dissemination. Additionally, repository founders would be able to use these metrics to investigate the use of its contents. Reporting on this would provide evidence to secure future resources and identify areas for repository development. Reporting can be done automatically, generating infographics for example, which could be used to market and further publicise the repository. Workshop attendees requested that registered users should be able to add comments to a specific deposit to provide feedback and facilitate discussion. Such a facility would also enable users to comment on how the research contribution had been used in practice. This would help to demonstrate research impact and provide UK author(s) with valuable insight and evidence to support Research Excellence Framework (REF) submissions [27]. This will be possible, although not in the initial version of the repository. Considerable interest in supporting the development of the repository was expressed by influential partners. These included the Government-funded KTN, The Chartered Society of Forensic Sciences, the Forensic Science Regulator [28], FIT-IN, What Works representatives from the College of Policing (CoP) and an employee from the National Crime Agency.

Jisc initially provided funding for the development and management of the proposed Research4Justice repository [29] due to their involvement as an Open Research Hub pilot [30]. However, the justice community will need to provide Jisc with fees to continue provision, use and continued development of their services.

4. Discussion

At present, the dissemination of student research beyond the host HEI varies with respect to qualification level, mechanism, frequency, subject specialism and geographic location. Doctoral research is more frequently disseminated through poster and oral presentation at professional conferences and subjected to more rigorous scientific peer-review through written publication. Master’s and undergraduate research outputs are less frequently disseminated through these mechanisms. More often, such research is used to support further research within a HEI and/or is disseminated to relatively small, targeted audiences, such as collaborating practitioner-based organisations. Comparatively small numbers of undergraduate studies are written up for professional magazine publication, such as in CSEye or Digital Forensic Magazine. These articles may or may not go through a peer-review process, and publication success is typically due to supervisor support [6,7].

In addition, there are no publications we are aware of that quantify the proportions of published practitioner research. Some practitioner research is published by certain individuals or organisations and/or in specific subject areas. However, anecdotal evidence indicates that most casework-related, practitioner-led research is not written up, and is not widely disseminated beyond the practitioner’s discipline. As a result, the feasibility and requirements-gathering phases of this project (outlined in sections 2, 3) identified both a mandate and set of requirements to support the creation of an OA justice system repository. This section discusses the findings from the primary research (section 3) in more depth. We identify some of the potential benefits and caveats of such a repository. This is considered within the global context of existing OA repositories and the requirements of intended repository users, including key justice system stakeholders. We begin by reviewing existing repositories and preprint services and how they are used. We look at how the proposed repository could operate, with particular emphasis on access, content, intellectual property, and quality assessment. We consider repository management, focusing on governance, maintenance and sustainability, and we take a look forward at potential future developments.

4.1. Existing repositories and preprint services

There is a drive and need to make research more openly accessible and transparent [4,14]. Published research outputs can be sourced through self-archiving platforms (green open access). These may include individual organisations’ websites, institutional repositories, and generic research-sharing platforms including Zenodo and ResearchGate. Then there are the more traditional publishing (gold open access) routes using peer-reviewed subject-specific conference proceedings and journals. Since the early 1990s, subject-specific sharing platforms known as ‘preprint servers’ have been developed worldwide to hold pre-review versions of articles [31]. These servers are now available across a range of traditional subject areas including physics, mathematics and computer science (arXiv), biology (BioRxiv) and chemistry (ChemRxiv). Ref. [32] identify 10 rules to support submission into preprint servers and review the benefits and limitations to both researchers and the public. They highlight that early dissemination of research through a preprint service can improve the timeliness of information exchange. It can also improve our ability to assign order of priority to research findings. Additionally, Bourne et al. advocate that access to informal non-peer-reviewed content can help to aid discussion and interpretation of research data. These services are therefore seen to complement rather than replace traditional publishing routes.

Some repositories focus on disseminating country/continent-specific research data including the Australian National Data Service and AfricArXiv (Africa). There are already subject-specific databases in policing that share ongoing and/or completed research taking place between academia-practitioner interfaces, including the UK CoP’s Research Map and the Global Policing Database. The former details and geographically plots police and crime scene-focused postgraduate research (Master’s level and above) in the UK. The latter contains published and unpublished evaluations of policing interventions around the world through a UK-Australia partnership. The KTN Innovation Database also highlights challenges in forensic science practice aiming to instigate new research in the sector. However, subject categorisation and segregation does not provide an easy solution. In particular, it is not helpful in sharing or searching for research that crosses traditional subject disciplines, has an interdisciplinary design, and/or covers international jurisdictions. A simple search of OpenDOAR (Directory of Open Access Repositories) in 2017 identified two policing databases, but none under the search term ‘forensic’. In 2019, a repeated search of OpenDOAR using variations on the terms ‘police’, ‘forensic’ and ‘criminology’ came up with zero items. This suggests such topics may be encapsulated amongst the eight ‘social science’ and/or 95 ‘science’ repositories. ‘Law’ however, did result in 20 specific repositories, suggesting this is a more focused and defined subject area. Differences in search results for policing-focused databases over time demonstrate that changes to categories and/or search/discovery algorithms can also result in missed opportunities. Therefore, any new repository will need to have well considered categories, provide continued maintenance and be prioritised by repository hosts to facilitate usage (see section 4.3 below for more on maintenance).

National and international justice systems make up a complex multidisciplinary system. The diversification of repositories increases this complexity. Likely results include increased chances of missing research affecting continual monitoring of emerging practices, conduct of systematic reviews and future research planning. Bringing justice-related research content together could potentially provide a more user-friendly, straightforward and flexible solution for both contributors and information-seekers [26,[33], [34], [35]]. It is this perceived benefit that suggests the need for a new central, OA platform. People from all sectors of the justice system could use this platform to gain access to resources from across relevant disciplines. To increase awareness and access to resources across traditional subject boundaries, an online toolkit of external OA resources has been launched on the Research4Justice platform (www.research4justice.ac.uk). One option is also to launch the new research repository here to deposit and share previously inaccessible research outputs from HEI-based students and practitioners. This new repository would have the potential to act as an interdisciplinary, justice-system-focused preprint server. In time, it could significantly expand its initial remit beyond forensic science-based theses and dissertations. Early posting of research findings would also create new mechanisms for ECRs to gain feedback on their work before submitting a manuscript. Further, it would provide opportunities for practitioners and researchers to contribute to research through informal or formal collaboration.

The implications for creating such a preprint server/research repository from the perspectives of academics, funders and journals have been reviewed by Ref. [31]. Berg et al. focused on life sciences around the world. In a justice system context, similar benefits, challenges and recommendations apply with preprint servers. They provide a complementary mechanism for research dissemination to formal peer-reviewed publication as they serve different purposes. At present, most journal publishers, including Elsevier and Wiley, do not consider an academic thesis/dissertation [36,37] or material in a preprint server [38] as previously published. This is regardless of the type of peer-review process conducted (single, double or triple blind). Therefore, disseminating research outputs within a preprint, subject-based repository like Research4Justice should not have a negative impact on publication potential. However, a minority of journals and publishers, like Springer, may consider electronic dissemination of such research outputs as previously published material. This means they will not accept such articles for journal publication. As a result, it is vital that authors review publishers’ policies before submitting research outputs to any OA repository. It will therefore be the depositing author’s decision as to how and whether they wish to use this new platform. Further discussion of repository content and assessment is addressed in section 4.2.

Contributing to a repository is a form of OA publishing (section 4.1) that can be particularly useful in filling gaps left by the constraints of traditional publication. For example, the formal publication system is heavily skewed towards positive findings, while repositories will hold research with null results, which can be scientifically useful [39]. In addition, some journals are not designed to publish interdisciplinary research, whereas a repository can cross topic and disciplinary boundaries. Although small-scale student research is not usually regarded as worthy of formal publication, repositories can also support the sharing of this emerging and contemporary research. This can represent as much as 10% of the national scientific output [40]. Much potentially useful research and data is never made available through repositories, and so cannot be used by professionals or the public [41]. Some influential bodies, such as HEIs and research funders, are working to counteract this by making it mandatory for grant recipients to contribute their work to a repository [5]. Eighty-three per cent of research funders either require or encourage OA archiving of research outputs in such repositories [42]. However, mandates are not necessarily complied with [26,43]. Also, the requirements for publishing OA articles and archiving their research data are generally less stringent [42].

There are several types of repository [33], including:

• 1.

  Institutional repositories holding information generated by a single institution, such as a university;

• 2.

  National repositories, holding information of relevance and interest at national level;

• 3.

  Subject-based repositories holding information for a specific community.

There is an ongoing debate about the relationship between institutional and subject-based repositories [44]. Information held in an institutional repository can be difficult to find and access for people outside that institution [33,34], whereas this is easier when using subject-based repositories [45]. To maximise visibility for their work, researchers may be best advised to contribute data or metadata to both their institutional and a relevant subject-based repository (ibid). However, there are barriers to contribution, which can be an onerous process, or can seem onerous when time is at a premium [33,34]. Additionally, there are many other reasons why some researchers are unwilling to share data [46]. In part, this may be because there is currently no recognised system for students or academics who create data to gain credit for sharing that data (ibid). However, community-based approaches to sharing information and resources are increasing [47]. European surveys conducted in 2009–10 and 2013-14 show that individuals in Europe seem to be both more willing to share data and actually sharing more data [48].

Rates of contribution to subject-based repositories have been found to be significantly higher than to institutional repositories [44]. A survey of over 3000 respondents in Europe found 46% would prefer to use a subject-based repository, compared to 22% who preferred an institutional repository (ibid). However, only 37% knew of a subject-based repository that they could use (ibid). As we have seen, in the case of the justice system, there is no such repository. Contributors are more likely to use subject-based repositories because the value of doing so is more evident [26,33]. This value includes: making ideas and results accessible; providing greater opportunities for collaborative working; offering scope for thrifty reuse of data; and delivering better education [21,35,46]. Furthermore, subject-based repositories make material accessible swiftly and worldwide [31]. Contributors are also more likely to use repositories if the process of contributing material is easy, and they trust that the repository will be sustained [26]. This further emphasises the need for the material in the repository to be preserved with technological advances and/or replacement.

In summary, the potential remit and scope of the proposed OA repository has broadened significantly. It is now possible to envisage the repository as a preprint server. Submission criteria could be extended beyond higher education students to include practitioner research and indeed any previously unpublished research relevant to a justice system. This provision would improve capacity for research dissemination where authors do not have the resources to make their research openly accessible. It could also facilitate knowledge sharing where valuable research may not meet the scope or prerequisites required for successful journal publication in highly competitive climates. Material that could be deposited into the repository includes (industrial) placement reports, posters, electronic presentations, factual documentaries, dissertations/theses, data, pilot studies and written discussions. The repository is designed to complement, not to replace, existing peer-reviewed journal publication. There remains a clear need for peer-reviewed processes to validate research findings [31].

4.2. Repository operation

Repository users include those who contribute data and those who search for information, such as students and researchers [35]. Although the research outlined in sections 2, 3 above focussed predominantly on justice communities, the range of potential users should have been much broader. Key information-seekers who will use a repository of research relevant to the justice system include a range of professionals. In particular, it is likely to be used by forensic scientists, law enforcement, solicitors, barristers, and judges [49]. It could also be used by health professionals, criminologists, sociologists and policymakers. Such repositories can also be an open educational resource for academia, providing access to research and data in a way that supports teaching and learning [47,50].

Any repository established would need users to be categorised according to access level requirements. A repository user would not have to login to access materials, as most materials would be OA. However, to deposit material a login would be required to create, modify and remove metadata and material for which the depositor is responsible. Based on the broader scope of the proposed repository, depositors could be academics/research supervisors, postgraduate researchers, practitioners, HEI/organisational administrators, or others. At least in the early phases of repository release, some of the authors of this article (i.e. the repository founders), would take a ‘superuser’ role. This would involve being named as approvers within the repository’s access hierarchy. Approvers would have access to metadata and content stored in all depositors’ accounts and could see both hidden and OA materials across organisations. The Code of Conduct for an approver would be to act as a repository gatekeeper. They would receive, review, accept or reject (with reasoning) any submissions into the repository to ensure relevance of the topic and compliance with deposition criteria. Approvers themselves would not edit/modify other users’ materials, deposits or associated metadata. However, they would be able to hide/unhide content on request and engage with the depositor to resolve any problems identified by the user community.

Since the workshop, it has been identified as beneficial for repository users (including depositors and approvers) to use a free, existing unique personal identifier provision. A common example is ORCiD. This would enable login to the service and increase the ability to cross-link and share metadata between online materials/records stored in existing external repositories. The repository service should also have three core components: storage, preservation, and reporting. The repository would enable material to be stored. That material should remain accessible regardless of future software developments through preservation. Users should have the capacity to report on the growth, usage, performance, value and future sustainability of the repository within the community over time.

It is important for repository users to value the content of, and continue to use, that repository. For this to happen, the authors, research contributors and depositors of uploaded material need to stand behind the data and interpretations they have made [32]. During the initial questionnaire and subsequent workshop, the user group discussed the assessment and rating of quality for uploaded material (section 3.2.2). In the questionnaire, most respondents wanted to see an independent assessment of the material prior to upload (section 3.1). During the workshop, the concept of ‘independent assessment’ was discussed in detail. Understandably, those working outside academia were not fully aware that research outputs are usually independently assessed as part of standard HEI quality assurance procedures. It was decided that research supervisors rather than the students themselves should be responsible for uploading research outputs. This would act as an appropriate review mechanism to ensure:

• •

  Any metadata and material uploaded into the repository was relevant and high quality;

• •

  All research partners were acknowledged and/or consented to be named on the submission;

• •

  The conducted research complied with the HEI ethical review procedures, copyright and IP rights;

• •

  The material did not contain sensitive or personally identifiable information that should not be openly shared.

Workshop attendees agreed that HEIs alone could be responsible for assessing the potential contribution of student research and/or data into the repository. Where necessary, it was suggested that the RAG scale (section 3.2.2) could be used to support deposition by a non-expert. The HEI could then stipulate their own criteria to enable administrative depositors to assess whether only limited metadata or the full research output is uploaded. Such a workflow would also support the sustainability of the project, in terms of time, human and financial resourcing (section 4.3). During discussions it was also suggested that the RAG scale would help those working outside of academia to assess the quality of research. Whilst we agree that could be valuable, it may also be misleading. Further, it could adversely affect (accidentally or otherwise) individual or institutional reputations due to the range of possible causes for assigning an unfavourable rating. As a result, we would not suggest assigning a RAG scale early in the repository’s life but would consider its later inclusion.

Before we discuss IP and the other issues outlined above, let us consider how a repository may document adherence to ethical guidelines. Although this topic was outside the scope of our original workshop, it is an important consideration. There is an increasing need to prove that research has been conducted in line with the country’s and/or organisation’s ethical requirements. Therefore we propose that any documentation evidencing the research’s ethical considerations, including information briefing(s) and blank consent forms, are submitted as hidden, supplementary material. This would support document preservation and potential for future ethical audit [51]. It appears that such protocols have not been considered in previously published literature. However, they may become an important feature within OA repositories in the future.

Understanding the IP rights in the context of student research can be quite complex. It is not as simple as whether the HEI or student owns the IP, especially when industry partners are involved with the project. A full discussion of the issue is beyond the scope of this article. However, we can say that appropriating IP rights between all research partners should always be agreed [52] in writing, before any research begins [21]. This negotiation is challenging as research is commonly conceived between student researcher and their supervisor, especially at undergraduate level. Sometimes the supervisor provides more intellectual input in designing the research question, establishing the research method, and aiding the interpretation of data. Other times, the student may be more independent and therefore feel they hold a more substantial proportion of IP regardless of the HEI policy. Such open discussion and initial written agreement will therefore help to ensure that all parties know where they stand [47]. Each party should provide written consent for the upload of any research outputs into an OA repository. Such consent is especially important where a HEI IP policy indicates that the student retains IP rights for the research. Resolving these issues is crucial for the effective and ethical sharing of research material [25].

A mechanism is also required for reporting and taking down material that may have accidentally or deliberately breached licencing, ethical guidelines, copyright, IP rights etc. Such notices can be reported by any user, provided a detailed justification for the takedown is submitted for the repository superuser to liaise directly with the author/depositor concerned. Material can be either temporarily or permanently removed from public view [53]. A supervisor, researcher and/or industry partner may be concerned about a range of potential problems. For example: the standard of writing, data accuracy, interpretation quality, commercial sensitivity, copyright, plagiarism, IP within the output, or adverse effect on future publication. In such cases, the depositor may choose to only list the metadata associated with the research [40]. Inclusion of such metadata could include research authors and contributors, organisations, title, keywords and abstract summarising the data. This would enable other researchers and practitioners to assess the type and extent of research being done, even without access to the research itself. It would also offer a mechanism for instigating conversation and research collaboration. And it would link to authors’ ‘related works’ within this and other repositories through minted DOIs, which is a novel provision for end users [24].

4.3. Governance, maintenance and sustainability

Workshop participants identified that repository governance needs to be established, including lines of accountability, marketing, and providing an Advisory Board. The need for repository governance is supported by Ref. [32] in the field of biomedical sciences. However, this requires the input of a wide range of stakeholder engagement. Therefore, a governance structure can be established, implemented and evolved over time, once the repository has begun to be used more extensively. For example, ASAPbio has been running as a preprint service since 2016, establishing a governance structure after two years of service. As ASAPbio are leaders in OA repository governance implementation it would be useful to consider their approach when developing governance for any justice-system focused repository. It is proposed that a single HEI/organisation would take legal responsibility for and ownership of the repository. This organisation would set up the stakeholder Advisory Board and provide clear usage Terms and Conditions (T&Cs) and Codes of Conduct. It would also establish policies relating to data deposition, takedown (section 4.2), downloading, licensing and copyright, and disclaimers for repository content usage. These documents should clearly explain that it would be the responsibility of the user to fully comply when agreeing to use the service. Full details of these policies are beyond the scope of this article; however, they should be provided electronically through the repository platform when the service goes live. This would be in accordance with other self-archiving OA preprint services and repositories, such as arXiv and Zenodo.

The infrastructure of UK repositories is typically well networked [33]. However, repositories are diverse in nature [54], and the existence and quality of relationships between repositories varies internationally. Ideally, repositories should be integrated with ‘interoperability’ to enable mutual contributions and searching across repositories rather than being isolated and inflexible [21,24,25,40,47]. However, interoperability can be challenging to achieve [25].

Future repository sustainability could be further supported through online accessibility certification such as WCAG2.1 AA 2018. This would help to ensure all users can appropriately access and view the repository’s content. The workshop decided this would be done later because it is resource-intensive and initially may be more of a barrier than an enabler.

Workshop attendees and the literature are clear that it is essential to market the use of a new repository to potential contributors and information-seekers [21,55]. Resourcing of any new repository must also be a significant consideration. We and our wider working group were keenly aware of this need. As a precursor to Research4Justice, Jisc’s Jorum service was selected by the group in 2015 to start to openly share digital resources. The aim was to make undergraduate and Master’s forensic science research available to criminal justice practitioners. However, less than six months after the project launched and had begun to take off, the Jorum service funding was cut. This left the initiative unrealised with no digital platform on which to host a repository. Resourcing and marketing are essential to promote repository growth, value and sustainability for the justice system community.

The innovative Jisc Open Research Hub (JORH) could provide the appropriate system and service for helping to manage the proposed repository. Since the workshop in 2017, Jisc have gained ISO27001 certification for JORH and started to pursue accessibility certification. If JORH was selected, accessibility would be governed, maintained and developed by Jisc as part of a Service Level Agreement with the repository host organisation. This would help towards the sustainability of the repository. For this initiative to become successful, the repository needs to remain free for all end users with maintained sustainability [25,47]. Such provision will require significant financial and human resources, and continued use by contributors and information-seekers [33,43].

4.4. Future developments

The first release of a repository should provide the community with a good user experience, successfully storing, approving, depositing and discovering content. Over time and with continued stakeholder input, the repository can develop to offer additional functionality that would otherwise adversely affect the initial repository release. Through stakeholder interactions outlined above (section 3), the following future developments have already been suggested, including:

• •

  The ability to comment on deposited resources if logged in;

• •

  Storing geolocation metadata for depositors and superusers to report on frequency of repository access and downloads by country;

• •

  Tools for automatic creation and sharing of infographics within the user community and through social media.

Such functionality would support individual researchers, institutions and organisations in assessing the reach and potential impact of depositing research outputs. It would also help with analysis of the repository’s international accessibility, significance, value and sustainability from both justice system and higher education perspectives.

The ability to measure and evaluate how information within the repository is used, particularly by actors within the justice system, is essential. This will be a vital marketing tool, providing evidence to support the value of the repository and for increasing awareness across international jurisdictions. Improving the capacity to access and analyse metadata will enable evaluation of the reach and impact of research contained within the repository. High quality metadata is essential for ease of location and retrieval of digital contributions [24,41,53].

Jisc has been supporting work to model the application profile of a thesis within the repository user community. This includes working towards a consensus on thesis metadata to feed directly into the data model for Research4Justice. Unfortunately, the scientific community lacks awareness about the importance of metadata [41]. Also, student metadata is variable in quality, even within single academic fields [40]. The proposed repository, and the service/platform on which it is based, will therefore need to be both clear and uncompromising about metadata requirements. This will help to ensure that the repository is valued and used [33,56].

To understand the true value(s), and cost, of establishing and maintaining such a repository, further research needs to be conducted. Such research should:

• •

  Interrogate and analyse associated repository metadata;

• •

  Investigate the attitudes and usage behaviours of repository users, including key justice system stakeholders and the public;

• •

  Track the application and use of repository content on international policy and practice.

Implementing an effective feedback loop will be vital to the success and perpetual growth of this initiative. Empowering users to directly contribute to the development of the repository, through interaction with service providers, will help users understand the value of their contribution. This will make it more likely that investment in the service will continue, and increase the potential for users to recommend the repository to others.

5. Conclusion

At present, a significant proportion of research undertaken by pre-doctoral students and casework-focused practitioners remains inaccessible to those outside the researcher’s organisation. This situation has been caused by a combination of factors, including:

• •

  The time and investment needed to disseminate research effectively through traditional publication routes;

• •

  The extensive range of sources whereby research can be disseminated;

• •

  The historic nature of siloed working within disciplines;

• •

  The lack of a relevant subject-specific repository for the justice community.

As a result, this study has presented the rationale for establishing a new platform for justice system professionals, higher education providers and research institutions. This new platform would increase the ability to access, share and initiate justice-related research across traditional subject and geographic boundaries. Furthermore, it could also support aligned sectors, such as defence and security industries.

Finding ways to support the sharing of research findings and datasets offers widespread benefits to science and society, both now and in the future [24,41]. Justice system practitioners can face knowledge gaps and operational challenges. As a result, there is a need for collaborative work to develop holistic solutions that effectively address issues arising in casework and court. This can be done by pooling research findings and datasets that are available across both policing and forensic science communities [49]. This will clearly also have benefits for academia. Therefore, there is significant value in evaluating and rapidly reporting findings that address the relatively small but important questions arising through the investigative process. For example, determining the frequency of certain types and colours of fibre and background levels of contact trace materials.

It is important to be aware that the outputs within this type of repository will not have undergone the rigorous peer-review process of journal publication. Also, there are potential limitations in directly using student research in casework and/or court (section 4). Nevertheless, those outputs do have potential value to practitioners and other researchers. For example, a repository of these types of outputs could:

• 1.

  Act as a subject-based preprint server covering the extensive range of disciplines of relevance to the justice system;

• 2.

  Be a valuable tool for researchers conducting literature/systematic reviews of ongoing and completed research across the world;

• 3.

  Support networking and future collaboration between academia and industry, particularly in niche areas and multi/interdisciplinary projects;

• 4.

  Provide a source of research yielding negative research findings, which would otherwise be unpublished and unnecessarily replicated;

• 5.

  Facilitate graduate recruitment within the sector and the commissioning of specialised research;

• 6.

  Enable swift information-sharing worldwide.

Failure to achieve evidence-based practice in the justice system (whether within investigations and policing, or in the forensic analysis of exhibits) can have undesirable consequences. These may include miscarriages of justice [49,57,58], or criminals going unidentified, or not being apprehended and so being free to commit further crimes. An OA repository alone cannot prevent these problems. However, it can provide research findings to help establish evidence bases for identifying, classifying and interpreting forensic science evidence [59]. In turn, this could potentially contribute to increased detection and prevention of crime.

To provide further evidence and a fuller understanding of the true values and costs, of such a repository, further research still needs to be conducted. Such research will need to:

• •

  Interrogate and analyse associated repository metadata;

• •

  Investigate the attitudes and usage behaviours of repository users, including key justice system stakeholders and the public;

• •

  Track the application and use of repository content on international policy and practice.

This evidence could build confidence in the quality, significance and awareness of the research contained within the repository. Thereby it could facilitate growth, development and future sustainability of such an initiative to increase access to student and practitioner research within international justice communities.

Conflict of Interest

The authors confirm that they have no conflicts of interest.

Acknowledgements

We would like to thank Jisc for funding and supporting our research and the writing of this article. We are grateful to three anonymous reviewers who made helpful comments on earlier versions. And we couldn’t have done this without the survey respondents and workshop participants. Thank you all.

Footnotes

Appendix A. Supplementary data

The following is the Supplementary data to this article:

References

1. Schöpfel J., Vanacker S., Kergosien E., Jacquemin B. Master’s theses and open scholarship: a case study. Digital Libr. Perspect. 2018;34(4):276–287. doi: 10.1108/DLP-07-2018-0021. [CrossRef] [Google Scholar]

2. Meakin Georgina E., Jamieson Allan. A response to a response to Meakin and Jamieson DNA transfer: review and implications for casework. Forensic Sci. Int.: Genetics. 2016;22:e5–e6. [PubMed] [Google Scholar]

3. Casey David G., Clayson Nicola, Jones Sarah, Lewis Jennie, Boyce Maggie, Fraser Isla, Kennedy Finlay, Alexender Karen. A response to Meakin and Jamieson DNA transfer: review and implications for casework. Forensic Sci. Int.: Genetics. 2016;21:117–118. [PubMed] [Google Scholar]

5. Abrams Stephen, Katz John, Simms Stephanie, Strong Marisa, Willett Perry. Dash: data sharing made easy at the University of California. Int. J. Digit. Curation. 2016;11(1) doi: 10.2218/ijdc.v11i1.408. [CrossRef] [Google Scholar]

7. Hollmann Malen, Borrell Carme, Garin Olatz, Fernández Esteve, Alonso Jordi. Factors influencing publication of scientific articles derived from masters theses in public health. Int. J. Public Health. 2015;60(4):495–504. [PubMed] [Google Scholar]

9. Mnookin Jennifer L., Cole Simon A., Dror Itiel E., Fisher Barry AJ. The need for a research culture in the forensic sciences. UCLA Law Rev. 2010;58:725. [Google Scholar]

10. Morgan Ruth M. Conceptualising forensic science and forensic reconstruction; part I: a conceptual model. Sci. Justice. 2017;57(6):455–459. doi: 10.1016/j.scijus.2017.06.002. [PubMed] [CrossRef] [Google Scholar]

11. Committee on Identifying the Needs of the Forensic Science Community, National Research Council . National Research Council of the National Academies; Washington, DC: 2009. Strengthening Forensic Science in the United States: A Path Forward. [Google Scholar]

12. Roux C., Talbot-Wright B., Robertson J., Crispino F., Ribaux O. The end of the (forensic science) world as we know it? The example of trace evidence. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 2015;370(1674) doi: 10.1098/rstb.2014.0260. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

15. National Audit Office . National Audit Office; London: 2014. The Criminal Justice System: Landscape Review. [Google Scholar]

17. The Law Commission . The House of Commons; London: 2011. Expert Evidence in Criminal Proceedings in England and Wales. [Google Scholar]

18. Silverman B. Home Office; London: 2011. Research and Development in Forensic Science: A Review. [Google Scholar]

19. Science and Technology Committee . The Stationery Office Ltd; London: 2011. The Forensic Science Service: Seventh Report of Session 2010-12. [Google Scholar]

21. OECD . Organisation for Economic Co-operation and Development; Paris, France: 2007. Principles and Guidelines for Access to Research Data from Public Funding. [Google Scholar]

22. Robson Colin, McCartan Kieran. fourth ed. John Wiley & Sons; 2016. Real World Research. [Google Scholar]

23. IPO . Intellectual Property Office; Newport, Wales: 2011. Intellectual Asset Management for Universities. [Google Scholar]

24. Abrams S., Cruse P., Strasser C., Perry W., Boushey G., Kochi J., Laurance M., Rizk-Jackson A. DataShare: empowering researcher data curation. Int. J. Digit. Curation. 2014;9(1):110–118. doi: 10.2218/ijdc.v9i1.305. [CrossRef] [Google Scholar]

25. Peer Limor, Green Ann. Building an open data repository for a specialized research community: process, challenges and lessons. Int. J. Digit. Curation. 2012;7(1):151–162. doi: 10.2218/ijdc.v7il.222. [CrossRef] [Google Scholar]

26. Swauger Shea, Vision Todd J. What factors influence where researchers deposit their data? A survey of researchers submitting to data repositories. Int. J. Digit. Curation. 2015;10(1):68–81. [Google Scholar]

27. REF (2021) Initial Decisions on the Research Excellence Framework 2021. Bristol: REF.

29. Kara H. Zenodo; December 6, 2017. ForSci Repository Requirements Gathering. Funded by Jisc as part of the Research Data Shared Service project. [CrossRef] [Google Scholar]

31. Berg J.M., Bhalla N., Bourne P.E., Chalfie M., Drubin D.G., Fraser J.S., Greider C.W., Hendricks M., Jones C., Kiley R., King S., Kirschner M.W., Krumholz H.M., Lehmann R., Leptin M., Pulverer B., Rosenzweig B., Spiro J.E., Stebbins M., Strasser C., Swaminathan S., Turner P., Vale R.D., VijayRaghavan K., Wolberger C. Preprints for the life sciences. Science. 2016;352(6288):899–901. doi: 10.1126/science.aaf9133. [PubMed] [CrossRef] [Google Scholar]

32. Bourne Philip E., Polka Jessica K., Vale Ronald D., Kiley Robert. Ten simple rules to consider regarding preprint submission. PLoS Comput. Biol. 2017;13(5) doi: 10.1371/journal.pcbi.1005473. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

33. Armbruster Chris, Romary Laurent. Comparing repository types: challenges and barriers for subject-based repositories, research repositories, national repository systems and institutional repositories in serving scholarly communication. Int. J. Digit. Libr. Syst. 2010;1(4):61–73. [Google Scholar]

34. Plale Beth, McDonald Robert H., Chandrasekar Kavitha, Kouper Inna, Konkiel Stacy, Hedstrom Margaret L., Myers James, Kumar Praveen. SEAD virtual archive: building a federation of institutional repositories for long-term data preservation in sustainability science. Int. J. Digit. Curation. 2013;8(2):172–180. [Google Scholar]

35. Zhang Tao, Maron Deborah J., Charles Christopher C. Usability evaluation of a research repository and collaboration web site. J. Web Librariansh. 2013;7(1):58–82. doi: 10.1080/19322909.2013.739041. [CrossRef] [Google Scholar]

39. Banks David. Reproducible research: a range of response. Stat. Politics Policy. 2011;2:4. doi: 10.2202/2151-7509.1023. [CrossRef] [Google Scholar]

40. Schöpfel Joachim, Zendulkova Danica, Fatemi Omid. Electronic theses and dissertations in CRIS. Procedia Comput. Sci. 2014;33:110–117. [Google Scholar]

41. Tenopir Carol, Allard Suzie, Douglass Kimberly, Aydinoglu Arsev Umur, Wu Lei, Read Eleanor, Manoff Maribeth, Frame Mike. Data sharing by scientists: practices and perceptions. PLoS One. 2011;6(6) [PMC free article] [PubMed] [Google Scholar]

43. Neugebauer Tomasz, Murray Annie. The critical role of institutional services in open access advocacy. Int. J. Digit. Curation. 2013;8(1):84–106. doi: 10.2218/ijdc.v8i1.238. [CrossRef] [Google Scholar]

44. Creaser Claire, Fry Jenny, Greenwood Helen, Oppenheim Charles, Probets Steve, Spezi Valérie, White Sonya. Authors’ awareness and attitudes toward open access repositories. New Rev. Acad. Librariansh. 2010;16(S1):145–161. doi: 10.1080/13614533.2010.518851. [CrossRef] [Google Scholar]

45. Akers K.G., Green J.A. Towards a symbiotic relationship between academic libraries and disciplinary data repositories: a dryad and University of Michigan case study. Int. J. Digit. Curation. 2014;9(1):119–131. doi: 10.2218/ijdc.v9il.306. [CrossRef] [Google Scholar]

46. Callaghan Sarah, Donegan Steve, Pepler Sam, Thorley Mark, Cunningham Nathan, Kirsch Peter, Ault Linda. Making data a first class scientific output: data citation and publication by NERC’s environmental data centres. Int. J. Digit. Curation. 2012;7(1):107–113. doi: 10.2218/ijdc.v7i1.218. [CrossRef] [Google Scholar]

47. Yuan Li, MacNeill Sheila, Kraan Willem G. Open Educational Resources-Opportunities and challenges for higher education. 2008. http://sprite.bolton.ac.uk/290/1/iec_reports-1.pdf Educational Cybernetics: Reports. Paper 1.

48. Tenopir Carol, Dalton Elizabeth D., Allard Suzie, Frame Mike, Pjesivac Ivanka, Birch Ben, Pollock Danielle, Dorsett Kristina. Changes in data sharing and data reuse practices and perceptions among scientists worldwide. PLoS One. 2015;10(8) [PMC free article] [PubMed] [Google Scholar]

49. Howes Loene M. The communication of forensic science in the criminal justice system: a review of theory and proposed directions for research. Sci. Justice. 2015;55(2):145–154. [PubMed] [Google Scholar]

50. Hemingway Angell, Angell C., Hartwell H., Heller Richard F. An emerging model for publishing and using open educational resources in public health. Perspect. Public Health. 2011;131(1):38–43. doi: 10.1177/1757913910391034. [PubMed] [CrossRef] [Google Scholar]

51. Kara H. Facilitating procedural ethics: establishing the research ethics application database at Oxford University. In: Tolich M., editor. Qualitative Ethics in Practice. Left Coast Press; Walnut Creek, CA: 2016. [Google Scholar]

52. Ilyas M. Second LACCEI International Latin American and Caribbean Conference for Engineering and Technology (LACCEI’2004), Challenges and Opportunities for Engineering Education, Research and Development, Miami, Florida, USA. 2-4 June 2004. Best practices in Industry-University Collaboration. [Google Scholar]

53. Tenopir C., Sandusky R.J., Allard S., Birch B. Research data management services in academic research libraries and perceptions of librarians. Libr. Inf. Sci. Res. 2014;36:84–90. doi: 10.1016/j.lisr.2013.11.003. [CrossRef] [Google Scholar]

54. Baker Karen S., Yarmey Lynn. Data stewardship: environmental data curation and a web-of-repositories. Int. J. Digit. Curation. 2009;4(2):12–27. [Google Scholar]

55. Hider P., Mitchell P., Galatis H., McDowell K. Department of Education and Training; Canberra, Australia: 2017. Developing an Effective, Accessible and Sustainable Digital Repository of OLT Learning and Teaching Resources. Final Report. [Google Scholar]

56. Durcikova Alexandra, Fadel Kelly J. Knowledge sourcing from repositories: the role of system characteristics and psychological climate. Inf. Manag. 2016;53(1):64–78. [Google Scholar]

57. Bolton-King Rachel S. Preventing miscarriages of justice: a review of forensic firearm identification. Sci. Justice. 2016;56(2):129–142. doi: 10.1016/j.scijus.2015.11.002. [PubMed] [CrossRef] [Google Scholar]

58. Savage Stephen P. Restoring justice: campaigns against miscarriages of justice and the restorative justice process. Eur. J. Criminol. 2007;4(2):195–216. doi: 10.1177/1477370807074855. [CrossRef] [Google Scholar]

59. Morgan Ruth M. Conceptualising forensic science and forensic reconstruction; part II: the critical interaction between research, policy/law and practice. Sci. Justice. 2017;57(6):460–467. doi: 10.1016/j.scijus.2017.06.003. [PubMed] [CrossRef] [Google Scholar]

Articles from Forensic Science International: Synergy are provided here courtesy of Elsevier