Evaluation of the Canadian Centre for Justice Statistics Program
(2011/2012 to 2015/2016)

Summary

Canada's criminal justice system is highly complex: federal, provincial, territorial and municipal agencies and organizations all play a part, but no agency or jurisdiction has control or ownership of the entire system. Through the CCJS, the efforts of the NJSI are directed toward the production of justice statistical data to inform the legislative, policy, management and research agendas of federal, provincial and territorial partners, and to inform the public.

"Building and maintaining an effective criminal justice system requires reliable national information on the nature of crime, on crime trends, and on what actions work" (Auditor General of Canada 2002).^{Footnote 12} Findings from the document and literature reviews indicate that integration of data and information sharing are essential to maintaining the Canadian justice system, given the independence of criminal justice agencies. Meeting these needs aligns with Statistics Canada's broad mandate to collect, compile, analyze and ultimately publish statistical information on all dimensions of Canada's economy, society and environment, and particularly with sections 26 to 29 of the Statistics Act regarding responsibilities for "criminal statistics."^{Footnote 13}

Evaluation findings demonstrate that the CCJS produces data that are used by a wide range of users such as federal departments; provincial, territorial and municipal governments; academic communities; individuals from the general public, including the media; and international organizations such as the Organization for Security and Co-operation in Europe (OSCE) for hate crimes and the United Nations Office on Drugs and Crime (UNODC).

User needs are also varied. For example, the Department of Justice and Public Safety Canada, together with provincial and territorial jurisdictions in policing, courts and corrections, use CCJS information to draft legislation; to develop policies, programs and services for Canadians in the areas of justice and public safety; and to prepare statistical reports such as the Trafficking in Persons annual report. Other federal departments that work closely with the CCJS include Status of Women Canada, particularly on issues related to victimization, and the Department of Canadian Heritage (PCH), on issues related to hate crime.

CCJS data are also used by consortia of different partners and stakeholders who have policy interests in particular justice and public safety areas. For example, the Family Violence Initiative, led by the Public Health Agency of Canada (PHAC), brings together 15 partner departments and agencies to respond to and prevent family violence. The CCJS provides statistical advice and analysis to produce an annual statistical profile that integrates different data sources (police-reported data, homicide data and victimization survey data), which helps to understand the nature and extent of family violence in Canada. Also as part of the Family Violence Initiative, the biennial Transition Home Survey provides information on the shelter services available to abused women and their children, as well as a profile of the individuals whom these facilities serve.^{Footnote 14}

Furthermore, as a subject-matter expert, the CCJS is often called upon by House of Commons and Senate committees to present information on justice and public safety to advance the work of government. In the five years covered by the evaluation, the CCJS provided information to four Senate or House of Commons committees, including three committees that required information from the CCJS in the last two fiscal years examined (2014/2015 and 2015/2016):

• In November 2014, the CCJS provided information to the House of Commons Standing Committee on the Status of Women to begin its study Promising Practices to Prevent Violence Against Women.
• In February 2016, the CCJS provided information to the Standing Senate Committee on Legal and Constitutional Affairs related to court delays in Canada.
• In April 2016, the CCJS provided information for a presentation to the House of Commons Standing Committee on the Status of Women related to violence against young women and girls in Canada.^{Footnote 15}

National and community-level information produced by the CCJS on crime and justice is reported regularly in the media. CCJS survey results were cited 1,141 times in the media between April 1, 2012, and March 31, 2016.^{Footnote 16} This included information related to the crime rate and the Crime Severity Index (CSI), adult and youth court caseloads and sentencing, the correctional population, and incarceration rates, as well as victims and victimization.

The Government of Canada is committed through different international agreements to fighting crime and ensuring human rights. Because of the CCJS's expertise and the nature of its activities, the program contributes to different international initiatives. For instance, the CCJS is a point of contact to fulfill information-sharing commitments with the Organization for Security and Co-operation in Europe for hate crimes, and the United Nations Office on Drugs and Crime for the UN's annual Survey of Crime Trends and Operations of Criminal Justice Systems.

Almost all key informants believed that there would be a substantial negative impact with the absence of justice statistics from the CCJS. They emphasized the loss of an evidence base that is used to understand the justice system in Canada and to inform decision making related to policies and programs. Without the CCJS, almost all key informants believed that valid, reliable and comparable national data would not be available.

About half of key informants reported that without justice statistics from the CCJS, comparisons between jurisdictions would be very challenging. This information is critical to understanding how a certain jurisdiction is performing relative to others, or how a jurisdiction is performing over time in specific areas (e.g., types of crime, court cases). Also, cross-jurisdictional data help jurisdictions understand whether certain trends are regional in nature or are part of a national trend. This allows provinces to explore whether there are lessons to be learned from other jurisdictions. Without cross-jurisdictional data, key informants expressed concerns that the system would become "parochial," and that "our vision would become very limited and narrow."

Some key informants also commented that the ability of government, non-governmental organizations and academics to conduct research would be impacted if CCJS data were not available. Such research benefits government and the justice system by identifying trends, impacts of legislation or policies, or potential issues of concern.

If CCJS data were unavailable, some key informants believed their organizations would try to fill the gap themselves, but would have limited ability to do so with current staff or financial resources. While certain statistics that are derived from centralized sources (e.g., corrections data) might be easy to produce, data that are compiled from many different agencies (e.g., national, provincial and municipal police forces) would be very difficult or impossible to reproduce. A few key informants reported trying to obtain data from provincial jurisdictions or hiring contractors to collect the data. However, they indicated that data from provincial jurisdictions were difficult to obtain, had numerous gaps, and were variable in terms of quality, and that hiring contractors was an expensive alternative.

Summary

There is a long-standing federal commitment to address crime and ensure that the justice system is fair, efficient and accessible to all Canadians. Speeches from the Throne during the evaluation period and recent budgets highlight some of these commitments:

• In the 2015 federal budget, the Government took further steps through additional funding to ensure that victims of crime are at the heart of Canada's judicial system, to assist Canadians in getting the information they need to protect themselves and their families online, and to strengthen the ability of law enforcement agencies to combat cybercrime.^{Footnote 17}
• In the 2016 federal budget, the Government remained committed to ongoing work with provinces and territories on a number of important initiatives, including supporting them in their efforts to take illegal guns off the streets and reduce gang-related violence.^{Footnote 18}
• In the coming years, the CCJS may provide information to support 15 current ministerial mandate letter priorities, including the National Inquiry into Missing and Murdered Indigenous Women and Girls, and a review of the changes in the criminal justice system and sentencing reforms.^{Footnote 19}

All these initiatives contribute to the Whole-of-Government Framework outcome area "A safe and secure Canada," which includes program activities that aim to maintain the safety and security of Canada and its citizens through crime prevention, law enforcement, the securing of Canadian borders, and emergency preparedness.^{Footnote 20}

The governing body of the NJSI is comprised of the federal, provincial and territorial (FPT) deputy ministers responsible for justice. The Chief Statistician of Canada or the Assistant Chief Statistician responsible for the CCJS program attends deputy minister meetings when required.^{Footnote 21} As the operational arm of the NJSI, the CCJS is continually working to provide information that aligns with government priorities. The CCJS annual operational and strategic plans are determined jointly and approved by the deputy ministers.^{Footnote 22} The CCJS promotes initiatives with its partners to build and maintain a statistical system flexible enough to respond to new, specific and relevant justice-related topics for all levels of government.

The evaluation found a number of strategic priorities of Statistics Canada in which the CCJS is directly involved detailed in various agency documents. The following are some of the CCJS key initiatives that support the agency's priority of responding to the emerging and evolving information needs of data users and stakeholders:^{Footnote 23}

• Expertise in linking justice and crime-related data: To understand the extent of the re-contact of individuals with the Canadian criminal justice system and their pathways through it, the CCJS is analyzing administrative data from police, courts and corrections facilities. The program is able to add new content to existing surveys, and steps are in place to collect sufficient information to allow data linkages (e.g., linking information such as personal identifiers of names, date of birth) and help evaluate the sociodemographic characteristics of people who have come in contact with the justice system.
• Police performance metrics: Collaborative work is ongoing in this pilot project, which examines opportunities and challenges in collecting national-level data on calls for service. In 2016/2017, key indicators of police performance will be identified, and a framework will be developed for police performance metrics. In 2017/2018, pending funding, systems will be built to extract and collect data on police calls for service.
• New National Fire Information Database: Statistics Canada, in collaboration with the Canadian Association of Fire Chiefs, will be collecting, for the first time, national fire incident and loss data. The CCJS will aggregate, map and model 10 years of provincial and territorial fire data, leading to the first national-level analysis of fire incidents in Canada. This will assist fire services in making timely and critical operational decisions, help guide policy and prevention measures in the development of appropriate and efficient methods of fire response, and help promote community safety through greater public awareness about the dangers of fire.

Summary

Justice in Canada is a shared responsibility between the federal government and the provinces and territories. However, the compilation of justice-related statistics for Canada is a federal responsibility and is specifically delineated under sections 22, 26, 27, 28 and 29 of the Statistics Act.^{Footnote 24} While Statistics Canada has a duty to collect, compile, analyze, abstract and publish statistics in relation to law enforcement and the administration of justice and corrections, it also has the responsibility to collect information on criminal statistics and offences. Under the direction of the NJSI, Statistics Canada has a long history of producing key statistics on the three main CCJS justice domains—policing, courts and corrections—and also information on victims.

About half of key informants did not perceive any issues regarding areas of ambiguity, areas of overlap, gaps in responsibility or conflicting priorities between the CCJS's mandate and the mandate of other statistical programs or institutions, either public or private. Although a few explained that there are other organizations that produce data—such as provinces—there are no other data "integrators" that compile data from across Canada to produce data that allow for comparisons across jurisdictions. Some of the remaining key informants believed that there are areas of ambiguity or gaps in how the CCJS fulfills its mandate, but their comments did not reflect a desire that these areas be handled by other statistical programs or institutions. Rather, key informants commented on particular gaps or ambiguities in the CCJS mandate or collaborations. For example, stakeholders suggested that justice statistics should be more cross-sectoral, or that there should be greater linkages between the CCJS and local statistical agencies or partnering departments, or that the CCJS should provide more or better information in certain areas (e.g., courts or corrections).

When asked whether the mandate of the CCJS could be fulfilled more effectively and efficiently by another organization within or outside the federal government, almost all respondents believed that the CCJS is the most appropriate to fulfill this role. Their comments provided a clear sense of their level of respect and the confidence they have in the CCJS fulfilling this mandate:

• Expertise and authority: Some of these key informants focused on the CCJS's experience and expertise, which come from its involvement in creating the program and building extensive relationships across multiple jurisdictions. The CCJS, as part of Statistics Canada, has credibility (the agency is renowned for rigorous data quality and security) and the authority it needs—via the Statistics Act—to collect data from jurisdictions across Canada. There is doubt that an external agency could be given the sort of access needed to collect this kind of data, and such a move would likely require legislative changes.
• Effective and efficient: Some key informants suggested that other organizations would be less effective and efficient in fulfilling this mandate. There is a perception that the current CCJS executive is forward-thinking in planning and experienced in collaborating, and this would be lost if the program were moved to another organization. Additionally, it is felt that having the responsibility for justice statistics be within Statistics Canada is efficient, because program staff can consult with other statistical experts within the agency.

Because of its mandate, the CCJS is also considered to be the only national provider of justice statistics, as it is the focal point of a federal–provincial–territorial partnership for the collection of information on the nature and extent of crime and the administration of civil and criminal justice in Canada.

Document and literature reviews indicated that Statistics Canada is, for some government programs, the only source of justice information. For instance, in an evaluation report completed in 2010, Citizenship and Immigration Canada (now IRCC) mentioned that "there are no other sources of national, police-reported data on hate crime in Canada. There was wide support for CCJS delivering the Data Collection Strategy and no strong evidence of viable delivery alternatives."^{Footnote 25} Also, at the department now called Indigenous and Northern Affairs Canada (INAC), the Social Development Program's performance measurement strategy mentions that "Canada has two main sources of national data on crime: the Uniform Crime Reporting (UCR) Survey, and the General Social Survey (GSS) on Victimization (…) The Program relies on this data as its primary source of 'violence' information and does not collect rates of family violence on reserve."^{Footnote 26}

Summary

According to the agency's 2009 quality guidelines,^{Footnote 28} for a data product to be accurate, appropriate methods and tools must be used to produce the results. These include choosing sampling methods, encouraging accurate responses, following up on nonresponse to surveys, dealing with missing data, and measuring response rates and sampling error.

In the case of the CCJS, most of the surveys are administrative and mandatory. Therefore, typical measures such as coefficients of variation (CVs) or response rates were not used to assess accuracy. For the CCJS, post-release corrections were examined as an indication of accuracy. Of the 90 products released by the CCJS from 2011/2012 to 2015/2016, there was one post-release correction due to accuracy (1%). It pertained to the Adult Correctional Services (ACS) Survey of the Correctional Services Program and occurred in December 2012. The error was a result of revised data received from a jurisdiction post-release and the subsequent identification of a data capture error.^{Footnote 29}

The program areas within the CCJS have rigorous data processing systems and well-established processes for data verification and sign-off for their surveys. Many verifications and certification points occur during the data production process, and it is through these processes that issues related to data quality and accuracy are addressed.^{Footnote 30}

Most key informants interviewed were satisfied with the accuracy of the CCJS's statistical information. They expressed their appreciation for the review and verification process and the use of footnotes to indicate where caution should be exercised when using certain statistics. Most respondents to the user survey were also satisfied with the accuracy of the data (118/125; 94%).

Evaluation findings also show that the CCJS has an effective work-in-progress (WIP) review process for the Liaison Officers Committee of the National Justice Statistics Initiative (LOCNJSI) that allows stakeholders to review Juristat articles prior to release for accuracy, coherence and interpretability.

Summary

Most CCJS products were released as scheduled. Of the 90 products released by the CCJS over the fiscal years 2011/2012 to 2015/2016, most products (80/90; 89%) met the release dates established by the program. Those that did not meet the pre-established release dates were related to policing surveys (5), courts (2), corrections (1), and GSS analysis (2). Of the releases that did not meet the program-established release dates, most (8/10) were released in 2011/2012 or 2012/2013, demonstrating an increase in the rate of meeting established release dates in more recent years.

Results from key informants interviewed and respondents to the user survey regarding timeliness were mixed. Respondents to the user survey were largely satisfied with the timeliness of the data (105/141; 74%). However, only a minority of key informants interviewed were satisfied with the timeliness of the CCJS information.

Overall, key informants want the CCJS data in a more timely manner. Jurisdictions desire the most up-to-date information possible to make appropriate decisions on programs and policy and to respond to immediate issues in the media.

Some key informants also connected the timeliness of data to the relevance of data. It was noted that the data released, particularly in the area of courts and corrections, reflected trends occurring up to a year ago, or more. A few of these key informants reported that data that were over six months to a year old were less relevant for making policy and program decisions and planning.

It was also mentioned that timeliness may be contingent on certain survey cycles (e.g., the GSS on Victimization, which is conducted only every five years). There is also an interest at the international level in Canada releasing victimization information more frequently. For instance, some countries study victimization on a shorter time frame (yearly or every two years).^{Footnote 31}

Unlike the UCR Survey, the GSS on Victimization collects data on criminal incidents whether or not they have been brought to the attention of the police. This survey is the only national survey of self-reported victimization and is collected in all provinces and territories. It allows for estimates of the numbers and characteristics of victims and criminal incidents. As not all crimes are reported to the police, the survey provides an important complement to officially recorded crime rates. It measures both crime incidents that come to the attention of the police and those that are unreported. It also helps with understanding the reasons behind whether or not people report a crime to the police.^{Footnote 32}

Challenges to timeliness

Key informants spoke about the issues contributing to challenges with timeliness. A few attributed these to the jurisdictions providing data, while others mentioned that it takes time to properly aggregate and analyze data from across the country. It was also recognized that timeliness may be tied to the slow pace at which the justice system operates, which makes it difficult to stay on top of immediate or emerging issues.

Internal key informants also acknowledged that the release of information in a timely manner is dependent on the timeliness of the inputs received from jurisdictions. They noted that some jurisdictions may not be able to respond as quickly as needed. This may be due to conflicting priorities, the fact that not all jurisdictions have adequate systems in place (e.g., technology) to provide timely information, or a lack of financial resources. It was noted that, previously, there had been funding to help jurisdictions develop capabilities, but that such funding is no longer available.

There is a heavy reliance in the CCJS on data providers, and, in that context, data collection is mostly out of the statistical agency's control. Receiving data on time is critical for key releases. This can have an impact on relevance, since the data may be well out of date by the time they are released.

The timeliness issue aligns well with the literature on the use of administrative data and their limitations. It indicates that many factors during the process of collection can hinder a provider from making the data available in a timely manner. They can be classified in three categories, as depicted in Figure 1: the financial aspect (funds), the human aspect (training and knowledge) and the political aspect (willingness of jurisdictions). Solutions to enhance timeliness rely on a good understanding of the elements at play: actors (effectiveness and buy-in), the system in place (only as fast as the slowest provider) and the data themselves (completeness, amount and policies in place).^{Footnote 33}

Figure 1 Factors contributing to data delays

Source: National Association for Public Health Statistics and Information Systems (NAPHSIS), 2013.

Despite these challenges with timeliness, key informants appreciated that the CCJS is upfront about why proposed deadlines are not possible and reported that when the CCJS commits to providing data, the organization respects this commitment.

Summary

The CCJS produces and disseminates close to 120 CANSIM data tables annually and releases related products, such as Daily releases, summary tables, publications, and information for analysts and researchers, to promote and facilitate access to crime and justice information. The CCJS also prepares custom data on a cost-recovery basis for various clients, including government departments, NGOs, academia and the media.

A client service email and phone number are set up to respond to public enquiries on data availability and access, and on data concepts, definitions and usage. Furthermore, the CCJS provides opportunities for work-in-progress reviews and data validation of CCJS analytical products to NJSI partners. Access to pre-release information is given via the agency's Electronic File Transfer Service.

Statistics on access (i.e., visiting the website, viewing products, and downloading and accessing data) show that there is interest in CCJS statistical products (see Table 2, below). Between 2012/2013 and 2015/2016, crime and justice statistics pages (CANSIM tables, publications and summary tables) were viewed 4.7 million times, compared with 64.8 million page views for Statistics Canada as a whole. Juristat was by far the most popular publication, representing 78% of all CCJS publication page views and 9% of all Statistics Canada publication page views for the 2012/2013-to-2015/2016 period.

Over the same period, the number of downloads of CANSIM tables and publications for the CCJS and Statistics Canada totalled 0.871 million and 13.6 million, respectively. Of the various publications, Juristat was the most downloaded, followed by Family Violence in Canada: A Statistical Profile, the Crime and Justice Research Paper Series, and the Canadian Centre for Justice Statistics Profile Series.

Almost all key informants agreed that CCJS information is functional, innovative and user-friendly. Some key informants specified that CANSIM tables and Juristat are helpful and easy to use, that the products are available in multiple formats (e.g., HTML, PDF and Excel), that CANSIM tables can be customized to address many different questions, and that email notifications were useful for providing information about new material uploaded to the website.

The user survey confirmed the generally high level of satisfaction with most methods of accessing CCJS products. The majority of survey respondents (88% or more) were satisfied with access to common sources of CCJS information on the website, such as The Daily, CANSIM tables, publications and reference documents, as well as the ease of retrieval from electronic sources and the convenience of output formats.

Only a minority of the 156 survey respondents reported that they had accessed data through other means, such as public use microdata files (PUMFs) (n=32), research data centres (RDCs) (n=37), Real Time Remote Access (RTRA) (n=10), or the Data Liberation Initiative (n=4). Among survey respondents who had accessed CCJS information through these means, satisfaction was generally high (70% or more).

Customized information and support services

From 2011/2012 to 2015/2016, a total of 282 contracts for customized products were delivered on, for total revenue of $9,198,325.

Federal government departments and agencies represented almost half of the custom data users (49%), followed by postsecondary institutions (30%) and provincial governments (8%). The remaining custom data users included municipal governments (3%); the media (3%); and other users, such as non-governmental organizations, the private sector and private individuals (7%).^{Footnote 34}

Survey respondents generally reported that they were satisfied with the accessibility of custom data tabulations that had been adapted to their needs (74/84; 88%). Among the survey respondents who had contacted Statistics Canada for additional services (e.g., customized data, support or training), 35% (25/71) paid for the services. Among those who paid for the services, most were satisfied with the services received (24/25; 96%).

Some of the external key informants also reported that they had contacted the CCJS for additional services, support, or customized data tabulations or analysis. All reported that they were satisfied with the services provided, although a few mentioned that it had taken a long time to fulfill the request for customized information.

Comparison with other data providers

About half of key informants did not have experience with other data providers, suggesting that CCJS data are a key source of justice-related information. Among the remaining key informants interviewed, most reported that dissemination and access were very good, and that the CCJS was a source of advanced national data with cross-jurisdictional information that was accessible to the public free of charge.

However, it was also noted that it was necessary to rely on provincial and territorial information for jurisdiction-specific information because the CCJS does not collect that level of detail.

A few key informants mentioned that other international jurisdictions, such as Australia, the United Kingdom, the United States and New Zealand, provide more user-friendly access to online information and effectively use data visualization techniques.

Challenges to accessibility

Some external key informants reported challenges with the accessibility of CCJS information, such as difficulty finding information or products (such as Juristat) on the website. A few key informants also specifically mentioned having difficulties using the search function on the Statistics Canada website, or downloading and using online CANSIM tables. Most respondents also provided suggestions to improve dissemination.

• Website: Suggestions were related to improving the search feature to increase the findability of information and to provide more information on the website about the types of data that are available.
• Notification and communication: Key informants suggested more communication about CCJS products. Specifically, it was suggested that the CCJS raise awareness of its data products, possibly through social media or by using push notifications. Although the CCJS has a notification service, it appears that not all key informants are aware of, or have access to, this service.
• Products: Improvements to CCJS products were suggested, including more effective use of visual representations of data, more timely release of court data and a more user-friendly design for CANSIM tables.
• Data-sharing mechanisms: Suggestions were made to improve the accessibility of data through RDCs. They included
  • increasing access to some types of data not currently available to researchers (e.g., certain details within the UCR Survey data)
  • promoting knowledge about CCJS products in RDCs (e.g., by providing information about RDCs at conferences)
  • improving access to RDCs through initiatives such as implementing RDCs within each provincial government
  • promoting use of a common information technology platform within all jurisdictions.

Internal key informants reported that researchers are notified of CCJS data in the RDCs through notifications sent to distribution lists at local universities, although it was unclear to what degree that information was communicated to faculty and students from there. Information on RDC data is also promoted through the Canadian Research Data Centres Network (CRDCN), through information posted on its website and during annual CRDCN conferences. Key informants acknowledged that improvements could be made to reach more academics and increase awareness of RDC information. It was also noted that researchers had expressed interest in having access to more information through RDCs, such as more geographic information available in CCJS RDC files, information on trends over time and record linkages that would provide the capacity to study recidivism outcomes.

Initiatives to improve accessibility

To further increase the access to and in-depth analyses of justice data by the academic community, the CCJS piloted several surveys through the RDC Program, an initiative by Statistics Canada, the Social Sciences and Humanities Research Council, and university consortia to help strengthen Canada's social research capacity and to support the policy research community.

Surveys available in the RDCs (as of March 2017)^{Footnote 35} include the following:

• UCR Survey: An update to the RDC data for the UCR Survey in January 2017 includes data up until 2015.
• UCR Survey Hate Crime Module: The hate crime module data are available through a pilot project from 2015 to 2017. To date, only one research proposal has been received and approved for access.
• Homicide Survey: The one-year pilot for the Homicide Survey was intended to end on March 31, 2015. However, there was less research activity than anticipated from the five approved research projects. The pilot was extended to March 2017, and new proposals will be accepted under Phase 2 of the pilot.
• Integrated Criminal Courts Survey: Six years of ICCS data files (2005/2006 to 2011/2012) will be available through an RDC pilot project.

Furthermore, some CCJS data are available through the Real Time Remote Access (RTRA) tool, including the UCR and Homicide surveys. The RTRA allows academic, federal, provincial and territorial researchers to securely work with microdata remotely from their desktops while preserving the security of confidential information.

Summary

Statistics Canada is tasked with leading the national statistical system, and working collaboratively with other federal departments and other levels of government to avoid duplication of effort and to promote coherence of statistical information.^{Footnote 36}

Most key informants indicated that they were satisfied with the coherence of CCJS statistical information. They reported that the CCJS effectively applied common methodologies to the data, which allowed for comparability and the clear identification of methodological changes. A few other key informants mentioned that coherence has been improving over time. Satisfaction with data coherence was also high among respondents to the user survey (120/127; 94%).

The CCJS is involved in several activities and working groups to maintain and enhance coherence of CCJS statistical information. In 2014, the CCJS established an internal Data Quality Secretariat to ensure high data quality, as a result of the high volume of products released by the CCJS and the increasing demand for detailed information from stakeholders. The objective of the secretariat is to identify any data quality vulnerabilities (including coherence) of existing surveys, develop action steps to address any risks, and identify best practices.^{Footnote 37}

The CCJS also has a work-in-progress review process for the Liaison Officers Committee of the National Justice Statistics Initiative (LOCNJSI), which allows the stakeholders to review Juristat articles for accuracy, coherence and interpretability.

The CCJS is also engaged in other initiatives with NJSI partners to improve data quality and coherence, such as^{Footnote 38}

• Police performance metrics: In partnership with Public Safety Canada, the CCJS is leading the development of a national framework around police performance metrics.
• UCR Survey data quality and comparability: The CCJS supported the work of the POLIS Working Group on Data Quality to identify areas of over-scoring, and led the work of POLIS's Organized Crime Working Group aimed at improving quality of reporting on the UCR organized crime flag.

Summary

The document and literature review showed that CCJS statistical information is complex, robust and relevant and that program information is available to assist users in interpreting the data. CCJS products meet the requirements of the Policy on Informing Users of Data Quality and Methodology, and are documented within Statistics Canada's Integrated Metadatabase.

For CCJS statistical products, the user can find descriptions of data sources and methodology, definitions of the concepts and variables that are measured, and indicators of data quality. The information on the status, frequency, record number, data release, reference period, collection period and subjects covered by the CCJS is generally up to date and accessible.

The administrative data review indicated that the metadata were generally up to date. In 2015, most CCJS metadata were fully compliant or had only minor deficiencies (see Table 3). However, the relative percentage of minor deficiencies was higher for the CCJS (69%) than the average across Statistics Canada overall (20%). Two of the CCJS surveys also had medium deficiencies in 2015: the Youth Court Survey and the Homicide Survey.

The majority of survey respondents were satisfied with the interpretability of the data (122/138; 88%). Most key informants also indicated that they were satisfied with the interpretability of CCJS statistical information. They reported that information regarding definitions and explanations was available through various sources on the website, such as data dictionaries, and footnotes in CCJS documents. A few key informants commented that if they could not find an explanation or definition within a publication, they were able to find that information on the Statistics Canada website or by calling the CCJS's staff. However, some also commented that they were not sure whether the general public would find the material equally interpretable.

Summary

Engagement and consultation process

The CCJS responds to the needs of federal, provincial and territorial governments through an effective consultation process, which is supported by a clear governance structure, including several committees and working groups. Some improvement would be required, such as involving other groups during consultations.

Some key informants—when asked about the extent to which CCJS justice statistics met current and emerging needs—responded by sharing positive observations about how the CCJS consults with jurisdictions to understand data needs, the CCJS's flexibility and active role in identifying and addressing data gaps, and the chance to review CCJS material prior to its being published. Internal interviewees also highlighted that the CCJS governance model is considered within Statistics Canada one of the best and should be applied to other programs.

Through the collaborative work between NJSI partners and the CCJS, needs and opportunities are better identified so that the stakeholders have access to the data they need, in a form that facilitates their effective use. Consultation roadmaps and operational plans are developed to set the priorities and initiatives to ensure the relevance of the program.

Key informants who referred to collaborative and consultative aspects of their partnership with the CCJS frequently referred to the formal governance structure, such as roles in specific committees (e.g., POLIS), subcommittees (e.g., CORIS) or working groups, and the role of liaison officers in connecting ministries with the CCJS. The consultative aspects of these collaborative relationships include sharing advice, developing analytical measures and approaches to ensure consistent reporting, voting on the selection of specialized studies, and having jurisdictions agree to pilot methodologies or identify alternative analytical approaches.

Key informants reported that the CCJS supports effective collaboration by building strong relationships and being inclusive, helpful and responsive. For example, the CCJS assists provincial and territorial jurisdictions in understanding changes to data collection, provides the opportunity (if applicable) to contribute to new pilot projects, provides information to better understand CCJS datasets or jurisdictional data quality issues, and provides opportunities to suggest possible research areas for further investigation.

Some key informants referred to the establishment of processes that promote collaboration between the CCJS and jurisdictional partners, such as the use of committees to communicate with the CCJS about various issues (e.g., types of information available, potential to customize a data product). A few national-jurisdiction key informants referred to the successful use of "deemed employees" from Justice Canada who were assigned to the CCJS (via an MOU). This arrangement was praised because it allowed the Justice Canada personnel to learn more about what data are available through the CCJS, while providing CCJS staff with additional resources and expertise in the criminal justice system.

Although most key informants spoke positively about the collaboration between the CCJS and their organization, they also identified some challenges or issues related to collaboration:

• Communication: The need for improved communications between the CCJS and partner jurisdictions was raised. Some key informants reported requiring more consultation to improve survey responses and methodologies in their jurisdiction, more time to review and consult on materials, more communication with senior-level provincial and territorial staff (e.g., deputy ministers), or more alignment between provincial and federal processes to ensure that deadlines are coordinated. A few key informants also desired more frequent communications via teleconference between the biannual face-to-face meetings. This approach would help reduce the demand to "address issues all at once" during the biannual meetings.
• Provincial and territorial resource limitations: Limited resources among provincial and territorial data providers are a challenge that affects collaboration. This includes limited human and budget resources, information technology, and statistical expertise to implement the changes needed to provide data to the CCJS.
• Liaison Officers Committee (LOC): More senior-level staff, specifically those who report directly to deputy ministers, should be involved in the LOC to better articulate deputy ministers' priorities to the CCJS. A few key informants also noted that the LOC would benefit from having a representative from Correctional Service Canada (CSC).^{Footnote 40}

Key informants also suggested the possibility of engaging with a broader range of stakeholders, either through formal engagement in NJSI governance or through general consultation on issues of interest to the CCJS.

• Some external key informants suggested more inclusion of family/civil justice representatives on the LOCNJSI. A few key informants supported the idea of involving academics from a variety of fields, including criminology, sociology and psychology, or others specializing in criminal justice issues.
• Other external key informants supported the idea of engaging in consultations with a broader range of stakeholders outside the governance model, such as think tanks, municipalities, academia, non-governmental or community organizations, and local statistics organizations. Specifically, a few key informants believed it was too early for these groups to be included in governance, but suggested that there might be beneficial aspects to consulting these groups about issues of interest and the use of CCJS data, and providing assistance in the use of the data.
• Some internal interviewees also suggested that the involvement of some groups, such as academics, should be encouraged. Extensive consultation through RDCs and pilot projects, where academics could provide their advice on quality control, was suggested.

Responsiveness to stakeholders' current and emerging needs

Overall, the CCJS is fulfilling jurisdictional needs and priorities for crime and justice statistics, although some gaps were identified. The results of the user survey indicate that 49% of respondents (71 out of 144) believed that the CCJS completely or to a great extent covers the issues of most importance, while 47% (68 out of 144) indicated it does so to some extent.

Key informants noted that CCJS data play a key role in developing policies, planning for programs and services, understanding how provinces and territories compare with each other and nationally in terms of crime statistics and trends (for research and performance measurement purposes), conducting research for internal or external briefings, and fulfilling government needs to keep the public informed on justice issues.

The majority of respondents to the user survey believed that justice statistical information responds to emerging trends, with 20% (18/90) of survey respondents indicating that it responds completely or to a great extent, and 64% (58/90) indicating that it responds to some extent.

Key informants identified several gaps in current or emerging information needs:

• Data linking and relationships: Some key informants would like more linking or connectivity between criminal and family/civil justice datasets (including police, courts and corrections) to understand how people are flowing through the justice system. Additionally, these key informants believed that linking justice data to non-justice datasets—such as health or hospital data—would enable studies of the interaction between social issues and justice system involvement, and outcomes for individuals.
• Costing and justice system resources: Some key informants wanted more data on justice system costs, which would allow jurisdictions to better understand the financial cost of an individual going through the various components of the justice system. This could include costs of police activities, resources, expenditures and personnel for the Crown prosecutor service, such as the information previously collected through the Courts Resources, Expenditures and Personnel Survey, or downstream costs related to maintenance enforcement. Justice system spending was also identified as an emerging issue by survey respondents.
• Police data: Some key informants reported that there are gaps in data on police because certain police activities (e.g., calls for service that do not involve criminal charges) are not tracked through the Uniform Crime Reporting (UCR) Survey, even though these types of calls for service may represent the majority of police activity. As a result, there is a lack of data on police early intervention activities and an underreporting of police workloads. Respondents also wanted more information on CCJS national data by police jurisdiction (e.g., municipal or provincial police or RCMP), information on the outcomes of police diversion/cautioning activities, and national Crown counsel data on police charges received and charges approved.
• Better data from courts: Some key informants expressed a desire for improved data from various types and levels of courts, including mental health courts, domestic violence courts, drug treatment courts, civil courts and superior courts.

A few respondents also reported gaps in the following areas:

• Civil/family law: Key informants mentioned themes related to maintenance enforcement, custodial outcomes for children and self-represented litigants.
• Organized crime: A few key informants indicated that there is a gap in data for organized crime, which is described as complex and multilayered.
• Victimization: Victimization data should be available more frequently (other than the current five-year cycle), and more information should be available for smaller geographic areas and for vulnerable populations, such as Aboriginal communities.
• Aboriginal representation in the justice system: Aboriginal identity is not available in CCJS data, although the CCJS is working with partners to resolve these gaps. These data are needed to better understand the issue of overrepresentation in the justice system; to support research being undertaken by Justice Canada; and, more generally, to support the mandate of the current federal government.
• Release (bail) and remand: A few key informants reported that there is a lack of data on bail, remand and release to better understand trends in admission, length of time spent in remand, turnover rates and conditions imposed on bail.

A few other key informants identified gaps in information on cybercrime (fraud, victimization and underreporting), legal representation (use of legal aid, counsel or self-representation) and youth justice, and the need for an improved corrections survey and increased analytical capacity for corrections data. Some survey respondents also reported emerging needs related to data on crimes and offences as well as better information on demographic, geographic and social trends. In fact, 38% (51/135) of respondents to the user survey also reported dissatisfaction with the level of disaggregation of observations.

Evidence from the document review shows that stakeholders consulted a number of justice-related data and research topics they considered important through the Justice Information Roadmap (JIR) 2015^{Footnote 41} processes, with the most commonly mentioned including justice-related information on mental health, re-contact with the justice system, and family violence and victimization.

Canadian Centre for Justice Statistics initiatives to enhance data relevance

The CCJS has been taking action to fill some data gaps. The following emerging issues were considered in planning CCJS operations for 2014/2015 and 2015/2016:

• changes to legislation (Criminal Code updates)
• Canadian Association of Chiefs of Police (CACP) resolution to conduct data quality audits with police services
• economics of policing (including cost of services)
• access to justice
• data gaps (e.g., mental health, re-contact with the justice system).

Further, the CCJS is working on the following initiatives:

• Collaborative work has been underway on a pilot police performance metrics project. In 2015/2016, a feasibility study was launched to examine opportunities and challenges in collecting national-level data on calls for service.
• To understand the extent of the re-contact of individuals with the Canadian criminal justice system and their pathways through it, the CCJS is analyzing administrative data from police, courts and corrections facilities, and will be integrating these data with other sources under the Social Data Linkage Environment to help evaluate the sociodemographic characteristics of people who come into contact with the justice system. In 2016/2017, the agency will study new jurisdictions and will produce a set of standardized indicators and products on the topic of re-contact by 2017/2018.
• The CCJS launched a pilot project whereby participating police services focused on the quality of the organized crime data for a select number of UCR Survey violations. The goal of the pilot was to identify obstacles in processes or systems to increase data quality and to identify best practices in data reporting. As a result of that pilot project and subsequent consultations with selected police services, a number of best practices were identified for collecting and verifying police service data on organized crime. Recommendations were proposed to enhance data quality, such as raising the priority of collecting organized crime data, performing ongoing quality control, and identifying dedicated personnel with appropriate access to be responsible for incident reporting on an ongoing basis. Recommendations were endorsed by the Canadian Association of Chiefs of Police Board of Directors in 2016.^{Footnote 42}
• Over the course of the next two years, the agency will be redesigning various justice-related surveys to improve their relevance and to meet new priorities in the area of policing and the administration of justice (the Integrated Criminal Court Survey, the Uniform Crime Reporting Survey, the Police Administration Survey, the Homicide Survey, the Transition Home Survey, the Integrated Correctional Services Survey, and the Legal Aid Survey).^{Footnote 43}

Some key informants also indicated that the CCJS is fulfilling emerging needs by being innovative; exploring ways to improve data on specific justice issues, such as organized crime, calls for police service (non-criminal), and Aboriginal victims and offenders; and developing a better understanding of the relationships between mental health, addiction and substance abuse, and the criminal justice system. A few informants specifically praised the re-contact study in Saskatchewan for linking police, court and corrections data to better understand the nature, prevalence and frequency of re-contact among a cohort of individuals in this province.

Summary

The evaluation explored the extent to which CCJS data are being used to inform debate, research and analysis, as well as decision making. The scope of issues covered by the program, combined with the fact that this information is readily available, free of charge, from the Statistics Canada website, creates numerous possibilities for data use. Consequently, while the evaluation findings shed some light on this matter, it is beyond the scope of this evaluation to capture the full range of potential uses of statistical information.

Who uses Canadian Centre for Justice Statistics program data, for what purpose, and how?

Based on the review of Statistics Canada's administrative databases, program documents and literature, the bibliometric and webometric assessments, surveys and interviews results, the evaluation identified a variety of uses and users in various areas of activity from the public and private sectors.

Findings suggest that the primary users of information produced by the CCJS are the Department of Justice and Public Safety Canada, together with provincial and territorial jurisdictions for policing, courts and corrections. These users draw on the information to draft legislation and to develop policies, programs and services for Canadians in the areas of justice and public safety, as well as to prepare statistical reports such as the Trafficking in Persons annual report. Other key federal departments include Status of Women (particularly on issues related to victimization) and Immigration, Refugees and Citizenship Canada (on issues related to hate crime).^{Footnote 44}

Results of a webometric analysis of links to CCJS products also suggest that CCJS information is used by the private non-profit and private sectors, including media organizations, and some international organizations such as the Virtual Knowledge Centre to End Violence Against Women and Girls (a program of the United Nations Entity for Gender Equality and the Empowerment of Women), as well as the Organization for Security and Co-operation in Europe. According to the results of the interviews and a survey of users, justice data sources are used for a variety of purposes, including conducting research and policy analysis, informing program and service delivery, performing jurisdictional comparisons and understanding national trends, producing internal publications, and responding to media questions.

World Wide Web

Citations in peer-reviewed scientific literature

According to the bibliometric assessment conducted as part of the evaluation, a total of 38,222 citations referring to Statistics Canada in peer-reviewed articles were found between 2000 and 2014, of which 936 (2%) refer to CCJS releases.

References to CCJS releases for the periods from 2000 to 2006 and from 2008 to 2014^{Footnote 46} grew more than fourfold. However, even this impressive rate of increase is still below the rate of growth for the number of citations to Statistics Canada documents overall (more than fivefold) for the same comparison period.

The document types most cited for the CCJS are publications (788, or 84%), surveys (75, or 8%), The Daily (39, or 4%), CANSIM tables (29, or 3%), and summary tables (5, or 1%). The two most cited publications were Family Violence in Canada: A Statistical Profile (45%) and Juristat (30%).

Survey respondents were asked about the contribution of CCJS information to the understanding of justice issues and decision making. Most respondents agreed (131/156; 84%) that the information produced by the CCJS allowed them to better understand population justice trends and issues in Canada (7% disagreed; 9% didn't know). Evidence from the survey also confirmed the contribution of CCJS products to informed decision making; 72% (112/156) of CCJS users surveyed reported that CCJS products helped them make more informed decisions (10% disagreed; 18% didn't know).

Overall, CCJS products are used by a variety of users to meet their needs related to research, policy development, planning and reporting; to facilitate an understanding of justice issues; and to make informed decisions.

1. Policing Services Program:

Subject-matter lead for information related to the policing sector of the justice domain and responsible for the following surveys:

Uniform Crime Reporting (UCR) Survey

The UCR Survey is an annual administrative data survey collecting police-reported crime data from police services, who are the survey respondents, across Canada since 1962. The microdata version of the survey (UCR2), covering over 99% of police-reported criminal incidents in Canada, captures information on each incident (e.g., violation, location, weapons involved) and the victim and the accused (e.g., age, sex, victim–accused relationship, level of injury to the victim). A few small police services provide only aggregate information to theUCR Survey. Many statistical indicators are produced from the UCR Survey, including the annual crime rate, Crime Severity Index (CSI), trends of the rate and the CSI, number of clearances (i.e., number of criminal incidents solved by the police), number and rate of charges, and detailed information on violation types. Information on police-reported hate crime and cybercrime has been included in the UCR Survey data collection since 2004.

Homicide Survey

This annual administrative data survey has collected police-reported data on the characteristics of all murder incidents, victims and accused persons since 1961 and all homicides (including murder, manslaughter and infanticide) since 1974. The survey produces data on annual homicide rates, the methods through which the homicide was committed, the relationship between the accused and the victim(s), and some characteristics of the accused and victim(s).

Police Administration Survey (PAS)

The PAS is an annual survey. It collects summary, aggregated administrative data from each police service on the number of police officers and civilians by rank, sex and major function, as well as high-level operating expenditure data. Through funding from the Police Sector Council, a new, enhanced police personnel survey was developed and first implemented for collection of 2012 data. The new survey captures information such as years of service, number of officers by age and number eligible to retire.

Transition Home Survey (THS)

The THS is a biennial administrative data survey. Every two years, the survey collects information on all residential agencies providing services to abused women and their children across Canada. It provides information on residential services for abused women and their children during the previous 12 months, as well as a one-day snapshot of the clientele being served on a specific date.

Victim Services Survey (VSS)

The VSS was a biennial aggregate administrative data survey collected through a cost-recovery contract with the Policy Centre for Victim Issues. The purpose of this survey was to collect information on victim service agencies that provided services directly to primary or secondary victims of crime during the 12-month reference period, as well as to provide a one-day snapshot of the clientele being served on a specific date. Funding for the VSS ended in 2012. At present, discussions are underway with a focus on exploring a redesign.

Document review

The document review consists of a review, synthesis and analysis of data from relevant documentation to provide a comprehensive understanding of the program and address the evaluation questions. The documents include, but are not limited to

• documents specific to the CCJS such as operational plans, agreements and MOUs, publications, special studies, Client Relationship Management System (CRMS) requests, stakeholder consultation reports, committee and federal–provincial–territorial consultation minutes, presentations, project/survey files, and other types of documents (such as news release inventories by the CCJS) related to this evaluation
• program- and policy-related documentation such as Memoranda to Cabinet, Treasury Board Submissions, strategic plans, the CCJS Performance Measurement Strategy, audit and evaluation reports (from Statistics Canada or other departments), as well as other documents that make reference to the CCJS (e.g., documents related to ministerial reviews, Speeches from the Throne, federal budgets, legislation, policy statements, OAG reports, mandate letters)
• targeted reports and publications from the federal government (e.g., Immigration, Refugees and Citizenship Canada; Indigenous and Northern Affairs Canada; Correctional Service Canada) and other Canadian institutions to serve as a line of evidence for all the evaluation issues and questions (this also includes publications by international partners, such as the United Nations Office on Drugs and Crime, to assess the quality of Statistics Canada's statistical information compared with international standards and identify potential alternatives).

Financial and administrative data review

The following sources, among others, were examined as part of the data review:

• financial documents and administrative documents/databases linked to the operations and activities of the CCJS, such as the financial request system (FRS)
• data linked to the publication and dissemination of CCJS products
• data related to the quality of the outputs produced by the CCJS
• information about data requests and support services provided to users by the CCJS
• information about the exposure of the statistical information in the traditional media and social media
• agreements and MOUs related to data sharing and cost-recovery activities.

This review is undertaken to provide a complete description of the program and its resources; assess the achievement of its outputs and outcomes, including its efficiency and economy; and inform the analysis and final reporting.

Key informant interviews

Forty-two interviews with 47 key informants were conducted to gather more in-depth information on the subjective perspectives of senior program managers as well as of key data users and stakeholders (within federal, provincial and municipal organizations) to complement, further explain and augment the validity of evaluation findings. The selection of participants was based on their expertise and role in the management and delivery of the CCJS.