Background

Since 1981, the federal, provincial and territorial deputy ministers responsible for the justice system in Canada have been working together, along with the Chief Statistician of Canada, on the National Justice Statistics Initiative (NJSI). The mandate of the NJSI is to provide information to the justice community and the public on the nature and extent of crime and victimization, as well as on the administration of criminal and civil justice in Canada.

Through the Canadian Centre for Justice Statistics (CCJS), a division of Statistics Canada, the efforts of the NJSI are directed toward the production of data to inform the legislative, policy, management and research agendas of federal–provincial–territorial partners, and to inform the public. The focus of the CCJS is the development, collection, integration and analysis of data that reflect trends in Canada and on the development of national- and jurisdictional-level indicators.

There are three statistical programs within the CCJS: the Policing Services Program, the Courts Program and the Correctional Services Program. Each program is responsible for the following flagship survey:

• Policing Services Program: The Uniform Crime Reporting (UCR) Survey collects data from police services across Canada on incidents, victims and persons accused of police-reported crime, including hate crimes.
• Courts Program: The Integrated Criminal Court Survey (ICCS) collects detailed microdata on every appearance in adult criminal and youth court, which are aggregated into charges and cases.
• Correctional Services Program: The Integrated Correctional Services Survey collects microdata on adults and youth under the responsibility of the federal and provincial–territorial correctional systems.

In addition to these three programs, CCJS also includes other groups and units:

• The Analysis Section is responsible for working with members of the Liaison Officers Committee of the National Joint Statistics Initiative (LOCNJSI) to undertake analysis on special topics of interest to the NJSI. This section also undertakes analysis on a cost-recovery basis for initiative partners and for other government departments on justice-related issues.
• The Data Access and Dissemination Unit works with program areas and other areas within Statistics Canada to increase access to CCJS data products, such as making justice data available through the research data centres.
• The Data Development Unit (formerly the Corporate and Special Projects Unit) is responsible for working with members of LOCNJSI on a multi-year project to examine re-contact with the justice system. This project will create a series of sector-specific indicators that will give policy makers and researchers information on the extent of re-arrest, re-conviction and re-involvement with the policing, courts and corrections sectors in Canada, as well as the capacity to analyze cross-sectoral factors that influence re-contact. It is also responsible for a number of other projects to maintain the relevancy of the reports produced by CCJS in future years.
• Outside of the work performed by the agency's Quality Secretariat, which exists to promote sound quality management practices for Statistics Canada program areas based on the agency's Quality Guidelines, the Data Quality Secretariat (DQS) was recently created within CCJS to carry out data-quality evaluations and to support its program areas in enhancing data-quality procedures. The intent of the DQS is to continue to evaluate the quality of data, assess key risks and document best practices for the various CCJS data programs; develop an implementation plan for addressing key areas of vulnerability with respect to data quality; and begin to implement recommended data quality initiatives.

The three CCJS programs rely on the reception of administrative data by justice partners, including policing services, courts and correctional facilities—referred to as 'data providers.' To ensure the quality of CCJS data and data products, a number of governance committees and working groups—comprising key CCJS personnel, data providers and other judicial stakeholders—have been established. For example, the LOCNJSI oversees the work of CCJS on behalf of the NJSI. The members of the LOCNJSI include departmental officials appointed by deputy ministers, the Statistics Canada director general responsible for CCJS, and a representative of the Canadian Association of Chiefs of Police. The committee provides insight into topics of interest within the community, reviews work-in-progress Juristats and other reports, and provides a communication channel between the data providers and CCJS to discuss data quality issues.

Audit objectives

The objectives of the audit were to provide the Chief Statistician and the Departmental Audit Committee with assurance that

• Statistics Canada has established an adequate governance framework to support the quality of Justice Statistics data
• effective control mechanisms have been established and are consistently applied to ensure the release of quality data in accordance with the agency's Quality Guidelines.

Scope

The scope of this audit included an examination of the adequacy and effectiveness of the quality assurance framework over Justice Statistics. Specific areas examined included whether

• governance, roles, responsibilities and accountabilities for the quality of data within CCJS are clear and well communicated
• management identifies and assesses the risks that may preclude the achievement of quality products
• effective control mechanisms have been established and are consistently applied to ensure the quality of data from data providers for the ICCS and UCR Survey, as well as for the quality of information released as part of the Juristat and The Daily release processes.

The scope of the audit work originally included an assessment of the activities related to the production of Justice Statistics products from January 1, 2014, to December 31, 2014. This scope was extended to also include an assessment of the activities related to the production of Justice Statistics products pertaining to the Courts Program from January 1, 2013, to December 31, 2013, as the Courts Program has not processed any data since 2013 because of its current transition to the Social Survey Processing Environment.

The audit criteria for this audit are presented in Appendix A.

Approach and methodology

The audit work consisted of a comprehensive review and analysis of relevant documentation, interviews with key management and staff, and testing to assess the effectiveness of processes in place.

The field work included a review and testing of CCJS processes and procedures in place related to the quality and accuracy of the Justice Statistics information.

The audit focused on the assessment of the key controls and processes established for two of the three key CCJS surveys: the UCR Survey and the ICCS.

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors' International Professional Practices Framework.

Authority

The audit was conducted under the authority of the approved Statistics Canada integrated Risk-Based Audit and Evaluation Plan 2014/2015 to 2018/2019.

Governance in support of the quality of Justice Statistics

Given the number of internal and external stakeholders, a robust governance framework is essential to ensure the quality of Justice Statistics. Roles, responsibilities and accountabilities for the quality of data within the Canadian Centre for Justice Statistics (CCJS), including the Data Quality Secretariat (DQS), should be clearly documented and well understood. As well, appropriate and effective oversight bodies should be established to monitor the quality of CCJS data, as it pertains to the internal and external environments of CCJS.

Roles, responsibilities and accountabilities are well understood within CCJS and by key external stakeholders; however, opportunities exist to improve the formalization of their roles, responsibilities and interdependencies

A key component of governance structures is the establishment of roles, responsibilities and accountabilities of stakeholders. The audit team noted that although the roles, responsibilities and accountabilities of the key personnel responsible for the quality of Justice Statistics seem appropriate and well understood within CCJS and by key external stakeholders, they are not formally documented. Rather, the only documentation outlining roles and responsibilities at the personnel-level are job descriptions, which are either generic and do not fully capture all of the roles and responsibilities of key personnel, or are out of date.

The governance structure for CCJS is unique in that it reports not only to internal stakeholders, but also to external stakeholders, through CCJS' membership in the National Justice Statistics Initiative (NJSI), which works with CCJS to produce useful information to inform the legislative, policy, management and research agendas of federal–provincial–territorial partners, and the public. These external stakeholders are important partners in ensuring the quality of CCJS' data and data products. As a direct result of the NJSI, several governance committees have been established to monitor the quality of CCJS data. These governance committees include the Liaison Officers Committee of the National Justice Statistics Initiative and the Police Information and Statistics (POLIS) Committee (a subcommittee of the committee for the Canadian Association of Chiefs of Police) and include representatives of the relevant CCJS programs and external stakeholders, such as government and data-provider representatives. The committees' membership structure fosters a consistent understanding of data product quality expectations, data definitions and jurisdictional data quality requirements, among others. Although formal terms of reference have been developed for the various governance committees, there is no formally documented accountability framework outlining the various roles, responsibilities and accountabilities of all governance committees, including their reporting structure and other interdependencies. Since many of the key personnel within CCJS have been in place for some time and have considerable organizational knowledge, there has not been a perceived need for documenting this.

However, because CCJS does have a unique and interlinked governance framework that includes oversight at both the departmental level and oversight through the various external partner organizations, there is the potential for overlap or gaps in coverage without a formal accountability framework in place. This would also provide a mechanism to ensure that key personnel and external stakeholders are aware of their responsibilities and accountabilities regarding data quality.

The roles, responsibilities and accountabilities for the CCJS' Data Quality Secretariat have not yet been formally defined and communicated

The DQS was created in 2014 for CCJS to address quality concerns stemming from the fact that specific Juristats^{Footnote 1} had to be retracted because of data errors. The DQS, which is part of CCJS' Data Access and Dissemination Unit, is described in the 2015/2016 CCJS Operational Plan as a "new secretariat [that] has been formed to carry out data quality evaluations and to support program areas in enhancing data quality procedures."

Although the DQS has been working on the completion of a number of quality reviews within CCJS, their role, responsibilities and accountabilities outside of this type of reporting have not yet been well-defined or documented. As well, interviewees noted confusion regarding the role of CCJS' DQS versus that of the agency-wide Quality Secretariat, which recently performed a quality review of CCJS' Courts Program. According to the Statistics Canada intranet, the agency's Quality Secretariat is dedicated to supporting the development and implementation of policies and procedures that promote sound quality management practices, designing and managing studies related to quality management, and providing advice and assistance to program areas on quality management.

The DQS currently comprises only the chief of the unit, and no other positions within that unit have been staffed. In the interim, the work undertaken by the unit is currently being performed by the chief and representatives from the Data Access and Dissemination Unit.

Without the formal documentation of the expected role and responsibilities of the DQS, there is an increased risk that they may not be meeting the expectations of both senior management and stakeholders in improving data quality throughout the CCJS division. Additionally, there is an increased risk of duplication of efforts, especially with that of the agency's Quality Secretariat. The development of an accountability and governance framework, as mentioned above, would provide an opportunity to outline the roles and responsibilities of the DQS, as well as to formalize how the DQS' role differs from that of the agency's Quality Secretariat.

Risk management in support of the quality of Justice Statistics

An effective risk management framework needs to be established to identify, assess, mitigate and monitor the potential and emerging risks that may affect the quality of the Canadian Centre for Justice Statistics' (CCJS') data and data products.

No formal risk management process exists at either the CCJS divisional or individual program level to identify and assess risks posed to the quality of data

CCJS management participates in the agency's risk management exercise, which is to periodically identify and assess the high-level risks facing the achievement of their objectives as part of the Corporate Risk Profile exercise. However, no formal risk management processes have been established to regularly identify and assess CCJS' divisional or program-level risks.

Although various documents outline high-level risks facing the CCJS division, the detailed risk register for the division has not been updated since March 2012. The chiefs and the director of CCJS are in the process of reviewing and updating the risk register to reflect the risks currently facing the three CCJS programs; however, at the time of the audit, this exercise had not been completed.

The audit further noted that, although certain risks and issues are being informally mitigated and monitored for each program, no formal process has been established for developing divisional or program-level risk mitigation strategies and their monitoring. The previous risk register, dated March 2012, provides a rating for the effectiveness of current mitigation strategies; however, the risk register does not include specific information on what these mitigation strategies are or how they will address the identified risks. Additionally, no documentation has been maintained as evidence of the ongoing monitoring of the identified risks and the effectiveness of the risk mitigation strategies.

Without a formal process in place for the systematic and ongoing identification and assessment of the risks to the quality of CCJS data and its related products, there is an increased possibility that these potential risks will not be appropriately escalated and mitigated in a timely manner by the (identified) appropriate owner of the risk or risks.

Control mechanisms over data processing

The Statistics Canada Quality Guidelines note that when using administrative data, such as those that are relied upon by the Canadian Centre for Justice Statistics (CCJS), the key quality indicators are that data are relevant, accurate, timely and coherent. Controls over the processing of the data received from data providers ensure that the data meet these quality indicators. Adequate guidance and training should be provided to data providers to support the collection of reliable, complete, accurate and timely data and channels should be established for data providers to communicate any changes to their systems and processes or any changes to administrative data definitions that affect the data provided to CCJS. Effective data cleansing and verification processes should also have been established to ensure the reliability and relevancy of the data collected, and an effective validation process should have been established to confirm the reliability, completeness and accuracy of the data provided.

Adequate guidance and training is provided to data providers to support the collection of reliable, complete, accurate and timely data

In conjunction with data providers, CCJS has developed National Data Requirements for each of the surveys reviewed. They outline the approved data definitions and parameters for each data field required for both the Uniform Crime Reporting (UCR) Survey and the Integrated Criminal Court Survey (ICCS), and are relied on by data providers and program staff to ensure a consistent understanding of the data definitions. In addition to the National Data Requirements, the Policing Services Program has also developed a UCR Incident-Based Survey Manual to provide additional guidance to data providers in the discharge of their responsibilities relative to the quality of the data they provide.

Because of the significant number of data providers (over 100 covering over 1,000 policing service detachments) they rely on, the Policing Services Program has further developed standardized training, which is available to all data providers, and created the role of National Training Officer, whose main responsibilities include providing guidance, support and training to data providers.

Given the limited number of data providers and the variety of information systems used to collect and transfer data to CCJS, the Courts Program has not developed standardized training for data providers outside of the National Data Requirements. Rather, the program relies on their ongoing relationship with their 13 data providers and their participation on the Liaison Officers Committee of the National Joint Statistics Initiative (LOCNJSI) Working Group on ICCS Data Quality to provide ongoing guidance and support to data providers.

The audit noted that both the Policing Services Program and the Courts Program have established adequate channels for data providers to communicate any changes to either their systems and processes or administrative data definitions that affect the data provided to CCJS. Specifically, the audit noted that both programs have mechanisms in place to foster strong working relationships with data providers through the LOCNJSI, the Police Information and Statistics Committee and various subcommittees of each, as well as through ongoing contact with individual data providers.

Adequate data cleansing and verification processes have been established to ensure the reliability and relevancy of the data collected; however, verification practices between programs varied

Since both the UCR Survey and ICCS rely solely on the reception of administrative data from data providers, there is a need for the effective cleansing and verification of the data received through the use of edit and imputation programs. Data cleansing and verification processes help detect and correct (or remove) corrupt, duplicate or inaccurate records from a record set. For both the UCR Survey and ICCS, edit and imputation programs have been developed in conjunction with Statistics Canada's Methodology Branch to achieve the level of quality required for the purpose of the data.

Specific to the UCR Survey, two types of data providers exist. They are

• UCR1: Respondents are unable to provide digital microdata because of systems that are incompatible with CCJS' systems, and instead provide aggregate counts by paper to CCJS
• UCR2: Respondents generally use one of two data management systems that are compatible with CCJS' systems, and provide microdata to the program via a data extract from their system.

In 2011, to leverage the expertise within Statistics Canada, the agency's Operations and Integration Division (OID) started playing a role in processing both UCR1 and UCR2 data. Specifically, OID is responsible for the data entry of the aggregate counts received by paper by UCR1 respondents and the processing of the data received by UCR2 data providers.

Based on a sample of 5 UCR1 and 10 UCR2 selected respondents, it was noted to be a general practice that the data be manually captured by a representative from OID. Although the audit team noted that some controls have been embedded within the data entry process, such as batch total checks, no peer or supervisory data entry verification is being performed. Given that UCR1 respondents represent only a small percentage of data providers (approximately 4%), the conduct of a peer or supervisory data entry verification could help decrease the risk of data integrity issues going undetected.

UCR2 data are processed using a SAS-based edit and imputation program. This program performs data cleansing and verification edits and imputations, and highlights fields that fall outside of expected parameters for verification purposes. The processing system will then generate reports logging any edits, imputations or warnings resulting from the processing of the data, called an Edit and Imputation (E & I) report. Because deletions, edits and imputations performed on the data records can affect the overall quality of the data, the Policing Services Program indicated that follow-up with data providers is deemed critical in addressing certain potential data integrity issues.

The audit team was able to review evidence of the conduct of follow-up activities performed by the program and OID; however, varying levels of documentation were provided for 7 out of the 10 E & I reports sampled, therefore follow-up activities are not being consistently captured and a standardized process or template for UCR analysts to track the status of these follow-ups with data providers has not yet been developed. Further, there is currently no requirement for management within the program to review the actions undertaken by the analysts when conducting this follow up with data providers.

Without the consistent and formal tracking of follow-up activities undertaken with UCR2 data providers based on the results of the E & I reports generated by the processing system, and without the review by CCJS program management of the activities undertaken by the analyst to address data issues, there is an increased risk that follow-up activities may not be performed or issues not remediated per expectations.

The ICCS data, similar to UCR2, are received electronically in the form of microdata, and then processed through a SAS-based program. Again, similar to the UCR2 data, reports are automatically generated by the system to provide CCJS with a summary of the edits and imputations performed by the program and any subsequent warnings that are flagged.

When deemed material, analysts are following up on the edits, imputations and warnings performed or flagged by the system with the ICCS data providers. The program indicated that, historically, deleted or imputed records within ICCS data have represented only a small percentage of received data records. The audit team noted, however, that no formal parameters (i.e., threshold numbers or percentages) have been established for the Courts Program team responsible for the ICCS to trigger when processing edits, imputations, warnings or trends should be investigated further. Rather, the need for follow-up activities is left at the discretion of the responsible analyst, based on the review of a tracking spreadsheet of the number and type of edits, imputations and warnings that were generated by the system for each data provider in past years.

Without the establishment of formal parameters (numbers and/or percentages) for the investigation of ICCS processing edits, imputations, warnings and trends, there is an increased risk that significant data integrity issues may not be investigated or key data may be deleted from the population, affecting the quality of data products.

An effective validation process has been established to confirm the reliability, completeness and accuracy of the data provided

To improve the reliability, completeness and accuracy of the data, CCJS requests that data providers validate their data sets annually before the production of Justice Statistics publications. As part of the process, CCJS will prepare data tables outlining the data provider's complete data set for the year, along with explanations of the tables provided and their significance. Data providers are then given a period of time to review the data packages and to provide the applicable CCJS program with any corrections or explanations for unexpected variances. Once these changes are made, data providers are expected to sign-off on the data set.

The audit team noted that data providers are consistently signing off on the reasonability of the data, and when possible, its accuracy and completeness. Based on a sample of 18 data providers for the UCR Survey and ICCS, all of the annual validation packages were found to be signed off for the year in question.

Control mechanisms over the production of Justice Statistics products

Effective control mechanisms over the production of Justice Statistics data products help enable the Canadian Centre for Justice Statistics (CCJS) to meet stakeholders' quality expectations. In accordance with the Statistics Canada Quality Guidelines, it is critical to have controls in place to ensure that the data released by CCJS are relevant, accurate, timely and coherent. To achieve this, an effective review process should be established to identify and address potential data errors in data products in a timely manner. Additionally, an effective framework should be established to ensure that any data errors identified post-review are also communicated and addressed in a timely manner.

An effective review process has been established to identify data errors within draft data products in a timely manner; however, evidence of the conduct of the internal and external reviews is not being consistently maintained

Once all data providers have signed-off on the data sets, both the Courts Program and the Policing Services Program use these validated data sets to create data products (via Juristat and The Daily). CCJS program analysts are responsible for creating the data products and, once completed, the data products undergo a robust internal review process, which includes (1) a peer review whereby all of the data are recalculated and analyzed; (2) a review by the survey manager and program chief, which includes questioning the analysis undertaken by the analyst and any unexpected data trends; and (3) reviews by the chiefs of the other CCJS programs and the director or director general for overall reasonableness of the data and consistency of their appearance with other CCJS data products.

In addition to the internal review process, to improve the reliability of the data reported, the data products are also submitted to external stakeholders, including data providers for review and validation. Specifically, the Integrated Criminal Court Survey (ICCS) Juristat is submitted to the associated liaison officers (LOs) and the Uniform Crime Reporting (UCR) Juristat is submitted to Police Information and Statistics Committee representatives and the representing LOs to validate that the information enclosed in the publication appears accurate.

Although the audit team is satisfied with the level of internal and external reviews, it was noted that an inconsistent level of documentation is maintained to demonstrate the conduct of the internal and external reviews, including how issues and errors have been addressed and resolved.

No internal formal processes have been established to communicate and remediate data errors identified post-review (pre- and post-release)

The processes to be followed in the event of the identification of an error post-review (both pre- and post-release) have not been formally documented. Rather, the audit noted that informal processes are being followed when an error is identified internally or communicated by external stakeholders. Specifically, Statistics Canada's Dissemination Division has developed an error log to track the errors discovered in released publications. This error log includes the results of an assessment of the error's impact, if the data have been corrected, and the reload date of the publication. However, the audit noted that for identified errors, inconsistent documentation is maintained to indicate how these errors were communicated, documented and addressed. For some identified errors, a communications plan was developed and maintained to describe the error and its impact, as well as to compare the original data to the revised data and provide a strategy for correcting the error and communicating the impact to the applicable stakeholders. However, for other errors, no documentation evidencing the internal communication and escalation of the issue was maintained.

Once errors have been corrected and publications have been reloaded on the Statistics Canada external website, an explanation note is included on the website stating the nature of the error that has been corrected and the date the publication was reloaded.

The audit also noted that CCJS is addressing and communicating errors identified post-review, and that the Statistics Canada Directive on Corrections to Daily Releases and Statistical Products is followed when addressing or escalating errors identified on production day, as well as post-release.

Appendix A: Audit criteria

Appendix B: Acronyms

Background

Health Statistics Division (HSD) at Statistics Canada has the mandate to provide accurate, timely and relevant information about the health of Canadians. HSD provides statistical information about the health of the population, the determinants of health, and the scope and utilization of Canada's health care resources. This information is used to assist and support health planners and decision makers at all levels of government, to sustain demographic and epidemiological research, and to report to the Canadian public about their collective health and health care system. HSD works in partnership with provincial and territorial vital statistics registrars and cancer registries as well as data providers and users at the federal level (Health Canada and the Public Health Agency of Canada), provincial level (provincial ministries of health), and regional level (health regions).

To achieve its mandate, HSD enters into statistical data-sharing agreements (DSAs) with other organizations under the authority of sections 11 and 12 of the Statistics Act. These agreements cover nearly all of the business surveys and a majority of household surveys, and enjoy certain exceptions regarding the release of confidential respondent information either with or without the respondent's consent, provided that the legal requirements for the provision of data-sharing information, consent rights and confidentiality protection are respected by all parties. In general, data sharing for statistical purposes occurs when statistical and information inquiry is initiated by joint survey partners, or where a common data resource is equally and jointly owned by two or more partners. Data sharing is exercised when there are significant reductions in response burden and compliance costs for data-sharing partners, as well as improvements in statistical data accuracy, coverage, relevance and timeliness.

In recent years, DSAs have become a key business process because ensuring confidentiality and protection of data can pose challenges. Currently, Statistics Canada has two agreements with Manitoba Health covering health surveys, under the authority of section 12 and Sub-section 17(2) of the Statistics Act. Health surveys for the Canadian Community Health Survey (CCHS), National Population Health Survey (NPHS) and Survey on Living with Chronic Diseases in Canada (SLCDC) are included in the first agreement. The second agreement is specific to the disclosure of information on the nutrition component from the CCHS.

Through its mandate, the CCHS program collects information related to health status, health care utilization and health determinants for the Canadian population. The first component of the CCHS program is an annual survey that relies upon a large sample of respondents and is designed to provide reliable estimates at the health region level. The second component focuses on specific health-related topics such as nutrition, mental health or healthy aging and is conducted approximately every three years. The uniqueness of the annual survey arises from the regional nature of both content and survey implementation.

The NPHS is a longitudinal survey providing unique information about the health of Canadians. The final cycle of this survey covered the period of 2010/2011. This survey, which was performed every two years, consisted of the same individuals providing current and in-depth information on their physical and mental health status, use of health care services, physical activities, life in the workplace and social environment. It collected information related to the health of the Canadian population and related socio-demographic information. The last release will provide researchers access to nine cycles of Canadian longitudinal health data to examine the dynamics of population health from 1994/1995 to 2010/2011.

The SLCDC is a cross-sectional survey sponsored by the Public Health Agency of Canada that collects information related to the experiences of Canadians with chronic health conditions. The SLCDC takes place every two to three years, with two chronic diseases covered in each survey cycle. The objectives of the survey are to

• assess the impact of chronic health conditions on quality of life
• provide more information on how people manage their chronic health conditions
• identify health behaviours that influence disease outcomes
• identify barriers to self-management of chronic health conditions.

The last survey was performed in 2014.

The data are used extensively by the research community and other health professionals. Federal and provincial departments of health and human resources, social service agencies and other types of government agencies use the information collected from the respondents to plan, implement and evaluate programs to improve health and the efficiency of health services. Non-profit health organizations and academic researchers use the information for research on ways to improve health.

Audit objective

The objective of the audit is to provide assurance to the Chief Statistician and Statistics Canada's Departmental Audit Committee that the terms and conditions of the data-sharing agreements between Statistics Canada and Manitoba Health are met.

Scope

The scope included an examination for compliance to the terms and conditions (T&Cs) prescribed in the DSAs to ensure that confidentiality of information and the sensitive nature of the information collected are protected. The audit focused on the confidentiality and security (physical access, IT storage and transmission, physical storage, information copying and retention, and record management) safeguards at Manitoba Health to ensure that data are protected and confidentiality is maintained.

The scope of the audit included examining all third-party agreements and contracts entered into by Manitoba Health since the signing of the two new DSAs.

Approach and methodology

The audit work consisted of an examination of documents, interviews with key senior management and personnel, and a review of compliance with relevant policies and guidelines. (For details, see Appendix A: Audit criteria.)

The field work included the following:

• a review and assessment of the processes and procedures outlined in the T&Cs of the DSAs with Manitoba Health, with emphasis on whether the security requirements are in place and complied with, and confidentiality of data is maintained
• testing of system application controls and authentication, and access procedures
• review of the agreement with the only third party recipient of Statistics Canada data at the time of the audit.

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing.

Authority

The audit was conducted under the authority of the approved Statistics Canada integrated Risk-Based Audit and Evaluation Plan 2014/15 to 2018/19.

Control environment for the management of the data-sharing agreements

Authorities, responsibilities and accountabilities should be clearly defined and understood at all levels to support effective management of the terms and conditions (T&Cs) of the omnibus data-sharing agreements (DSAs). Monitoring of practices as outlined in the T&Cs of the omnibus DSAs should be in place to detect unwanted disclosures that would otherwise increase operational risk.

Authorities are defined

Statistics Canada exercises its mandate to enter into statistical DSAs with other organizations under the authority of sections 11 and 12 of the Statistics Act. The Directive on Data Sharing under sections 11 and 12 sets out the roles and responsibilities for the development, implementation and monitoring requirements of DSAs. The directive notes that Information Management Division (IMD), in consultation with Legal Services, is responsible for drafting data-sharing agreements when requested from directors of statistical programs. IMD is also required to support managers during the development of new or modified data-sharing agreements with receiving parties, pursuant to section 12 of the Statistics Act. Subject-matter divisions are responsible for communicating with recipient organizations during the negotiations and drafting the agreements.

Roles and procedures related to the management of Statistics Canada data are clearly established at Manitoba Health but need to be documented

Statistics Canada confidential information is managed by two branches within Manitoba Health—the Health Information Management (HIM) Branch and the Information Systems Branch (ISB)—as follows:

• The executive director of HIM Branch at Manitoba Health has overall responsibility for the life cycle of Statistics Canada data. The manager of Management Information Systems, Data Management and Development, who reports to the executive director, HIM Branch, is the data custodian responsible for managing the DSAs and the Statistics Canada data.
• The database administrator, who reports to the data custodian, performs the duties of data handling, storage, controlling access to the UNIX server where the Statistics Canada data are stored, and preparing the transmission of Statistics Canada to be sent to third parties.
• Two senior statistical analysts reporting to the director of Analytics and Research within the HIM Branch are responsible for analyzing Statistics Canada data and sometimes providing aggregate data to other internal departments as required.
• The ISB of Manitoba Health is responsible for the maintenance and security of the UNIX server.

HIM Branch at Manitoba Health has been a stable group for a number of years. There has not been any significant employee turnover in the functions related to the management of Statistics Canada data and employees understand their roles. However, these roles, responsibilities and procedures have not been formally documented. While this lack of documentation has not proven to be a problem to date, the audit revealed that the data custodian announced that she will retire in the coming year. Therefore, knowledge transfer and documentation of roles and responsibilities will be a key success factor before her retirement.

Responsibilities of the data custodian prescribed in the DSAs have been implemented but a confidentiality document needs to be signed

As stipulated under Appendix C of the DSAs, the data custodian is responsible for the following three key requirements:

1. preparing a Confidentiality Document and ensuring that all individuals who will access the Statistics Canada data sign it
2. maintaining a register of data files received from Statistics Canada
3. maintaining a register of all individuals who have been granted access to the data files.

Appendix C stipulates that the data custodian will "prepare a document for the use of the Receiving Party's employees and contractors, outlining the T&Cs governing the use of the Information, as well as the procedures to send, receive, handle and store the Information (hereinafter the "Confidentiality Document")." Prior to granting access to Statistics Canada information, the data custodian must ensure that every employee and contractor who will access the Statistics Canada data has agreed in writing to comply with the terms of the DSAs by signing and acknowledging that they have read, understood and agree to comply with the T&Cs of the DSA as highlighted in the Confidentiality Document.

The audit revealed that a Confidentiality Document, which outlines the T&Cs governing the use of the information, exists and has been signed by all employees who have been granted access to the Statistics Canada information, with one exception: the data custodian had incorrectly assumed that certain documents signed with Statistics Canada at the time of becoming the data custodian replaced the need for a signed Confidentiality Document.

The register of data files received from Statistics Canada and the register of access granted to Statistics Canada data files have been implemented and are in use at Manitoba Health.

The DSAs' compliance requirements were only recently communicated to the relevant staff

The ISB of Manitoba Health is responsible for the maintenance and security of the UNIX server where the Statistics Canada data are stored. ISB must abide by Manitoba Health's overall security requirements for restricted information; this reduces the risk of non-compliance with the security requirements of the DSAs. The ISB employees were only recently informed of the existence of the DSAs and the security requirements stipulated therein, which could result in a potential breach of the DSAs going unnoticed.

Processes exist for activating access to the UNIX server and for sharing aggregate data with internal departments within the HIM Branch

Interviews revealed that the creation of a new UNIX account for new employees is approved by the executive director, HIM Branch, and requested from ISB by the database administrator in writing. This was not tested as there has not been a new employee requiring access to UNIX in several years. Once a new UNIX account is created by ISB, the database administrator has the capability of activating or disabling employees' access to the UNIX server. All access to the UNIX server is logged by the database administrator.

Primary users of the Statistics Canada data within the HIM Branch are two senior statistical analysts. A review of the access log for the past two years indicated that access was granted to the analysts only three times during that timeframe. The audit revealed that when the analysts request activation of their access to the UNIX server, the request is made verbally directly to the database administrator without the need for a supervisor's written authorization. Because of the close proximity of their offices, the analysts simply walk across the hall to the database administrator's office and request access. While not necessarily a requirement of the DSAs, it is prudent practice to implement more formal controls for authorizing access to the Statistics Canada data.

The purpose of the analysts' access to the Statistics Canada data on the UNIX server is to analyze and provide aggregate data internally within Manitoba Health, such as when meeting its mandate to participate in the Community Health Assessments that take place every five years. Interviews revealed that the analysts will sometimes ask a colleague or their supervisor to review the data before they send it internally via email, and the supervisor is copied on the email. However, there is no formal requirement for the analysts to obtain a supervisor's approval before the aggregate data are sent. While not a requirement of the DSAs, it would be a prudent practice to obtain a supervisor's written authorization before the distribution of aggregate data. A review of an aggregate data report provided internally indicated that there were no personal identifiers.

Manitoba Health's third party agreements include a clause for inspections or audits

Clauses with respect to monitoring are prescribed by Statistics Canada in the DSAs. The DSAs prescribe that third party agreements entered into by Manitoba Health "shall contain a clause stipulating the right of Statistics Canada or the Receiving Party to review compliance with the terms of this Agreement."

Paragraph 6.2.4 of the DSAs allows Manitoba Health to share information with the Manitoba Centre for Health Policy (MCHP) at the University of Manitoba. MCHP is not a legal entity in its own right but is a research unit of the University of Manitoba and the only third-party recipient of Statistics Canada data.

In November 2006, Manitoba Health signed a contract called "Information Sharing and Protection of Privacy Agreement" with the University of Manitoba, allowing Manitoba Health to share information with MCHP. MCHP is intended to support Manitoba Health in coordinating health research, planning, evaluation and monitoring by acting as the focus for a program of data analysis for researching, planning, monitoring and evaluating the provision of health care, emerging health issues and the broader determinants of health primarily in Manitoba.

In addition to the above contract, in October 2014, Manitoba Health and the University of Manitoba signed a letter of agreement specifically for data transfer to and use of Canadian Community Health Survey data by the MCHP. The letter of agreement requires MCHP to comply with the confidentiality and security requirements of the DSAs signed between Manitoba Health and Statistics Canada in February 2014; these requirements are attached as appendices to the letter of agreement.

The Information Sharing and Protection of Privacy Agreement between Manitoba Health and the University of Manitoba includes an audit clause that allows Manitoba Health access to MCHP's premises and the authority to inspect, review or audit MCHP's records or information, privacy practices, policies, procedures or security arrangements. However, Manitoba Health has not established a plan to exercise its right to inspect or audit MCHP's compliance.

Data stewardship

Internal protocols and controls for the sound management of data should be in place to ensure the protection and safeguarding of Statistics Canada health survey information over the full life cycle of the information.

Processes are in place and monitored to fulfill the requirements stipulated in the data-sharing agreements

The audit revealed that processes, although not formally documented, are in place at Manitoba Health to fulfill the requirements stipulated in the data-sharing agreements (DSAs), and that employees in the Health Information Management (HIM) Branch understand and comply with the processes for the receipt, storing, handling and transmission of Statistics Canada data. These are longstanding employees who are very familiar with their duties.

Data files are sent by Statistics Canada via electronic file transmission (e-FT) directly to the data custodian at Manitoba Health. The files are password protected and encrypted during transfer. Once a data file is received from Statistics Canada, the data custodian is notified that a file is in the e-FT vault and requests a password from Statistics Canada to access the file. Then the data custodian sends Statistics Canada an acknowledgement of file receipt and updates the register of all files received from Statistics Canada.

The data file is downloaded by the data custodian onto a secure network drive in a separate Statistics Canada folder designated for the HIM Branch. The database administrator accesses these files from the network folder and saves them in the UNIX database where all the Statistics Canada data are maintained. The database administrator is the only one who can activate access to the UNIX server and he or she maintains a register of all access granted to employees.

Third party sharing of Statistics Canada data

All information shared with external parties must be approved by Manitoba Health's data custodian, who physically arranges for the transmission of Statistics Canada data via encrypted CD to the Manitoba Centre for Health Policy (MCHP) of the University of Manitoba. The database administrator zips and password protects the files on an encrypted CD. The data custodian then arranges for the delivery of the CD to MCHP via bonded courier. Once the CD has been received by the contact at MCHP, he or she must contact Manitoba Health's data custodian to obtain a password to open the file.

Although under Paragraph 6.2.4 of the DSA Manitoba Health is allowed to share personal identifiers with MCHP, testing of the Canadian Community Health Survey 2012 share file sent to MCHP in 2013 (the only one shared in the past two years) revealed that there were no personal identifiers present.

A risk management process is in place at Manitoba Health to identify and monitor risks

The audit revealed that a risk management process has been implemented at Manitoba Health over the last two years. The Management Services Branch is responsible for providing the overall risk management framework to the organization and for supporting the various branches in preparing the risk management section of their annual work plans. The process entails each branch identifying its risks and the related action plans to mitigate those risks.

Each branch's executive director and reporting assistant deputy minister must sign a 'Risk Management Plan Accountability Statement to the Deputy Minister' attesting to the fact that the risk assessment performed has considered all categories of risks, appropriate action plans and mitigation strategies, and that the assistant deputy minister has been debriefed and will ensure that the branch will update its risk assessment cyclically.

Currently, Manitoba Health's risk assessment process is at the stage where each branch identifies its risks but they have not yet consolidated the risks from a corporate level standpoint. There is no risk management committee and no formal corporate risk management policy; this process is still in its early development. Management indicated that they are in the process of considering these tools and evaluating corporate-wide risks.

Management within the HIM Branch is proactive in identifying and assessing risks of non-compliance with the DSAs

The HIM Branch mitigated its risk of non-compliance with the security requirements of the DSAs by performing a thorough analysis of the updated requirements of the new omnibus DSAs and, where required, updated their internal operating procedures to ensure compliance with the terms and conditions of the DSAs. Furthermore, the HIM Branch requested that MCHP perform a similar analysis to assess whether they were also complying with the DSAs. A review of these two analyses showed how each organization met each of the security requirements of the DSAs.

Policies and ongoing training exist at Manitoba Health to mitigate exposure to risk

A key document used at Manitoba Health—one that outlines policies and procedures employees must follow under the Personal Health Information Act (PHIA)—is the PHIA Manual. This is a document of policies in use at Manitoba Health to protect sensitive personal information within the organization. For example, Policy VIII: Security of Personal Health Information lists the procedures and safeguards in place to protect personal information. Another is Policy IX: Corrective Procedures to Address Security Breaches Involving Personal Health Information, a policy on breaches. Through it, Manitoba Health, as a trustee under the PHIA, can address complaints and conduct investigations related to personal health information security breaches, in accordance with the requirements of PHIA and the Personal Health Information Regulation. The policy outlines the steps to be taken under the various types of breaches, how to proceed with a security breach investigation, and what forms to complete. If a breach occurs, a Security Breach Reporting Form needs to be completed by the branch involved and provided to the assistant deputy minister and the Legislative Unit, along with the findings of the investigation. Interviews revealed that there have not been any breaches related to Statistics Canada data.

The Legislative Unit of Manitoba Health provides guidance to the department in ensuring that it is in compliance with the many statutes for which it is responsible, including thePHIA. Mandatory PHIA training is provided to all new employees and refresher training to all existing employees every three years.

Physical and information technology security

Control and protection of information, either physically or electronically, should be executed in a manner that protects against loss, theft, compromise and improper disclosure. Access to the data should be given only to employees or contractors on a 'need-to-know' basis as part of their duties.

Logical access controls are in place at Manitoba Health

Testing of logical access controls with the database administrator revealed that a password was required to enter into the Manitoba Government network and a separate password was necessary to enter the UNIX server where the Statistics Canada data are maintained. Access to the UNIX server is restricted to a limited number of employees and access is disabled by the database administrator when staff members are not using the data. The register of user access is updated by the database administrator each time access to the data in UNIX is granted, as required under the data-sharing agreements (DSAs).

Access to the Statistics Canada shared folder in the Manitoba Health network drive should be restricted to those individuals who need access to the Statistics Canada data

When the data custodian receives a data file from Statistics Canada, the file is downloaded onto a temporary secure network drive in a designated Statistics Canada folder within the Health Information Management (HIM) Branch directory. The database administrator accesses the file from the folder and saves it in the UNIX database where all the Statistics Canada data are maintained. Once saved on the UNIX server, the database administrator deletes the data file from the folder in the network drive. Normally, the length of time between the files being placed temporarily in the network drive folder by the data custodian and their being deleted from the folder by the database administrator is no more than a day or two. During that time, however, the data portion is accessible to approximately 20 employees in the HIM Branch.

The audit testing revealed that two data files had not been deleted from the shared network folder and data were accessible to all individuals within the HIM Branch. Upon discovery during the audit, the two files were deleted from the shared folder. Access should be restricted to individuals who need to be involved with the Statistics Canada data and who have signed a confidentiality agreement.

Physical access is secure

Statistics Canada data resides in two locations at Manitoba Health—the server room within the Information Services Branch (ISB) where the UNIX server and Statistics Canada data are stored, and the HIM Branch where the data are accessed.

Physical access to the Manitoba Health premises is controlled by physical card access systems. Each secured area (ISB and HIM Branch) is protected by locked doors with card scanners. Staff must swipe their cards to enter the restricted area. Visitors must sign in at the security desk in the main lobby, obtain a visitor's card, and be escorted by authorized people at all times.

Interviews revealed that security guards are on duty weekly from 6:00 a.m. to 12:00 a.m. (18 hours) in the main lobby of the Manitoba Health building. Building security is responsible for monitoring building access and maintaining a register of all visitors. When building security is not on duty, the building is locked and only accessible to people with access cards. Access logs are reviewed by building security and there are security cameras in the hallways. In the event of employee termination, managers must follow the Checklist – Departing Staff and all access to the systems, as well as physical access to the premises, must be terminated.

The UNIX server where the Statistics Canada data are stored resides in the server room, which is accessed through the command centre. Access to these rooms is limited to approximately 30 ISB staff and sometimes to a few registered contractors and vendors. All visitors upon entry must sign a visitor log located in the command centre.

Cameras are positioned outside each of these secure locations to monitor individuals entering these areas. Alarms detect anyone breaking into these rooms and notify building security. A pass card access log to the command centre is generated and reviewed monthly by ISB's IT security officer.

Security measures exist for information copying and retention, and records management

Policy VIII of the Personal Health Information Act Manual requires that whenever sensitive data are removed from a secure environment, they must be encrypted. Furthermore, Manitoba Health's policy on Secondary Use and Disclosure Processes for Ad hoc Requests states that data can be transmitted only at aggregate levels. If quasi identifiers are present, then data must be transmitted via encrypted CD.

Statistics Canada data cannot be directly accessed or removed from the servers since the servers do not have USB ports. The server security includes multiple layers of firewall, choke routers, switchers, intrusion detectors, virus scanning and network monitoring. The UNIX server is backed up daily on encrypted tapes that are housed on site and sent off site each week, to a private storage company.

There are three levels of data security at Manitoba Health: public, internal and restricted. Statistics Canada data are classified as restricted (highest level of security). All shredding and disposal of documents is in accordance with restricted information as outlined in the Manitoba Government Electronic Media Disposal Standards and Procedures. A private shredding company is used for the secure disposal of confidential information.

Clauses for termination and return or destruction of the shared data no longer in use are included in the DSAs as well as in the agreement with the University of Manitoba's Manitoba Centre for Health Policy.

Appendix A: Audit criteria

Appendix B: Acronyms

Background

Statistics Canada, as the national statistical agency, is required to "collect, compile, analyse, abstract and publish" statistical information on Canada's economy and society that is fundamental to support evidence based decision-making and inform debate.

The Daily is Statistics Canada's official release vehicle and supports the mandate to publish statistical information on Canada. Official Release (OR) is governed under the Policy on Official Release at Statistics Canada. Under this Policy, every new cycle of every statistical program, whether based on surveys or administrative data, whether funded through estimates or cost recovery, must be released in The Daily before it can be disseminated to the public. Until Official Release, the dataset or product has "Protected" status as defined by the Government Security Policy. The minimum requirement for OR is a statement in The Daily that a dataset is available. This practice ensures that all potential users have access to statistical program results and products at the same time. It creates a level playing field that reinforces the Agency's reputation for neutrality.

Under the Policy on Official Release, there are certain circumstances in which Statistics Canada can release information to external parties in advance of OR in The Daily. These circumstances occur when:

• Federal officials have advance access to a small number of key releases to prepare appropriate responses for ministers, given that ministerial comment can have repercussions on markets.
• Work-in-progress datasets, analytical products and information products may be provided in advance of OR to designated individuals of external organizations for the purposes of data validation.
• Analytical products may be provided in advance of OR to an individual or external organization for the purpose of institutional and peer review.
• Microdata files or confidential aggregates from which datasets are obtained may be shared with a third party prior to the OR.
• Advance release (AR) for information applies to collaborative programs (cost-recovery programs, administrative data files and common governance structure).
• The Chief Statistician may authorize dissemination of information in advance of release in special circumstances in which the benefits justify the exception.

Oversight of the application of the Policy on Official Release falls under the responsibility of Communications Division within the Agency. Directors of program areas are responsible for ensuring that protected information is securely transmitted when provided outside the agency and that it is covered by an acknowledgement of confidentiality requiring the receiving organization to comply with requirements described in the Policy. Additionally,outside organizations that receive AR information must acknowledge receipt of the protected information they receive, safeguard the information, limit access and must destroy all protected information once the review has been completed.

Audit Objectives

The objectives of the audit were to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that:

• Statistics Canada has adequate governance structures and a risk management framework in support of the Official Release.
• Effective control mechanisms have been established and are consistently applied to ensure that the advance release of Agency products is compliant with the Policy on Official Release and other applicable Statistics Canada policies and directives.

Scope

The scope of this audit included an examination of the adequacy and effectiveness of the controls for OR. Specific areas that were examined include:

• Governance and monitoring practices that support clear accountability, oversight, scrutiny and challenge functions;
• Effective risk management processes for the identification of the key risks for OR, as well as the development and monitoring of risk management strategies; and
• Operational processes and controls in place to ensure the confidentiality and integrity of sensitive statistical information in the advance release of information.

The scope of the audit work included an assessment of AR activities from January 1, 2013 to July 1, 2014.

Approach and Methodology

The audit work consisted of a comprehensive review and analysis of relevant documentation, interviews with key management and staff and a review for compliance with relevant policies and guidelines.

The field work included a review and testing of the OR processes and procedures in place to ensure the protection of sensitive statistical information until dissemination of the information in The Daily.

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors (IIA) International Professional Practices Framework.

Authority

This audit was conducted under the authority of the approved Statistics Canada Integrated Risk-Based Audit and Evaluation Plan 2013/14 to 2017/18.

Governance and Risk Management

A robust governance and risk management framework is essential to ensure that Official Release materials are adequately protected, both internally to Statistics Canada and when information is transmitted to third parties. Roles, responsibilities and accountabilities for Official Release should be clear, communicated and understood. Appropriate and effective oversight bodies should be established and their roles and responsibilities formalized. Employees should understand their obligations under the Values and Ethics code; it should be easily accessible and explain how to proceed should any potential wrongdoing be observed.

Roles, responsibilities and accountabilities for staff working on OR are defined and communicated.

One of the expected results of the Policy on Official Release is that governance structures, mechanisms and resources are in place to ensure the continuous and effective management of Statistics Canada pre-release information. Oversight of the application of the Policy on Official Release falls under the responsibility of Communications Division within the Agency. As per the Policy, Directors of program areas are responsible for ensuring that protected information is securely transmitted when provided outside the agency and that it is covered by an acknowledgement of confidentiality requiring the receiving organization to comply with requirements described in the Policy.

The audit examined several job descriptions and procedures documents and found that these adequately outlined the various tasks associated with the preparation of OR materials.

Oversight bodies have clearly defined mandates to govern the OR process.

The Editorial Board of The Daily is responsible for overseeing The Daily editorial and guides The Daily team on strategic and operational issues related to The Daily. A review of meeting minutes and interviews with staff indicated that the Editorial Board was fulfilling its mandate as a forum to ensure a coherent and uniform approach to writing and publishing ORs. Moreover, review of the mandate and minutes revealed that  the Board helps serve as an oversight function, given that the content and the format of The Daily has greatly evolved over the past few years. The audit found that the Communications Division has a step-by-step 'Process Flow for Daily Releases', which helps the Subject Matter Divisions (SMDs) understand the steps to follow and approvals required when writing a release for their division.

The EFT Governance Steering Committee (EFT-DGC) was established to comply with the requirements stated in the Directive on the Transmission of Protected Information. Interviews with members of the committee noted that they serve as a second level of review to ensure that a request was placed for the creation of an EFT safe-box, but that the key control remains with the SMDs, which retain responsibility for ensuring that all requirements are met when providing AR information to third parties as per the Policy on Official Release. The audit determined that although the EFT-DGC is in place and operational, a formal mandate and membership list have not been developed, nor does it post minutes of its meetings. Without a formal mandate, the effectiveness of the intended oversight may be reduced.

The Corporate Risk Management framework requires updating.

Management at Statistics Canada is required to periodically identify and assess risks to their division and this is done through the annual risk register exercise. Communications Division has identified risks associated with OR in its risk register. The risk register also notes that the protection of pre-release information is critical, and that access to pre-release information should be restricted throughout the process and that management periodically conducts compliance checks within the Communications Division to ensure that access is restricted.

The audit noted that with respect to OR, subject matter divisional risk registers examined were not detailed enough to capture specific risks to OR. Similarly, corresponding SMD's divisional Business Continuity Plans assessed were at too high a level to address specific risks to OR objectives, and did not provide guidance on action plans should risks to OR occur.

During audit interviews, a number of risks were identified informally by staff at both the management and operational levels within SMDs. Risks identified included: the risk posed to the timeliness of OR when working with tight timelines and small teams; the risk that last minute changes to The Daily text may change the meaning of the release or may not be captured appropriately when translated into the second official language; the risk of the unauthorized release of information before the OR; and the risk posed by outdated access rights within divisions.

The audit found that the Official Release process encompasses activities in several divisions as well as outside organizations. The risk management framework currently used throughout the Agency does not compel managers to consider risk inter-dependencies, nor provide an integrated lens to risk management. As a result, significant risks noted by SMDs are not considered in the Communications Division risk register, therefore possibly limiting the effectiveness of potential mitigation strategies.

Recommendations:

It is recommended that the Assistant Chief Statistician of Census, Operations and Communications ensures that:

• A formalized mandate, and membership list is put in place that includes the roles and responsibilities of the EFT Dissemination Governance Committee.

Management Response:

Management agrees with the recommendation.

• The Director, Dissemination Division will develop a formalized document including mandate, membership and roles and responsibilities for the EFT Dissemination Committee.
  
  Deliverables and Timeline: A formalized document including mandate, membership and roles and responsibilities for the EFT Dissemination Committee by June 2015.

It is recommended that the Corporate Risk Officer ensure that:

• The Corporate Risk Management framework is updated to enable the integration of risks inter-dependencies across departmental divisions.

Management Response:

Management agrees with the recommendation.

• The risk registry and corporate risk profile have been conducted annually and have evolved with improvements year over year. The Corporate Risk Officer will undertake a review of current business processes to more efficiently and effectively capture operational and strategic risks. This review will lead to an improved risk framework and introduce more rigor during the development of the risk registry, including the identification of interdependencies that impact outcomes.
  
  Deliverables and Timeline: A documented business process for the integrated risk management framework; the development of a risk framework and risk registry; and tools to support the development of the risk registry will be developed by March 2016.

Objective 2: Effective control mechanisms have been established and are consistently applied to ensure that the Advance Release of the Agency products is compliant with the Policy on Official Release and other applicable Statistics Canada policies and directives.

Control in place to protect advance release information provided to external organizations

Effective control activities include IT security practices and other activities that effectively manage third parties, prescribe how associated tasks should be performed, prohibit inappropriate action, and provide the limit of acceptable action that third parties should clearly understand and follow.

The system used to ensure that advance release information is protected and sent only to authorized external parties requires improvement.

Under the Policy on Official Release, all new Statistics Canada datasets, analytical and information products are released through The Daily, which is the OR vehicle for the Agency. The Policy notes that there are 5 conditions that allow information to be disseminated prior to OR. The conditions under which AR is permitted are:

• Authorization by the Clerk of the Privy Council on the advice of the Chief Statistician;
• Authorization by the Chief Statistician;
• Work-in-progress agreements (for data validation purposes);
• AR for information purposes^{Footnote 1} (cost recovery and administrative data back to the source organization); and
• Common Governance.

To obtain an authorization by the Clerk of the Privy Council for an AR, a letter is sent from the Chief Statistician to the Clerk, requesting authorization for a department. The Clerk of the Privy Council sends a letter back to Statistics Canada, authorizing ARs under this condition. As at June 2014, there were 6 federal departments or agencies that had authorization by the Clerk of the Privy Council for AR for the various mission critical surveys and major economic indicators.

There were no AR arrangements in place under authorization of the Chief Statistician.

The majority of ARs fall under the category of work-in-progress (91 submissions) and AR for information purposes (47 submissions). In all cases, except Common Governance (8 submissions), when organizations request AR of information, requests must be approved by the SMDs as well as the Director General of Communications Division.

A Common Governance Structure Recognition Submission form must be prepared for a formal collaborative program that has been designated by the Executive Management Board in a recognized common governance structure with partner organizations. For the audit period there were 8 common governance recognitions in place.

The Policy requires that for approved AR submissions, external partners must acknowledge their responsibilities. For this purpose, the Acknowledgement of Confidentiality (AoC) forms have been created for recipients to sign. The terms in these forms are:

• acknowledging receipt of the information
• safeguarding the confidentiality of the protected release information provided to them
• limiting access to the protected information to those designated officials within their organizations for work-related purposes (respecting the 'need-to-know')
• undertaking not to further disseminate the protected release information
• destroying the protected information once the review is completed, prior to official release.

Additionally, the Policy notes that when AR is used to brief a Minister, the briefing cannot take place prior to 5 pm EST on the business day preceding the day on which the information will be published in The Daily.

For ARs under the conditions of information purposes and work-in-progress agreements, new AoCs must be signed every 15 months. The audit was advised by staff in Communications Division that there is no renewal requirement for AR under Authorization of the Clerk or for Common Governance. The audit noted that the terms in the AoCs have changed over time. As a result, if there is no requirement for a renewal of AoCs with recipients, should respondent responsibilities on AoCs change (i.e. AoC terms are changed), recipients will not have acknowledged these new responsibilities, and will not have been periodically reminded of their responsibilities for the protection of sensitive statistical information.

The Policy notes that Communications Division is responsible for keeping a registry of all ARs and to periodically collect information on the results and benefits of previously approved ARs from program managers in the organization. This is to ensure continuous and effective management of Statistics Canada's advance release information as required in the Policy. CommunicationsDivision also keeps copies of signed AoCs.

The audit examined the AR registries in place in Communications Division for 2012-13 and 2013-14 and found that a registry was in place for all AR conditions except for AR under authorization of the Clerk of the Privy Council.

While the Office of the Chief Statistician had a list of all departments receiving information under Authorization of the Clerk, Communications Division did not have a registry in place for advance releases under authorization of the Clerk of the Privy Council as required by the Policy. As a result, the management of AR under this condition is not fully effective. The audit noted in the two major indicators sampled, in one case, incorrect forms were signed and not all those designated to receive the information had signed AoCs. For one SMD who provided AR information under authorization of the Clerk of the Privy Council, no AoCs were in place at all (Table 1).

The Policy on Official Release requires that the results and benefits of previously approved advance releases are documented to ensure that there is a demonstrable benefit to allowing outside organizations to have advance release materials. The Policy also requires that Communications Division periodically collect this information from subject matter divisions. The audit noted that subject matter divisions contacted do not document this information and Communications Division does not collect this information. In the absence of documented evidence of the benefits of allowing protected information to be accessed in advance of official release, the organization may not be able to adequately complete the risk/reward exercise and could lead to unnecessarily providing advance releases with little or no benefit to the organization.

During the annual update of the advance release registries, Communications Division undertakes an exercise in which subject matter divisions provide a justification for continued advance release under common governance, Work In Progress (WIP), and advance release for information purposes. The audit noted that these justifications are at a high level and included rationales such as: recipient is a partner; recipient provides validation; provided for information purposes. Justifications are high level and generic in nature, and do not provide enough information to prove a demonstrable benefit.

The audit tested AR submissions and the corresponding AoCs under each of these conditions for the period of January 1, 2013 to July 1, 2014. As a result, 13 of the 14 AR submissions tested under the 4 conditions where the audit found AoCs, SMDs had recipients sign the correct forms.  The audit also tested to determine if all submissions under work-in-progress, information purposes and common governance had the required signed Acknowledgements of Confidentiality in place (Table 1). Overall testing results revealed that some of the AoCs were not in place. More specifically, under each advance release condition, audit testing found:

•  Work-In-Progress agreements - Of the 5 WIP submissions sampled, 4 submissions had the required number of Acknowledgements of Confidentiality in place.
•  Advance release for information purposes, 4 of the 4 submissions sampled had all of the required Acknowledgements of Confidentiality on file.
• Common Governance – 2 of the 2 submissions sampled had all of the required Acknowledgments of Confidentiality in place.
• Advance release Under Authorization of the Clerk, of the 2 submissions sampled, neither used the correct AR submission forms, and one had the required number Acknowledgements of Confidentiality in place.

Further, the audit noted in 2 of the signed AoCs in place, the AR recipients had changed the terms on the acknowledgements without approval from Communications Division.

While there is a process in place for external parties to formally acknowledge their responsibilities as they relate to sensitive statistical information, the current process is not fully effective which may increase the risk to the protection of sensitive statistical information.

The media lock-up room is secure

When there is a major economic, Census or National Household Survey release, journalists holding a valid Parliamentary Press Gallery Pass may attend a media lock-up. The Media lock-up provides an environment where journalists can access OR materials approximately one hour in advance of the official release to prepare an article.

The media lock-up room was last renovated in September 2013 and the room is protected against all wireless or radio-frequency communications. Additionally, except for the telephone on the Media Relations staff monitoring desk, all other landlines are  disabled during lockup. During a media-lockup, the room is locked from the start of lock-up (usually 7:30 am) until the official release time of 8:30 am. When in lock-up mode and the door to the room is closed, no communication with external parties is permitted until release time.  Journalists are required to turn in all cell phones which are placed in a secure lead box. During the lock-up period, journalists receive a copy of the OR material after the room has been secured and prepare their articles. At exactly 8:30 am (synchronized with the standard time of the National Research Council), the lock-up room is unsecured and journalists are able to submit their articles for publication.

The audit tested the media lock-up room and its operating procedures and found that the security features of the room are effective and ensure the protection of information.

Recommendations:

It is recommended that the Assistant Chief Statistician of Census, Operations and Communications ensures that:

• Communications Division works with SMDs to enhance the registry for Advance Release Submissions to enable effective management of submissions and ensure AR materials are limited to authorized individuals only.
• All Advance Releases should be managed by a single authority (i.e. Communications Division), and requirements for Acknowledgements for Confidentiality should be made uniform to ensure that periodic updates to Acknowledgments of Confidentiality take place which would ensure confirmation of recipients understanding and their responsibilities with respect to protection of protected information.
• There is a demonstrable benefit associated with Advance Releases for Work in Progress (WIP) purposes and they should be documented by SMDs and collected by Communications Division.

Management response:

Management agrees with the recommendations.

• The Director General, Communications will work with the Directors of Subject-matter Divisions to confirm the accuracy of the information in the registers, collect signed copies of acknowledgement of confidentiality for all advance release agreements and add them to Statistics Canada's register of advance release and common governance agreements.

Deliverables and Timeline: An enhanced and accurate registry for Advance Release Submissions by June 2015.

• The Director General, Communications will ensure that a register for the Advance Releases authorized by the Clerk of the Privy Council is established and maintained by the Communications Division.
  
  Deliverables and Timeline: A register for Advance Releases authorized by the Clerk of the Privy Council maintained by Communications Division by June 2015.
• The Director General, Communications will ensure that the Policy on Official Release is modified to specify periodic updates for the Acknowledgements of Confidentiality for the Advance Releases authorized by the Clerk of the Privy Council and for Common Governance Structure agreements. The forms "Acknowledgement of Confidentiality for Advance Releases authorized by the Clerk of the Privy Council" and "Acknowledgment of Confidentiality for Common Governance" will be modified accordingly.
  
  Deliverables and Timeline: A modified Policy on Official Release by June 2015.
• The Director General, Communications will ensure that the Communications Division works with Subject Matter Divisions to ensure that the benefits associated with Work-in-progress agreements (advance release for validation purposes) are stated specifically in the register.
  
  Deliverables and Timeline: Documentation of the benefits of WIPS, with greater specificity in the register by June 2015.

Controls within Statistics Canada to ensure the confidentiality of pre-release information

Effective control mechanisms and processes support the organization in the management of risks and the achievement of the established objectives by ensuring that assets are safeguarded and that actions and decisions of the organization are in compliance with applicable laws, policies and directives.

Access privileges to shared folders require regular updating.

The Directive on the Security of Sensitive Statistical Information and the Security Practices Manual note that Statistics Canada considers all sensitive statistical information under its care and control as Protected B. As such, this information should be controlled to protect against loss, theft, compromise or improper disclosure. Within Statistics Canada, this information may only be circulated to individuals on a need-to-know basis.

Subject Matter Divisions noted that all protected information is housed on secure shared drives. Shared drives are on secure Network A and access to these drives is restricted on a need-to-know basis. The audit confirmed that when a review of information is to take place, a hyperlink to the information is circulated via an email, and that electronic versions of the information are not sent.

The audit examined file folders and the associated permissions for the pre-release information for the Labour Force Survey (LFS), Monthly Retail Trade Survey (MRTS) and Communications Division to determine if file access is restricted to only those with a need to know. Although access was found to be restricted in the prerelease folders, the audit found that permissions were not regularly updated and that in both SMDs pre-release folders examined, some individuals with permission to these folders were no longer working in the division.

The Smart Daily application allows Subject Matter Divisions to load pre-release Daily texts into an application which formats the official release materials. All SMDs which have Official Releases use this application; however access is to be restricted to authorized users only within divisions. Communications Division manages the access and permissions for this application.  The audit examined the Smart Daily application to ensure that SMD employees who are authorized to use the Smart Daily application have access to only their own materials and that pre-release information from other areas is restricted. The audit found that employees in Communications Division limit access and users can only access their own information and cannot view information from other programs or divisions.

Media spokespersons for information releases have not all followed the mandatory media training.

Prior to speaking to the media, individuals designated to be media spokespersons are to have taken the "Encountering the Media" course. This training is designed to help Statistics Canada staff to be prepared for media requests and ensure that spokespersons understand their responsibilities when designated as media spokespersons.

The Agency has adopted a policy to accept media requests for interviews and to provide comments on program issues and data interpretation, and for each OR, will designate certain individuals within SMDs as media spokespersons or information contacts. According to both the Policy on Official Release and the Policy on Media Relations: Spokespersons and Response to the Media, Communications Division is responsible for providing media training to information contacts for OR purposes and directors in SMDs must ensure identified spokespersons have taken the required training.

A training and development plan exists for the handling and management of the media; however the audit took a sample of 25 individuals who were named as media spokespersons over the audit period and found that 28% of them had not completed the required media training. Interviews with Communications employees who provide the training noted that refresher courses and mock interviews can be arranged with Communications Division on an ad-hoc basis, but that these informal courses are not considered as a substitute for the formal course.

When media spokespersons have not taken the required formal media training there is an increased risk that employees will not understand their roles and responsibilities with respect to being an official spokesperson for the Agency.

Monitoring the transfer of information from the secure network to the public network ensures IT malfunctions are dealt with in a timely manner.

The IT infrastructure that is used to prepare and package OR materials is under the responsibility of Shared Services Canada. At the same time, Statistics Canada staff are responsible for ensuring that the OR materials (the Daily) are disseminated at the OR time of 8:30 am EST (Eastern Standard Time). Because the IT infrastructure is no longer under Statistics Canada control, staff from Administrative and Dissemination Systems Division (ADSD) monitor the release process on each release day beginning at 7:00 am until OR at 8:30 am.

The IT environment is complex, given that there are several servers and the movement of information is time-sensitive. Additionally, some servers move information in and out of the protected zone into the Public Access Zone, which is where information becomes available to the public at the OR time of 8:30 a.m.

ADSD staff has developed a series of protocols and automated notifications at each movement of data to ensure that staff is made aware of any malfunctions in the process and to advise them of where the malfunction took place. The protocol also notes at each step what manual intervention must be activated in the event of an IT malfunction. These protocols are evergreen and are updated when there are changes to the IT infrastructure.

ADSD maintains a schedule of employees who are responsible for ensuring that adequate monitoring always takes place. Additionally, the chief responsible for monitoring the movement of information noted that there are always 2 individuals on site. It was noted that this is an important control. Should a malfunction occur, one individual can focus on initiating the manual controls and the other employee can ensure the communication of the issue with management, subject matter and Shared Services staff.

The audit observed the monitoring of transfer of information and found that the process is followed as outlined in the protocol, the automated notifications are generated as described and the ADSD monitor validates that the correct information has been transferred which is an effective method to mitigate the risks of an IT malfunction.

Contracting practices for the procurement of translation services require strengthening.

All official release materials are disseminated in both official languages. Statistics Canada outsources translation services to professional translators. Government of Canada security and contracting policies require that when protected information is accessed by external parties, several security measures are required. This includes ensuring that the individuals accessing protected information have the appropriate security clearance and that the facility and document safeguarding capabilities correspond to the level of security of the information received. As part of the procurement process, departments send the Security Requirement Checklist (SRCLs) to Public Works and Government Services Canada (PWGSC) for confirmation. SRCLs outline the classification of the information to be handled and PWGSC confirms whether the proposed contractor holds the appropriate security clearances commensurate with the level of security of the information to be accessed. PWGSC also conducts the document and facility security inspections of facilities that will be handling Government of Canada protected information.  Additionally, the Government of Canada Policy on Contracting notes that, "the government security policy is to be applied equally to procurement contracts as it is to internal operations."

The audit examined translation contracts to determine if the appropriate security requirements were included and that the contractors had the appropriate security clearances in place. Testing found 28% of translation contracts sampled (10 of 36) did not include security requirements. Given this, no protected or secret information should be sent out for translation under these contracts. The audit noted that in 66% (6 of 9) of sub-sampled contracts that did not contain security requirements, protected information had been provided to contractors who did not have the appropriate facility and document safeguarding or, in some cases, a personal security clearance.

Project authorities are responsible for establishing terms & conditions within contracts, including SRCLs when applicable. Interviews with the contracting authority and departmental security noted that they do not provide a challenge function to project authorities on the necessity of including security requirements in contracts.

The audit found several other contracting weaknesses and inconsistencies. Certain, but not all, translation contracts require that individuals having access to protected information sign an acknowledgement of values and ethics and the Oath of Secrecy. There was no evidence of any of these in any of the contracts sampled. For some translation contracts, the project authority noted that contractors must sign a "security statement" in lieu of having security clearances. The "security statement" document was developed internally at Statistics Canada, with the collaboration of departmental security, as a means to compensate for the absence of security requirements in certain contracts. There was no evidence of any of these in any of the contracts sampled.

The audit found significant control weaknesses for the procurement of translation services and identified several instances of non-compliance, which considerably increases the risk to the confidentiality of sensitive statistical information.

Recommendations:

It is recommended that the Assistant Chief Statistician of Census, Operations and Communications ensures that:

• Prior to release, subject matter divisions and media relations staff verify that media contacts listed for Official Release have been provided with Media Training and are aware of their roles and responsibilities under the Policy on Media Relations: Spokespersons and Response to the Media.
• Contracts issued for the procurement of translation services include security requirements commensurate with the level of security of the information that will be accessed by contractors.

Management agrees with the recommendations.

Management response:

• The Director General, Communications, in collaboration with Directors of subject-matter divisions will ensure that all spokespersons take formal media training prior to speaking to media. This is accomplished with:
  • Quarterly emails to directors by the Director General, Communications, with follow-ups as required;
  • Media training sessions provided by Communications Division throughout the year;
  • Support for mandatory training confirmed by the Learning and Development Committee.
  In cases where a spokesperson does not take the required training, the Director of the subject-matter division will act as the spokesperson.
  
  Deliverables and Timeline: All spokespersons have taken formal media training prior to speaking to media by July 2015.
• Management agrees with the recommendation and acknowledges that the security clearance of some of its translation service providers was issued by departmental security and not by PWGSC as it should have been. Although this was a lapse in procedures, the audit did not identify any specific case where the information that had been sent for translation was used inappropriately.
  
  Upon being informed of the situation, two mechanisms were established to remedy the situation:
  • Protected B documents are sent to suppliers who have the required security clearance;
  • Within Communications Division, a process to confirm the security clearance of service providers has been established.
  The Requests for Proposals (RFP) to select suppliers for 2015-16 will include the proper security requirements. The required security clauses (received from PWGSC) will be included in all contracts.
  
  Deliverables and Timeline: Implementation of mechanisms to ensure translation service providers have the required security clearances was implemented on December 5th, 2014 and RFPs to select suppliers for 2015-16 will include the proper security requirements by April 2015.

It is recommended that the Assistant Chief Statistician of Corporate Services (and CFO) ensure that:

• Access privileges to shared folders containing protected information are regularly updated to ensure only those with a need-to-know requirement can access information.

Management response:

• Management agrees with the recommendation.
  
  A reminder stressing the importance and necessity to review and update access privileges to shared folders containing protected information will be circulated by the Chief Information Officer (CIO) to all executives. Options to automate the management of access privileges will also be explored.
  
  Deliverables and Timeline: A generic reminder will be circulated by the CIO annually, effective immediately. A working group, in conjunction with the Administrative Processes Review and Automation (APRA) project, will be set-up to explore options to automate the management of access privileges. A recommendation will be presented to the Information Management Committee by September 2015.

Appendix A: Audit criteria

Appendix B: Acronyms