Audit Report

November 13, 2012
Project Number: 80590-72

• Executive Summary
• Introduction
  • Background
  • Audit Objectives
  • Scope
  • Approach and Methodology
  • Authority
• Findings, Recommendations and Management Responses
  • Information Technology Security
  • Physical Security
  • Administration of Microdata Research Contracts and Confidentiality Vetting
• Appendices
  • Appendix A: Audit Criteria
  • Appendix B: Acronyms

Executive summary

The University of Alberta Research Data Centre (RDC) is one of twenty-four RDCs located in university campuses across Canada. These RDCs were established through the efforts of Statistics Canada, Social Sciences and Humanities Research Council, Canadian Institutes of Health Research and university consortia, to strengthen Canada's social research capacity, and support the policy research community. The University of Alberta RDC facility was part of the group of 9 RDCs to open its doors in 2001 and is located on the third floor in the Rutherford Library at the University of Alberta. In 2011, the University of Alberta RDC experienced a growth of almost 44% in the number of contracts at the facility. In an average month in 2011, approximately 59 researchers had access to data at the University of Alberta RDC.

RDCs are staffed by Statistics Canada employees, and are operated under the provisions of the Statistics Act. As such, RDCs must have security measures in place to safeguard the confidentiality of data to the same degree as in Statistics Canada's offices.

The objective of this audit was to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that the RDC at the University of Alberta:

• Complies with applicable Treasury Board Secretariat (TBS) and Statistics Canada policies and standards regarding Information Technology (IT) and Physical Security, to ensure that confidentiality of data is protected in the delivery of services.
• Has effective practices and mechanisms in place to ensure that the confidentiality of data is protected in the delivery of services.

The audit was conducted by Internal Audit Services in accordance with the Government of Canada's Policy on Internal Audit.

Key findings

The University of Alberta Research Data Centre has effective IT and physical security measures to ensure access, identification and protection of the confidentiality of sensitive statistical information. Access to the University of Alberta RDC is restricted to authorized personnel, including: deemed employees, researchers with valid Microdata Research Contracts (MRCs), and University of Alberta IT support staff.

The audit revealed that there are opportunities to strengthen the IT control environment by ensuring that the login timeout feature on researcher workstations is functional in order to maintain the effectiveness of this control.

The audit noted that procedural documents related to the use of personal electronic devices within the RDCs were not consistent. Multiple directives highlighted a range of procedures, from access to complete prohibition. RDC staff are left to decide which control is applicable.

Roles, responsibilities, and accountabilities are defined and communicated in the following areas: administration of Microdata Research Contracts, confidentiality vetting, as well as physical security and IT security within the RDC facility.

Authority is formally delegated at the program and operations level. Processes and procedures for confidentiality vetting are in place and requests for vetting are carefully administered and effectively screened by the RDC analyst to ensure that confidentiality of data is not compromised. Applicable physical security measures and IT access, identification and authentication safeguard measures are in place and adhered to for safeguarding and protecting Statistics Canada confidential data.

Overall conclusion

Physical security measures and IT access, identification and authentication safeguard measures are compliant with applicable TBS and Statistics Canada policies and standards. The administration of Microdata Research Contracts is appropriately managed and the research proposal process is effective. The processes and procedures for confidentiality vetting of output in place are adequate and effective to ensure the confidentiality of sensitive statistical information.

The audit results highlighted opportunities for improving the practices and mechanisms that are in place to ensure that confidentiality of data is protected in the delivery of services. Areas to be strengthened are: 1) ensuring that the login timeout feature on inactive workstations is functional; and, 2) ensuring consistency in the procedures and practices related to personal electronic devices usage in RDCs.

Conformance with professional standards
The audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Quality Assurance and Improvement Program.

Patrice Prud'homme
Chief Audit Executive

Introduction

Background

Decision-makers need an up-to-date and in-depth understanding of Canadian society to help them respond not only to today's needs, but to anticipate tomorrow's as well. This need is underlined by a growing demand for analytical output from the rich sources of data collected by Statistics Canada.

In 1998, the Canadian Initiative on Social Statistics studied the challenges facing the research community in Canada. One of the recommendations of the national task force report on the Advancement of Research using Social Statistics, was the creation of research facilities to give academic researchers improved access to Statistics Canada's microdata files.

The Research Data Centres (RDCs) are part of an initiative by Statistics Canada, the Social Sciences and Humanities Research Council (SSHRC), Canadian Institutes of Health Research (CIHR) and university consortia, aimed at strengthening Canada's social research capacity and supporting the policy research community. SSHRC is the federal agency that promotes and supports university-based research and training in the social sciences and humanities. CIHR is the major federal agency responsible for funding health research in Canada.

Twenty-four RDCs are located in secure settings on Canadian university campuses. These RDCs provide researchers with access to microdata from population and household surveys without requiring researchers to travel to Ottawa to access Statistics Canada microdata. A Federal Research Data Centre (FRDC) located in Ottawa provides microdata access to researchers from federal policy departments.

The RDCs provide opportunities to generate a wide perspective on Canada's social landscape; provide social science research facilities across the country in both larger and smaller population centres; train a new generation of Canadian quantitative social scientists; and expand the collaboration between Statistics Canada, and the stakeholders - SSHRC, the Canadian Research Data Centre Network (CRDCN), CIHR and academic researchers.

The RDCs are staffed by Statistics Canada employees and are operated under the provisions of the Statistics Act, including the requirement to operate in accordance with Statistics Canada's confidentiality rules and accessibility restrictions (i.e. facilities are accessible only to researchers with approved projects who have been sworn in under the Statistics Act as deemed employees).

There is a growing demand from academic and government researchers outside Statistics Canada for access to microdata. Microdata Access Division (MAD) provides restricted access to confidential microdata through RDCs at universities across the country and the federal RDC in Ottawa. MAD is responsible for ensuring the confidentiality of information provided by Canadians.

The Statistics Canada Risk-Based Audit Plan requires Internal Audit Services to complete an audit of one RDC per year. In 2010, an audit was undertaken to look at the overarching governance of all RDCs which included site visits to 5 RDC locations. In 2011, the Prairie Research Data Centre (University of Calgary and University of Lethbridge) was audited. The University of Alberta RDC is the second audit that has focused on a specific RDC. This audit will add value by ensuring that confidentially measures are followed and effective.

Audit objectives

The objectives of this audit are to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that the RDC at the University of Alberta:

• Complies with applicable Treasury Board Secretariat (TBS) and Statistics Canada (StatCan) policies and standards regarding Information Technology (IT) and Physical Security, to ensure that confidentiality of data is protected in the delivery of services.
• Has effective practices and mechanisms in place to ensure that the confidentiality of data is protected in the delivery of services.

Scope

The scope of this audit included a detailed examination of the systems and practices of the University of Alberta RDC for the protection of data, use of technology, and physical security.

The audit focused on disclosure risk analysis and vetting of data output by the on-site Statistics Canada employees; deemed employee status and security clearance requirements for access to microdata; research proposal process for RDC; microdata research contract administration; physical security of the RDC site in compliance with applicable TBS and Statistics Canada policies and standards and IT protection in compliance with applicable TBS and Statistics Canada policies and standards.

Approach and methodology

The field work was performed in two stages beginning with interviews and review and assessment of the processes and procedures to ensure physical security, use of technology and the protection of data. The second stage consisted of a site visit to the University of Alberta RDC to test controls for safeguarding microdata files including logical access and computer security controls, and to perform compliance testing of the centres to assess the physical security measures in place. The administration of MRCs and confidentiality vetting were also tested to ensure appropriate controls were in place.

This audit was conducted in accordance with the Standards for the Professional Practice of Internal Auditing as per The Institute of Internal Auditors (IIA), and the TBS Policy on Internal Audit.

Authority

The audit was conducted under the authority of Statistics Canada Integrated Risk-Based Audit and Evaluation Plan 2012/13-2014/15 recommended by the Departmental Audit Committee, April 2012 and subsequently approved by the Chief Statistician.

Findings, recommendations and management responses

Line of Enquiry No. 1: The University of Alberta RDC complies with applicable TBS and Statistics Canada policies and standards regarding Information Technology Security and Physical Security to ensure that confidentiality of data is protected in the delivery of services.

Information technology security

Information technology security in RDCs should be compliant with applicable TBS policies, such as the Operational Security Standards: Management of IT Security and Statistics Canada's Security Practices Manual. Roles, responsibilities, and accountabilities should be clearly defined and communicated. In the context of the RDCs, IT security should include system and communications protection: security controls that support the protection of the information system itself as well as communications with and within the information system; Access control: security controls that support the ability to permit or deny user access to resources within an information system; and identification and authentication: security controls that support the unique identification of users and the authentication of these users when attempting to access the information system.

Roles and responsibilities

The audit found that at the program level, functional authority is formally delegated to the Manager/Director of the RDC Program, and at the regional level the RDC analyst. The University of Alberta RDC analyst reports to the RDC regional manager who is located in Winnipeg. The statistical assistant at the University of Alberta facility reports to the full-time RDC analyst.

Departmental Security and Information Technology Services at Statistics Canada head office provide guidance and directives on IT and physical security requirements. They perform the physical and IT security inspections of the RDC sites and provide recommendations to the Director/Manager of the RDC program. The first IT and physical security inspection of the University of Alberta RDC took place just prior to the facility opening in 2001. A second inspection was conducted in February, 2011. Periodic inspections have been scheduled for each RDC every 4 years.

System and communications protection safeguards

The computing environment at the University of Alberta RDC has been recently updated and the new server uses a MAC operating system. It is located in a secured closet located within the RDC. The server is a stand-alone setup with open directories, using Access Control List (ACL industry standard) to grant permissions. Apart from the wide-area network (WAN), the server has no external connection. As a result, remote access to the server outside of the RDC is not possible. Data on the server is also written in Redundant Array of Independent Disks (RAID) which is a data storage method whereby data is broken down into blocks and stored on multiple hard drives. RAID provides an enhanced level of security; should one or more of the server's hard drives be stolen, it would be impossible to replicate data.

There are eight stand-alone workstations available for use by the researchers. These are not connected to the internet (internet access is only available to RDC employees in the RDC analyst office), and do not have an operating system or programs installed. Workstations are part of the thick client system that had been virtualized. Upon start-up, the Basic Input/Output System (BIOS) sends a query directly to the server to get an image from the server.

The audit examined workstation configurations, and found the USB ports have been disabled and Ethernet ports are configured to workstation serial numbers. As a result, should another device be plugged into an Ethernet port, the port automatically deactivates and must be manually reset. Researcher workstations are not configured to print and the audit determined that researchers would not be able to change the printer configuration as it is locked.

Login timeouts were found to be set to 10 minutes. However, the audit found that the lockout function was not operational; after 10 minutes a screensaver came on but did not lockout the user. As a result, reactivation did not require a password; increasing the risk of unauthorized access to Statistics Canada confidential microdata on unattended workstations.

Access, identification and authentication safeguards

Procedures specify that user accounts should be created only when a contract is approved and becomes active; access should be removed if the account is not active and password configuration should meet Statistics Canada standards. Creation of user accounts and the granting of access to microdata files were substantiated by approved active contracts in all sampled contracts.

Administrative privileges rest with the RDC analyst and IT support staff assigned to the RDC. Upon requisition of the RDC analyst, IT support creates userIDs, accounts and permissions to data sets. Tests are completed by the RDC analyst and IT support to ensure only researchers can access data set out in the MRC. Researchers are not able to view data that is not associated with their MRC, and cannot move files between projects. Password configuration within the University of Alberta RDC complies with Statistics Canada standards.

The audit tested access control in three ways, by examining

• UserIDs
• Active Control Listing by survey
• Active Control Listing by project numbers.

There were seventy-four active userIDs at the University of Alberta RDC, seven were for administrative purposes and the remaining sixty-seven were validated to ensure userIDs were only associated with active contracts. The audit team found that when researchers were associated with more than one research project, a separate userID had been created corresponding to each project.

A sub-sample of twenty-nine researcher names were chosen from the 71 researcher names found in the twenty-six contracts sampled to ensure that only researchers with active contracts had a valid userID and could access only data noted in MRC. The audit found that associated with the twenty-six contracts were nine active userIDs (eight were associated with the data sets listed in active MRCs and one account was set up but not associated with any dataset as they were waiting for the final signed contract). Another seven researchers from the sub-sample had active contracts where the principal researcher was from the University of Alberta, but because they worked out of other RDCs they did not have a userID at the University of Alberta facility. The remaining thirteen names were associated with completed contracts and did not have active userIDs.

Active Control Listings by survey name were examined to ensure that only researchers with MRCs associated with the surveys were validated. Two surveys were chosen to validate. The Active Control Listings for these surveys indicated that access was restricted to userIDs associated with authorized researchers only.

A final validation of the Active Control Listing by project number was conducted to determine if project numbers had been set up to access only data associated with the project. The audit examined two separate project numbers and the userIDs associated with the projects. The audit confirmed that users would not be able to access data not associated with their project.

The audit determined that applicable IT security measures are in place and adhered to for safeguarding and protecting Statistics Canada confidential data. IT access, identification and authentication safeguard measures are in place at the RDCs and are working as intended.

Recommendations:

It is recommended that the Assistant Chief Statistician of Social, Health and Labour Statistics should ensure that:

• The login timeout of researcher workstations in all RDCs is functional to ensure the security of sensitive statistical information.

Management response:

Management agrees with the recommendations.

• The Director, MAD will ensure that workstations are reconfigured so the login timeout is functional.
  
  Deliverables and Timeline: Validation that workstations have functional login timeouts. This has been completed.
• The Director, MAD will ensure that the IT checklist included in the IT Security Verification document which provides relevant information on what and how to verify security settings on the domain, servers and workstations is updated to include verifying that workstation login timeouts are functional.
  
  Deliverables and Timeline: The IT checklist is updated to include verification of the functionality of workstations login timeouts. This has been completed. Administration of the checklist will take place once per academic semester.
• The Director, MAD will ensure that RDCs that have been migrated to the single master domain will have automatic application of an automatic login timeout of 10 minutes by the application of a Group Policy Object at the domain level.
  
  Deliverables and Timeline: Centralized management of login timeout will be completed by March 2014.

Physical security

Physical security in RDCs should be compliant with applicable TBS policies, such as the Government Policy on Security (GPS) and Statistics Canada's Security Practices Manual. Roles, responsibilities, and accountabilities should be clearly defined and communicated. In the context of RDCs, physical security should include controls such as: physical access, intrusion detection and monitoring activities.

Roles and responsibilities

The audit found that at the program level functional authority is formally delegated to the Manager/Director of the RDC Program, and at the regional level the RDC analyst. The University of Alberta RDC analyst reports to the RDC regional manager who is located in Winnipeg. The statistical assistant at the University of Alberta facility reports to the full-time RDC analyst. Departmental Security at Statistics Canada head office provides guidance and directives on physical security requirements. They perform the physical inspections of the RDC sites and provide recommendations to the Director/Manager of the RDC program. The first physical security inspection of the University of Alberta RDC took place just prior to the facility opening in 2001. A second inspection was conducted in February, 2011. Periodic inspections have been scheduled for each RDC every 4 years.

Perimeter security controls

The University of Alberta RDC is located on the third floor of the Rutherford Library at the University of Alberta. The audit noted that the facility is in compliance with TBS and Statistics Canada's requirements for perimeter security for 'shared floor occupancy' i.e. wall separation and construction; and solid-core wood doors with heavy-duty hardware and accessories.

Entry security controls

Physical access in and out of the University of Alberta RDC is through a single entry point to allow for effective screening and monitoring by the RDC staff. Each single entry door is equipped with an electronic intrusion alarm and deadbolt lock for which only the RDC staff and campus security have the key. This is in compliance with TBS and Statistics Canada requirements.

Access security controls

To comply with TBS and Statistics Canada requirements, the RDC has in place an electronic swipe card access system consisting of an ID card which contains electronic information identifying the cardholder to control access to the facility. The system registers all entries, exits and failed attempts for the RDC and records the card number which is linked to the user. RDC staff can view and print out a record of all access, which notes the card number, the time, and the outcome if required. Unauthorised visitors are not allowed past the single entry door of the RDC facilities. The audit tested the system and examined the access registry for 4 separate days between May and July 2012, and found evidence that all entrants and exits and failed attempts were logged to specific access cards.

Telecommunications wiring and restricted-access area controls

The audit noted that IT-related wiring is channelled through the walls and ceiling of the RDC in secure conduits. There is a secure server room, which is kept locked. The facility has locked storage cabinets for storing archived CDs and researchers' files to protect confidential, classified, and protected information. Network B access is only available in the RDC analyst office. There is a separate conference room within the facility for researchers and RDC staff to use. The printer, fax/photocopier are housed within the facility, in a separate room with a closed door. The fax and scanning functions are further restricted, as they require a password which only the RDC analyst and statistical assistant have to access these features.

Intrusion detection and monitoring activity controls

Campus security provides 24/7 monitoring of the RDC facilities. They have an access card and security code to the alarm system. A motion sensor system is also in place to enhance the security. The motion sensor system is activated between the hours of 6:00 pm and 7:00 am. Should the alarm or motion sensor system be triggered, campus security would be notified as first response. The RDC analyst and the Academic Director would be subsequently be notified.

RDCs cannot be left unattended; if the RDC analyst or the statistical assistant is required to leave the RDC for a period of time during regular hours, researchers are required to leave the RDC, the door is then locked and a sign is placed on the door.

Based on the physical inspection of the University of Alberta RDC, the audit determined that applicable physical security measures are in place and adhered to for safeguarding and protecting Statistics Canada confidential data and access to the RDC facility is restricted to authorized personnel, i.e. deemed employees, such as researchers and university IT support staff.

Cleaning and maintenance service activities controls

In compliance with TBS and Statistics Canada requirements, the audit confirmed with RDC staff that maintenance and cleaning personnel do not have access cards to the RDC. Cleaning personnel can only access the RDC facility during regular working hours and are escorted by RDC staff.

Use of electronic devices in the RDCs

The audit noted that RDC authored documentation related to the use of personal computing devices (such as cell phones, PDAs and laptops) within the RDC is inconsistent. The document entitled Security in the RDC notes that "researchers will not be allowed to bring any personal computing devices into the centre." The internal document entitled, Description of security measures to protect data notes that, "researchers are prohibited from having any electronic devices in the vicinity of their workstation." Finally the Researcher Guide which is distributed to all researchers with a signed MRC states that, "researchers may carry electronic devices with them in the RDC, however, researchers are not to operate any text editing or messaging or image/test capture devices at the workstation or inside the RDC secure area. A cell phone should not be operated in the RDC... In case of emergencies, researchers may leave their cell phones on while in the centre. To answer a call, the researcher must do so away from the workstation and conduct the conversation outside of the RDC secure area."

The procedures guidelines related to the use of electronic devices in the RDCs by researchers would benefit from consistency and clarity related to the handling of cell phones and other electronic devices within the RDC facility. Confusion related to the correct handling of personal electronic devices may increase the risk to the confidentiality of information held in the RDC.

Recommendations:

It is recommended that the Assistant Chief Statistician of Social, Health and Labour Statistics should ensure that:

• Policies and directives related to the usage of cell phones and other electronic devices in the RDC are consistent, and clearly outline the approved practice.

Management response:

Management agrees with the recommendations.

• The Director, MAD will ensure that older RDC documentation has been updated to ensure that these documents are consistent with the Researcher Guide with respect to cell phone usage in the RDCs.
  
  Deliverables and Timeline: Older documentation is updated to reflect the cell phone practices set out in the Researcher Guide. This has been completed.
• The Director, MAD will ensure Regional Managers will remind RDC Analysts of the correct procedures for proper cell phone and electronic device usage during their regional calls.
  
  Deliverables and Timeline: Regional Managers will communicate correct cell phone and electronic device procedures to all RDC analysts. Communication will take place with all RDC Analysts by November 30, 2012.

Line of Enquiry No. 2: The University of Alberta RDC has effective practices and mechanisms in place to ensure that the confidentiality of data is protected in the delivery of services.

Administration of Microdata Research Contracts and confidentiality vetting

Administration of Microdata Research Contracts (MRCs)

Administration of MRCs is a combination of assigned responsibilities and procedures to control and protect information held in RDCs. Practices include restricting access to the facility to only those researchers with valid security clearances and current contracts, and the establishment and maintenance of an inventory of administrative information related to each research project.

Authority

The University of Alberta RDC operates under the provisions of the Statistics Act in accordance with all the confidentiality rules and requirements that govern Statistics Canada. The RDC is accessible only to researchers with approved projects who have been sworn in under the Statistics Act as deemed employees.

Roles and responsibilities

The roles and responsibilities for the management of the MRCs, access to confidential microdata and confidentiality vetting are defined and communicated to stakeholders in policies, guidelines, standards and detailed guides. At the program level, authority is formally delegated to the RDC Manager in Statistics Canada's Security Practices Manual, which states that the RDC manager "is responsible for establishing and maintaining an inventory of administrative information on research projects involving deemed employees for Headquarters, the regional offices and the research data centres. Information includes research proposals and other information throughout the life-cycle of the project and certification that required procedures have been followed."

All RDC contract information has migrated to the Client Relationship Management System (CRMS). This database is used to manage information about MRC contracts and individuals who are authorized to have access to microdata in the RDCs and headquarters. Information includes contract status, approval dates, names of researchers, reviewers and review outcomes, contract end dates and data approved for access.

Additionally, the Policy on the Security of Sensitive Statistical Information assigns to Directors, "the responsibility for controlling and protecting all sensitive statistical information obtained or held by their respective areas in the pursuit of their program objectives. When access to sensitive statistical information is provided in a Research Data Centre or equivalent, the Manager, Research Data Centre Program, assumes these responsibilities."

Contract processing procedures

RDCs are operated under the provisions of the Statistics Act in accordance with confidentiality rules. This mode of access is appropriate when a research question can only be answered using inferential statistical analysis on the confidential microdata. The researcher must also be willing and able to become a deemed employee of Statistics Canada and conduct the data analysis in the RDC secured computer lab. RDC access is appropriate when:

• access to sensitive variables not provided in the Public Use Microdata File (PUMF) is required for the analysis; or
• a PUMF does not exist; or
• longitudinal data are required for the analysis; or
• the analytical work is complex in nature and not suitable for other forms of data access.

If an RDC is determined to be the appropriate mode of data access, the researcher must prepare a proposal requesting data access. The proposal defines the scope of the work proposed by the researcher and the data sets to be used in his/her analysis. When preparing the proposal, researchers are asked to define the research question(s) and objective(s), as well as statistical methods and software requirements.

Prospective researchers submit proposals that must be approved by one of two peer review procedures: the Social Sciences and Humanities Research Council (SSHRC); or for policy- related work, "Provincial and Territorial Statistical Focal Points" of the province within each ministry for federal-provincial-territorial governments. A Statistics Canada institutional review accompanies each of these application types.

After the review process, for a SSHRC peer review project, the Principal Investigator receives a letter from SSHRC notifying the researcher of the status of the proposal, and providing a copy of the reviewers' comments.

If the proposal is approved, the letter from SSHRC asks the Principal Investigator to contact the RDC analyst to begin the administrative process to gain access to an RDC. Researchers have 12 months from the date of the letter to start the project; whereupon the approval of the proposal will expire and the researcher will need to re-apply to SSHRC. The approved proposal is part of the MRC (or contract) between the Principal Investigator and Statistics Canada.

While the proposal defines the scope of the research to be done under the contract, the contract specifies the following terms of access:

• Purpose and scope of the research project as outlined in the approved research proposal
• Agreement by Statistics Canada to provide access to confidential microdata
• Agreement of the researchers to abide by the RDC security and confidentiality requirements
• Agreement of the work to be done in the RDC by the researchers and the results produced are to correspond to the objectives identified in the research proposal
• Agreement on the length of the contract
• Agreement of the Principal Investigator to provide a final product to Statistics Canada at the end of the contract.

Prior to obtaining Statistics Canada's signature on the MRC, researchers wishing to access the RDC are required to become 'deemed employees' and undergo a reliability security screening pursuant to sub-sections 5(2) and 5(3) of the Statistics Act, and take an oath or affirmation of office and secrecy, pursuant to sub-section 6(1) of the Statistics Act. They must also sign Statistics Canada's Values and Ethics Code for the Public Service. These actions are completed prior to the MRC being signed by Statistics Canada. Once a researcher has successfully completed these requirements and attended an orientation session, they are officially a "deemed employee" of Statistics Canada. RDC researchers and other deemed employees must reaffirm their oath of secrecy under two conditions: 1) when a researcher wishes to regain data access when there has not been an active contract for one year or more or; 2) when the current security clearance expires.

The Microdata Research Contract is also signed by Statistics Canada, either by the Manager of the RDC Program or the Director of the survey division. Upon receipt of this signature, a researcher can then be given access to confidential microdata and commence data analysis in the RDC.

The audit tested compliance of the contract processing procedures by reviewing a sample of twenty completed and in-progress contracts, as well as six proposals in progress, approved but awaiting final contract or withdrawn, for the University of Alberta RDC. The audit noted that for all the in-progress and completed contracts sampled, records of the project proposal with a listing of the data sets, and signed copies of the MRC and its respective amendments and revision(s) were on file and complete. For proposals awaiting contracts or proposals withdrawn, evidence was found of proposal review and approval. The audit also found evidence of cases in which proposals were approved but final outputs were rejected or had to be amended, indicating that the confidentiality risk assessment continues throughout the life of the contract.

The audit tested that all required documentation was in place, valid and signed for the seventy-one researchers that fell under the twenty-six MRCs selected. The audit found evidence that all researchers had signed oaths and values and ethics forms and held valid security screenings.

Confidentiality vetting

RDCs are repositories of Statistics Canada microdata files that are accessible to researchers with approved projects. Effective and appropriate processes and procedures for confidentiality vetting should be in place and adhered to in order to significantly reduce the risk of unwanted disclosure. Confidentiality vetting should be carefully administered by the RDC analyst, as per the established protocols, to ensure that confidentiality of data is not compromised.

Confidentiality vetting is the process of screening research outputs, syntax or any confidential data-related material to assess the risk of a prohibited disclosure. This is done by analysing whether obvious identification of individual cases or whether information about individual cases can be inferred or deduced from the statistical output. There are three types of disclosures: identity disclosure, attribute disclosure, and residual disclosure.

Roles and responsibilities

The audit revealed that the full-time RDC analyst at the University of Alberta performs confidentiality vetting for the RDC. She has been at the RDC since its opening in 2001 and has knowledge and experience of statistical sampling techniques and software. Documentation notes that the RDC Analyst's primary responsibility in confidentiality vetting is to ensure confidentiality is not breached when allowing research outputs to leave the RDC. The Analyst reviews all the materials that the researcher would like to remove from the RDC, and the final responsibility and decision to release the output rests with the Analyst.

Confidentiality vetting is conducted using the survey-specific guidelines for all surveys housed in the RDCs. Questions or concerns with regards to the vetting process or related to unfamiliar statistical techniques are addressed with the RDC regional manager or with the RDC Vetting Committee.

Researchers are provided training during their orientation session related to the confidentiality vetting process, required documentation including the various analytical methods and completion of the disclosure request form for every output request.

Processes and procedures

A detailed and comprehensive draft document Disclosure Risk Analysis Guide for RDC Analysts provides detailed step-by-step instructions with illustrations and flow-charts on how to conduct and perform confidentiality vetting. Guidelines on disclosure risk analysis for various data types and descriptive or tabular output and variance-covariance and correlation matrices, graphs, models, and an example of a 'disclosure request form' are also included.

An important part of the process is for researchers to complete the "Vetting Request Form" (formerly known as Disclosure Request Form) which provides the required information for the Analyst to conduct and document the vetting request. Information required from the researcher includes the name of the output file, the name of the survey and cycles used in the analysis, the characteristics of the population being analyzed, the statistical procedure, the weights used, and if applicable, to verify the extent of residual disclosure risk. Researchers also must provide a description of the variables on the request form or in a separate document and are asked to produce the same outputs both weighted (by applying the survey weights) and un-weighted (outputs from the raw data) as supporting documentation for confidentiality vetting. Analysts rely on the un-weighted outputs to confirm that the results meet the thresholds of the minimum number of respondents necessary to produce releasable estimates. After the vetting is completed, the weighted outputs are released from the RDC to the researcher.

Confidentiality vetting guidelines and processes are found in the Researcher Guide. Printing is directed to the network printer which is controlled by the RDC analyst. Coloured paper is used for printing and anything on coloured paper cannot be taken out of the facility. This control enables RDC staff to visually detect what is being removed from the RDC.

The audit tested sixteen active and completed contracts to ensure that processes and procedures for confidentiality vetting are in place and effective. In three of the contracts sampled, no data had been sent for vetting. Of the thirteen contracts that required vetting of output, appropriate documentation was found in place. That is, a confidentiality vetting request, a list of variables, weighted and unweighted outputs, previously vetted data as well syntax (in some cases) was in place. The audit found that associated vetted folders contained evidence that confidentiality vetting was completed for all thirteen contracts, including checks of minimum cell counts, removal of unweighted output and suppression of weighted output in cases where confidentiality was at risk.

The audit determined that the RDC analyst ensures that Statistics Canada confidential data is not compromised by carefully administering and screening all confidentiality vetting requests.

Appendices

Appendix A: Audit criteria

Appendix B: Acronyms

Objectives, uses and users
Concepts, variables and classifications
Coverage and frames
Sampling
Questionnaire design
Response and nonresponse
Data collection and capture operations
Editing
Imputation
Estimation
Revisions and seasonal adjustment
Data quality evaluation
Disclosure control

1. Objectives, uses and users

1.1. Objective

The Monthly Retail Trade Survey (MRTS) provides information on the performance of the retail trade sector on a monthly basis, and when combined with other statistics, represents an important indicator of the state of the Canadian economy.

1.2. Uses

The estimates provide a measure of the health and performance of the retail trade sector. Information collected is used to estimate level and monthly trend for retail sales. At the end of each year, the estimates provide a preliminary look at annual retail sales and performance.

1.3. Users

A variety of organizations, sector associations, and levels of government make use of the information. Retailers rely on the survey results to compare their performance against similar types of businesses, as well as for marketing purposes. Retail associations are able to monitor industry performance and promote their retail industries. Investors can monitor industry growth, which can result in better access to investment capital by retailers. Governments are able to understand the role of retailers in the economy, which aids in the development of policies and tax incentives. As an important industry in the Canadian economy, governments are able to better determine the overall health of the economy through the use of the estimates in the calculation of the nation’s Gross Domestic Product (GDP).

2. Concepts, variables and classifications

2.1. Concepts

The retail trade sector comprises establishments primarily engaged in retailing merchandise, generally without transformation, and rendering services incidental to the sale of merchandise.

The retailing process is the final step in the distribution of merchandise; retailers are therefore organized to sell merchandise in small quantities to the general public. This sector comprises two main types of retailers, that is, store and non-store retailers. The MRTS covers only store retailers. Their main characteristics are described below. Store retailers operate fixed point-of-sale locations, located and designed to attract a high volume of walk-in customers. In general, retail stores have extensive displays of merchandise and use mass-media advertising to attract customers. They typically sell merchandise to the general public for personal or household consumption, but some also serve business and institutional clients. These include establishments such as office supplies stores, computer and software stores, gasoline stations, building material dealers, plumbing supplies stores and electrical supplies stores.

In addition to selling merchandise, some types of store retailers are also engaged in the provision of after-sales services, such as repair and installation. For example, new automobile dealers, electronic and appliance stores and musical instrument and supplies stores often provide repair services, while floor covering stores and window treatment stores often provide installation services. As a general rule, establishments engaged in retailing merchandise and providing after sales services are classified in this sector. Catalogue sales showrooms, gasoline service stations, and mobile home dealers are treated as store retailers.

2.2. Variables

Sales are defined as the sales of all goods purchased for resale, net of returns and discounts. This includes commission revenue and fees earned from selling goods and services on account of others, such as selling lottery tickets, bus tickets, and phone cards. It also includes parts and labour revenue from repair and maintenance; revenue from rental and leasing of goods and equipment; revenues from services, including food services; sales of goods manufactured as a secondary activity; and the proprietor’s withdrawals, at retail, of goods for personal use. Other revenue from rental of real estate, placement fees, operating subsidies, grants, royalties and franchise fees are excluded.

Trading Location is the physical location(s) in which business activity is conducted in each province and territory, and for which sales are credited or recognized in the financial records of the company. For retailers, this would normally be a store.

Constant Dollars: The value of retail trade is measured in two ways; including the effects of price change on sales and net of the effects of price change. The first measure is referred to as retail trade in current dollars and the latter as retail trade in constant dollars. The method of calculating the current dollar estimate is to aggregate the weighted value of sales for all retail outlets. The method of calculating the constant dollar estimate is to first adjust the sales values to a base year, using the Consumer Price Index, and then sum up the resulting values.

2.3. Classification

The Monthly Retail Trade Survey is based on the definition of retail trade under the NAICS (North American Industry Classification System). NAICS is the agreed upon common framework for the production of comparable statistics by the statistical agencies of Canada, Mexico and the United States. The agreement defines the boundaries of twenty sectors. NAICS is based on a production-oriented, or supply based conceptual framework in that establishments are groups into industries according to similarity in production processes used to produce goods and services.

Estimates appear for 21 industries based on special aggregations of the 2012 North American Industry Classification System (NAICS) industries. The 21 industries are further aggregated to 11 sub-sectors.

Geographically, sales estimates are produced for Canada and each province and territory.

3. Coverage and frames

Statistics Canada’s Business Register ( BR) provides the frame for the Monthly Retail Trade Survey. The BR is a structured list of businesses engaged in the production of goods and services in Canada. It is a centrally maintained database containing detailed descriptions of most business entities operating within Canada. The BR includes all incorporated businesses, with or without employees. For unincorporated businesses, the BR includes all employers with businesses, and businesses with no employees with annual sales that have a Goods and Services Tax (GST) or annual revenue that declares individual taxes.  annual sales greater than $30,000 that have a Goods and Services Tax (GST) account (the BR does not include unincorporated businesses with no employees and with annual sales less than $30,000).

The businesses on the BR are represented by a hierarchical structure with four levels, with the statistical enterprise at the top, followed by the statistical company, the statistical establishment and the statistical location. An enterprise can be linked to one or more statistical companies, a statistical company can be linked to one or more statistical establishments, and a statistical establishment to one or more statistical locations.

The target population for the MRTS consists of all statistical establishments on the BR that are classified to the retail sector using the North American Industry Classification System (NAICS) (approximately 200,000 establishments). The NAICS code range for the retail sector is 441100 to 453999. A statistical establishment is the production entity or the smallest grouping of production entities which: produces a homogeneous set of goods or services; does not cross provincial boundaries; and provides data on the value of output, together with the cost of principal intermediate inputs used, along with the cost and quantity of labour used to produce the output. The production entity is the physical unit where the business operations are carried out. It must have a civic address and dedicated labour.

The exclusions to the target population are ancillary establishments (producers of services in support of the activity of producing goods and services for the market of more than one establishment within the enterprise, and serves as a cost centre or a discretionary expense centre for which data on all its costs including labour and depreciation can be reported by the business), future establishments, establishments with a missing or a zero gross business income (GBI) value on the BR and establishments in the following non-covered NAICS:

• 4541 (electronic shopping and mail-order houses)
• 4542 (vending machine operators)
• 45431 (fuel dealers)
• 45439 (other direct selling establishments)

4. Sampling

The MRTS sample consists of 10,000 groups of establishments (clusters) classified to the Retail Trade sector selected from the Statistics Canada Business Register. A cluster of establishments is defined as all establishments belonging to a statistical enterprise that are in the same industrial group and geographical region. The MRTS uses a stratified design with simple random sample selection in each stratum. The stratification is done by industry groups (the mainly, but not only four digit level NAICS), and the geographical regions consisting of the provinces and territories, as well as three provincial sub-regions. We further stratify the population by size.

The size measure is created using a combination of independent survey data and three administrative variables: the annual profiled revenue, the GST sales expressed on an annual basis, and the declared tax revenue (T1 or T2). The size strata consist of one take-all (census), at most, two take-some (partially sampled) strata, and one take-none (non-sampled) stratum. Take-none strata serve to reduce respondent burden by excluding the smaller businesses from the surveyed population. These businesses should represent at most ten percent of total sales. Instead of sending questionnaires to these businesses, the estimates are produced through the use of administrative data.

The sample was allocated optimally in order to reach target coefficients of variation at the national, provincial/territorial, industrial, and industrial groups by province/territory levels. The sample was also inflated to compensate for dead, non-responding, and misclassified units.

MRTS is a repeated survey with maximisation of monthly sample overlap. The sample is kept month after month, and every month new units are added (births) to the sample.  MRTS births, i.e., new clusters of establishment(s), are identified every month via the BR’s latest universe. They are stratified according to the same criteria as the initial population. A sample of these births is selected according to the sampling fraction of the stratum to which they belong and is added to the monthly sample. Deaths occur on a monthly basis. A death can be a cluster of establishment(s) that have ceased their activities (out-of-business) or whose major activities are no longer in retail trade (out-of-scope). The status of these businesses is updated on the BR using administrative sources and survey feedback, including feedback from the MRTS. Methods to treat dead units and misclassified units are part of the sample and population update procedures.

5. Questionnaire design

The Monthly Retail Trade Survey incorporates the following sub-surveys:

Monthly Retail Trade Survey - R8

Monthly Retail Trade Survey (with inventories) – R8

Survey of Sales and Inventories of Alcoholic Beverages

The questionnaires collect monthly data on retail sales and the number of trading locations by province or territory and inventories of goods owned and intended for resale from a sample of retailers. The items on the questionnaires have remained unchanged for several years. For the 2004 redesign, the general questionnaires were subject to cosmetic changes only. The questionnaire for Sales and Inventories of Alcoholic Beverages underwent more extensive changes. The modifications were discussed with stakeholders and the respondents were given an opportunity to comment before the new questionnaire was finalized. If further changes are needed to any of the questionnaires, proposed changes would go through a review committee and a field test with respondents and data users to ensure its relevancy.

6. Response and nonresponse

6.1. Response and non-response

Despite the best efforts of survey managers and operations staff to maximize response in the MRTS, some non-response will occur. For statistical establishments to be classified as responding, the degree of partial response (where an accurate response is obtained for only some of the questions asked a respondent) must meet a minimum threshold level below which the response would be rejected and considered a unit nonresponse.  In such an instance, the business is classified as not having responded at all.

Non-response has two effects on data: first it introduces bias in estimates when nonrespondents differ from respondents in the characteristics measured; and second, it contributes to an increase in the sampling variance of estimates because the effective sample size is reduced from that originally sought.

The degree to which efforts are made to get a response from a non-respondent is based on budget and time constraints, its impact on the overall quality and the risk of nonresponse bias.

The main method to reduce the impact of non-response at sampling is to inflate the sample size through the use of over-sampling rates that have been determined from similar surveys.

Besides the methods to reduce the impact of non-response at sampling and collection, the non-responses to the survey that do occur are treated through imputation. In order to measure the amount of non-response that occurs each month, various response rates are calculated. For a given reference month, the estimation process is run at least twice (a preliminary and a revised run). Between each run, respondent data can be identified as unusable and imputed values can be corrected through respondent data. As a consequence, response rates are computed following each run of the estimation process.

For the MRTS, two types of rates are calculated (un-weighted and weighted). In order to assess the efficiency of the collection process, un-weighted response rates are calculated. Weighted rates, using the estimation weight and the value for the variable of interest, assess the quality of estimation. Within each of these types of rates, there are distinct rates for units that are surveyed and for units that are only modeled from administrative data that has been extracted from GST files.

To get a better picture of the success of the collection process, two un-weighted rates called the ‘collection results rate’ and the ‘extraction results rate’ are computed. They are computed by dividing the number of respondents by the number of units that we tried to contact or tried to receive extracted data for them. Non-monthly reporters (respondents with special reporting arrangements where they do not report every month but for whom actual data is available in subsequent revisions) are excluded from both the numerator and denominator for the months where no contact is performed.

In summary, the various response rates are calculated as follows:

Weighted rates:

Survey Response rate (estimation) =
Sum of weighted sales of units with response status i / Sum of survey weighted sales

where i = units that have either reported data that will be used in estimation or are converted refusals, or have reported data that has not yet been resolved for estimation.

Admin Response rate (estimation) =
Sum of weighted sales of units with response status ii / Sum of administrative weighted sales

where ii = units that have data that was extracted from administrative files and are usable for estimation.

Total Response rate (estimation) =
Sum of weighted sales of units with response status i or response status ii / Sum of all weighted sales

Un-weighted rates:

Survey Response rate (collection) =
Number of questionnaires with response status iii/ Number of questionnaires with response status iv

where iii = units that have either reported data (unresolved, used or not used for estimation) or are converted refusals.

where iv = all of the above plus units that have refused to respond, units that were not contacted and other types of non-respondent units.

Admin Response rate (extraction) =
Number of questionnaires with response status vi/ Number of questionnaires with response status vii

where vi = in-scope units that have data (either usable or non-usable) that was extracted from administrative files

where vii = all of the above plus units that have refused to report to the administrative data source, units that were not contacted and other types of non-respondent units.

(% of questionnaire collected over all in-scope questionnaires)

Collection Results Rate =
Number of questionnaires with response status iii / Number of questionnaires with response status viii

where iii = same as iii defined above

where viii = same as iv except for the exclusion of units that were contacted because their response is unavailable for a particular month since they are non-monthly reporters.

Extraction Results Rate =
Number of questionnaires with response status ix / Number of questionnaires with response status vii

where ix = same as vi with the addition of extracted units that have been imputed or were out of scope

where vii = same as vii defined above

(% of questionnaires collected over all questionnaire in-scope we tried to collect)

All the above weighted and un-weighted rates are provided at the industrial group, geography and size group level or for any combination of these levels.

Use of Administrative Data

Managing response burden is an ongoing challenge for Statistics Canada. In an attempt to alleviate response burden and survey costs, especially for smaller businesses, the MRTS has reduced the number of simple establishments in the sample that are surveyed directly and instead derives sales data for these establishments from Goods and Service Tax (GST) files using a statistical model. The model accounts for differences between sales and revenue (reported for GST purposes) as well as for the time lag between the survey reference period and the reference period of the GST file.

For more information on the methodology used for modeling sales from administrative data sources, refer to ‘Monthly Retail Trade Survey: Use of Administrative Data’ under ‘Documentation’ of the IMDB.

Table 1 contains the weighted response rates for all industry groups as well as for total retail trade for each province and territory. For more detailed weighted response rates, please contact the Marketing and Dissemination Section at (613) 951-3549, toll free: 1-877-421-3067 or by e-mail at retailinfo@statcan.

6.2. Methods used to reduce non-response at collection

Significant effort is spent trying to minimize non-response during collection. Methods used, among others, are interviewer techniques such as probing and persuasion, repeated re-scheduling and call-backs to obtain the information, and procedures dealing with how to handle non-compliant (refusal) respondents.

If data are unavailable at the time of collection, a respondent's best estimates are also accepted, and are subsequently revised once the actual data become available.

To minimize total non-response for all variables, partial responses are accepted. In addition, questionnaires are customized for the collection of certain variables, such as inventory, so that collection is timed for those months when the data are available.

Finally, to build trust and rapport between the interviewers and respondents, cases are generally assigned to the same interviewer each month. This action establishes a personal relationship between interviewer and respondent, and builds respondent trust.

7. Data collection and capture operations

Collection of the data is performed by Statistics Canada’s Regional Offices.

Weighted Response Rates

Respondents are sent a questionnaire or are contacted by telephone to obtain their sales and inventory values, as well as to confirm the opening or closing of business trading locations. Collection of the data begins approximately 7 working days after the end of the reference month and continues for the duration of that month.

New entrants to the survey are introduced to the survey via an introductory letter that informs the respondent that a representative of Statistics Canada will be calling. This call is to introduce the respondent to the survey, confirm the respondent's business activity, establish and begin data collection, as well as to answer any questions that the respondent may have.

8. Editing

Data editing is the application of checks to detect missing, invalid or inconsistent entries or to point to data records that are potentially in error. In the survey process for the MRTS, data editing is done at two different time periods.

First of all, editing is done during data collection. Once data are collected via the telephone, or via the receipt of completed mail-in questionnaires, the data are captured using customized data capture applications. All data are subjected to data editing. Edits during data collection are referred to as field edits and generally consist of validity and some simple consistency edits. They are used to detect mistakes made during the interview by the respondent or the interviewer and to identify missing information during collection in order to reduce the need for follow-up later on. Another purpose of the field edits is to clean up responses. In the MRTS, the current month’s responses are edited against the respondent’s previous month’s responses and/or the previous year’s responses for the current month. Field edits are also used to identify problems with data collection procedures and the design of the questionnaire, as well as the need for more interviewer training.

Follow-up with respondents occurs to validate potential erroneous data following any failed preliminary edit check of the data. Once validated, the collected data is regularly transmitted to the head office in Ottawa.

Secondly, editing known as statistical editing is also done after data collection and this is more empirical in nature. Statistical editing is run prior to imputation in order to identify the data that will be used as a basis to impute non-respondents. Large outliers that could disrupt a monthly trend are excluded from trend calculations by the statistical edits. It should be noted that adjustments are not made at this stage to correct the reported outliers.

The first step in the statistical editing is to identify which responses will be subjected to the statistical edit rules. Reported data for the current reference month will go through various edit checks.

The first set of edit checks is based on the Hidiriglou-Berthelot method whereby a ratio of the respondent’s current month data over historical (last month, same month last year) or auxiliary data is analyzed. When the respondent’s ratio differs significantly from ratios of respondents who are similar in terms of industry and/or geography group, the response is deemed an outlier.

The second set of edits consists of an edit known as the share of market edit. With this method, one is able to edit all respondents, even those where historical and auxiliary data is unavailable. The method relies on current month data only. Therefore, within a group of respondents, that are similar in terms of industrial group and/or geography, if the weighted contribution of a respondent to the group’s total is too large, it will be flagged as an outlier.

For edit checks based on the Hidiriglou-Berthelot method, data that are flagged as an outlier will not be included in the imputation models (those based on ratios). Also, data that are flagged as outliers in the share of market edit will not be included in the imputation models where means and medians are calculated to impute for responses that have no historical responses.

In conjunction with the statistical editing after data collection of reported data, there is also error detection done on the extracted GST data. Modeled data based on the GST are also subject to an extensive series of processing steps which thoroughly verify each record that is the basis for the model as well as the record being modeled. Edits are performed at a more aggregate level (industry by geography level) to detect records which deviate from the expected range, either by exhibiting large month-to-month change, or differing significantly from the remaining units. All data which fail these edits are subject to manual inspection and possible corrective action.

9. Imputation

Imputation in the MRTS is the process used to assign replacement values for missing data. This is done by assigning values when they are missing on the record being edited to ensure that estimates are of high quality and that a plausible, internal consistency is created. Due to concerns of response burden, cost and timeliness, it is generally impossible to do all follow-ups with the respondents in order to resolve missing responses. Since it is desirable to produce a complete and consistent microdata file, imputation is used to handle the remaining missing cases.

In the MRTS, imputation is based on historical data or administrative data (GST sales). The appropriate method is selected according to a strategy that is based on whether historical data is available, auxiliary data is available and/or which reference month is being processed.

There are three types of historical imputation methods. The first type is a general trend that uses one historical data source (previous month, data from next month or data from same month previous year). The second type is a regression model where data from previous month and same month previous year are used simultaneously. The third type uses the historical data as a direct replacement value for a non-respondent. Depending upon the particular reference month, there is an order of preference that exists so that top quality imputation can result. The historical imputation method that was labelled as the third type above is always the last option in the order for each reference month.

The imputation methods using administrative data are automatically selected when historical information is unavailable for a non-respondent. The administrative data source (annual GST sales) is the basis of these methods. The annual GST sales are used for two types of methods. One is a general trend that will be used for simple structure, e.g. enterprises with only one establishment, and a second type is called median-average that is used for units with a more complex structure.

10. Estimation

Estimation is a process that approximates unknown population parameters using only part of the population that is included in a sample. Inferences about these unknown parameters are then made, using the sample data and associated survey design. This stage uses Statistics Canada's Generalized Estimation System (GES).

For retail sales, the population is divided into a survey portion (take-all and take-some strata) and a non-survey portion (take-none stratum). From the sample that is drawn from the survey portion, an estimate for the population is determined through the use of a Horvitz-Thompson estimator where responses for sales are weighted by using the inverses of the inclusion probabilities of the sampled units. Such weights (called sampling weights) can be interpreted as the number of times that each sampled unit should be replicated to represent the entire population. The calculated weighted sales values are summed by domain, to produce the total sales estimates by each industrial group / geographic area combination. A domain is defined as the most recent classification values available from the BR for the unit and the survey reference period. These domains may differ from the original sampling strata because units may have changed size, industry or location. Changes in classification are reflected immediately in the estimates and do not accumulate over time. For the non-survey portion, the sales are estimated with statistical models using monthly GST sales.

For more information on the methodology for modeling sales from administrative data sources which also contributes to the estimates of the survey portion, refer to ‘Monthly Retail Survey: Use of Administrative Data’ under ‘Documentation’ of the IMDB.

The measure of precision used for the MRTS to evaluate the quality of a population parameter estimate and to obtain valid inferences is the variance. The variance from the survey portion is derived directly from a stratified simple random sample without replacement.

Sample estimates may differ from the expected value of the estimates. However, since the estimate is based on a probability sample, the variability of the sample estimate with respect to its expected value can be measured. The variance of an estimate is a measure of the precision of the sample estimate and is defined as the average, over all possible samples, of the squared difference of the estimate from its expected value.

11. Revisions and seasonal adjustment

Revisions in the raw data are required to correct known non-sampling errors. These normally include replacing imputed data with reported data, corrections to previously reported data, and estimates for new births that were not known at the time of the original estimates. Raw data are revised, on a monthly basis, for the month immediately prior to the current reference month being published. That is, when data for December are being published for the first time, there will also be revisions, if necessary, to the raw data for November. In addition, revisions are made once a year, with the initial release of the February data, for all months in the previous year. The purpose is to correct any significant problems that have been found that apply for an extended period. The actual period of revision depends on the nature of the problem identified, but rarely exceeds three years. Time series contain the elements essential to the description, explanation and forecasting of the behaviour of an economic phenomenon: "They are statistical records of the evolution of economic processes through time."¹ Economic time series such as the Monthly Retail Trade Survey can be broken down into five main components: the trend-cycle, seasonality, the trading-day effect, the Easter holiday effect and the irregular component.

The trend represents the long-term change in the series, whereas the cycle represents a smooth, quasi-periodical movement about the trend, showing a succession of growth and decline phases (e.g., the business cycle). These two components—the trend and the cycle—are estimated together, and the trend-cycle reflects the fundamental evolution of the series. The other components reflect short-term transient movements.

The seasonal component represents sub-annual, monthly or quarterly fluctuations that recur more or less regularly from one year to the next. Seasonal variations are caused by the direct and indirect effects of the climatic seasons and institutional factors (attributable to social conventions or administrative rules; e.g., Christmas).

The trading-day component originates from the fact that the relative importance of the days varies systematically within the week and that the number of each day of the week in a given month varies from year to year. This effect is present when activity varies with the day of the week. For instance, Sunday is typically less active than the other days, and the number of Sundays, Mondays, etc., in a given month changes from year to year.

The Easter holiday effect is the variation due to the shift of part of April’s activity to March when Easter falls in March rather than April.

Lastly, the irregular component includes all other more or less erratic fluctuations not taken into account in the preceding components. It is a residual that includes errors of measurement on the 1. A Note on the Seasonal adjustment of Economic Time Series», Canadian Statistical Review, August 1974.  A variable itself as well as unusual events (e.g., strikes, drought, floods, major power blackout or other unexpected events causing variations in respondents’ activities).

Thus, the latter four components—seasonal, irregular, trading-day and Easter holiday effect—all conceal the fundamental trend-cycle component of the series. Seasonal adjustment (correction of seasonal variation) consists in removing the seasonal, trading-day and Easter holiday effect components from the series, and it thus helps reveal the trend-cycle. While seasonal adjustment permits a better understanding of the underlying trend-cycle of a series, the seasonally adjusted series still contains an irregular component. Slight month-to-month variations in the seasonally adjusted series may be simple irregular movements. To get a better idea of the underlying trend, users should examine several months of the seasonally adjusted series.

Since April 2008, Monthly Retail Trade Survey data are seasonally adjusted using the X-12- ARIMA2 software. The technique that is used essentially consists of first correcting the initial series for all sorts of undesirable effects, such as the trading-day and the Easter holiday effects, by a module called regARIMA. These effects are estimated using regression models with ARIMA errors (auto-regressive integrated moving average models). The series can also be extrapolated for at least one year by using the model. Subsequently, the raw series—pre-adjusted and extrapolated if applicable— is seasonally adjusted by the X-11 method.

The X-11 method is used for analysing monthly and quarterly series. It is based on an iterative principle applied in estimating the different components, with estimation being done at each stage using adequate moving averages3. The moving averages used to estimate the main components—the trend and seasonality—are primarily smoothing tools designed to eliminate an undesirable component from the series. Since moving averages react poorly to the presence of atypical values, the X-11 method includes a tool for detecting and correcting atypical points. This tool is used to clean up the series during the seasonal adjustment. Outlying data points can also be detected and corrected in advance, within the regARIMA module.

Lastly, the annual totals of the seasonally adjusted series are forced to the annual totals of the original series.

Unfortunately, seasonal adjustment removes the sub-annual additivity of a system of series; small discrepancies can be observed between the sum of seasonally adjusted series and the direct seasonal adjustment of their total. To insure or restore additivity in a system of series, a reconciliation process is applied or indirect seasonal adjustment is used, i.e. the seasonal adjustment of a total is derived by the summation of the individually seasonally adjusted series.

12. Data quality evaluation

The methodology of this survey has been designed to control errors and to reduce their potential effects on estimates. However, the survey results remain subject to errors, of which sampling error is only one component of the total survey error. Sampling error results when observations are made only on a sample and not on the entire population. All other errors arising from the various phases of a survey are referred to as nonsampling errors. For example, these types of errors can occur when a respondent provides incorrect information or does not answer certain questions; when a unit in the target population is omitted or covered more than once; when GST data for records being modeled for a particular month are not representative of the actual record for various reasons; when a unit that is out of scope for the survey is included by mistake or when errors occur in data processing, such as coding or capture errors.

Prior to publication, combined survey results are analyzed for comparability; in general, this includes a detailed review of individual responses (especially for large businesses), general economic conditions and historical trends.

A common measure of data quality for surveys is the coefficient of variation (CV). The coefficient of variation, defined as the standard error divided by the sample estimate, is a measure of precision in relative terms. Since the coefficient of variation is calculated from responses of individual units, it also measures some non-sampling errors.

The formula used to calculate coefficients of variation (CV) as percentages is:

CV (X) = S(X) * 100% / X
where X denotes the estimate and S(X) denotes the standard error of X.

Confidence intervals can be constructed around the estimates using the estimate and the CV. Thus, for our sample, it is possible to state with a given level of confidence that the expected value will fall within the confidence interval constructed around the estimate. For example, if an estimate of $12,000,000 has a CV of 2%, the standard error will be $240,000 (the estimate multiplied by the CV). It can be stated with 68% confidence that the expected values will fall within the interval whose length equals the standard deviation about the estimate, i.e. between $11,760,000 and $12,240,000.

Alternatively, it can be stated with 95% confidence that the expected value will fall within the interval whose length equals two standard deviations about the estimate, i.e. between $11,520,000 and $12,480,000.

Finally, due to the small contribution of the non-survey portion to the total estimates, bias in the non-survey portion has a negligible impact on the CVs. Therefore, the CV from the survey portion is used for the total estimate that is the summation of estimates from the surveyed and non-surveyed portions.

13. Disclosure control

Statistics Canada is prohibited by law from releasing any data which would divulge information obtained under the Statistics Act that relates to any identifiable person, business or organization without the prior knowledge or the consent in writing of that person, business or organization. Various confidentiality rules are applied to all data that are released or published to prevent the publication or disclosure of any information deemed confidential. If necessary, data are suppressed to prevent direct or residual disclosure of identifiable data.

Confidentiality analysis includes the detection of possible "direct disclosure", which occurs when the value in a tabulation cell is composed of a few respondents or when the cell is dominated by a few companies.