Audit of the Business Register

Table of contents

• Executive Summary
• Introduction
  • Background
  • Audit Objectives
  • Scope
  • Approach and Methodology
  • Authority
• Findings, Recommendations and Management Response
  • Framework Supporting the Business Register
  • Control Mechanisms for the Business Register
  • Data Processing
  • Maintenance of Business Register data
  • Information Provided to Clients
• Appendices
  • Appendix A: Audit Criteria
  • Appendix B: Acronyms

Executive Summary

The Business Register (BR) is the central repository of information on businesses in Canada. The BR is used as the principal frame for the economic statistics program of Statistics Canada ("the agency"). The BR maintains a complete, up-to-date and unduplicated list of all active businesses (six million) in Canada. The role of the BR is to provide the frame for Statistics Canada's data collection activities (i.e., surveys) and a comprehensive listing of business entities from which survey samples are selected.

Business records maintained in the BR are classified as either simple or complex business structures. The complex portion of the BR represents approximately 1% of the total active businesses in the database, and it accounts for approximately 52% of total economic activity in Canada. The simple portion represents approximately 99% of the total active businesses in the database, and it accounts for approximately 48% of total economic activity in Canada.

One of the primary methods of updating the BR is through profiling. Profiling involves conducting Internet research, telephone calls and on-site visits to obtain legal and operating structure, business address, major activity, business number based on the North American Industry Classification System (NAICS), relationships, accounting practices, and all pertinent financial information about a company. Representatives of two programs—the BR Program and the Enterprise Portfolio Management Program in the Enterprise Statistics Division—conduct profiling activities.

Stakeholders across the agency play a significant role in maintaining the quality of the data in the BR. The agency's Administrative Data Division liaises with the Canada Revenue Agency (CRA) to gather tax administrative data and carry out validation procedures before uploads to the BR. Subject-matter divisions (SMDs), including regional offices, are responsible for working with the BR Program; they make updates to BR data, as well as receive adequate output files to determine samples and develop surveys.

The objectives of the audit were to provide the Chief Statistician and the Departmental Audit Committee with assurance of the following:

• Statistics Canada has established an adequate governance framework to support the quality of the BR.
• Effective control mechanisms have been established and are consistently applied to ensure the maintenance of quality data within the BR in accordance with the agency's Quality Guidelines.

The scope of this audit included an examination of the quality assurance framework for the data in the BR. The audit covered the period of April 1, 2015, to January 31, 2016.

Why is this important?

Statistics Canada continues to execute its mandate by providing Canadians with quality information about the state of Canada's economy. The BR is essential to the Statistics Canada to provide the frame for the economic statistics program. Thus, it is essential to establish an adequate governance framework and effective control mechanisms to ensure the maintenance of quality data within the BR in accordance with the agency's Quality Guidelines. To date an audit of BR has not been completed at the Agency.

Key Findings

Overall, roles, responsibilities and accountabilities of the key personnel responsible for the quality of the BR, including stakeholders outside the BR, are well understood; however, in some cases, these roles and responsibilities are not formally documented. Although many stakeholders are involved, and hold vested interests in the quality of the BR data, no overarching accountability framework has been established to integrate the roles, responsibilities, accountabilities and interdependencies of all stakeholders. Risks for the BR have not been formally identified and validated but are discussed at key management meetings on an on-going basis.

Formal processes have been established to validate and test updates to data prior to uploading to the BR; however, there is limited documentation on the results of this process, specifically issues identified and any follow-up procedures conducted for specific data.

Processes in place are adequate for ensuring that profiling of business activities and the resulting updates to the BR are performed. Guidance documents have been developed to support staff in performing their day-to-day activities. However, no specific criteria have been established to prioritize the notifications of updates (messages) to the BR received by profilers, nor have targets or timelines been developed for the resolution of the messages.

A quality assurance program has been established to assess the quality of key BR information, which continues to be reviewed and updated. There are opportunities to improve the following:

• Profiling activities within the BR should be monitored formally to ensure that activities and results are being documented and that approaches are consistent across supervisors.
• Appropriate quality assurance mechanisms should be applied to profiling activities across the agency.
• Although training and guidance processes are well defined and documented, a refresher course should be developed, and more practical examples and exercises should be included in the training content.

Processes exist and have been documented for validating production data files used for surveys; however, there is limited evidence regarding the results of the validation process, including tracking issues, communicating with SMDs and the final resolution.

Overall Conclusion

The BR Program has established a governance structure to ensure the ongoing quality of the BR. The increasing complexity of the BR, and the number of stakeholders who are responsible for contributing to the quality of the BR, have made it necessary to look for ways to improve governance and risk management. This could happen by formalizing the accountability framework outlining the roles, responsibilities, accountabilities and interdependencies of all BR stakeholders; as well as formally identifying and managing risks as part of the agency-wide risk management framework.

Overall, the audit identified that effective control mechanisms are in place, and are being consistently applied to ensure the ongoing quality of the BR. Strengthening the formality of specific elements of the quality assurance program would ensure greater efficiency and continued compliance with the Statistics Canada Quality Guidelines.

Conformance with Professional Standards

This audit engagement conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of a quality assurance and improvement program.

Sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the findings and conclusions in this report and to provide an audit level of assurance. The findings and conclusions are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria. The findings and conclusions are applicable to the entity examined and for the scope and time period covered by the audit.

Steven McRoberts
Chief Audit & Evaluation Executive

Introduction

Background

Quality is Statistics Canada's hallmark. The agency strives to build relevance and quality into all of its programs and products. The quality of official statistics is founded on the use of sound scientific methods adapted over time to meet changing client needs, as well as to the changing reality that the agency aims to measure, and it is dependent on the capacity or willingness of respondents to supply reliable and timely data. The Statistics Canada Quality Guidelines define quality in terms of six elements: Relevance, Accuracy, Timeliness, Accessibility, Interpretability and Coherence. Together, these elements help program managers to handle the multi-dimensional nature of quality.

The Business Register (BR) is the central repository of information on businesses in Canada. The BR is used as the principal frame for the economic statistics program of Statistics Canada. It maintains a complete, up-to-date and unduplicated list of all active businesses (approximately 6 million) in Canada that have a corporate income tax (T2) account, are an employer (PD7) or have a GST account, as well as sole proprietors who report business income (T1). The role of the BR is to provide the frame for Statistics Canada's data collection activities (i.e., surveys) and a comprehensive listing of business entities from which survey samples are selected. The BR includes key attributes such as industrial classification, revenues and number of employees, which are used to stratify survey samples to improve sampling efficiency. Approximately 200 business surveys use the BR to support their activities, which include establishing a survey frame, sampling, collecting and processing data, and producing estimates.

Business records maintained within the BR are classified as either simple or complex business structures. Complex business structures within the BR represent approximately 1% of the total active businesses in the database, and they account for approximately 52% of total economic activity in Canada. The simple structures represent approximately 99% of the total active businesses in the database, and they account for approximately 48% of total economic activity in Canada.

Data are updated through profiling activities. Profiling is the process of conducting in-depth telephone or on-site interviews with senior company representatives to obtain all pertinent financial information, relationships and structures about a business. Changes and corrections to frame data in the BR are also transmitted regularly by survey collection areas during the collection of economic survey data. Research-gathering tools such as the Internet, provincial gazettes, trade and business publications, and newspaper clippings are also used to update BR data.

Profiling activities for the BR are conducted primarily by two programs: BR profilers within the Statistical Registers and Geography Division (SRGD) and Enterprise Portfolio Management (EPM) profilers within the Enterprise Statistics Division (ESD). EPM profilers have been assigned approximately 350 of the largest enterprises in the BR. BR profilers are responsible for profiling all other enterprises classified as complex business structures. BR and EPM profilers are responsible for profiling and updating data for complex businesses based on administrative data and messages from subject-matter divisions (SMDs), in regional offices, and collection services indicating that the data need to be updated.

Stakeholders across the agency play a significant role in maintaining the quality of the data in the BR. The Administrative Data Division (ADD) is the liaison between Statistics Canada and the Canada Revenue Agency (CRA) for tax data and validation testing of changes in the administrative data provided by the CRA to ensure alignment with the BR System. The SMDs, including regional offices, are responsible for providing updated information to the BR. They either send messages to the profilers for complex businesses or make direct updates themselves for simple businesses, based on survey feedback. SMDs also work with survey account managers (SAMs) to ensure that the output files received from the BR data meet SAMs' needs and that SAMs can rely on the BR data for the creation of surveys.

The following diagram represents the primary BR business process.

A quality assurance framework has been established within the BR Program to validate and analyze the main input sources and main output. Error detection is performed on an ongoing basis using various methods, both internal and external to the BR Program. A Quality Assurance Survey is carried out bimonthly, and is designed primarily to draw conclusions on the quality of all North American Industry Classification System (NAICS) coding in the BR, specifically error rate (proportion of businesses coded incorrectly), volatility rate (proportion of businesses with an outdated NAICS) and death rate (proportion of inactive businesses). Error rates and death rates are produced at the industrial sector level based on the survey. Then, the classification and business status are updated accordingly in the BR. Ongoing quality measures have also been established and implemented within the BR Program. Specifically, the BR verifies 100% of updates made by new staff, and supervisors regularly perform spot checks on any changes to the BR made by BR profilers. An extensive training program is in place for those who are tasked with updating the BR.

Audit Objectives

The objectives of the audit were to provide the Chief Statistician and the Departmental Audit Committee with assurance of the following:

• Statistics Canada has established an adequate governance framework to support the quality of the BR.
• Effective control mechanisms have been established and are consistently applied to ensure the maintenance of quality data within the BR in accordance with the agency's Quality Guidelines.

Scope

The scope of the audit included the following areas, to assess whether:

• Governance, roles, responsibilities and accountabilities for the quality of data within the BR Program as well as for other users who access and update the BR are clear and well communicated;
• Management identifies and assesses the risks that may preclude the achievement of quality data for the BR;
• Effective processes have been established for the input of data into the BR, manual updates made to the BR data, and the production of data extracted from the BR and provided to clients for the development of surveys; and,
• Training and guidance provided to users of the BR is adequate to carry out responsibilities pertaining to the BR and for ensuring the quality of the data maintained in the BR.

The scope of this audit included an examination of the quality assurance framework for the data in the BR. The audit covered the period of April 1, 2015, to January 31, 2016.

The criteria for this audit are presented in Appendix A.

Approach and Methodology

The audit work consisted of a comprehensive review and analysis of relevant documentation, interviews with key management and staff, and testing to assess the effectiveness of processes in place.

The fieldwork included review and testing of the BR Program's processes and procedures in place related to the quality and accuracy of the BR information.

The audit assessed the BR Program against elements of the Statistics Canada Quality Guidelines. Based on the risk assessment, this audit was primarily focused on the accuracy and coherence elements of those guidelines.

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors International Professional Practices Framework.

Authority

The audit was conducted under the authority of the approved Statistics Canada Integrated Risk-Based Audit and Evaluation Plan 2015/2016 to 2019/2020.

Findings, Recommendations and Management Response

Framework Supporting the Business Register

By design, the BR Program relies on several organizations across the agency to support the maintenance of the quality of the BR data. This includes the ADD, which liaises with the CRA and validates the data received from the CRA, and SMDs, which make updates to the BR as well as receive output files for survey samples.

Given the number of stakeholders involved in maintaining the BR, a robust governance and accountability framework is essential to ensure the quality of the BR. Roles, responsibilities and accountabilities for the quality of data within the BR, including the ADD and SMDs, should be clearly documented and well understood, with formal oversight mechanisms to monitor the quality of BR data.

No formal framework has been established to outline key roles, responsibilities, accountabilities and interdependencies of the parties involved.

A key component of governance structures is the establishment of roles, responsibilities and accountabilities of stakeholders. The audit team noted that although the roles, responsibilities and accountabilities of the key personnel responsible for the quality of the BR are appropriate and well understood, these roles and responsibilities have been documented at different levels of formality.

The BR Program has developed, in addition to formal job descriptions, A Brief Guide to the BR, which outlines the various roles and responsibilities of the different sections in the BR. However, the guide does not identify the external stakeholders who influence data quality in the BR. The EPM Program also maintains its own documentation of profiling activities and associated responsibilities.

While ADD is responsible for performing validation procedures on the data files received from the CRA to identify errors or outliers within the data prior to their being uploaded to the BR, these roles and responsibilities are not formally documented.

A service level agreement (SLA) does exist between the BR and SMDs, including regional offices, which outlines the specific roles and responsibilities of SMDs regarding the BR. Although the SLA does not outline specific tasks on the quality of data, some of the duties described affect and improve data quality.

Two committees, the BR Liaison Committee and the BR Strategic Issues Committee, have been established, representing stakeholders across the agency, to support the quality of BR data, including the following:

• The BR Liaison Committee functions at an operational level, and is responsible for informing all SMDs of any changes related to the BR frame that might affect their survey programs. The committee is comprised of representatives from the SRGD, ADD, all client survey managers, methodologists from the Business Survey Methods Division, and systems representatives from the System Development Division.
• The Assistant Director of the SRGD chairs the BR Strategic Issues Committee. It is made up of assistant directors from the BR's clients, including SMDs. This committee meets monthly to discuss strategic issues affecting the BR, and discusses how to address these issues. SMD representatives play a role in representing the interests of their division and raise key issues and challenges faced.

There is a mandate document for each of these two committees, yet no formal Terms of Reference have been established for the committees outlining their responsibilities and authority. In each case, formal meeting minutes are maintained, and action items are taken by the BR Program to the management team for appropriate resolution and reporting back.

As outlined above, given the number of organizations involved in maintaining the quality of the BR data, no overarching framework has been established to define stakeholders and their responsibilities and interdependencies. Without a formal accountability framework that outlines individual roles and responsibilities, there is potential for overlap or gaps in coverage relative to the quality of the BR data, and key interdependencies may be overlooked.

In the BR Program, the management structure in place is leveraged to ensure ongoing management and oversight of the day-to-day quality of the BR.  Risks for the BR have not been formally identified and validated as part of the agency's risk management framework since 2014. The management team meets regularly to discuss and resolve operational risks and issues that have materialized. Risks and issues are also presented by other stakeholders through various governance committees and are brought back to the management committee for discussion and resolution.

Recommendation

It is recommended that the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure, ensure that:

• A formal accountability framework be developed and communicated for the BR, outlining the roles, responsibilities and accountabilities of all BR stakeholders, including relevant committees, and their interdependencies, as these pertain to the mandate of the BR. This accountability framework should outline the escalation processes for the resolution of issues and change requests.
• Risks are formally identified and managed as part of the agency-wide risk management framework.

Management Response

Management agrees with the recommendation.

• A formal governance and issue/change resolution framework will be defined and developed for the Business Register and validated and communicated with stakeholder managers and committees in the subject-matter, survey collection, methodology, national accounting and IT areas. Having a well-defined framework will enhance the benefits of the Register as a backbone to the economic statistics programs of Statistics Canada, and ensure that we can further enrich the coherence and efficiency of our statistical programs in general.
• A formal risk management process for the Business Register will be developed. Risks will be identified and documented and appropriate mitigation strategies will be developed.

Deliverables and Timeline:

• The Director General, Statistical Registers and Geography Division and the Director, Communication and Dissemination Branch will develop a new accountability framework and communication plan by June 2017.
• The Director General, Statistical Registers and Geography Division will work with the Integrated Risk Management coordination group to formalize the risk management process for the Business Register. Development of the risk management processes will take place in 2017-2018 with updated risks documented by March 31, 2018.

Control Mechanisms for the Business Register

Data Processing

ADD is the liaison between Statistics Canada and the CRA for tax data. It is therefore important for processes to be established for ADD to be informed of and to perform validation testing on changes in the administrative data provided by the CRA, to ensure alignment with the BR System. An effective validation process should also be established to confirm the reliability, completeness and accuracy of the data provided by the CRA prior to its being integrated in the BR.

Overall, there is an effective process to validate changes in data formats received from the CRA.

A Memorandum of Understanding (MOU) exists between Statistics Canada and the CRA, outlining the roles and responsibilities of each party concerning the CRA's providing data to ADD. The MOU outlines a requirement for the CRA to inform ADD of any changes in the administrative data provided. These changes often include changes to the format of the files provided, including the data fields used, which affect the compatibility of the data with the BR System and require ADD to make edits to properly adjust the formatting. When ADD is notified of changes in the administrative data, validation procedures are required to ensure that the format of the data will continue to be compatible with the BR. As necessary, changes to the BR are required before new data files can be loaded and relied upon.

As an example, every month, the T2 Unit within ADD processes and loads new and reassessed information received from the CRA. Before loading these data into the BR, the T2 Unit runs several verification programs designed to check for specific errors and make the appropriate corrections, based on defined variables for T2 data. These variables are based on fields within the data, such as "end date" and "number of months." New variables for these data are received twice per year by the CRA, based on any changes made to the format of data files. The complexity of the new or deleted variables determines the turnaround time for ADD to put these variables into production. The audit reviewed the list of T2 variables added by the CRA for these data as of December 31, 2015, and confirmed that the variables had been updated for the current data released.

Limited documentation exists relative to the results of the validation of CRA data files.

The ADD is responsible for performing validation on the data files received from the CRA prior to their being uploaded into the BR, to ensure the quality and integrity of the data. This validation involves verifying record linkages, data variables, the length and number of records, and irregular fluctuations in revenue as well as ensuring that no errors occur when the data is converted to other formats.

The audit team was able to review documentation outlining the validation processes performed on the data, as well as the ways to confirm the accuracy of the data prior to its being uploaded to the BR. Every month, the ADD processes and loads new and reassessed information received from the CRA for approximately 220 T2 schedules. Before loading these data, the T2 Unit within ADD runs several validation programs designed to check for specific errors and make the appropriate corrections, and the audit noted that these validation procedures are documented.

The BR expects the ADD to provide usable quality data to be uploaded to the BR, and that the ADD perform all T2, GST values and edits required. Through the audit team's walkthrough of the validation procedures, review of resulting documentation, and discussions with BR Program management, the audit team confirmed that validation procedures carried out by ADD meet the BR's needs regarding the quality of data uploaded to the BR.

The ADD monitors its validation processes through a monthly monitoring report, which lists the counts of records that were affected by each edit process. The audit reviewed examples of the output tables created during the validation process. These tables are generated throughout the process, and reviewed and maintained; however, limited documentation exists around the outliers and issues identified from reviewing these tables and any follow-up procedures performed, specifically for T2 data. This could result in issues not being appropriately followed up on and remediated for an appropriate resolution. Results with respect to GST data are documented, and issues are formally tracked.

Maintenance of Business Register data

To align with the Statistics Canada Quality Guidelines and provide the most accurate business data to stakeholders, there must be control mechanisms in place to ensure that all updates made to the BR records are appropriate. An adequate quality assurance program should be in place to assess the quality of the information maintained within the BR. Further, adequate training and guidance should be provided to all users of the BR in support of reliable, complete, accurate and timely updates made to these data.

Processes and associated guidance documents have been established to support day-to-day profiling activities.

The EPM program is responsible for performing profiling activities for over 350 of Canada's largest and most complex business enterprises. This program has a team of 10 profilers, and each has a portfolio of 32 to 35 businesses, with the requirement to complete 8 to 10 full parent BR profile reviews during a year.

Profilers in the BR program are responsible for profiling all other complex businesses (approximately 32,000 businesses), and each profiler is assigned a portfolio of businesses. Each profiler is expected to complete approximately 30 to 40 business profiles annually and is responsible for responding to notifications of changes or messages relative to his or her portfolio.

The BR Profiling section has developed several guides and training manuals to support profiling activities. For example, a Checklist for the Profilers document describes the steps for updating the BR. Another guide is Interviewer's Manual for the Quality Assurance Profiling describes the procedures for Quality Assurance Survey on NAICS. Feedback obtained during the audit from profilers and supervisors indicated that the guidance documents provided are adequate and support daily activities. Overall, the profiling activities undertaken align with the Statistics Canada Quality Guidelines.

Limited criteria have been established for profilers to prioritize and address notifications of changes or updates (messages) to the BR data, other than messages affecting monthly or annual surveys.

Messages represent all requests for updates and information received during collection of data for complex structures. The collection of this information may originate from various sources, such as collection centres, regional offices, SMDs and the CRA. Messages sent are automatically routed to the profiler assigned to that portfolio, to perform required manual updates. A message can be marked as either "critical" or "non-critical," which is selected by the originator. Types of messages include request to change legal name, change of address affecting the province, cessation of the operation entity (or bankruptcy), and changes to the NAICS.

Based on interviews and walkthroughs performed within the BR Program, when a message is received, the profiler carries out research or contacts the business owner to confirm the information in the message. If necessary, the update is made in the BR and the message status is changed to "update applied," with the justification documented in an entry journal in the BR System. Based on the information gathered or research conducted, if the profiler finds that the proposed change is inaccurate or inappropriate and, therefore, not required, no update is made. The message is then cancelled, and the justification is documented in the journal. Supervisors review the messages periodically, but no documentation is maintained to demonstrate that a review was performed.

Profilers receive a significant number of messages and have responsibility to determine how to prioritize these messages. As of March 31, 2016, 56,204 unanswered messages remained in the BR. Although profilers try to give priority to messages affecting monthly surveys and ensure that all messages affecting annual surveys are addressed by December 31, limited criteria have been established for the prioritization of the messages received by profilers. Further, no targets or timeframes have been outlined for the resolution of the messages. This increases the risk that necessary changes are not made to the BR data in a timely manner, thus reducing the quality of information contained in the BR.

A formal quality assurance program has been established to assess the quality of key BR information. As this program continues to evolve, there are opportunities to improve the consistency of specific elements.

Two levels of quality assurance are applied to evaluate the quality of the data maintained within the BR. The first level is performed at the micro level within the BR profilers section and the EPM program. The second is performed at a macro level by the Concepts, Quality Assurance and Training (CQAT) section. Thus, the Quality Assurance team ensures that the quality of the data in the BR System is adequate through the establishment of several quality assurance mechanisms. The results of the quality assurance testing performed are used to draw conclusions about the quality of all NAICS coding in the BR. These results are reported to the methodologists and then communicated to management through dashboard reports.

The monitoring (i.e., spot checks) of the BR profiling activities is not formalized and standardized.

Profiler supervisors have responsibility to review the profiling activities of profilers. For new profilers, supervisors review all profiles completed by the profiler. Supervisors perform spot checks on more experienced profilers carrying out telephone surveys and making updates to the BR. Then, supervisors provide comments to profilers on the adequacy of the activities carried out. However, no evidence is maintained for the review/spot checks performed with respect to indicating how the profiles that were spot checked were selected, and the outcome of the spot checks performed.

The audit conducted a walkthrough of spot checks performed by a BR supervisor on BR profiling activities, and confirmed that spot checks were performed adequately and that any issues were escalated and mitigated appropriately. However, no documentation was maintained to support these quality assurance activities. During the audit, supervisors confirmed that there is no standard frequency or consistent process to perform spot checks among supervisors. As a result, it is at the supervisors' discretion to determine how they conduct spot checks and how many they conduct.

Review of profiling activities conducted by EPM profilers differs from those conducted by BR profilers, since EPM profilers deal with the approximately 350 largest and most complex businesses; they are also responsible for activities such as collection and surveys. Spot checks are not performed when EPM profilers do manual updates in the BR. Instead, unit heads review the profiling activities once a profile is complete for an enterprise and document the results in an enterprise report document. The audit confirmed that any comments or questions raised by unit heads during their review are adequately followed up and documented in the report.

Quality assurance profiling for the NAICS is performed adequately.

Bimonthly, the Quality Assurance Profiling (QAP) team performs telephone surveys for approximately 650 businesses in the BR to determine whether the NAICS coding of active businesses in the BR is accurate or requires updating. The sample is selected by the Methodology team and based on active businesses that have not been contacted over the past two years. The QAP team is made up of a separate group of profilers who contact the businesses to obtain the required information and determine whether a change is required to the NAICS coding.

The audit tested a sample of 20 businesses that had already been tested under the QAP. Audit testing indicated that a spreadsheet is maintained showing all businesses tested, all updates required, and comments provided on the information obtained. More detailed comments on procedures performed, as well as detailed information on the industry and structure are documented in the journal within the BR. The audit work confirmed that when a change was required to the NAICS, it was accurately updated in the BR. For all updates made, a log shows which field was updated, who made the update, and when the update was made. Based on the testing performed by the audit team, the QAP is performed adequately and the results are sufficiently documented.

Review of incoherence reports results were followed up, and corrections were made in the BR.

Quality assurance is provided at a macro level by the CQAT through the review of incoherence reports to ensure that the General Survey Universe File (G-SUF) is adequate for the production of various surveys. Incoherence reports are generated monthly using the final G-SUF and running queries on various types of BR data to identify irregularities in specific key data fields. The G-SUF provides a preview of the population of units in scope for survey programs and is generated on a monthly basis for SMDs to review when developing their surveys. To be able to provide the most accurate data possible to SMDs, incoherence reports are run on BR data to identify potential anomalies within the data and to show where updates may be required.

Incoherence reports are run for the following:

• legal operating entity, part of a consolidation, that have GST but no NAICS
• T4 parents in a consolidation
• PD7 (employees) incoherencies
• income tax returns incoherencies
• NAICS incoherencies
• inactive reporting entity
• operating entity with a single direct child
• participants in a consolidation with no NAICS
• cost centre production with no revenue

Only incoherencies appearing on two of the nine reports (T4 parents in a consolidation, and inactive reporting entity) need to be addressed before the production of the final G­-SUF, because those have a great impact on the quality of data for the production of surveys. The remaining incoherence reports are considered non-critical and are provided to the BR profilers, EPM and specific SMDs to address. Audit testing confirmed that for the two critical incoherence reports, results were followed up and corrections were made in the BR.

With the aim of continuing to evolve and refine the quality assurance program within the BR, the CQAT has developed Enterprise Quality Assurance Indicators to assess the enterprise quality by looking at various indicators of profiling, including messages, incoherencies, T4 data and the profile date. The CQAT indicated that the new indicators will help to better prioritize the profiling activities and allow the quality of the profiling work performed for each enterprise in the BR to be assessed.

Quality assurance processes to oversee profiling activities and changes made by SMDs who have Direct Updater access are not subject to the same level of quality assurance monitoring as for BR profilers.

The majority of profiling activities and changes to the BR are performed by BR and EPM profilers; however, the audit team observed that many SMD users had Direct Updater access, with the ability to make edits not only to the data but also to the enterprise structures—consistent with BR access given to profilers. It was confirmed that, by design, specific SMDs are performing direct profiling activities to ensure that survey requirements are being met and to allow for more timely and accurate updates to the BR.

Although the changes made by SMDs directly to the BR would be subject to some of the quality assurance activities, others that are specifically applied to profilers (i.e., spot checks) would not be applied to the SMDs, which could result in inaccurate updates to the BR.

Training and guidance processes are well defined and documented. In general, profilers and users of the BR have received proper training before being granted access to the BR.

The BR Program has established mandatory training to obtain access to the BR System and has developed training courses adapted for users according to their roles, responsibilities and required access to the BR.

The Chief of the BR Profiling section is responsible for training for all staff. The CQAT provides or coordinates training for all other users of the BR, including the regional offices that perform profiling activities.

A previous audit identified a finding about the training provided to employees in regional offices. The associated recommendation suggested that training be tailored to address specific needs and weaknesses of regional employees. As this included training on the BR, the BR Program considered this recommendation; training has since been streamlined to focus on key concepts of the BR.

The audit team found that the training programs and associated training materials are well defined and documented, and appropriate tracking for completion of training is maintained. Interviews with several users of the BR confirmed that sufficient training and training materials have been received, and they are useful for conducting profiling activities. It was noted, however, that providing a refresher course for users periodically would be useful, especially to support SMD and regional office users of the BR. Further, feedback from regional offices identified the opportunity to include more practical examples and exercises within the training content.

Information Provided to Clients

Effective control mechanisms for the production of BR data products help the BR Program to meet stakeholders' quality expectations. In accordance with the Statistics Canada Quality Guidelines, an effective review process should be established to identify and address potential data errors, to ensure that the data released by the BR is relevant, accurate, timely and coherent.

Processes have been established for validating the production files used for the creation of surveys; however, the results have not been formally documented.

The survey account manager (SAM) team performs validation on the G-SUF, since SMDs choose their survey sample from this file and SAMs must ensure that samples have been pulled correctly and adhere to the characteristics of the survey. A Survey Processing Rules document was developed in March 2014 that provides an overview of the process, and gives examples to assist SMDs in determining which rules would be most beneficial for their surveys. This document also includes a template for noting the applicable rules, which is sent back to SAMs.

The audit reviewed examples of the files generated and how validation procedures are performed on these files. Results of the procedures applied and the issues identified, as well as correspondence with SMDs to resolve these issues, are not formally documented. The BR Program places heavy reliance on the knowledge and experience of the SAMs to identify and resolve issues. Without formally tracking and documenting the communication and resolution of the identified issues, there is the potential that issues might not be resolved, which would affect the quality of the final production files.

Recommendations

It is recommended that the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure ensure that:

• A standardized process with accompanying tools be established for the tracking of all outliers and issues identified as a result of the validation process carried out by ADD, including documentation of correspondence between the CRA and the BR, and the resolution of the issues.
• The quality assurance program is strengthened through :
  • The establishment of criteria for profilers to automatically prioritize messages received and targets should be established and monitored to ensure messages are being addressed accordingly;
  • the establishment of a standardized monitoring approach for BR supervisors and formal documentation of monitoring activities;
  • a consistent level of quality assurance activities are being applied to profiling activities across the organization by SMDs;
  • the inclusion of more practical examples and exercises as part of the existing training curriculum.
• A standardized process with accompanying tools be established for the SAM team which outlines the validation procedures to be carried out; specifically how to track and document the communication and resolution of identified system errors.

Management Response

Management agrees with the recommendations.

The development of new tools and processes will be done as part of the approved Continuity and Quality Maintenance (CQM) items that are planned for the T2 redesign that will take place from April 2017 to March 2020. By March 2017, ADD will review the existing tools and procedures and specify the required changes.

In the interim, shorter-term mechanisms will be identified and implemented that can address the needs prior to having the fully integrated solutions to be developed as part of the CQM.

There is a system to establish priorities that is communicated to employees. However, there is no system that allows the program to monitor the handling of the messages according to the various priorities.

A more automatic means of identifying priority messages will be created. It could be through the creation of a priority index for messages that could then be used to set targets/quotas for profilers and updaters. This will ensure the focus on those messages that have potential to impact statistical aggregates for key industry-geography domains.

Procedures and guidelines will be reviewed and improved for checking updates made by profilers and direct updaters in the various divisions (i.e. Statistical Registers and Geography Division profilers, EPM profilers and SMDs). Newly formalized guidelines and procedures will be documented, communicated and implemented. This should ensure all divisions are using a consistent approach.

New training exercises and materials that are being piloted in the regional offices, will be reviewed and finalized taking into consideration the audit recommendations.

Procedures and documentation for identifying, communicating and resolving issues related to the processing of records by Survey Account Managers (SAMs) and SMDs will be reviewed. This will build on existing procedural documentation and also newly-implemented Service Level Agreements with SMDs. Issue resolution will be in line with the framework to be implemented as part of the outcome of recommendation related to formal accountability framework.

Deliverables and Timeline

The Director of the Administrative Data Division will:

• Identify and implement short term solutions by March of 2017;
• Review the existing tools and procedures and specify the required changes by March 2017; and,
• Implement the fully-integrated tools and procedures of the CQM including the T2 redesign by March 2020.

The Director General, Statistical Registers and Geography Division will:

• Implement a method for prioritizing messages and settings quotas by March 2017;
• Document, communicate and implement newly formalized procedures and guidelines for checking updates made by profilers and direct updaters in the various divisions by March 2017;
• Implement the training examples for regional interviewers by June 2017; and,
• Develop documented and standardized procedures and tools for validating production files by December 2017.

Appendices

Appendix A: Audit Criteria

Appendix B: Acronyms