Audit of Equalization Certificates

Key findings

The equalization certificate program is a small but complex undertaking for Statistics Canada. Equalization certificates require over 100 data series from eleven divisions within the department. In 2012, National Accounts Integration and Development Division (NAIDD) was delegated the responsibility for coordinating and preparing the equalization certificates. Within NAIDD and the data supplying divisions, roles and responsibilities for those preparing the information are clearly communicated and understood.

While the processes and tools in place with NAIDD are effective and help to ensure risks to the quality and timeliness of the certificates are minimized, the audit revealed that there are opportunities to strengthen the certificate framework environment by ensuring that systems access permissions within NAIDD are periodically monitored to ensure access is provided on a "need-to-know" basis. Additionally, NAIDD should actively monitor the operating environment within Statistics Canada to assess and respond to risks external to the division that may impact the equalization certificates.

Within NAIDD, effective data quality control points have been integrated into the certificate preparation business process. The audit noted that the quality management framework developed by NAIDD could be improved by ensuring validation of the formulas in the final certificates takes place and configuring the templates to protect the data to ensure that data and formulas cannot be inadvertently overwritten.

Five of the six data supplying divisions (DSDs) examined in the audit have effective procedures documents that have been tailored to meet the needs of the specific programs as they respond to the requirements of the equalization program and evidence of quality review was found. However, in one DSD examined, the audit found no process documentation in place for the preparation and quality review of the data. The audit noted that this division has been subject to data anomalies in past submissions. The implementation of processes and procedures documentation for the preparation, quality review and sign-off of data submissions would reduce the risk of future data anomalies.

The majority of data supplying divisions (DSDs) examined had effective quality management practices in place to ensure the appropriate validation information on the divisional templates. Assessment of internal quality review, program director challenge and sign-off indicated that the framework could be strengthened by ensuring these processes are formalized and evidence of quality review and challenge is documented. Additionally, DSDs should ensure that segregation of duties is in place between data preparation and data validation, and that contingency plans are in place to ensure DSD submissions can be accurately completed.

The certificate preparation cycle takes place over several months and divisional data requests are timed to correspond with specific survey data releases. Although the certificate process is designed to ensure data provided are final, the audit noted that inconsistencies in the reporting of data revisions to NAIDD during the annual certificate process underscore that there is some uncertainty within DSDs related to the definitions and reporting requirements outlined in the Directive on Corrections to Daily Releases and Statistical Products. This elevates the risk that data on the certificates may not reflect the most accurate and up-to-date information. The audit team found that opportunities exist to strengthen the quality management practices by ensuring that DSDs develop procedures to advise NAIDD when data revisions or corrections occur to the data supplied on the divisional templates during the equalization certificate cycle.

Overall conclusion

The audit determined that the equalization certificates prepared by Statistics Canada in 2011 and 2012 were accurate and the tools, people, resources and quality assurance mechanisms in place support the completion of certificates in accordance with the FPFAA. Although the audit found minor data anomalies on the source and base certificates prepared in 2011 and 2012, these discrepancies did not impact equalization information for the provinces or territories for the years reviewed.

The certificate framework and quality management controls in place within NAIDD are generally effective; however, opportunities exist to strengthen these control frameworks within the certificate preparing division.

The certificate and quality management frameworks within DSDs could be enhanced by ensuring that within all DSDs process documents are developed and strengthening and documenting the quality assurance mechanisms in place.

Conformance with professional standards

The conduct of this engagement conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Quality Assurance and Improvement Program. Sufficient testing was carried to support the findings and related recommendations.

Patrice Prud'homme
Chief Audit Executive

Background

The equalization program has been in place since 1957 and is part of the federal-provincial fiscal arrangements legislation. Equalization has changed significantly since the original implementation and continues to undergo periodic review. The next review of the Federal-Provincial Fiscal Arrangements Act (FPFAA) and regulations is scheduled to take place in 2013.

In general, equalization is a federal transfer payment program that has been designed to assess the fiscal capacity of the provinces and smooth out disparities. It is embedded in Section 36(2) of The Constitution Act (1982) which states:

Parliament and the Government of Canada are committed to the principle of making equalization payments to ensure that provincial governments have sufficient revenues to provide reasonably comparable levels of public services at reasonably comparable levels of taxation.

The equalization program examines, on a per capita basis, a province's ability to generate own source revenues to the fiscal capacity for all provinces. In addition to equalization for the provinces, the FPFAA includes Territorial Formula Financing, an annual unconditional transfer to the three territorial governments. Territorial financing enables the territories to provide public services comparable to those provided by provinces while taking into consideration the higher costs of providing programs and services in the territories.

The statutory authority for equalization and territorial financing is found in the Federal-Provincial Fiscal Arrangements Act and regulations. This Act outlines the role of the Chief Statistician as it pertains to fiscal equalization payments to provinces and territorial financing. Statistics Canada is responsible for preparing the revenue base and revenue source certificates. The regulations associated with the Act provide specific interpretation and outline data requirements and calculations.

The equalization certificate program is a small but complex undertaking for Statistics Canada; over 100 data series are collected from eleven divisions to prepare the equalization certificates. Reviews of data are completed by the eleven data supplying divisions, the certificate preparing division, Provincial and Territorial Focal Points, and the Department of Finance. The equalization program has a large impact on the country: in 2012, approximately 18.5 billion dollars were transferred to provinces under the FPFAA.

Until 2012, Public Sector Statistics Division (PSSD) within the National Accounts and Analytical Studies Field compiled the legislated input required for the administration of the FPFAA. Beginning in 2012, responsibility for certificate preparation was transferred to the National Accounts Integration and Development Division (NAIDD) within the National Accounts assumed responsibility for the preparation of the certificates for Statistics Canada. Effective April 1, 2013, NAIDD was subsumed into Industry Accounts and Expenditure Division and moved into the Economics Statistics Field.

Audit objectives

The objectives of the audit were to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that:

• Statistics Canada has the tools, people and other resources necessary to develop timely, accurate and complete certificates in accordance with the requirements of the Federal-Provincial Fiscal Arrangements Act.
• Statistics Canada has appropriate quality assurance mechanisms in place over the equalization certificate program to ensure compliance with the requirements of the Federal-Provincial Fiscal Arrangements Act.

Scope

The scope of this audit included an examination of the adequacy and effectiveness of the controls in place to prepare the equalization certificates. Specific areas examined included the preparation and quality assurance controls for the data and actual certificates, and examination of the adequacy and effectiveness of controls in place to ensure compliance with the data required by the FPFAA regulations. The audit covered statistical information for certificates prepared in 2011 and 2012.

Approach and methodology

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors (IIA) International Standards for the Professional Practice of Internal Auditing.

The audit work consisted of an examination of documents, interviews with key Senior Management and personnel, and a review for compliance with relevant policies and guidelines.

The methodology consisted of a review of applicable policies, procedures, and information related to quality, accuracy and sign-off of the certificates and source data. Testing of a sample of quality assurance mechanisms in place as they relate to the certificates and data submissions also took place. Audit testing included: validation to ensure that data supplied on the certificates matched current equalization regulations on the source and base certificates; confirming that all formulas on finalized source and base certificates were equivalent for each province and territory; ensuring that final certificate data matched data supplied by DSDs; validating that formulas within spreadsheets that were linked to equalization certificates were coherent and accurate; and validating that English and French versions of the certificates were equivalent, and units of measure were accurately denoted. The audit also examined evidence confirming that quality assurance activities took place within DSDs and NAIDD.

Authority

The audit was conducted under the authority of Statistics Canada Integrated Risk-Based Audit & Evaluation Plan 2012/13 to 2016/17, approved April 2012, by the Departmental Audit Committee.

Certificate framework

An effective certification framework should ensure that appropriate processes and procedures are in place so that individuals preparing and validating the information have the tools necessary to develop quality equalization certificates. This includes detailed process documents outlining how data should be prepared and what data validation is required to ensure the quality of the information. Roles and responsibilities should be clearly defined and communicated and the business and operating environments should be monitored for change. Controls in place should ensure files are restricted to authorized users only.

Certificate framework: process overview

Equalization certificates are developed based on over 100 data series received from eleven divisions within Statistics Canada. National Accounts Integration and Development Division (NAIDD) has been delegated responsibility for the coordination and preparation of the equalization certificates, which are comprised of revenue base, revenue source and population certificates.

NAIDD requests data from data supplying divisions (DSDs) using structured templates specifying the data to be provided. Requests are timed to follow the official release of data. Staff in DSDs prepare the required information, carry out quality assurance over the information prepared, and obtain program director sign-off on the completed template.

Information transmitted back to NAIDD is compiled into the certificates and reviewed. NAIDD performs basic quality checks over the information provided by DSDs and coordinates Work-in-Progress (WIP) reviews with Provincial/Territorial Focal Points and the Department of Finance. Once reviewed and finalized, the certificates are sent to the Director General of NAIDD and the Assistant Chief Statistician for review, and subsequently to the Chief Statistician for sign-off. Final signed certificates are sent to the Department of Finance, where provincial equalization and territorial funding payments are calculated using these data.

Figure 1: Certificate Preparation Process

Certificate framework: Certificate Preparing Division

Certificate framework: Data Supplying Divisions

The audit examined six of the eleven divisions which provide data for the equalization certificates. Divisions were chosen to ensure coverage of all fields that provide data to the certificate program. Additionally, divisions in which data anomalies had been detected in past submissions were included.

Quality management

The quality management framework for the equalization program should include documented guidance for quality review and approval activities, clear segregation of duties between data preparation and quality review, and formal review and challenge prior to approval and sign-off on data quality. Quality management processes should be integrated into the business practices in place for the preparation of equalization certificates. The Statistics Canada Quality Guidelines note that certification or validation of statistical information should include a challenge of the data rather than a rationalization of the information. This challenge should ensure that data movements, unusual events or uneven growth are understood and can be explained.

Quality management framework: Certificate Preparing Division

The audit found that the data quality control points within NAIDD are effective and have been integrated into the certificate preparation business process. The certificate preparing division typically requests data that have been released in the Daily. Requesting finalized data ensures that the information have been subject to internal quality assurance practices within each DSD. When requesting the data, the NAIDD provides structured templates to each data supplying division. Templates specify the data to be provided, and include a section for DSDs to note methodological, classification or other changes that may impact the data. The audit confirmed that these templates are used by data supplying divisions.

Upon receipt of completed templates, NAIDD performs a number of basic validity checks, including checking against previous submissions, previous year data (growth analysis), and where applicable, validation against CANSIM, other publications and a check of totals. A quality review checklist is in place to support this activity. Unusual movements, variations and other issues are queried with the data supplying division and documented.

The certificate preparing division coordinates three Work-in-Progress reviews: one review with Provincial/Territorial Focal Points, and two reviews with the Department of Finance. Focal Points review provincial and territorial data to be included in the revenue source certificate and the Department of Finance reviews point-in-time versions of the revenue base and revenue source certificates in September and November. These reviews provide a check of coherence and reasonableness. The audit found evidence of items raised through WIP reviews including data challenges, requests for explanations and notices of updates for missing information being queried with DSDs, and responses documented.

Finalized templates are sent by the Assistant Chief Statistician, System of National Accounts Branch to the Office of the Chief Statistician (OCS) for signature. The audit confirmed that the OCS relies upon the data reviews completed within NAIDD and DSDs. This places further importance on the practices put in place by NAIDD and DSDs to mitigate potential risks to the quality of information provided by Statistics Canada to meet the organization's obligation with respect to the FPFAA and regulations.

The audit conducted testing to confirm that the data being supplied by Statistics Canada for the revenue base and revenue source certificates correspond to the data series specified in the FPFAA regulations. The audit tested each reference made to the regulations in the 2011/2012 revenue base certificate template, and in the provincial tabs supporting the 2009/2010 revenue source certificate templates. Each reference made in the templates was validated against the corresponding section of the regulations.

The audit examined the data provided in the base and source certificates for coherence, completeness and accuracy. The audit tested and found that the data on the final certificates matched the DSD templates. The audit also confirmed that the data found in the French and English versions of the certificates are equivalent, units of measure (e.g. kilowatts, tonnes, dollars etc.), and table stub blocks, header blocks and notes blocks have been correctly denoted.

The finalized certificates are linked to specific worksheets. These worksheets contain the data and specific calculations based upon the regulations. The audit tested the calculations on the base and revenue source certificates prepared in 2011 and 2012 and found that the computations on the linked data sheets were coherent and accurate, and that provincial and territorial formulas were equivalent.

The audit found that NAIDD does not validate that the formulas in each cell of the equalization certificates are referenced to the appropriate linked source data sheet or that they have not been overwritten which increases the risks to the quality of the information. Audit testing of the formulas on the revenue base and revenue source certificates prepared in 2011 and 2012 revealed instances of incorrect references and overwritten formulas on the finalized equalization certificates. While data formula and data anomalies were found on the final certificates, the audit noted these inconsistencies did not impact final figures for the years examined.

Overall, NAIDD has implemented effective quality control points within the certificate preparation process to ensure the quality of the information. Opportunities exist to strengthen the quality management practices by ensuring that validation of the formulas within the certificates takes place and ensuring the certificates are configured to protect the data and ensure formulas and data cannot be inadvertently overwritten.

Quality management framework: Data Supplying Divisions

The Statistics Canada Quality Guidelines notes that certification or validation of statistical information should challenge data rather than rationalize the information to help ensure information quality. Additionally, the Guidelines recommend that validation/certification should be carried out by analysts not involved in the production of the data. The audit revealed that the quality management practices followed within DSDs for preparation, validation, and sign-off varied. In five of the six DSDs examined, segregation between data preparation and data validation is in place. In one division in which data anomalies occurred in the 2011 submission, processes rely on a single person for both data retrieval and validation and only one individual in the division has the requisite knowledge to provide the information. There is no contingency plan in the event of an absence. This person-dependent process exposes the data to potential quality or timeliness risks.

Data supplying divisions are responsible for the quality of the information they provide. In five of the six divisions, the audit found evidence that data validation takes place and is documented. Evidence of validation consisted of spreadsheet comparisons to previous years, documented explanations for data large movements, validation against other data sets or provinces, and checks of totals. In the division without process documents there was no documented evidence of data validation occurring.

Program directors are required to sign the divisional template as acknowledgement of their approval of the information being submitted by their division. In 5 of the 6 DSDs examined, the management review of divisional templates is informal and undocumented, but management noted that prior to sign-off they must have assurance that the data are of sufficient quality. This informal review process varied: in some cases the review was general, whereby directors asked only if quality assurance took place (no specific challenge) prior to sign-off, while others provided a more detailed challenge function which included confirming what types of checks were done and how the data differed from previous years prior to providing sign-off.

In one of the six divisions examined, the management review is formalized. A formal presentation is made to the program director, outlining the data, major changes, largest contributors and reasons for large variations. The program director provides a challenge function over the information presented. Queries made by the program director during the challenge are documented, and the analyst must subsequently respond in writing to these queries. When the director is satisfied with the explanations, the templates are signed. The audit found that signatures were in place on all divisional templates in 2011 and 2012 for the six DSDs examined in the audit.

The audit found that the majority of the DSDs examined had effective quality management practices in place to ensure the appropriate validation information on the divisional templates. Assessment of internal quality review, program director challenge and sign-off indicated that the framework could be strengthened by ensuring these processes are formalized and evidence of quality review and challenge is documented. Additionally, DSDs should ensure that segregation of duties is in place between data preparation and data validation, and that contingency plans are in place to ensure DSD submissions can be accurately completed.

Appendix A: Audit criteria

Appendix B: Acronyms