Archived – Audit of Annual Population Estimates Certification

Audit Report

March 31, 2011
Project Number: 80590-62 ( Document (PDF, 341.09 KB) )

• Executive Summary
  • Key Findings
  • Overall Conclusion
• Introduction
  • Background
  • Objectives
  • Scope and Approach
  • Authority
• Findings, Recommendations and Management Responses
• Quality Assurance Framework
  • Governance
  • Policies and Programs
  • Citizen-Focused Service
  • Results and Performance
• Risk Management
• Stewardship
• Appendices
  • Appendix A: Audit Criteria
  • Appendix B: Glossary

Executive Summary

Statistics Canada's Population Estimates Program (PEP) has a statutory obligation to provide the federal government with the official and certified annual population estimates as key variables in the formulas for calculation of revenue transfers and contributions under various federal statutory programs. The annual population estimates are used to allocate approximately $60 billion dollars from the federal government annually. According to the most recent Regulations (2008) of the Federal-Provincial Fiscal Arrangements Act, these estimates have to be submitted to the department of Finance in the form of Population Certificates as of June 1^st and as of July 1^st. The Population Certificates are produced by the Demography division, part of the Social, Health and Labour Statistics Field ^{Footnote 1}, and within the Annual Population Estimates Certification Statistical Business Process (APEC SBP) of its Population Estimates Program (PEP).

The APEC SBP is a data integration activity, in which data are estimated, modeled, or otherwise derived from existing statistical data sources that come from the multiple internal and external suppliers and in which data quality problems can be inherited from the data suppliers or created internally. The impact of errors in the estimates could lead to errors in the determination and collection of differences in payments of transfers and contributions.

The objective of this audit is to provide the Chief Statistician and the Departmental Audit Committee (DAC) with assurance that the Quality Assurance Framework (QAF) for the Annual Population Estimates Certification (APEC) is in place and aligned with Statistics Canada Quality Assurance Framework, associated Quality Guidelines and Treasury Board Core Management Control criteria (TBS CMC).

The audit was conducted by the Internal Audit Services of Statistics Canada and the evidence was gathered in compliance with the Internal Audit Standards for the Government of Canada and the International Professional Practices Framework (IPPF) of the Institute of Internal Auditors.

Key Findings

The APEC SBP has in place a strategic governance regime for the continuous improvement of its quality assurance program, which is aligned with Statistics Canada QAF and Guidelines. While APEC SBP has sound strategic methodological publications and quality assurance mechanism, processes are not formally supported by procedures manuals or checklists. The APEC SBP facilitates access to its data with an effective communications strategy and Customer Service to guide communications with the provincial and territorial focal points, federal government departments and other data users. Quality evaluation methodologies and indicators are applied to all components of population estimates. Quality indicators are subject to review and active monitoring, and results are reported.

The Population Estimates Program's involvement in risk management has been limited to the department-wide risk assessment conducted during the Threat and Risk Assessment (TRA) exercise and the Quadrennial Program Review (QPR) processes. These activities focused on the broad data supply, IT, HR. As such, the specific risks for the population estimates program are not fully identified, assessed, and mitigated through these processes.

The APEC SBP has put in place quality measures which include a functional mix of preventive, detective, automated, manual and interactive quality controls. However, the design and operating effectiveness of the APEC SBP quality assurance methodologies has not been formally monitored on the regular basis.

Overall Conclusion

The audit found that the Population Estimates Certification Statistical Business Process (APEC SBP) is aligned with Statistics Canada QAF and Guidelines, and for the most part, meets the Treasury Board Core Management Control Criteria. The APEC SBP should be strengthened, by improving the documentation related to the quality assurance process; bolstering the risk management framework; and monitoring the program's quality assurance design and effectiveness.

Introduction

Background

Statistics Canada's Population Estimates Program was established in the 1970s and it has consistently met its legislative mandate to provide certified population data to the policy departments. The program's annual operating budget is approximately $2.1 million and involves 28 FTEs. According to Parts V and V.1 of the Federal-Provincial Fiscal Arrangements Act and sections 27-28 of the associated Regulations (2008), Statistics Canada is required to submit to the Minister of Finance, no later than September 30^th of each year, Certificates that set out the population of each province and territory as follows:

• A Population Certificate as of June 1^st, covering all provinces and territories, with total current population counts, for the Canadian Health Transfer Program, the Canadian Social Transfer Program, the Wait Times Reduction Transfer and the Equalization Program purposes;
• A Population Certificate as of July 1^st, covering all provinces and territories, with total current population counts, for the Territory Formula Financing Program purposes.

The Population Certificates include the current year estimates and the data for seven prior fiscal/calendar years by estimate type (intercensal (ID), and postcensal (PP, PR and PD)), as well as data to which historical revisions were applied. The annual population estimates are used to allocate approximately $60 billion dollars from the federal government annually

The Population Certificates are produced by the Demography division, which is part of the Social, Health and Labour Statistics Field ^{Footnote 2} within the Annual Population Estimates Certification Statistical Business Process (APEC SBP) of the Population Estimates Program (PEP). The APEC SBP has the following characteristics:

• it is classified as a derived statistical activity, in which data are estimated, modeled, or otherwise derived from existing statistical data sources, that come from multiple internal and external suppliers;
• it is subject to Federal-Provincial-Territorial relations, and respective consultative and data validation processes;
• data quality problems can be inherited from the data suppliers or created internally, which requires systematic communication and feedback mechanisms among PEP and its suppliers and stakeholders;
• quality assurance/certification practices are embedded into the production of estimates and are conducted internally by the PEP.

The production of Population Certificates in the form of special tabulations to the Department of Finance is a bi-product of the APEC SBP, which produces and publishes a wide range of quarterly and annual population estimates available to the general user community. The format and quality assurance of Population Certificates are guided by regulatory requirements, Statistics Canada QAF (2002), associated Guidelines (2009) and the instructions, received from the Public Sector Statistics Division of the System of National Accounts on the basis of bilateral and multilateral consultations with policy departments.

Objectives

The objective of this audit is to provide the Chief Statistician and the Departmental Audit Committee (DAC) with assurance that:

• The Quality Assurance Framework for the Annual Population Estimates Certification is in place and aligned with Statistics Canada Quality Assurance Framework, associated Quality Guidelines and Treasury Board Core Management Control criteria.

Scope and Approach

The audit engagement was conducted in conformity with the Government of Canada and the IIA standards on internal auditing, and the Treasury Board of Canada's Policy on Internal Audit. All work was conducted in collaboration with key senior managers of the Demography Division, the managers and staff of the Population Estimates Section and the Development and Demographic Methods Section. The audit approach was inspired by the Treasury Board Core Management Control Guidelines issued by the Office of the Comptroller General.

Assessing the Quality Assurance Framework (QAF) for the APEC SBP involved comprehensive examination of the degree of development and alignment of existing quality measures vis-à-vis the principles of Statistics Canada Quality Assurance Framework (2002), Quality Guidelines (2009) and Treasury Board Core Management Controls in the areas of governance, risk management and control (see Audit Criteria, Appendix A). The audit focuses on the assessment of the APEC SBP QAF for the production of current counts of annual population estimates for Canada and its provinces and territories, which are subject to annual certification in implementation of the Federal-Provincial Fiscal Arrangements Act and its Regulations (2008).

The scope of the audit was based on the universe of annual population estimates used in the annual Population Certificates provided to the Department of Finance at the end of September each year for the last 12 years (2010-1998), by their types, components and dimensions. These estimates are produced as of June 1^st and as of July 1^st. The audit universe consists of the certified postcensal and intercensal annual population estimates, which constitute a series of estimates revisions produced at various time intervals to account for both accuracy and timeliness.

The audit covered 95% of the respective population estimates components and APEC SBP process steps, and included components that are embedded in APEC SBP on a cyclical basis (i.e. Revisions, Intercensal estimates, Censal population counts). Based on a risk assessment, the components for estimation and certification of deaths and returning emigrants were excluded from the scope of the audit.

To perform the audit work, the following methods were used:

• interviews with senior managers, PEP management and production staff;
• walkthroughs and examinations;
• Population Certificates Quality Assurance Test;
• examination of samples of ESTIME and Non-ESTIME specifications by population estimate component based on risk;
• review of various procedures documents, and minutes.

Authority

The audit was conducted by IAS under the authority of Statistics Canada Risk-Based Audit Plan (RBAP) for the fiscal years of 2010/11-2012/13. The RBAP was approved by the Departmental Audit Committee on April 15, 2010.

Findings, Recommendations and Management Responses

The APEC SBP QAF was assessed with respect to the six groups of Treasury Board Secretariat Core Management Controls criteria. The controls were examined taking into account their effectiveness and level of development according to the IIA Control Maturity Model (see Figure 1).

Description: Figure 1. APEC SBP QAF Assessment IIA CMM Ranking Profile

The APEC SBP is aligned with Statistics Canada QAF and Guidelines and generally meets the Treasury Board Core Management Control Criteria.

To strengthen the Annual Population Estimates Certification Statistical Business Process, additional investments are required to improve the documentation related to the quality assurance process; strengthen the risk management framework; and monitor the program's quality assurance design and effectiveness.

All recommendations and management response and action plans that follow in the sections below should be considered within the existing Statistics Canada management structure.

Quality Assurance Framework

Governance

A sound governance regime should include effective oversight bodies to provide strategic guidance; clearly defined and communicated strategic objectives; operational plans aimed at achieving the program's objectives; and on-going monitoring of internal and external environments.

The Demography Division has established a strategic governance regime for determining methodological, operational, and regulatory requirements for the APEC SBP to ensure data quality. For strategic guidance, the APEC SBP relies on an adequate and effective network of external and internal committees, such as the Federal-Provincial-Territorial Committee on Demography. For guidance on Population Certificates, the APEC SBP relies on departmental regulatory oversight bodies and processes for the production and submission of certificates such as the Public Sector Statistics Division and bilateral and multilateral consultations between policy departments and Statistics Canada on the data requirements for Population Certificates in fulfillment of the Federal-Provincial Fiscal Arrangements Act. An appropriate governance structure and mechanisms for quality assurance are in place, ensuring separation of quality assurance responsibilities between production and methodology. However, several vacancies exist within the division and key operational knowledge, as well as the responsibilities surrounding both production and intermediate data evaluation mostly resides with one staff member, and is not well documented.

The APEC SBP has a strategic Quality Assurance Framework for population estimates with clearly established, communicated and specific quality assurance directions and objectives. They are based on the departmental QAF and Guidelines. Therefore its QAF is aligned with these reference documents. A Review of Statistics Canada's QAF and Guidelines revealed that they are built on the concepts of the Total Quality Management (TQM), which embed quality evaluation at every step of the statistical business process. This TQM culture is also ingrained in the statistical, econometric and methodological training of Statistics Canada staff. Moreover, in 2009 Statistics Canada adopted the OECD Standard Generic Statistical Business Process Model (GSBPM) as a reference quality assurance framework, which has the same TQM principles.

While the APEC SBP has sound strategic methodological publications and quality assurance mechanisms, there is no integrated document on the operational APEC SBP QAF or the quality assurance manual. In fact:

• Statistics Canada's QAF is a voluntary and flexible guideline, and serves as a reference to subject-matter divisions in designing quality assurance programs, tailored to specific needs and objectives;
• There exists a number of targeted QA policies related to the processes of dissemination and metadata management, in lieu of integrated departmental policy on the quality assurance;
• Certificate requirements are determined in a series of bilateral or multilateral consultations.

ESTIME is the statistical platform used for the production of population estimates. The APEC SBP has formal plans for the development of quality-improving methodological studies, and remaining ESTIME specifications and user guides. Program employees use production and dissemination planning tools such as release calendars by reference date publications (quarterly and annual), generic process roadmaps, Priority Management documents, and system improvement project plans. Project and process management documents have been produced, such as an integrated Microsoft Project tool for the APEC SBP which schedules evaluation and certification steps; an ESTIME User Guide for the production of one of the components of population estimates (Births); brief descriptions of the validation and certification processes in the document “Processes of Evaluation”; a calendar of quality assurance priorities implementation; and “To Do Lists” for some elements of APEC SBP. However, these tools do not incorporate quality assurance protocols or checklists.

The Demography division has established an effective monitoring regime of external and internal users and stakeholders environments for the purposes of improving the APEC SBP QAF in the areas of methodological research, alternative data sources and methods, and data validation. During the meetings with data providers, Demography employees discuss the impact of their source files on APEC estimates and the impact of changes in legislation and various data estimation processes.

The APEC SBP has an effective functional information technology (IT) governance framework for the achievement of its quality objectives. The governance structure, aimed at the development and maintenance of the ESTIME platform, rests on a system of committees and working groups, both formal and informal.

Policies and Programs

A sound and effectively designed program should embed a disciplined approach to evaluation, where results are considered, approved, and integrated to improve the program. Key stakeholders are involved in the integration of program changes. Program activities are monitored on a regular basis to identify strengths and areas where improvements can be made.

The APEC SBP has established formal processes for the consideration, approval and usage of results from quality consultations, and research and analysis to coordinate and integrate changes to the existing QAF. The processes are conducted with provincial/territorial and other external and internal stakeholders, on an annual or quarterly basis, and results are incorporated into the quality assurance processes. Key stakeholders are involved in the integration and coordination of changes to existing QAF, via quarterly and annual meetings with the statistical focal points, data providers, and Advisory Committee on Demographic Statistics and Studies.

Methodological monitoring of the quality of population estimates is linked to the Census five-year cycle, and results are reported to the required management level, through various analytical and information products. Data quality monitoring is conducted via the production of internal reports, as well as various types of metadata, which are reviewed and approved by senior management. The APEC SBP conducts quality evaluation activities and consults with the Development and Demographic Methods Section to identify APEC strengths, weaknesses and impacts, as well as alternative ways of designing SBP or the outputs. Senior managers are systematically involved in the review of results from consultations, research and analysis, and participate in consultative, and R&D review processes related to the APEC SBP.

Citizen-Focused Service

Access to data should be ensured through compliance with relevant policies and an effective communication strategy aimed at current and potential users.

The APEC SBP has a number of measures in place which facilitate user access to its data, and complies with Statistics Canada's Policy on Informing Users of Data Quality and Methodology. Users are able to access APEC data via multiple service delivery channels, such as the Daily, CANSIM and IMDB. Pre-release and dissemination activities are handled mainly by the Client Services unit of the Demography Division, and involve ongoing interactions with the Dissemination Division, the Communications Division, and statistical focal points. The Client Services Unit has an effective communication strategy in place; it manages various information requests from the media, focal points, and various user organizations and records client queries. A Client Service unit has been established by the APEC SBP to support communication activities with provincial and territorial statistical focal points as the key stakeholder group, as well as with other users of population estimates.

The external Statistics Canada File Transfer Protocol (FTP) Remote Access site provides statistical focal points with the time-series of the pre-release population estimates and related metadata. It also provides numerous opportunities for the validation of estimates and their quality indicators; access to studies and results of consultations and discussions with specific provinces and territories; as well as to past, current and future methodological, estimation and quality control practices. An examination of the information available on these sites, along with interviews with Population Estimates Program management and production staff, demonstrate that quality standards are communicated and understood internally, and externally, based on Customer Satisfaction Survey results. The main methodological publication documents, developed to support of the production of population estimates, are in compliance with Statistics Canada's Policy on Informing Users of Data Quality and Methodology.

Results and Performance

An effective results and performance management requires appropriate performance indicators, which are monitored and periodically reviewed for relevance. Evaluation methodologies are in place and applied and results are documented and reported to management.

The APEC SBP conducts systematic reviews and updates of quality indicators, which are documented and reported both internally and externally, and demonstrate active monitoring of quality indicators through various formal and informal methods. Several reports are produced with comprehensive analysis of quality indicators for the population estimates. The key methodological document - "Population and Family Estimation Methods at Statistics Canada", is published every 5 years.

The Development and Demographic Methods Section has a quarterly reporting process, in which the Progress Report on methodological and quality assessment studies and projects is disseminated to other sections of Demography Division and Statistics Canada. The Client Satisfaction Survey (CSS) is an external process for the collection of user opinions on the quality of population estimates and is conducted biannually by the Statistics Canada Client Services Division. Results of the CSS are reviewed internally by the Demography Division's senior management.

The APEC SBP conducts active monitoring of quality indicators on a five-year basis, timed with the Census cycle. Indicators are monitored internally by the Demography Division, and validated externally by the Federal-Provincial-Territorial Committee Meetings consultation and discussion forums.

Recommendation No. 1

It is recommended that the ACS for Social, Health and Labour Statistics Field ensure that processes surrounding methodological publications and quality assurance mechanisms are formally documented and supported by a procedures manual.

Management Response

Management agrees with the recommendation. A complete procedures manual will be developed, documenting the main steps involved in producing quarterly and annual total population estimates. Separate chapters will be developed for each of the components of population growth. This document will be targeted at junior to intermediate production staff.

Deliverable

Complete procedures manual, including chapters for each component of population growth.

Accountability

The work involved will be assigned to the Head of the production unit, within Demographic Estimates section.

Periodic updates will be provided to the Departmental Audit Committee.

Timeline

Given that manuals will be developed by the same staff responsible for quarterly production, this work will be spread over 2 years beginning in April 2011.

March 31, 2012: At least two components of population growth (e.g. births and immigration) will be completed.

March 31, 2013: The remaining chapters will be completed.

Risk Management

An effective risk management framework requires formal processes to identify and assess risk. Risk mitigation strategies should be documented and communicated. Information obtained on risk should be used to support business planning.

The Populations Estimates Program's involvement in risk management has been limited to the strategic department-wide risk assessment conducted during the Threat and Risk Assessment (TRA) exercises, and the Quadrennial Program Review (QPR) processes. These activities focused on the broad data supply, IT, HR. As such, the specific quality risks for the population estimates program are not fully identified, assessed, and mitigated through these processes. Occasionally, a risk assessment is undertaken for a particular event impact analysis, such as ESTIME system conversion or Census redesign.

Nevertheless, since the 1970s, the APEC SBP has been able to meet the legislative mandate for the production of population estimates. This is explained by the fact that there are quality risk controls embedded in the program design. In fact, the program is not restricted to produce estimates strictly according to the published methodology. In the event that the official source data are not yet available, alternative data and contingency solutions can be used. In addition, there are a number of informal quality risk management practices in place surrounding the evaluation of data sources, such as: meetings and exchanges with provincial and territorial statistical and other data providers; automatic preventive quality controls embedded in the ESTIME system; and evaluations of the time-series of source data.

There is a practice of ensuring systematic and formal responses to identified APEC quality risks. The Development and Demographic Methods Section provides research proposals for methodological improvements at the annual Federal-Provincial-Territorial Committee on Demography meetings. There are two groups of methodological risk mitigation solutions:

• Alternative processes and methods evaluation and development; and
• Alternative data sources evaluation acquisition/development.

As such, when deemed applicable, an alternative method or application is available for use. These alternatives are carefully identified and tested. Furthermore, provinces and territories can also be consulted to provide an alternative source of measurement and estimation of population estimates. Provinces and territories use different population accounting methods, based on their administrative health-care files. These alternative sources of information produce significant benefits in both the evaluation of the quality of APEC data, and as contingency source databases.

Operational APEC staff have an intuitive approach to managing its risks. Components of population estimates are prioritized based on size and proportion. However, there is no documented protocol for quality risks mitigation for the APEC SBP.

Demography Division's key information on quality risks is embedded in divisional performance reports conducted every four years. The APEC SBP does not have formal and comprehensive sectional operational plans or performance reports that embed information on quality risks for the production and certification processes.

The Development and Demographic Methods Section works on long-term strategic quality risk issues for all programs of the Demography Division. With respect to the APEC SBP, discussions with provincial and territorial stakeholders, with regards to methodological plans to improve data quality and to address known quality risks, result in the development of formal sectional work plans for methodological studies and operational tools, such as specifications for ESTIME. The section also provides an assessment of quality risks, when APEC embarks on new quality improvement initiatives, for example the use of the American Community Survey to improve the quality of emigration estimates. Although methodology improvements are always targeted at quality improvements, there are no annual management documents which demonstrate quality risk identification, assessment, or action plans.

There is no practice of developing formal business continuity plans, embedding quality risks for the APEC SBP, which is heavily dependent on the smooth operation of its new production platform ESTIME. Although there are no formal sectional business continuity plans, there are various departmental standards for the diagnosis of IT systems risks, as reported in the governance section of this audit report. Finally, there is no indication that information on Certification quality risks is used to guide business continuity planning of the APEC SBP.

Recommendation No. 2

It is recommended that ACS for Social, Health and Labour Statistics Field ensure that the APEC SBP strengthen the risk management framework by embedding information on quality risks for the components of production and certification processes; and that information on the Certification quality risks is used to guide business continuity planning of the APEC SBP.

Management Response

Management agrees with the recommendation. The annual planning process for the population estimates program, which currently focuses on setting and monitoring the highest priority work, will be adjusted to include an explicit risk assessment component. A risk register will be established, mitigation strategies identified and risks will be monitored over the year.

Since the highest risks to the population estimate program are related to the supply of data, work currently underway to renew relationships with data suppliers will be continued.

Deliverable

Risks register, including risk mitigation strategies.

Establishment of service level agreements with main data suppliers (including Health Statistics Division, Income Statistics Division and Citizenship and Immigration Canada).

Accountability

Responsibility for establishing and maintaining risks register will be assigned to the Chief of the Demographic Estimates group.

Responsibility for service level agreements will be assigned to the Assistant Director of the Demography Division

Periodic updates will be provided to the Departmental Audit Committee.

Timeline

March 31, 2012: Development of a risks register.

March 31, 2012: Service level agreements with internal data suppliers will be in place.

Stewardship

The effectiveness of internal controls requires a balanced mix of preventive, detective, automated and manual controls. Internal controls are tested for effectiveness on a periodic basis.

System of Internal Controls

The APEC SBP has a functional mix of preventive, detective, automated, manual and interactive quality controls embedded in all steps of the production and quality evaluation processes, in both ESTIME and non-ESTIME production environments for the population estimates.

Production and intermediate data quality evaluation processes, quality assurance methodology and controls, surrounding the Revisions component, requires exporting data and graphics from ESTIME into the Excel-based analytical tools, as well as holding informal discussions. The Excel-based tools used are adequate and effective to support the quality evaluation of each component of population estimates and aggregates. However, quality assurance analysis and decision-making processes are not formally documented, and the ESTIME program codes are not consistently updated to incorporate changes in methodology and to provide a proper match to the levels of data that are being processed.

For certification, there is a balanced mix of preventive and detective quality assurance controls, such as: guidance provided by the Public Sector Statistics Division on Population Certificates templates; periodic meetings of APEC SBP management with external and internal data suppliers; and carrying out a comprehensive final quality assurance assessment.

Manual controls are used when the Senior Analyst/Certification Chief needs to carry out independent calculations to verify evaluation results, presented by the Production Chief. Interactive controls include informal discussions with production staff, methodologists, suppliers and statistical focal points. Controls for the final approval of the Population Certificates by the Director of Demography Division are informal. The Population Estimates Quality Assurance Test revealed an identical match between the data appearing in all databases and a principal match of the format of the issued certificates. Data for the July 1st Population Certificate match those appearing on all other dissemination channels – i.e. Daily, IMDB, CANSIM, and publications. Data for the June 1st Population Certificate are special tabulations, and are not published.

ESTIME and Non-ESTIME specifications contain controls to ensure correctness of the production processes, and prescriptions for the design of automated controls. Specifications themselves represent automated production quality controls. However, there is no evidence of the presence of automated quality assurance controls in the sample of specifications examined. Methodological interactive controls include FTP processes; weekly meetings with APEC SBP managers for methodological planning and solution generation; as well as periodic meetings with data providers.

For FTP Remote Site (Data Return Facility) and Pre-Release QAF, there is a balanced mix of controls in place. There are automated controls for access and communication security on the FTP site, where ITSD controls software permissions and micro-data access verification. Uploading and archiving of data on the FTP site is a manual process, for which security safeguards exist to prevent errors and correct them. Interactive controls relate primarily to the documentation of questions and responses; archiving of e-mails; and proceedings from meetings.

Quality Assurance

For the APEC SBP Dissemination and Post-Release QAF at the departmental level, there is an extensive mix of policies and protocols on pre-release, dissemination and post-release quality assurance for the data and associated information products. These processes are coordinated by the Dissemination and Communication divisions. At the divisional level, there is a mix of preventive and detective quality assurance controls to ensure the quality of dissemination processes and disseminated materials. However, there is no integrated operational APEC SBP manual which would include descriptions of processes for the pre-release, dissemination and post-release quality assurance.

The initial operational testing of the ESTIME was conducted by the production team. However, there was no evidence of formal periodic testing and review protocols for the quality assurance in ESTIME system or other operational APEC SBP quality assurance methodologies. In fact, the APEC SBP has not established a formal and systematic process for the periodic testing of the design and operating effectiveness of its quality assurance methodologies.

Recommendation No. 3

It is recommended that the ACS Social, Health and Labour Statistics Field ensure that the design and operating effectiveness of the APEC SBP quality assurance methodologies are monitored on the regular basis.

Management Response

Management agrees with the recommendation. For each component of population growth, the ensemble of quality controls currently in place will be reviewed. As necessary, new quality control indicators and check-lists will be developed and incorporated within the production process of quarterly and annual population estimates.

Deliverable

A review of current quality controls.

A quality control framework, which will include quality control indicators and check-lists.

Accountability

Responsibility for developing a quality control framework will be assigned to the Head of the analysis and dissemination unit within the Demographic Estimates section.

Timeline

Given that this review will be conducted by the same staff responsible for quarterly production, this work will be spread over 2 years, beginning in the summer of 2011.

June 2011: Review of quality controls and development of a quality control framework will begin.

September 2012: Partial implementation is planned for the production of population certificates.

September 2013: The feasibility of full implementation by this date will be assessed once the project has begun.

Appendices

Appendix A: Audit Criteria

Appendix A: Audit Criteria

Lines of Enquiry	Criteria
Governance: The Annual Population Estimates Certification(APEC) Statistical Business Process (SBP) has in place essential conditions to support development and continuous improvement of its quality assurance program.	Effective oversight bodies are established to provide strategic guidance. The QAF embeds quality assurance directions and objectives, aligned with the PEP mandate, STATCAN QAF and associated guidelines. The APEC SBP has in place operational plans and project management documentation aimed at ensuring quality of the statistical business process. External/Internal user and stakeholder environments are monitored to obtain information that may signal a need to re-evaluate the APEC QAF. An IT governance framework has been developed and implemented to guide the use of IT in the achievement of APEC SBP quality assurance objectives
Results and Performance: Management of Annual Population Estimates Certification SBP has identified appropriate target quality indicators, and performs monitoring of actual vs. target results.	Quality evaluation methodologies are in place and are applied to all estimates. Quality indicators are reviewed on a periodic basis and updated as required. Results of quality reviews and quality indicators are documented and are reported to required authorities. Active monitoring of quality indicators is demonstrated.
Risk Management: Management of Annual Population Estimates Certification SBP identifies, assesses, manages and communicates risks to the quality of estimates, and integrates it into planning and resource allocation.	Formal processes and guidelines exist to identify and assess risks and controls to mitigate them. A formal response to the risk is documented and communicated to all necessary parties. Information on the quality risks is embedded in the key planning and performance reports. Information on the quality risks is used to support business continuity planning.
Policies and Programs: The APEC SBP has a formal and rigorous approach to quality assurance framework design and implementation, including monitoring of related activities in a regular and timely manner.	The APEC SBP embeds the discipline of quality evaluation into production and certification cycles. Results from consultations, research and analysis are considered, approved and used to coordinate and integrate changes to existing QAF. Key stakeholders are involved in the integration and coordination of changes to existing QAF. Quality monitoring is conducted on a regular basis and results are documented and reported to the required management level. Quality evaluation activities are used to identify APEC strengths, weaknesses and impacts as well as alternative ways of designing SBP or its outputs. Senior management are involved in a regular review of the results from consultation, research and analysis.
Stewardship: Management of APEC SBP assesses the appropriateness of the mix of quality assurance measures in place and monitors their effectiveness on a periodic basis.	Quality measures include a balance of preventive and detective quality controls. Quality measures include a mix of automated and manual/interactive controls. The effectiveness of design of quality assurance methodologies/methods in mitigating quality risks is periodically tested.
Citizen-Focused Service: APEC SBP takes measures to facilitate access to its data.	Users are able to access the APEC data via multiple service delivery channels (incl. compliance to Policy on Informing Users of Data Quality and Methodology). APEC SBP has in place a communications strategy to guide communications with current and potential users of data (incl. compliance to Policy on Informing Users of Data Quality and Methodology). The annual populations estimates quality standards are communicated and understood internally and externally (incl. compliance to Policy on Informing Users of Data Quality and Methodology).

Appendix B: Glossary

GSBPM – OECD Standard Generic Statistical Business Process Model, approved as Statistics Canada Quality Management Reference Model on March 15, 2010. /concepts/gsbpm-msgpo-eng.htm

APEC – Annual Population Estimates Certification

SBP – Statistical Business Process

TBS CMC – Treasury Board Secretariat Core Management Control Criteria

QAF – Quality Assurance Framework.

IIA CMM – International Institute of Internal Auditors, Controls Maturity Model.

IAA CMM Ranking Scale is provided below.

IAA CMM Ranking Scale

Rank	Title	Description
0	Non-existent	No formal or informal controls.
1	Initial/ Ad Hoc	Controls are fragmented and ad hoc; generally managed in silos and reactive; lack of formal policies and procedures; dependent on the “heroics” of individuals to get things done; higher potential for errors; higher costs due to inefficiencies; not sustainable.
2	Repeatable but Intuitive	Controls are established with some policy structure; formal process documentation still lacking; some clarity on roles, responsibilities and authorities, but not accountability; increased discipline and guidelines support repeatability; high reliance on existing personnel created exposure to change.
3	Defined	Controls are well defined and documented, thus there is consistency even in times of change; overall control awareness exists; control gaps are detected and remediated timely; performance monitoring is informal, placing great reliance on the diligence of people and independent audits.
4	Mature/Managed & Measurable	Performance measurement indicators (systems, tools, processes) and monitoring techniques are employed to measure success; greater reliance on prevention versus detection controls; strong self-assessment of operating effectiveness by process owners; chain of accountability exists and is well understood.
5	World Class/ Optimized	Controls are considered “world class”, based on benchmarking and continuous improvement; the control infrastructure is highly automated and self-updating, thus creating a competitive advantage; extensive use of real-time monitoring and executive dashboards.

DDMS - Development and Demographic Methods Section of Demography Division.

PEP - Population Estimates Program, respective Population Estimates Section of Demography Division.

SDD - Systems Development Division.

ITSD - Informatics Technology Services Division.

CBA - Corporate Business Architecture.

Notes: