Audit of Quality: Monthly Survey of Manufacturing

Audit Report

Report Date: March 2012
Project Number: 80590-70

• Executive Summary
  • Key Findings
  • Overall Conclusion
  • Conformance and Professional Standards
• Introduction
  • Background
  • Audit Objectives
  • Scope
  • Approach
  • Authority
• Findings, Recommendations and Management Response
  • Quality Governance Environment of the MSM
  • Risk Management
  • Data Stewardship
• Appendix
  • Appendix A: Audit Criteria

Executive Summary

Quality is Statistics Canada's hallmark. The organization strives to build relevance and quality into all its programs and products. The quality of official statistics is founded on the use of sound scientific methods adapted over time to changing client needs, to the changing reality that the Agency aims to measure, and to the capacity or willingness of respondents to supply reliable and timely data.

The Monthly Survey of Manufacturing (MSM) is a mission critical survey at Statistics Canada and has been providing information about the manufacturing sector in Canada since 1947. Results from this survey are used by finance departments of the federal and provincial governments, the Bank of Canada, Industry Canada, the System of National Accounts within Statistics Canada, the manufacturing community, and various other domestic and foreign research organizations. As a result, data quality and accuracy are paramount to the survey.

The objective of this audit is to provide assurance to the Chief Statistician (CS) and Statistics Canada's Departmental Audit Committee (DAC) that:

• MSM production environment has a sound quality control framework to ensure that the quality of the data is in accordance with the accuracy component of the Statistics Canada Quality Guidelines.
• Manual controls undertaken by the MSM staff during the production phase of the survey are documented, authorized and validated.

The audit was conducted by Internal Audit Services in accordance with the Government of Canada's Policy on Internal Audit.

Key Findings

The Monthly Survey of Manufacturing has a strong quality control framework that embeds the six static dimensions of quality outlined in the Statistics Canada Quality Guidelines. The production environment has clearly outlined roles, responsibilities and authorities related to quality. Formal practices and procedures documents outlining the tasks for survey staff are in place and complied with to ensure that the quality of information is maintained. Effective quality targets and quality assurance mechanisms are in place with the MSM process; they are monitored and adjusted as required to ensure that data quality and accuracy are maintained.

The division has a formal risk identification process in place for the MSM and also monitors the internal and external environments on an on-going basis. Management indentifies and monitors the controls in place to manage risks to quality. Input into the quality assurance control of MSM information also comes from other divisions within Statistics Canada.

Assessment of manual controls within the MSM production environment revealed that opportunities exist to strengthen controls by ensuring that major manual adjustments are formally authorized and ongoing monitoring of the control environment is carried out to ensure that manual controls do not unintentionally override other automated controls.

Overall Conclusion

The existing quality governance framework for the MSM is effective. Roles and responsibilities are formally defined, communicated and complied with by MSM staff to ensure the quality and accuracy of MSM information. Effective quality assurance mechanisms and quality targets are in place and are monitored to ensure quality. Formal practices and procedures are in place and effectively ensure that management identifies and responds to risks to maintain the quality and accuracy of MSM information.

Opportunities to strengthen controls in the production cycle exist by 1) ensuring that when major manual adjustments are required, formal authorization is documented and 2) monitoring of the control environment to ensure that manual controls do not unintentionally override other automated controls in the production system.

Conformance and Professional Standards

The conduct of this engagement conforms to the International Standards for the Professional Practice of Internal Auditing and the Government of Canada Internal Auditing Standards. Sufficient testing was carried to support the findings and related recommendation.

Introduction

Background

Quality is Statistics Canada's hallmark. The organization strives to build relevance and quality into all its programs and products. The quality of official statistics is founded on the use of sound scientific methods adapted over time to changing client needs, to the changing reality that the Agency aims to measure, and to the capacity or willingness of respondents to supply reliable and timely data. The Statistics Canada Quality Guidelines define quality using six static dimensions: Relevance, Accuracy, Timeliness, Accessibility, Interpretability and Coherence. Together, these elements help program managers manage the multidimensional nature of quality.

The Monthly Survey of Manufacturing (MSM) is considered to be a mission-critical survey within Statistics Canada and has been providing information about the manufacturing sector in Canada since 1947. The MSM produces information on the values of sales of goods manufactured, inventories and orders. MSM data are used as indicators of the economic condition of manufacturing industries; as inputs to Canada's Gross Domestic Product (GDP); as components in the Statistics Canada composite indicator; as input to economic studies, and in econometric models (e.g. to determine market share, apparent domestic availability, etc.). Results from this survey are used by finance departments of the federal and provincial governments, the Bank of Canada, Industry Canada, the System of National Accounts within Statistics Canada, the manufacturing community, and various other domestic and foreign research organizations

Within the MSM survey process there are eight steps; sampling, collection, batch reception, production, estimation, aggregation, verification and publication. Among these stages, there are various interdependencies with other divisions which provide services to MSM staff during the survey process. For example, the Business Register and Business Survey Methods Divisions are involved with the sample design, Regional Operations does the collection, Industry Accounts Division is consulted during the aggregation step and Communications and Dissemination Division provides services for the publication of the information.

During the production phase of the survey there are both manual and automated controls to validate and ensure the accuracy of the information collected. Automated controls in a survey environment ensure that there is consistent implementation of processes. In contrast, when manual controls are employed in the survey process, there may be an elevated risk to accuracy as these controls may be based upon judgement of an individual. Various members of the MSM team may make modifications to incoming data under certain conditions. Team leads and unit heads validate manual changes made during production. There are task and check lists for survey staff to ensure that changes made do not compromise the quality or accuracy of the data.

Estimates are produced based on returns from a sample of manufacturing establishments in combination with administrative data for a portion of the smallest establishments. The survey sample includes 100% coverage of the large manufacturing establishments in each industry by province, plus partial coverage of the medium and small-sized firms. Beginning in 2004, GST data began being used to estimate sales of goods manufactured for some medium and small-sized establishments.

The MSM program publishes monthly series for the values of manufacturers' shipments, inventories and orders for up to 325 industries nationally and by province/territory with comparisons to U.S. data within 45 calendar days after the reference month.

Audit Objectives

The objectives of the audit are to provide the Chief Statistician and the Departmental Audit Committee with assurance that the:

• MSM production environment has a sound quality control framework to ensure that the quality of the data is in accordance with the accuracy component of the Statistics Canada Quality Guidelines.
• Manual controls undertaken by the MSM staff during the production phase of the survey are documented, authorized and validated.

Scope

The scope of this audit covered the quality assurance procedures with a specific focus on accuracy to ensure the quality control framework is in accordance with the Statistics Canada Quality Guidelines. The scope included an examination of the manual controls undertaken during the production phase of the MSM cycle to ensure that the quality of the information is maintained. The audit covered statistical information collected in 2011-2012.

The MSM process is undergoing changes at different steps within the survey process. For example, the sample is set to be restratified in 2012. Additionally, the automated survey processing system for MSM has been redeveloped and is scheduled to be rolled-out for the four branch mission critical surveys in late spring 2012. As a result, examination of the sample and automated processing system were not included in the scope of this audit.

Approach

The audit approach consisted of a review of applicable policies, procedures, and information related to quality and accuracy of the Monthly Survey of Manufacturing information. Interviews with MSM staff and management and an examination, review and testing of the processes and procedures in place within MSM and a validation of quality assurance mechanisms as they relate to the manual controls within the production step of the survey took place.

This audit was conducted following the Standards for the Professional Practice of Internal Auditing as per The Institute of Internal Auditors (IIA) and in accordance with the TBS Policy on Internal Audit.

Authority

The audit was conducted under the authority of Statistics Canada Multi-Year Risk-Based Audit Plan 2011/12-2013/14, approved March, 2011 by the Departmental Audit Committee.

Findings, Recommendations and Management Response

Line of Enquiry No.1:  The MSM production environment has a sound quality control framework to ensure that the quality of the data is in accordance with the accuracy component of Statistics Canada Quality Guidelines.

Quality Governance Environment of the MSM

A sound quality governance environment should have clearly defined roles, responsibilities and accountability to ensure the effective management of quality. Quality targets should be in place and monitored against planned outcomes and objectives.

Roles, Responsibilities and Accountability
The audit determined that roles and responsibilities as they relate to ensuring the quality of MSM data are formally defined, communicated, and practiced. Within the MSM unit, the senior economist is responsible for determining when the data are finalized. The director of the division holds the overall accountability for the quality of information produced by the division. Statistics Canada's Policy Committee performs a review of the final MSM outputs to ensure data coherence and relevancy.

For each team member in the production environment of the MSM there is a task list that staff must follow. These lists summarize specific procedures, responsibilities and detail at what point in the process activities should be completed.

• MSM subject matter officers (SMOs) are responsible for verifying, validating and analyzing data for the manufacturing industries assigned to them. The tasks outlined in the SMOs task list relate to the duties that must be completed prior to the finalization of the data for the month. Tasks related to quality for SMOs include: review of the on-going problem cases for their industry, review of media related to manufacturing industries to ensure incoming data reflect reality. As data come into the MSM environment from regional office collections, SMOs review and validate data by examining diagnostic reports at the individual industry level. They also analyze the data related to their industries at the provincial and national level to ensure the data are coherent and accurate. Diagnostic report review and analysis continues throughout the production process until all outstanding data issues have been resolved.
• Within the MSM team there are two team leads. One is responsible for manufacturing industries rolled up into durable industries and the other is responsible for the roll-up of non-durable manufacturing industries. They also provide validation of the data for industries which fall under their domain. Team leads review the biggest industries and establishments under their domain to ensure any manual controls have been appropriately applied. SMO problem files are reviewed and followed-up to resolve any outstanding issues. Team leads review data for the top ten movers in each industry for sales, inventories and unfilled orders at several quality control points during production. They also conduct reviews of provincial data for industries and respond to data quality concerns from SMOs and Industry Accounts Division.
• The unit head (senior economist) provides the final validation of the estimates at the industry, provincial and aggregate levels and is responsible for determining and authorizing when the data are final. The unit head of operations ensures that all diagnostic reports have been run at the appropriate control points. The unit head is responsible for examining the top 100 movers, reviews unadjusted national level manufacturing totals, and seasonally adjusted and unadjusted data for the previous three months to determine if large revisions have taken place. The unit head also reviews, analyses and validates the provincial data to ensure data coherence at the provincial industry level. The unit head must also respond to data quality questions and validate that all required manual controls have been completed.

The audit conducted testing to ensure that task lists and check lists have been consistently applied by staff. Examination of sign-offs and a review of monthly production reports revealed that the unit head of operations within the MSM keeps track of the diagnostic reports completed and items outstanding at each estimation phase for the MSM team. Each month the senior economist prepares a sign-off of the checklists which indicate requisite checks were completed at each estimation pass during the data production period.

Clearly defined roles and responsibilities and accountabilities are in place for the MSM to support the effective management of quality for the data. Tasks and responsibilities outlined in the checklists are complied with by MSM staff.

Monitoring
An effective quality management framework requires appropriate quality targets and quality assurance mechanisms to be in place. These targets and mechanisms should be monitored and adjustments should be made as required.

The main purpose of the Statistics Canada Quality Guidelines is to provide a comprehensive list of guiding principles and good practices in survey design. These Guidelines outline survey steps, processes and suggested quality assurance strategies for maximizing the six static elements of data quality: relevancy, accuracy, timeliness, accessibility, interpretability and coherence. The Guidelines state that detailed and up-to-date procedures documents should be in place and that editing of data should be monitored. Quality indicators suggested by the Guidelines include, edit rates, contribution of editing to key variables, imputation rates by variable, weighted and unweighted response rates and Coefficients of Variation for main estimates.

Within the MSM environment, procedures documents are in place and have been developed as task lists and effectively identify the duties and responsibilities of each team member. Tasks detail the necessary steps required to ensure data quality during the processing phase of the survey as suggested by the Statistics Canada Quality Guidelines.

MSM management has identified several quality indicators for internal use to ensure data quality at the collection and processing stages for MSM data. The audit found the following quality indicators to be in place and effective; during collection, quality indicators such as touch rates and completion rates are documented and monitored to ensure that collection is on time and priority units have been contacted. Within the data processing cycle, diagnostic reports which examine negative values, outliers, zero reporters, top movers and top contributors are created to monitor and validate the accuracy of the information for negative values, outliers, zero reporters and top movers.

Data quality measures are produced and reported externally for the MSM as per the Policy on Informing Data Users which prescribes minimum amounts of information on data quality and methodology that are to be provided to users.

Concerning accuracy, the Policy also specifies that all data releases are to be accompanied by information on three common sources of error: coverage, non-response, and sampling error, as well as any other significant sources of error (e.g., response errors, processing errors). The audit found that for the MSM , information on these sources of error is produced monthly. Quality measures produced include final response rates, edit and imputation rates and coefficients of variation (CVs) for the main estimates. Each month these quality indicators as well as information on the survey concepts, definitions, data sources and methodology are published on the Integrated Meta database (IMDB), which is maintained by Standards division.

Within the various steps of the MSM process there are a variety of quality assurance strategies in place, applied and effective. Reviews of the microdata are completed by subject matter officers within each estimation phase of the MSM production cycle.^{Footnote 1} At each estimation pass, subject matter officers run diagnostic reports on specific industries to ensure that the incoming data are valid and that edit and imputation strategies have been appropriately applied. For data anomalies, SMOs and team leads validate changes and recreate diagnostic reports to ensure the data are coherent and accurate. The senior economist provides a final review – runs diagnostic reports on micro data and validates the aggregate trends. Prior to data sign-off, an external review of the microdata is completed by Industry Accounts Division, to ensure data accuracy and coherence. A final review of aggregate information is provided by senior management within Manufacturing and Energy Division (MED) and at the Statistics Canada level.

Based on the existing quality processes, the audit determined that an effective quality management framework is in place and quality targets and quality assurance mechanisms are monitored and effective.

Risk Management

The existence of formal practices and procedures for managing risks related to the quality of MSM data ensures that management identifies and responds to risks for assuring the accuracy and quality of survey information.

This includes the identification of risks; the assessment of controls in place to mitigate risks; the implementation of risk mitigation strategies and action plans to manage outstanding quality risks; and periodic re-assessment of risks for relevancy and assessment of the effectiveness of mitigation strategies.

Divisional Risk Management
Within MED, the assessment of risk for the MSM was completed as part of the divisional Program Performance Review (2010), as well as during the Quality Assurance Review (2007). During these exercises, external and internal risks to the quality of MSM information were identified and mitigation strategies were developed. MED management noted that risk identification is also on-going and takes place informally as issues arise, however these discussions have not been documented.

MED risk management is conducted as a formal exercise which permits management to assess and monitor the internal environment (survey process) and external environment (manufacturing sector in general) for risks. Management identifies, evaluates and proactively monitors risks; mitigation strategies have been developed to reduce identified key risks to acceptable levels. Management reassess risks and the risk environment formally as part of program performance reviews, and informally on an on-going basis.

Overall, Manufacturing and Energy Division has an effective formal risk identification process in place for the MSM which monitors the internal and external environments on an on-going basis. Management indentifies and monitors the controls in place to manage risks to quality and input into the quality assurance control of MSM information comes from other divisions within Statistics Canada.

Line of Enquiry No. 2:  The manual controls undertaken by MSM staff during the production phase of the survey are validated, documented and authorized.

Data Stewardship

Key quality control points should be consistently applied by MSM staff and activities linked to quality in the task lists should be linked to the six static dimensions of quality outlined in the Statistics Canada Quality Guidelines.

The audit determined that the key quality control points have been integrated into the MSM process and consistently applied. The key quality control points in the MSM process occur at each estimation pass completed during production of the data. Results of the walkthrough with the unit head of operations and testing of sign-off of checklists completed by the senior economist showed that requisite diagnostic reports were created and monitored at each estimation pass. The senior economist validated that the required checks were completed at each control point and were consistently applied.

The Statistics Canada Quality Guidelines state that up-to-date detailed procedures documents should exist. These guidelines also note that Statistics Canada considers quality to be multidimensional and is defined in six dimensions and these dimensions can be linked to the survey process. The task lists for the MSM act as procedures documents and are detailed and effective. Although the MSM task lists for SMOs, team leads and the unit head do not specifically reference the six dimensions of quality, the audit team examined the task lists and Guidelines and found that processes and tasks detailed for each MSM production employee can clearly be linked with the six elements of quality.

The effectiveness of manual controls should include a balanced mix of preventive and detective controls and should be validated, documented and authorized. Manual controls should be tested for effectiveness on a periodic basis and a formal process to challenge survey assumptions should be in place.

The MSM processing system uses both manual and automated controls in the production of monthly MSM estimates. Automated controls in a survey environment ensure that there is consistent implementation of processes. For example, automated controls are used to implement on-going recurring data revisions such as the calendarization of data for respondents who cannot report monthly, on-going splits of data between establishments, and GST sales calculations.

The MSM has a functional mix of preventive and detective manual quality controls embedded in the production environment. Subject matter officers keep an on-going list of establishments which require manual validation or revision of data. Examples of preventive manual controls include incoming data requiring conversion into Canadian currency to ensure data are accurate; new reporting structures for an establishment that have not been programmed into the system; or in cases where there is prearrangement with a large, complex multi-location establishment to have the head office collect the data. These preventive controls ensure that the appropriate calculations and allocations of data are completed.

Detective manual controls are employed when diagnostic reports indicate a data anomaly. For example, zero reporters, negative values, blank records, top movers and largest firms within each specific industry would all trigger a manual review of the data for these establishments. The audit tests determined that when an establishment appears in a diagnostic report, the data are investigated and validated to ensure that the cause of data anomaly is justified.

The audit tested to confirm that manual controls were documented, validated and authorized. Testing included the verification that the rationale for manual edits was documented in the MSM processing system. A sample of 22 establishments requiring manual controls was chosen.

In the audit sample one large enterprise required currency conversion. The audit tested and confirmed that the conversion rates used for the currency change corresponded to the rates prescribed in the task and check lists for the industry, for the survey reference periods of September and November 2011. There were 11 instances in the sample chosen which required the data to be split to ensure that the correct sales, inventories and/or orders were allocated to the appropriate locations for the enterprise. The audit confirmed that these splits were accurately completed and documented in the system.

The audit examined 11 top movers among the three largest manufacturing industries in Canada. The audit confirmed that for all the top movers, MSM staff validated that the major change was the result of an economic event and was documented. In cases where no justification for the large change could be established, there was a documented short-term strategy for the situation, which also ensured that the establishment was moved onto the problem file to be monitored. No formal authorizations were in evidence for manual changes to the data. MSM staff stated that authorizations for major revisions are done verbally but not documented. Informal, undocumented authorizations increase the risk of unknown accountability and the loss of the underlying rationale for manual data revisions over time.

The sample included one case where the data for a respondent was derived from GST files and had been reported as zero value for sales for the month of November. Upon examination of the comments in the system it was noted that manufacturing sales were being imputed from tax data. Tax data files were then examined, and the audit team found that although GST sales were zero, tax data indicated that sales were reported for the unit and these were being used to estimate manufacturing sales for the month. It was noted that GST data should not have a zero value. This led the SMO to examine the BR, which classified the unit as a manufacturer. However, upon examination of the corporate tax file it was found that the unit was classified as a wholesaler and as a result should not be in the sample and could impact estimates for the specific manufacturing industry. Consequently, the SMO sent a request to the BR to validate the type of industry of the unit.

Examination of comments in the production system and of on-going problem files for the three largest manufacturing industries showed that both preventive and detective manual controls were effectively documented. The system also includes an audit trail which records any changes made to the data, the date of the changes and the user making the changes.

Validation of manual controls is ongoing throughout the MSM process. SMOs, team leads and the senior economist for the MSM re-run diagnostic reports at each estimation pass, perform validation and provide a challenge function of the manual controls completed in the month.

During the testing phase, the audit team found two instances within the sample where manual controls were found to override separate automated controls. More specifically, the production system had been programmed to automatically calendarize incoming data for these establishments because their data reporting periods were not on monthly basis. As a result of the manual entry of data, automated controls were overridden. This situation went undetected until July 2011, when a new SMO noticed the automated control was not being implemented. Calendarization for these two establishments is now calculated manually. Although this was corrected by MSM staff, there is no established procedure in place within the MSM process to ensure that manual controls do not impact automated controls.

The audit team found that both detective and preventive manual controls are in place and manual controls are effectively validated and documented. Opportunities exist to ensure that manual controls are formally authorized. Assessment of manual controls within the MSM production system indicated that the control framework should be strengthened to ensure that manual controls do not unintentionally override other automated controls.

Recommendations

It is recommended that the Assistant Chief Statistician Business and Trade Statistics Field should ensure that:

• Manual controls, when necessary, are formally authorized when major data revisions are required;
• Ongoing monitoring of the control environment takes place to ensure that manual controls do not unintentionally override other automated controls

Management Response

Management agrees with the recommendations.

• The Chief, MED will ensure the development and implementation of a control sheet to record events and approvals when manual controls are undertaken for MSM data; and will ensure that this new reporting requirement is communicated to all MSM staff and incorporated into the existing procedures document for Subject Matter Officers (SMO). Monitoring will take place to ensure control sheets are appropriately used.
  
  Deliverables and Timeline: Development and implementation of the control sheet by October 2012. Monitoring will be ongoing beginning in October 2012.
• The Project Manager of the Industry Statistics Branch Monthly Survey Systems Integration Project (ISBMSSIP), MED will ensure that implementation of the ISBMSSIP production system is completed, which will reduce the requirement for manual intervention, and allow for enhanced comments and notes in the system.
  
  Deliverables and Timeline: The implementation of the ISBMSSIP production system by June 2012.
• The Chief, MED will ensure the completion of the sample restratification, which will reduce the need for corrective manual controls in the MSM production environment.
  
  Deliverables and Timeline: The restratification of the MSM sample will be completed by March 2013.
• The Chief, MED will ensure that other MSM staff and Statistics Canada data users are made aware of the manual controls applied to the data to reduce the risk of overriding other controls.
  
  Deliverables and Timeline: Plan for ensuring that applied manual controls are communicated to other MSM staff and data users is ongoing.
• The Chief, MED will ensure that a log of records requiring manual controls is produced each month and these units will be monitored to ensure automated controls have not been unintentionally overridden.
  
  Deliverables and Timeline: Implementation of a manual control log by October 2012. Monitoring to ensure that manual controls do not unintentionally override other automated controls will be ongoing.

Appendix

Appendix A: Audit Criteria

Line of Enquiry/ Core Controls	Criteria
The MSM production environment has a sound quality control framework to ensure that the quality of the data is in accordance with the accuracy component of Statistics Canada Quality Guidelines.
Accountability	Authority, responsibility and accountability related to quality are clear and communicated.
Results and Performance	Management has identified appropriate quality measure linked to planned results. Management monitors actual performance against planned results and adjusts course as needed.
Risk Management	Management identifies the risks that may preclude the achievement of its objectives. Management identifies and assesses the controls in place to manage risks.
The manual controls undertaken by the MSM staff during the production phase of the survey are validated, documented and authorized.
Stewardship	Activities needed to achieve quality targets have been integrated into the MSM quality control process. A formal process is in place to challenge the survey assumptions and related manual quality controls.

Note