Audit Report
April 22, 2013
Project Number: 80590-71
- Executive summary
- Introduction
- Background
- Audit objectives
- Scope
- Approach and methodology
- Authority
- Findings, Recommendations and Management Responses
- Management Control Framework
- Stewardship
- Appendices
- Appendix A: 2011 Census Field Structure
- Appendix B: Pay Process per Form 49-E Finance Office Manual
- Appendix C: Audit Criteria
- Appendix D: Acronyms
Executive Summary
The Statistics Act requires Statistics Canada to perform a Census of Population (Census) every five years. The Census is the largest, the most complex and publicly visible program, which obtains information from the entire population of Canada. In June 2010, a decision was taken by the government to maintain the 2011 Census of Population as a basic short-form census. All remaining questions that were previously asked in the long form census were included in a new voluntary survey, the National Household Survey (NHS).
Collection operations for the Census/NHS program operated nation-wide, with 3 regional offices, 5 Regional Census Centres (RCC) and numerous Local Census Offices (LCO) located across the country. Non-Response Follow-Up (NRFU) activities were carried out over a period of 11 to 15 weeks. There are distinct and inherent challenges to managing NRFU activities of approximately 30,000 enumerators operating in every part of the country, while striving to achieve the desired response rate for both the Census and NHS. In order to meet Census program objectives, Census management had the task of striking an appropriate balance between applying vigilance and due care to cost controls while managing the complexities of collection operations, in order to obtain a satisfactory response rate within an inflexible timeframe.
The practice of timekeeping for the enumerators was new in 2011. In the past, enumerators were paid based on piece rate. In order to improve enumerator recruitment and retention for the 2011 Census, the approach was modified to pay enumerators on an hourly rate. Timekeeping became an important element and was integrated into the Field Management System (FMS). Field staff used the FMS to report work performed daily, and to enter their pay and expense claims. Total costs managed through FMS, amounted to approximately $128 million.
This audit took place after Census collection operations had wound down and focussed on key controls and processes in place to ensure the reasonableness of enumerator fees. It was meant to add value by assessing practices in place during the 2011 Census, verifying if identified risks from a management led review had materialized, and identifying potential for improvements as management takes on the planning phase for the 2016 Census.
This report is intended to provide the Chief Statistician and senior management of Statistics Canada with relevant information that supports engagement observations and recommendations. As such, the focus of this report lends itself to the areas which may need management's attention. Areas where performance was deemed satisfactory have been summarized and acknowledged as part of the overall conclusion.
Key Findings
Management has identified a number of risks that may preclude the achievement of its objectives and has assessed the existing controls in place. However, financial risks resulting from significant changes in the Census pay structure were considered but not included as part of the risk management framework.
The change to enumerator pay based on an hourly rate, rather than on piece rate, added complexity to financial planning of collection operations due to a lack of historical data. Regions were instructed to carry out field operations and report on program spending, with the understanding that additional funds would be allocated if required to complete both Census and NHS collection operations based on active monitoring of response rates. The absence of historical data, combined with the changes related to the new NHS, made it challenging for Census management to set informed financial targets for Census 2011. This vulnerability elevates the risk that regional management may have a diminished incentive to control NRFU costs for collections activities.
FMS and Management Information Systems (MIS) provided timely management information for case management and pay/expenses incurred by enumerators. The quality of information contained in these reports can be enhanced by identifying and explaining financial variances so that management can monitor actual performance against planned results and adjust its course as needed.
Training and directives with regard to cost control and the use of MIS reports identified roles and responsibilities but would have benefitted from clearer direction or guidance on the approach to carrying out oversight duties.
Planning and monitoring activities were mainly focused on meeting collection targets. Crew Leaders (CL) and Field Operation Supervisors (FOS) reviewed claims and productivity reports for reasonableness. This key control relied upon by Local Census Office Managers (LCOM) for approving pay and expense claims under Section 34 of the FAA, provided low assurance that costs were reasonable.
Results of audit tests performed demonstrated adherence to the Census Pay Process. All employees examined in the sample were deemed legitimate; all travel claims examined were pre-approved and signed by individuals with the proper authority; and payments were deposited in the bank accounts of the intended enumerators. Enumerator pay and expense claims were batched and approved by individuals holding authority under Sections 34 and 33 of the FAA. However, in most cases, there was no evidence of any analysis or enquiries regarding accuracy and reasonableness of batches, as required for Section 34 sign-off. Consequently, individuals approving payments under Section 33 did not always have sufficient evidence that procedures required under Section 34 had been performed.
Overall Conclusion
The Census is the largest, most complex and publicly visible program at Statistics Canada. It is a collaborative initiative involving both the national program and regional operations. Collection activities are carried out over a very short period of time. The Field Management System (FMS) was developed mainly to address communication problems encountered in 2006 Census. A significant change occurred for the 2011 Census; enumerator pay was based on an hourly rate, rather than on piece rate and timekeeping became an important element and was integrated to FMS.
While Statistics Canada has an adequately designed control framework in place surrounding expenses processed through FMS, opportunities exist to strengthen controls over NRFU costs.
Audit procedures did not reveal any instances of fraudulent activity, malfeasance, or misappropriation of funds.
Key controls relied upon by Local Census Office Managers (LCOM) for approving pay and expense claims under Section 34 of the FAA, provided low assurance that costs were reasonable.
Conformance with Professional Standards
The conduct of this engagement conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Quality Assurance and Improvement Program.
Patrice Prud'homme
Chief Audit Executive
Introduction
Background
The Statistics Act requires Statistics Canada to perform a Census of Population (Census) every five years. The Census is the largest, the most complex and publicly visible program, which obtains information from the entire population of Canada. It is different from other surveys, in that every household in Canada is legally required to complete the Census form; therefore, it attains higher level of coverage and accuracy than voluntary surveys. In June 2010, a decision was taken by the government to maintain the 2011 Census of Population as a basic short-form census. All remaining questions that were previously asked in the long form census were included in a new voluntary survey, the National Household Survey.
Changes to the Census questionnaire and the introduction of the National Household Survey also introduced changes to the sample decision that resulted in a significant increase in the number of dwellings receiving the Census and the National Household Survey questionnaires. The Census questionnaire was sent to 100% of Canadian dwellings with the National Household Survey distributed to 33% of Canadian dwellings.
In addition to initial funding received for the 2011 Census of Population, Statistics Canada received additional funding that was intended to be used for three purposes:
- Implement changes to the questionnaires and processing systems;
- Increased costs associated with the printing and mailing of additional questionnaires (both Census and the NHS)
- Increased follow up, if necessary, to ensure a 98% response rate on the Census
Statistics Canada hired approximately 30,000 enumerators under the Statistics Act to conduct field collection operations for the 2011 Census and NHS.
The timeframe in which Non-Response Follow-Up (NRFU) activities were carried out was very short. There are distinct and inherent challenges to managing the NRFU activities of approximately 30,000 enumerators operating in every part of the country. In order to meet Census program objectives, Census management had the task of striking an appropriate balance between applying vigilance and due care to cost controls while managing the complexities of collection operations, in order to obtain a satisfactory response rate within an inflexible timeframe.
The timelines for Census and NHS were as follows:
- Early Enumeration: from February 1, 2011 to May 20, 2011 (15 weeks)
- Census NRFU: from May 20, 2011 to August 5, 2011 (11 weeks)
- NHS NRFU: from June 8, 2011 to Aug 24, 2011 (11 weeks)
Collection operations for the Census/NHS program operated nation-wide, with 3 regional offices, 5 Regional Census Centres (RCC) and numerous Local Census Offices (LCO) located throughout the country.
One of the most important innovations introduced for the management of the 2011 Census was the Field Management System (FMS). This system was developed mainly to address the communication problems encountered in 2006, but also to include other features to support collection work in the field.
The practice of timekeeping for the enumerators was new in 2011. In the past, enumerators were paid based on piece rate. In order to improve enumerator recruitment and retention for the 2011 Census, the approach was modified to pay enumerators on an hourly rate. Timekeeping became an important element and was integrated to FMS. Field staff used FMS to report their productivity on a daily basis, and to enter their hours worked and expense claims.
Total costs managed through FMS, which consists of enumerators' salary fees, local KMs claimed and other costs, amounted to approximately $128 million.
Region | Claims | Salary ($) | Local travel ($) | Meals and expenses ($) | Total ($) |
---|---|---|---|---|---|
*Source: FOP (2011 Census - Field Operations Project) | |||||
East | 681,016 | 37,730,179 | 5,291,290 | 123,028 | 43,144,496 |
Central | 696,349 | 35,777,213 | 3,968,622 | 101,008 | 39,846,843 |
West | 705,949 | 39,199,572 | 5,672,516 | 331,447 | 45,203,535 |
TOTAL | 2,083,314 | 112,706,963 | 14,932,427 | 555,483 | 128,194,874 |
In November 2011 and February 2012, two management-led reviews were performed on the Census Pay Systems: Design and Implementation; and Operating Effectiveness. The reviews covered mostly IT controls imbedded within the FMS system as part of the overall assessment of internal controls for Census Pay. Results of these reviews revealed a number of significant weaknesses related to FMS; however no testing had been undertaken to verify whether existing compensatory controls were effective in mitigating these risks or if risks identified for 2011 had materialized. Management plans to address these concerns as part of the planning phase of the 2016 Census.
Aside from IT controls imbedded within the FMS, other key controls were in place to ensure the reasonableness of enumerator fees, such as reviewing enumerators' performance prior to approving hours and KMs claimed and closely monitoring progress. Monitoring results and promptly addressing performance issues were also key controls relied upon to ensuring program objectives were met. Throughout the Census collection period, the FMS also provided a wide range of reports and information, updated daily, and accessible to supervisors and program management for timely decision making.
The audit took place after Census collection operations had wound down. It was meant to add value by assessing practices in place during the 2011 Census, verify if identified risks had materialized, and identifying potential for improvements as management takes on the planning phase for the 2016 Census.
Audit Objectives
The objectives of this audit were to provide assurance to the Chief Statistician and Statistics Canada's Departmental Audit Committee on:
- The adequacy of the control framework in place surrounding financial planning, monitoring and reporting of expenses processed through FMS; and
- The effectiveness of key controls embedded within the pay process surrounding payments to employees hired under the Statistics Act during the 2011 Census/NHS, including those managed through the Field Management System (FMS).
Scope
The audit focussed on mechanisms in place to ensure the reasonableness of enumerator fees. This included compliance to procedures described in Form 49-E Finance Office Manual, as well as the quality of financial planning, monitoring and reporting practices critical to sound program financial management and reports produced for high-level program decision making.
Two regions were selected for the conduct of this audit: the Eastern and Western regions. These two regions offered the most national coverage for the Census collection process. Two site-visits were conducted during the execution of the audit. Regional headquarters are located in Montreal and Edmonton, and were the focus of this audit.
Approach and Methodology
This audit was conducted following the Standards for the Professional Practice of Internal Auditing as per the Institute of Internal Auditors (IIA) and in accordance with the TBS Policy on Internal Audit. The audit approach was inspired by the Government of Canada Management Accountability Framework (MAF) and the Treasury Board Core Management Control guidelines issued by the Office of the Comptroller General.
The audit work consisted of an examination of documents, interviews with key Senior Management and personnel of Statistics Canada, including the Collection and Regional Services Branch (CRSB), as well as former employees hired under the Statistics Canada Act, a review of the processes and procedures outlined in policies, various guidelines and regional manuals, and substantive testing based on a judgemental sample of transactions.
Authority
The audit was conducted under the authority of Statistics Canada Integrated Risk-Based Audit and Evaluation Plan 2012/13-2014/15 recommended by the Departmental Audit Committee, April 2012 and subsequently approved by the Chief Statistician.
Findings, Recommendations and Management Responses
Objective No. 1: Adequacy of the control framework in place surrounding financial planning, monitoring and reporting of expenses processed through FMS.
Management Control Framework
Management has identified a number of risks that may preclude the achievement of its objectives and has assessed the existing controls in place. However, financial risks resulting from significant changes in the Census pay structure were considered but not included as part of the risk management framework.
The absence of historical data related to the way enumerators were paid, combined with the changes related to the new NHS, made it challenging for Census management to set informed financial targets for Census 2011. This vulnerability elevates the risk that regional management may have a diminished incentive to control NRFU costs for collections activities.
FMS and MIS provided timely management information for case management and pay/expenses incurred by enumerators. The quality of information contained in these reports can be enhanced by identifying and explaining financial variances so that management can monitor actual performance against planned results and adjust its course as needed.
A sound management control framework should be in place to effectively manage Census operations and related risks. Review and monitoring of enumerator performance, deemed a key control, should be conducted on a daily basis and poor performers should be dealt with in a timely fashion. Ongoing reviews, analysis, and reports of financial results versus established budgets should occur to ensure performances of the regional operations are managed appropriately, and communicated to relevant stakeholders for program decision-making.
Risk Management
Roles, responsibilities and accountabilities for risk management were clearly defined at the Census Project Team (CPT) management level. Risks identified covered most applicable types of risks and considered both internal and external sources of risks. Risk events were also identified at the regional level, and captured in the Risk Management System (RMS) by FOP management at NHQ.
The Risk Management System (RMS), specifically created for the Census through a Census Master Risk Register (CMRR), served as a formal risk management system for the 2011 Census. Identified risks and associated mitigation strategies were formally captured in the RMS. Each risk was categorized according to its probability and impact, and a risk register was developed for all high impact risks faced by the program. Specific options and mitigation strategies were developed, discussed, approved and formally documented in advance by the Census project team.
For each identified risk, the following information was documented: risk owner, date risk was identified, category (internal or external), probability, impact, affected activities element and projects, escalated to which activity element (responsible manager), timeframe, detailed description of the risk, impact assessment, mitigation plan and file attachment for contingency plans. A closing response was also provided at the end of the Census.
The audit revealed that financial risks resulting from significant changes in the Census pay structure were not included in the RMS. However, these risks were identified by FOP and regional operations staff, and were documented as part of planning documents and analysis. Examples of risks documented by the program or mentioned by regional staff during interviews included: the inability to effectively manage costs under hourly pay system; potential for collusion between CLs and enumerators; lack of understanding of the importance of controls in place; and lack of resources responsible for financial administration of the program. Although efforts were clearly made to assess the impact of the change in payment from piece rate to hourly rate, these risks were not documented in the RMS.
Consequently, key financial risks were considered but not included in the formal risk management framework and did not benefit from the thoroughness of the Census RMS, where mitigation strategies would have been developed and communicated to staff, responsibility would have been assigned to a manager, and monitored for effectiveness.
Budgeting and Accountability
The Census Management Office (CMO) was responsible for the entire 2011 Census financial envelop. FOP was allocated approximately $130M for field operations and was responsible for assigning financial targets for Census field operations, and monitoring progress over program delivery, including total program spending. Census field operations were focused on attaining a response rate of 100% for Census and a sufficiently high response rate for NHS in order to ensure data quality.
A significant change occurred for the 2011 Census; enumerator pay was based on an hourly rate, rather than on piece rate. This change added complexity to financial planning of collection operations due to a lack of historical data. Regions were instructed to carry out field operations and report on program spending, with the understanding that initial financial targets were insufficient and that additional funds would be allocated to complete both Census and NHS collection operations, based on active monitoring of response rates.
Response rate targets for the NHS were communicated by CMO in July 2011. At that time, program costs in some regions were exceeding financial targets. Regions were being instructed by CMO to continue collection activities for Census and NHS until a satisfactory response rate across the country was achieved. Given the daily decision making required to achieve balanced response rates, regional management and the CMO tracked expenditure patterns and costs on a daily basis, but chose not to update the budgets on the same basis. Census management stated that costs were an integral part of the decision making process regarding follow up activities which were managed centrally with CMO, CRSB and Methodology Branch representatives.
The absence of historical data related to the way enumerators were paid, combined with the changes related to the new NHS, made it challenging for Census management to set informed financial targets for Census 2011. This vulnerability elevates the risk that regional management may have a diminished incentive to control NRFU costs for collections activities.
Field Management System & MIS Reports Used for Decision Making
Integrating performance targets throughout the reporting process yields better information for planning, forecasting and resource management. It also allows management to monitor actual performance against planned results and adjust its course if needed.
The FMS was developed following the 2006 Census to remediate problems related to the field operations in the areas of case distribution and management. The FMS's most critical functions were developed and tested; results were presented in the FMS Census Test Evaluation, in September 2009. They included the creation and management of NRFU assignments, pay entry and management features, access to and management of respondent action requests, access to management information, data capture of visitation records and the capture of collective dwelling information.
The FMS is a web based application that all field personnel could access using their own personal computer and Internet connection. It allowed field staff to have access to lists of dwellings requiring non-response follow-up, notification lists for questionnaires received at the Data Operations Centre, action requests, management information reports, and timely instructions and messages from management. Field staff also used it to input their productivity and to enter their pay and expense claims, on a daily basis.
The Management Information System (MIS) was the main source of information to monitor Census regional operations. While FMS was the system used to collect data on enumerators' productivity and time/expenses incurred, the MIS was the system used to compile results from FMS data. The majority of reports available in this data warehouse provided information on the progress of operations against set targets. Information on enumeration was entered in FMS, and pre-configured reports were accessible to supervisors and management at NHQ and in the regions through the MIS portal.
Reports which could be generated from this system provided information related to:
- Progress on recruitment, hiring and training hours,
- Cumulative number of hours claimed by enumerators,
- Progress on various activities of enumeration – including Early Enumeration and NRFU,
- Cumulative cost for each operation at the FOS and LCO levels, and
- Progress of shipments of completed paper forms.
MIS Summary reports were produced daily by ADs-Census for Regional Directors' review, and were used to monitor progress against operational targets set for the region, outliers trending above average on productivity and/or costs, and the percentage of allotted funds spent to date. However, these reports did not present cost items against any set financial targets. As an example, the Cost by Operation Key Indicator Report (KIR) provides the total daily and cumulative expenses, including: number hours and KMs incurred on a given day; the associated dollar value; other expenses; and the total amount of all expenses. These amounts are not compared to any target or expectation. The absence of historical data for remuneration based on an hourly rate precluded them from the benefit of using benchmarks or indicators as a basis to assess productivity. This impaired the ability to determine whether the cumulative amounts shown were within expectations or not, and explanations for budgetary deficits were not documented.
Field Operations Project (FOP) team was the bridge between regional operations and the Census program. FOP met with the regions on a daily basis via conference call to discuss results from MIS reports. FOP then met the Census Management Office (CMO) team on a weekly basis to provide updates on regional operations. The Operations MIS Committee, made up of all members of the Census Project Team, also reviewed MIS information as the operations unfolded to monitor the progress of the different projects. Although management reports were reviewed, key reports such as the Actual vs. Planned Costs and Cost by Operation KIR did not provide any explanation on the cause of financial variances or their impact on operations. This impaired management's ability to identify the source of problematic issues, take corrective actions, and possibly benefit from lessons learned in the future.
Overall, the FMS and MIS provided timely management information for case management and pay/expenses incurred by enumerators. The quality of information can be enhanced by identifying and explaining financial variances so that management can monitor actual performance against planned results and adjust its course as needed.
Recommendations:
It is recommended that the Assistant Chief Statistician of Census, Operations and Communications ensures that:
- Financial risks resulting from significant changes in the Census pay structure are included as part of the risk management framework;
- Financial and non-financial information gathered during the 2011 Census be used as benchmarks for planning and establishing targets at an appropriate level for the 2016 Census Program activities;
- The quality of information contained in MIS reports is enhanced to identify and explain financial variances so that management can monitor actual performance against planned results and adjust its course as needed.
Management Response:
Management agrees with the recommendations.
The Census of Population is the largest peace time operations in the federal government and by far the most complex program at Statistics Canada. The most challenging component of the program is the Field Operations, particularly the non-response follow up activity for millions of non-responding dwellings. Planning and executing non-response follow up activities requires a mature, dynamic and effective governance structure. Statistics Canada cannot determine at the outset of operations who will or will not respond and where, geographically, the non-responders will be located. Non-response activities are initially planned with response patterns from previous census cycles. The object of the field operations is to quickly, effectively and as efficiently as possible seek responses from those households that did not respond with initial contact.
Management must retain the flexibility to move its collection workforce and/or the workload around and to invest in follow up activities that ensure balanced response rates across the country. Based on years of experience conducting the Census, the Census Management Office, in collaboration with the Field Operations Project in Collection and Regional Services Branch determines productivity benchmarks and manage the financial risks in achieving a successful outcome at the corporate level.
In June 2010, a decision was taken by the government to maintain the 2011 Census of Population as a basic short-form census. All remaining questions that were previously asked in the long form census were included in a new voluntary survey, the National Household Survey. As a result, the complexity of field operations was augmented in 2011 with the introduction of the National Household Survey – the largest voluntary survey ever conducted by Statistics Canada.
Changes to the Census questionnaire and the introduction of the National Household Survey also introduced changes to the sample resulting in a significant increase in the number of dwellings receiving the Census and the National Household Survey questionnaires. The Census questionnaire was sent to 100% of Canadian dwellings – up from the planned 80% sample; with the National Household Survey distributed to 33% of Canadian dwellings – up from 20% sample for the long form Census.
In addition to initial funding received for the 2011 Census of Population, Statistics Canada received additional funding that was intended to be used for three purposes:
- Implement changes to the questionnaires and processing systems;
- Increased costs associated with the printing and mailing of additional questionnaires (both Census and the NHS)
- Increased follow up, if necessary, to ensure a 98% response rate on the Census
The additional funding was retained and managed by the Census Management Office. Approximately 50% of the fund was required to implement the changes and to fund additional printing and mailing costs. The remaining 50% was retained as a contingency and managed by the Census Management Office. The distribution of these funds was based on decisions taken by Policy Committee, on the recommendation of management of the Census Program. The recommendations were formulated based on daily discussions between Census Management Office, Collection and Regional Services Branch and Methodology Branch.
Given the daily decision making required to achieve balanced response across the country, it would be virtually impossible to update budgets for each Local Census Office on a daily basis. At the local level, managers are expected to manage productivity. However, expenditure control is an integral part of the decision making process to ensure there is continued return on investment. Expenditure patterns and costs are tracked on a daily basis by regional management in collaboration with the Census Management Office. Decisions on data collection priorities and budget allocations must be managed centrally with input from the Collection and Regional Services Branch, Methodology Branch and the Census Management Office.
The collection operations for the 2011 Census went extremely well and ultimately the contingency fund was not required to ensure the success of this collection activity. Achieving balanced response rates across the country then became the primary focus for the National Household Survey and the contingency funds were deployed for this purpose. Once it was determined that additional follow up would no longer improve the quality of the response rates, the decision to cease follow up activities was taken by Policy Committee and the remaining funds from the contingency were returned to the Treasury Board.
- The Census Management Office will ensure that financial risks related to the significant aspect of the census program will be included in Census risk management framework.
Deliverables and Timeline: The risk register will be kept up-to-date for the 2016 program, on an on-going basis. - As was done in the previous censuses, where data from the 2006 Census was used as benchmarks to support management of operations in 2011, the Census Management Office and the Research, Evaluation and Quality Management Project team will ensure that a similar approach will be developed for relevant operations for the 2016 program. The approach, while supporting sound financial management practices, must be flexible and targets established at appropriate levels to take into account the unpredictability of non-response operations Continuing the practice established in the 2011 collection operations, management from the Regional Offices, the Census Management Office and the Methodology Branch will work together to adjust daily operations priorities to achieve optimal response rates and to remain within allocated budgets.
Deliverables and Timeline: A detailed management information approach will be developed and tested in-time for the 2016 operations. - Census Management Office and the Field Operations Project team will ensure that based on the 2011 Census Program information, performance indicators/targets will be adjusted and financial MIS enhanced to provide better tools to improve the management of collection variable costs (e.g. alternative solutions will be considered to reduce kilometric costs).
Deliverables and Timeline: A detailed management information approach will be developed and tested for the 2016 operations. This task will take place during the 2014 test period.
Objective No. 2: Adequacy and effectiveness of key controls embedded within the pay process surrounding payments to employees hired under the Statistics Act during the 2011 Census/NHS, including those managed through the field management system (FMS).
Stewardship
Training and directives with regard to cost control and the use of MIS reports identified roles and responsibilities but would have benefitted from clearer direction or guidance on the approach to carrying out oversight duties.
Planning and monitoring activities were mainly focused on meeting collection targets. CL and FOS reviewed claims and productivity reports for reasonableness. This key control, relied upon by Local Census Office Managers (LCOM) for approving pay and expense claims under Section 34 of the FAA, provided low assurance that costs were reasonable.
Results of audit tests performed demonstrated adherence to the Census Pay Process. All employees examined in the sample were deemed legitimate; all travel claims examined were pre-approved and signed by individuals with the proper authority; and payments were deposited in the bank accounts of the intended enumerators. Enumerator pay and expense claims were batched and approved by individuals holding authority under Sections 34 and 33 of FAA. However, in most cases, there was no evidence of any analysis or enquiries regarding accuracy and reasonableness of batches, as required for Section 34 sign-off. Consequently, individuals approving payments under Section 33 did not always have sufficient evidence that procedures required under Section 34 had been performed.
The approval process for payments to enumerators included two key controls which were relied upon in the Census pay process: CLs' daily review of submitted claims, and the review and analysis of productivity reports conducted by the CLs, FOSs, Pay Supervisors, and LCOMs before the pay and expense claims were approved for payment. To do so, reports obtained for the review and monitoring of pay and expense claims must yield timely and accurate information for each level of supervision to identify errors or instances of unreasonable claims submitted by enumerators.
Control over Enumerator Pay and Expenses
For the 2011 Census enumerators were paid an hourly rate to improve enumerator recruitment and retention. Consequently, timekeeping became an important element and was integrated into FMS. Enumerators used FMS to report their productivity and to enter their hours worked and expense claims on a daily basis.
Each CL was responsible for supervising the work of between 8 to14 enumerators. CLs were expected to review each pay and expense claim before recommending them for approval in the FMS. The system included an optional function, where enumerators could enter details or justifications in support of their claims. To support the CLs' monitoring activities, the FMS system also flagged the top 25% highest amount of TPC in a given LCO.
As a result, heavy reliance was placed on CLs to ensure the reasonableness of enumerator claims, by reviewing enumerators' performance prior to approving hours and KMs claimed and closely monitoring progress. Monitoring results and promptly addressing performance issues were key controls relied upon to ensure program objectives were met. Effective cost control procedures are critical to providing assurance to individuals with delegated authority for Sections 34 and 33 of the FAA prior to approving payments to enumerators.
Expectations in regards to cost control to be undertaken by CLs were outlined in the Directive 08: Cost Control and Financial Management for Field Operations, as follow:
"Crew leaders are expected to hold review sessions with their enumerators shortly after the start of each operation. They have access to G-forms and MIS reports to monitor productivity and time per case. They should be in contact with their enumerators daily. It is the crew leaders' responsibility to communicate expectations to their enumerators. Should production fall behind expectations or claims for kilometres be unreasonably high, closer supervision is needed, corrective action must be taken, and if needed, an employee can be replaced when necessary. Crew leaders should not hesitate to turn to their FOS for help when a problem cannot be solved."
This directive identifies roles and responsibilities for cost control, but would have benefitted from clearer direction or guidance on the approach. The document indicates that discussions should take place for non-performers; however it does not explicitly address a need for control over hours charged, and does not provide clear guidance on what is considered reasonable and what is not.
Throughout the Census collection period, the FMS also provided a variety of management reports and information, updated daily, and accessible to supervisors and program management for timely decision making. Productivity reports were deemed to be a key monitoring tool to be used by CLs, FOSs, AMFOs and LCOMs. The reports generated from the FMS data included information by geographic region, including TPC, number of attempts, KMs claimed, cases completed, CLD TPC average, LCO TPC average. However, procedures documents and training material did not explain how to use the MIS reports in performing oversight duties for cost accuracy and reasonableness.
Through interviews and testing, the audit team assessed the CLs' review processes pertaining to pay/expenses and productivity reports, in order to determine their effectiveness in providing assurance that costs were controlled. Our examination of approximately 50 claims revealed that nearly two thirds of the claims did not include comments to explain hours/KMs, even in instances, for example, where more than 100 KMs were claimed and no questionnaires were collected.
The lack of justifications places the onus on the CL's to follow up with enumerators who were automatically flagged by the FMS as under performers. Furthermore, CLs interviewed stated that when they did contact enumerators to enquire about their time and expenses, they rarely denied claims because the enumerators were able to provide various plausible explanations (e.g. excessive traffic congestion, road construction, etc,) to justify claims that appeared excessive. In most cases, CLs were not able to challenge exaggerated claims, only those which appeared grossly inflated were investigated or received attention.
As mentioned above, the FMS automatically flagged high amounts of TPC, and generated a report which identified the top 25% highest amount of TPC in a given LCO. CLs had access to monitoring reports and were expected to use this information prior to recommending pay claims. The control was meant to quickly identify claims that appeared unreasonable, in comparison to other enumerators for the same area. Tests revealed that if an enumerator did not report any forms collected, hours charged trying to collect questionnaire would not be flagged by the system's reports. The system did not flag cases of high TPC until the questionnaires were collected within the collection unit.
After examination and interviews with regional staff, the audit team noted that enumerator's time and expenses vs. questionnaire information were not always uploaded into MIS simultaneously. Depending on the time of day at which the enumerators input their time into FMS, loading the case information did not always occur on the same day as the time and expense information was entered. This situation impacted the accuracy of TPC statistics for collection units, providing varying results. Some CLs interviewed mentioned that they did not always follow-up on flagged cases, as in many cases they noticed that the numbers eventually evened out. This affected the timeliness and integrity of information used to identify poor performers and reduced the effectiveness of the control. Census Evaluation results indicated that 95% of all claims were paid without any edits by CLs, demonstrating that few changes resulted from the CLs' monitoring of claims.
Field operations supervisors (FOS) are the front line supervisors in comparing productivity to cost by using the MIS reports related to each operation. Their other main role was to monitor reports from their crew leaders and provide guidance when needed. FOS supervised between 7-12 CL's, all working in different geographical locations and contexts. Interviews with FOSs revealed that they did not monitor the effectiveness of reviews conducted at the CLs' level and were mainly focussed on optimizing staff distribution according to productivity levels attained. Most importantly, they stated that the expectation that FOSs would undertake this analysis was not realistic considering the short timeframe for NRFU.
Audit testing showed that enumerators flagged by the system for low productivity remained active while no significant improvement occurred. There was no evidence that follow-up had been performed for any of these poor performers. Conversely, performance evaluation for these same enumerators indicated "met" or "exceeded expectations" – by making reference to their ability to collect questionnaires, never commenting on their productivity or value for money.
The audit revealed that planning and monitoring activities were mainly focused on meeting collection targets. The review of FMS claims and productivity reports by Crew Leader and Field Operation Supervisors, considered a key control relied upon by Local Census Office Managers (LCOM) for approving pay and expense claims under Section 34 of the FAA, provided low assurance that costs were reasonable.
Adherence to Census Pay Process
Authority for sign-off under Section 34 of the FAA for fees paid to enumerators resided with the LCOM. It was expected that procedures outlined in the Form 49-E Financial Office Manual would be followed and executed.
Once CLs recommended enumerators' claims, a summary of claims (batches) was prepared within each LCO for Section 34 approval by the LCOM, after which it was sent to the regional finance group and respective Pay Units for Section 33 approval. Details of the pay process in place during the 2011 Census can be found in Appendix B.
Attributes from Form 49-E Financial Office Manual and the Guide for Local Census Office Managers were used to develop our audit procedures. Testing was to provide assurance that controls were in place to ensure that:
- Enumerators who received payments were legitimate and were hired according to established procedures;
- Section 32 of FAA approval for Travel Authorization Forms was documented and supported;
- Section 34 of FAA approval was documented and supported;
- Section 33 of FAA approval was documented and supported; and
- Payments were made to the bank account of the intended enumerator.
A sample of 25 enumerators' pay claims were selected by the audit team for each region, covering two pay periods within the peak time of NRFU. Ten Crew Leader Districts (CLDs) were randomly selected within these regions, including enumerators that had claimed a significant number of hours. The sample included enumerators with a Time per Case (TPC) within the average and those exceeding the average.
The audit revealed the following:
In accordance with program directives, employee files were uploaded from the original application files created in the Census Automated Recruitment System (CARS), into FMS. In compliance with Form 46E - Directive 14: Recruitment for the 2011 Census, employees' application forms were subjected to effective segregation of duties throughout the approval process prior to being hired. Applicant records were traced back to CARS and examined to ensure that each level of approval was approved electronically by the right persons. Multiple-level approval processes are an effective measure in preventing the creation of fictitious employees. Original application forms, and other required documentation, which includes employee photos, were also on file. All employees examined in the sample were deemed to be legitimate employees.
Travel claims submitted by enumerators were pre-approved prior to travel. Travel Authority Request forms were signed by individuals having delegated authority under Section 32 of the FAA.
Enumerator pay and expense claims were batched and approved by individuals holding authority under Sections 34 and 33 of FAA. However, in most cases, there was no evidence of any analysis or enquiries regarding accuracy and reasonableness of batches, as required for Section 34 sign-off. Consequently, individuals approving payments under Section 33 did not always have sufficient evidence that procedures required under Section 34 had been performed.
Pay supervisors had the ability to modify employee pay account information in FMS. Testing found that payments were made to the bank accounts of intended enumerators. Prescribed forms required for amending pay account information for enumerators were on file and all required signatures matched.
Recommendations:
It is recommended that the Assistant Chief Statistician of Census, Operations and Communications ensures that:
- Training and directives with regard to cost control and the use of MIS reports are revised to provide clear direction and guidance on the approach to carrying out oversight duties.
- Key controls relied upon for approving pay and expense claims under Section 34 of the FAA, are reviewed for feasibility and effectiveness, taking into consideration the short time frame for carrying out NRFU activities;
- Procedures required under Section 34 of the FAA are consistently performed, and yield sufficient evidence and assurance for approval of payments under Section 33.
Management Response:
Management agrees with the recommendations.
- The Field Operations Project team will ensure that performance indicators/targets will be adjusted and processes refined to ensure managers have the tools they require to better manage collection variable costs. Training and directives on the use of MIS reports will be updated and responsibilities will be clarified to ensure managers understand and comply with their financial obligations.
Deliverables and Timeline: A detailed management information approach will be developed and tested for the 2016 operations, including training for those involved. This task will take place during the 2014 test period. - The Field Operations Project team will ensure that Section 34 authorities will be reviewed and adjusted as required. Key control processes will be clarified and appropriate training provided to ensure those individuals approving expenses under section 34 comply with their delegated authorities.
Deliverables and Timeline: A detailed management information approach is developed and tested for the 2016 operations, including relevant information in relation to FAA obligations. This task will take place during the 2014 test period. - The Field Operations team will ensure that Section 34 authorities will be reviewed and adjusted as required. Procedures required under Section 34 of the FAA will be clarified and appropriate training provided. In collaboration with the Finance Branch, individuals approving pay and expense claims will be monitored for compliance with their delegated authorities.
Deliverables and Timeline: A detailed management information approach will be developed and tested for the 2016 operations, including relevant information in relation to FAA obligations. This task will take place during the 2014 test period.
Appendices
Appendix A: 2011 Census Field Structure
Appendix B: Pay Process per Form 49-E Finance Office Manual
Chapter 3 - Filing
- Unless otherwise specified, standard regional office filing procedures will be followed.
- Statistics Act employees will have one file for all their documentation (e.g., appointment and pay forms). These files will be maintained in alphabetical order in the LCO within the AMA / Pay Unit. All account forms, hiring forms and support documents will be retained.
- Each Statistics Act employee file should contain:
- Hiring Checklist – Statistics Act Staff, Form 21B
- Statistics Act Candidate Screening Form, Form 22A
- Recommendation and Assignment of Statistics Act Staff, Form 23
- Oath or Affirmation of Office and Secrecy, Form 24
- Implication of the Oath or Affirmation, Form 24A
- Terms and Conditions of Employment (appropriate Form 25 series)
- Census Application for Employment, Form 26 (if application was not completed online), resume, CV and any other supporting notes produced during the selection process
- Selection Test, Form 27 or 27B / PSC Answer Sheet, Form PSC2256
- Non-supervisory Candidate Assessment Booklet, Form 27C/Answer Key – Selection Test, Form 27D
- Census Reliability Assessment, Form 26B
- FMS User Account Agreement Form, Form 26E
- Change of Address Form, Form 30B (if applicable)
- one photo
- Recommended Weekly Hours and Expenses Log, Form 32 (only needed when expenses have been claimed or when hours and productivity are entered by proxy)
- receipts are to be attached and filed with the Form 32
- Performance Evaluation, Form 38
- Direct Deposit Information, Form 39
- Deduction Declaration, Form 39B, applicable TD1, TP-1015.3-V and/or TD1-IN forms
- language test (if applicable)
- other relevant documentation (correspondence, notes of disciplinary meetings, etc.).
The contents of these forms are highly confidential and these forms must be filed in a secure, lockable cabinet with limited access, which must be kept locked after hours and when not in use.
- Files should be created on an ongoing basis and filing should be kept up to date in order to allow staff to locate documents quickly and easily.
- Batch payment authorization reports, payment confirmation report (CDFS) and summary report (CDFS) should be filed by date and in batch number sequence.
- The filing system must be maintained so that an audit trail can be followed by individual, by Batch Payment Authorization Report, or by Payment Confirmation Report.
- Form 31 and other documents preauthorizing expenditures will be sent to the regional census centre (RCC) for payment with a copy being retained and filed in the LCO where it originated. These files will be maintained in province, census district, and collection unit code sequence to enable easy access should they need to be retrieved at a later date.
- All protected information collected must be stored in a locked cabinet. These cabinets must be kept locked, especially at the end of each day, with the keys or combination stored in a secure and central location. If the need to store large volumes of protected material arises, contact Departmental Security.
Appendix C: Audit Criteria
Line of Enquiry / Core Controls / Criteria | Controls (existing and expected) | Policy Instrument |
---|---|---|
1) Adequacy of the control framework in place surrounding financial planning, monitoring and reporting of expenses processed through FMS. | ||
Risk Management | ||
1.1 Management identifies the risks that may preclude the achievement of its objectives, and assesses the existing controls in place. (RM-2 and 3) | 1.1.1 Management is aware of the potential impacts of changes in the Census structure (NHS), as well as significant operational changes (i.e. hourly pay) and has tools in place to manage these risks. | Management Accountability Framework RMS User Guide Census The 2011 CensusManagement Guide Statistics Act Form 46-E Directive 08: Cost Control and Financial Management for Field Operations |
Budgeting and Accountability / FMS & MIS Reports Used for Decision Making | ||
1.2 Financial management policies and directives are appropriately designed for Census field operations to ensure that budgeted and actual cost are reported and monitored at the program level. (ST-5) Reviews are conducted to analyze, compare and explain financial variances between actual and plan. (ST-15) |
1.2.1 Guidance is provided by Finance on the financial information structure, available reporting tools and financial management practices that need to be followed for the Census. | Management Accountability Framework MIS Portal Suite |
1.2.2 Reviews and analysis are performed over financial results and is compared to budgets in order to ensure that regional operations are managed appropriately. | Management Accountability Framework MIS Portal Suite |
|
1.2.3 Reports are prepared and communicated to relevant stakeholders on results of regional operations. | Management Accountability Framework MIS Portal Suite |
|
2) Adequacy and effectiveness of key controls embedded within the pay process surrounding payments to employees hired under the Statistics Act during the 2011 Census/NHS, including those managed through the Field Management System (FMS). | ||
Control over Enumerator Pay and Expenses | ||
2.1 Information used to approve enumerator claims is accurate and timely. Productivity reports are reviewed and analyzed by FOSs, Pay Supervisors, and LCOMs before the related pay and expense claims are approved for payment accuracy and timeliness. (ST-10/18) | 2.1.1 Review and monitoring of performance is conducted on a daily basis and poor performers are identified and are dealt with in a timely fashion. | Management Accountability Framework Financial Administration Act Analyzing NFRU Enumerator Productivity KIR Supervisor FMS User Guide and Forms Statistics Act |
2.1.2 Prior to recommending claims, supervisors either recommend the claims or consult with the employee to clarify and/or correct any discrepancies, as necessary. | Management Accountability Framework Financial Administration Act Analyzing NFRU Enumerator Productivity KIR Supervisor FMS User Guide and Forms Statistics Act |
|
2.2 Reports obtained for the review and monitoring of pay and expense claims could identify errors or cases of unreasonable claims submitted by enumerators. (ST-20) | 2.2.1 Claims are reviewed by supervisors and approved accordingly. | Management Accountability Framework Manual 46.39 Recommending Claims and Authorizing Payments by the FOS and LCOM |
2.3 Employees were provided with necessary training, tools, resources and information to support the discharge of their monitoring and oversight responsibilities. (PPL-4) | 2.3.1 Training for employees with supervisory functions in field operations provides direction on oversight over financial activities under their responsibilities i.e. review of pay and expense claims, productivity, S. 34 and S. 33. | Management Accountability Framework E-Library - en.STCwiki Statistics Act |
Adherence to Census Pay Process | ||
2.4 Pay and expense claims are reviewed for accuracy and reasonableness and approved by the appropriate individual (S. 34 of FAA and S. 33 of FAA). (ST-15/18) | 2.4.1 Section 34 of FAA review, and approval is documented and based on information allowing the identification of potential errors. | Management Accountability Framework Managers Financial Signing Authorities and the Procurement Process for Census Field Financial Administration Act Form 49E - Finance Office Manual Section 33 - Census Regional Finance Officers Form 32 Directive |
2.4.2 Segregation of duties for S. 33 of FAA is in place and monitored by FMOSD. | Management Accountability Framework Managers Financial Signing Authorities and the Procurement Process for Census Field Financial Administration Act Form 49E - Finance Office Manual Section 33 - Census Regional Finance Officers Form 32 Directive Form 46E - Directive 14: Recruitment for the 2011 |
Appendix D: Acronyms
Acronym | Description |
---|---|
AMFO | Assistant-Manager Field Operations |
CARS | Census Automated Recruitment System |
CL | Crew Leader |
CLD | Crew Leader District |
CMO | Census Management Office |
CPT | Census Project Team |
CRSB | Collections and Regional Services Branch |
DPC | Data Processing Center |
FAA | Financial Administration Act |
FMS | Field Management System |
FOP | Field Operations Project |
FOS | Field Operations Supervisor |
HQ | Headquarters |
IIA | Institute of Internal Auditors |
KM | Kilometres |
LCO | Local Census Office |
LCOM | Local Census Office Manager |
LOE | Line of Enquiry |
MAF | Management Accountability Framework |
MIS | Management Information System |
NHS | National Household Survey |
NRFU | Non-Response Follow-Up |
PSEA | Public Service Employment Act |
RCC | Regional Census Centre |
RMD | Resource Management Division |
RMS | Risk Management System |
TBS | Treasury Board Secretariat |
TPC | Time Per Case |