Audit of Collection and Regional Services

Background

Statistics Canada was established to ensure that Canadians have access to a trusted source of Canadian statistics to meet their highest priority needs. Access to trusted statistical information underpins democratic societies, as it supports evidence-based decision-making in the public and private sectors, and informs debate on public policy issues. Under the Statistics Act, Statistics Canada is required to "collect, compile, analyze, abstract and publish statistical information relating to the commercial, industrial, financial, social, economic and general activities and conditions of the people of Canada", which requires the organization to collect data from a wide variety of Canadian businesses and households.

At Statistics Canada, Collection and Regional Services Branch (CRSB) is responsible for the majority of data collection activities. Its mandate is to:

• Provide data collection services to Statistics Canada's statistical programs (Household surveys, Business surveys and Census collection);
• Provide the Management Structure for the Separate Employer – Statistical Survey Operations (SSO) – which comprises Statistics Canada's interviewers;
• Promote the availability and effective use of Statistics Canada's products and services through the Advisory Services and Communications program in the regions.

Within the branch, Collection Planning and Management Division (CPMD) is accountable for the coordinated planning of the collection capabilities, and the capacities of the collection infrastructure. They also act as the front door for most survey partners who require collection support. CPMD is the interface between the subject matter divisions and the regions and designs collection activities in response to client specifications, provides cost estimates, acquires requisite services from supplying divisions within the Field to execute the survey, monitors and reports on survey progress and conducts post-mortems of completed projects.

There are three regions that manage survey data collection operations: the Eastern Region, which has offices in Halifax, Montreal and Sherbrooke; the Central region, with offices in Ottawa, Toronto and Sturgeon Falls; and the Western Region and Northern Territories with offices in Edmonton, Regina, Calgary, Winnipeg and Vancouver. Within these regions, there are approximately 2000 interviewers and public servants providing services and conducting surveys.

The CRSB branch began piloting a branch renewal project in January 2014, with full implementation scheduled for April 1, 2014. Under this project, the roles and responsibilities for regional offices and CPMD will change. CPMD will transfer the management of surveys to the regions, and subject matter areas will communicate directly with the regions. CPMD will continue to do the planning and prepare the training materials for surveys, while regional offices will be fully responsible for the resolution of survey collection issues.

Audit objectives

The objectives of the audit were to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that:

• Statistics Canada has appropriate governance mechanisms in support of the planning and resource allocation of Collection and Regional services.
• Appropriate quality assurance mechanisms have been established and are consistently applied to ensure that Collection and Regional Services Branch are gathering quality data and are compliant with the Statistics Canada Quality Guidelines.

Scope

The scope of this audit included an examination of the adequacy and effectiveness of the quality controls in place for the collection of data conducted by Collection and Regional Service branch. Specific areas that were examined include the quality assurance controls currently in place within the Computer Assisted Telephone Interview (CATI) and Computer Assisted Personal Interview (CAPI) collection environments to ensure compliance with the Statistics Canada Quality Guidelines.

The audit also examined the effectiveness and adequacy of the governance and capacity planning for Collection and Regional Services. During the course of the audit, the audit team examined Collection Planning and Management Division at head office, as well as the regional offices in Edmonton, Sherbrooke and Sturgeon Falls. The audit covered the period from January 2013 to January 2014. The audit did not include infrastructure elements related to Shared Services Canada, as they will be examined in a separate engagement.

Approach and methodology

The audit consisted of a comprehensive review and analysis of relevant documentation, as well as interviews with key management and staff from CRSB at headquarters and regional offices. The field work included a review, assessment, and testing of the processes and procedures in place to ensure the Agency provides quality statistical information.

This audit included interviews with personnel in all of the regions, subject matter divisions, as well as interviews with CRSB staff located at headquarters. Document review included survey reports and documentation from each of the regions and headquarters. A random sample of interviewers was drawn from each of the sites visited. Testing included a random sampling of interviewer files from each of the three sites visited and examination of the Quality Control Feedback System (QCFS) reports for the specific months over the period under examination, and a validation to ensure training was provided to interviewers and that it occurred prior to collection activities. The audit also tested to determine the procedures in place to ensure that SSO employees were informed of and acknowledged the code of ethics and conduct for SSO employees. Additionally, because the Edmonton Regional office was the only office visited responsible for CAPI interviewers, a sample of CAPI interviewer files were examined.

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors (IIA) International Professional Practices Framework.

Authority

This audit was conducted under the authority of the approved Statistics Canada Integrated Risk-Based Audit and Evaluation Plan 2013/14 to 2017/18.

Governance

A robust governance framework is essential to ensure that Collection and Regional Services Branch (CRSB) provides quality data collection services to Statistics Canada's statistical programs. Authorities, responsibilities and accountabilities should be clearly defined and understood at all levels to support effective collection and capacity management, and a well-developed approach to risk management should be in place.

Responsibilities and accountabilities for CPMD and RO staff are defined and communicated

Responsibilities and accountabilities of the Collection Planning and Management Division (CPMD) and Regional offices (ROs) have been clearly defined in the Collection and Operations Services Agreement (COSA), which is a formalized agreement that has been developed to define roles, responsibilities and accountabilities of key stakeholders in the survey process.

The COSA describes the project, details the collection and operation activities, outlines the areas responsible for each activity and provides timelines. It also details what the deliverables are, and how they will be measured and monitored, including what management information reports will be used. The COSA includes details of the governance and communications processes that will be implemented to manage collection and operations activities. All new surveys are required to have a COSA completed and signed, but because COSA is new and is being implemented in stages, most ongoing surveys do not yet have an approved COSA in place.

Regional collections staff roles and responsibilities are outlined in the SSO employee handbook. It outlines key tasks, expectations and responsibilities for field and office interviewers, and senior interviewers.

Clearly defined mandates have been developed for collection oversight committees

Effective oversight bodies are important to ensure management's direction, plans and actions are appropriate and responsible. In order for oversight bodies to be effective, they should be provided with timely and accurate information to adequately fulfil their oversight function.

The audit noted that three oversight committees have been established for survey collections. These are:

• Collection Planning Committee (CPC);
• Social Collection Steering Committee (SCSC); and,
• Business and Agriculture Surveys Collection Steering Committee (BACSC).

The mandate of the Collection Planning Committee is primarily to review and recommend strategies for collection, respondent relations and operational efficiencies related to cost, quality and timeliness. The committee is also responsible for reviewing demands and priorities for collection services to recommend capacity adjustments. The CPC meets monthly and is chaired by two directors general. Membership is comprised of directors and assistant directors from within CRSB, as well as key stakeholder divisions from statistical infrastructure, informatics, and subject matter divisions. This committee reports to Executive Management Board at Statistics Canada.

The audit found that the committee is fulfilling its objectives with respect to collection strategies, respondent relations, quality, timeliness and reviews capacity demands. A review of the committee's minutes and interviews with key CPC members revealed that various issues related to the mandate are discussed and action items are assigned as required to ensure follow-up occurs. Additionally, representatives from working groups are invited to CPC meetings to make informational presentations regarding the status of initiatives or research.

The Social Collection Steering Committee and the Business and Agriculture Collection Steering Committee have clearly defined mandates. Both committees provide oversight of household, business and agriculture surveys collection and are required to report back to CPC on business or household survey collection issues, committee initiatives or decisions. SCSC and BACSC membership is at the director level or their delegates from subject matter and statistical infrastructure divisions, and representatives from CRSB are also members of these committees. The audit reviewed the committees' minutes during the last fiscal year that the SCSC and the BACSC are meeting the obligations noted in their mandates.

Effective capacity planning takes place

The Statistics Canada quality guidelines note that capacity planning is an important step in the survey process and effective capacity planning should be used as a tool to help ensure the quality of collection activities.

The audit found that capacity planning within CPMD is formalized and aligned with data collection planning in the regions. Interviews and document review revealed that business rules are in place and planning assumptions are documented. Current and future needs for resources are assessed on a regular basis and the information provided by headquarters is sufficient for regional offices to undertake their planning activities. Within the regions, labour market conditions are monitored to ensure offices have adequate staffing capacity to carry out collection activities.

CRSB and regional office risk management activities require further development

A well-developed approach to risk management should include identification and assessment of risks, development of mitigation plans to reduce the likelihood of risks, and the on-going monitoring of conditions to ensure risk management strategies are working as intended.

Within Statistics Canada, integrated risk management is performed at the program, division or branch level. Each year, areas identify the risks that may impact Statistics Canada's ability to accomplish objectives, along with the potential likelihood, impact and mitigation strategies.

The audit noted that risk management is developed at the Collection and Regional Services Branch level. Eight key risks facing the branch that may preclude the attainment of objectives were noted. Broadly, these risks can be organized into risks related to respondent relations, human resource risks, competing priorities, interdependencies, and the stewardship of information. High-level mitigation strategies have been identified, but do not outline the specific risk management activities to be performed, nor assign the accountabilities or timelines for implementation.

Recommendations:

The Assistant Chief Statistician, Census, Operations and Communications field should ensure that:

• Specific risk mitigation activities are established, accountability is assigned and timelines for implementation are documented.

Management Response:

Management agrees with the recommendations.

• The Director General, CRSB will modify the comprehensive Branch level risk profile to provide additional detail including specific risk mitigation activities, with appropriate designated accountabilities and specific timelines.
  
  Deliverables and Timeline: Revised CRS Branch Risk Profile will be completed by March, 2015.

Objective 2: Appropriate quality assurance mechanisms have been established and are consistently applied to ensure that Collection and Regional Services Branch are gathering quality data and are compliant with the Statistics Canada Quality Guidelines.

Survey Collection Management

The Statistics Canada Quality Guidelines note that data collections and capture operations have an impact on the accuracy of data. Given this, there should be a balance between quality and performance measurement tools used to manage data collections to help ensure the accuracy of the information. The Guidelines outline several reports that can be used as measures of quality and further notes that these measures help support decisions regarding the need to amend collection processes or redesign collection tools.

There are inconsistent approaches to survey collection management by Regional Offices and CPMD

Survey collection management includes the monitoring and controlling of collection activities. It should ensure that survey production plans are implemented and that necessary corrections or adjustments are made and communicated to subject matter and other relevant stakeholders. Regional office staff manages regional data collection activities and CPMD project officers manage collections, as well as monitor and report on overall survey progress.

Reports related to overall collection activities are prepared in the regions by data collection managers (DCMs), district managers and regional program managers (RPMs). Within CPMD, project officers are responsible for the preparation of reports to support survey management. The audit also found that, for some of the surveys examined, subject matter divisions also prepare and use collection reports. These reports analyze response rates, outcome codes, and other paradata to support survey collection management.

DCMs and RPMs within the regional offices prepare and review a number of system generated reports designed to monitor survey performance and interviewer productivity. These reports include outcome codes and daily progress reports, time per unit, and cost versus claims reports. The audit found inconsistencies in the use and frequency of the review of these reports. For example, some DCMs noted reviewing some or all of these reports and, in some cases, DCMs did not review any of these reports. The variation in survey management approaches within the regions increases the risk of discrepancies in the reports created by regions for CPMD and other partners, which may affect CPMD's ability for effective decision-making.

In addition, the reports used by CPMD for collection management purposes vary depending on the project officer. The audit found that some CPMD project officers rely solely on the daily response rate reports to monitor their surveys. Others indicated that they reviewed an array of reports, such as reports on results of conversion and tracing efforts. No procedural documentation exists outlining what reporting should be reviewed by CPMD project officers for effective collections management.

Interviews with subject-matter representatives noted that the differences in collection management approaches affect the usefulness of the information they are receiving regarding the status and quality of the survey's collection activities, resulting in quality issues that may go undetected.

Quality assurance tools are insufficiently leveraged in the management of collection activities

The Statistics Canada Quality Guidelines outline several reports that may be used as survey collection management tools by both CPMD and Regional Offices. These reports include processing error rates, follow-up rates, rates of non-response by reason, response rates, and capture/coding error rates.

Management noted that, for many surveys, response rates are declining. The audit found that, as a result of this decline, significant effort is placed on ensuring response rates for surveys, as these rates are indicators of both performance and data quality. At the same time, adequate response rates alone do not guarantee data quality. The Statistics Canada Quality Guidelines note that an effective collection management framework should also include tools specifically designed to monitor quality.

The audit team examined the reports prepared in CPMD and the regions. Within each of the three regions examined, the primary focus of the survey-reporting that is prepared and reviewed is to ensure that response rate targets are being met, and that interviewers are meeting productivity targets. Specific reports being used include: outcome codes, time per unit and cost versus claim reports. While these reports form part of an overall quality assurance suite of indicators, they are used as performance management tools and not sufficiently leveraged in the management of data quality.

Within CPMD, the audit found no indication that specific reporting on the quality of the collected data is being created or quality is being monitored. Reports designed to examine overall collections quality would help with the early identification and mitigation of issues affecting the quality of the data being collected.

The Quality Control Feedback System (QCFS) is used to monitor individual interviewers, but also has reporting capabilities that allow users to create reports, which enable analysis of the quality of data collections by creating and analysing reports by error type, errors by survey, trends in interviewer performance, etc. The audit noted that this type of analysis is not done within CPMD or in the regions, which would help interviewers understand survey requirements and be used as a tool to monitor interviewer collection performance.

In order to address survey management and quality reporting weakness, the audit noted that one subject matter division has been requesting additional reports from CPMD in order to conduct their own paradata analysis in an attempt to identify quality issues. Through this analysis, they have been able to provide feedback to CPMD on areas requiring investigation. This exercise has proven to be useful and, in one instance, led to the identification of serious interviewing gaps.

An effective process for feedback and the escalation of issues has been established

Communication of issues between Regional Office and CPMD takes place daily. Specifically, the audit found evidence that survey-specific issues with potential impacts on collection activities are reported and escalated within the regions and to CPMD for action. These issues include such things as, capacity and budgets, labour and environmental issues, technical issues, and survey-specific informational requests.

The audit found that CPMD and Regional offices have established an effective process for feedback and the escalation of issues. Communication occurs daily and issues escalation occurs at the appropriate level.

Recommendations:

The Assistant Chief Statistician, Census, Operations and Communications field should ensure that:

• A collection management approach, including the identification of reports to be produced, is established and communicated, to ensure consistency in the collection services provided to partners;
• Quality indicators, such as those outlined in the Statistics Canada Quality Guidelines, are included within the mandatory reporting requirements, and information from QCFS is regularly analyzed in order to identify training weaknesses and other survey issues.

Management Response:
Management agrees with the recommendations.

• The Director, Collection Planning and Research Division will develop a set of quality indicators and reports that can be used to identify issues during or after collection activities and to maintain or improve the quality of data collection. The development of these indicators will be done in collaboration with the Social Survey Collection Steering Committee and the Business and Agriculture Collection Steering Committee. Should the Branch be unable to absorb the costs associated with developing, implementing and maintaining the expanded quality indicators, a business proposal will be brought forward through the Department's long term planning process.
  
  Deliverables and Timeline: Development of the Collection Data Framework will be completed by December 2014.

Training and Monitoring in Support of Quality

Training and Tools

The Statistics Canada Quality Guidelines note that interviewer manuals and training must be carefully prepared and planned, since they provide the best way to guarantee data quality (e.g., high response rate and accurate responses), the comprehension of survey concepts and subject matter, and ensure proper answers to questions from respondents.

Effective basic and survey-specific training and tools for interviewers are in place, however, Business Register training should be enhanced to ensure interviewers understand requirements

The audit examined basic training given to all interviewers upon hire. This training effectively outlines Statistics Canada's mandate and objectives, the roles and responsibilities of interviewers and senior interviewers, and the importance of data collection. It also describes collection activities and quality control.

Survey-specific training is to be given to interviewers assigned to that specific survey. To ensure interviewers understand survey methods and concepts and the survey application, and have the opportunity to do mock interviews, training should take place prior to them beginning work on the survey. The audit examined two business surveys and three social surveys and found that training materials outline specific concepts, procedures and applications. In accordance with the Statistics Canada Guidelines, survey training also included several approaches to training, including home study, classroom training, and mock interviews. Audit testing confirmed that interviewers received basic and survey- specific training and that it took place prior to interviewers beginning any collection activities.

The Business Register is the survey frame for business and agricultural surveys. Prior to 2011, any changes to the frame were made by staff in the Business Register Division and were subject to validation with subject matter areas. Beginning in 2012, the responsibility for making live updates to the register was transferred to interviewers in the regions.  Interviewers in all regions visited noted that the Business Register training currently provided does not effectively prepare them to be able to make changes to the register with confidence. Interviews with subject-matter areas substantiated this by noting that there have been changes made to the frame by interviewers in the regions that were inaccurate, which have resulted in erroneous changes to the frame. The audit examined the Business Register training materials and noted that it assumes a level of knowledge of industrial organization and classification that staff in the regions may not have. Without effective training for interviewers, there is a risk that invalid changes made to the Business Register may go undetected and impact the quality of business and agricultural surveys.

Monitoring

The Statistics Canada Quality Guidelines note that the 'interviewing skills of interviewers should be monitored to ensure that they conform to a pre-established list of standards,' and that, '...monitoring should also be used to identify strengths and weaknesses in the interviewer's skill set, to provide feedback to the interviewers and to focus training on weaker areas.'

The QCFS is designed to capture the results of quality control monitoring operations for both Computer Assisted Telephone Interviewing (CATI) and Computer Assisted Personal Interviewing (CAPI), and to provide quality estimates and feedback reports to interviewers and managers. It also generates interviewer sampling plans required for the efficient administration of quality control procedures and monitoring. Each CATI and CAPI interviewer is assigned to a monitoring plan based on the interviewer's past performance. Interviewer plans and the required number of monthly monitoring sessions for CATI and CAPI interviewers in each plan are classified in QCFS as follows:

During a given monitoring session, the quality of the interviewer's work is evaluated against a pre-defined set of criteria. If an interviewer makes an error or shows poor interviewing practices for any question, the information is recorded in QCFS.

CATI interviewers work out of regional offices across the country, and using a computer survey application, telephone respondents to conduct the interview. Monitoring is carried out by senior interviewers who listen, observe, and assess the live interaction between the interviewer and the respondent, while using a telephone listening device and observe a simultaneous duplicate image of the interviewer's computer screen. Respondents are advised that the call may be listened to by a supervisor.

CAPI interviewers work in the field and go to respondent's residences to complete interviews. Survey responses are captured by the interviewer in a computer survey application. Monitoring is carried out by reviewing digitally recorded segments of interviews. Each recorded question is saved in a separate audio file and transmitted daily to the regional offices—where monitoring activities can take place. Respondents are asked by the interviewer if they consent to being recorded, allowing them the option to decline. If they do not to give consent, the recording is turned off.

The audit team observed eight monitoring sessions and confirmed that monitoring activities are performed according to the QCFS process. The audit noted that monitors actively listen to interviewers to ensure data are coded as required, interviewers ask questions as per the questionnaire and that the interviewers' conduct is professional. Errors and comments from each monitoring session were subsequently entered into the system, and for CATI monitoring sessions feedback was provided immediately after each monitoring session.

Monitoring of CATI interviewers should be enhanced to optimize quality

The audit assessed the monitoring activities within each of the sites visited to confirm whether monitoring occurred, if it was relevant and if feedback was given to interviewers. Of the 53 CATI interviewer files tested across the regions, only three interviewers were monitored as frequently as required by their monitoring plans. Interviews with regional management noted that senior interviewers have day-to-day responsibilities in addition to monitoring activities and that monitoring targets are still considered onerous, despite recent reductions in the required monitoring targets. Regional management also stated that interviewer plans for those requiring more monitoring due to identified quality issues, are not reviewed to ensure monitoring plans are implemented as required.

The audit noted that feedback from monitoring sessions is provided to CATI interviewers in a timely manner. However, regions do not use QCFS to identify performance trends for interviewers as QCFS is considered to be a coaching tool only and not a performance management tool. Management in the regions noted that the focus of QCFS monitoring is to meet monitoring targets set by head office and explained that, because feedback is given to interviewers after each monitoring session, quality is assured. Without analysis of monitoring results, the need for specific training or specific survey issues may not be identified or addressed.

CAPI monitoring is inconsistent across regions

There are several methods of monitoring CAPI interviews to ensure the quality of interviewer collections, including monitoring using the QCFS, validation monitoring and observation monitoring. Of the 15 CAPI surveys currently in the field, the QCFS is used to monitor CAPI interviewers for 2 CAPI surveys only—the Canadian Community Health Survey (CCHS) and the Survey of Household Spending (SHS). As a result, only CAPI interviewers working on these two surveys can be monitored using the QCFS.

The audit examined the QCFS monitoring consent-rate report for SHS produced in CPMD which detailed, for each CAPI interviewer, the rates of consent for recording of the interview. It showed that, on average, 29% of CAPI respondents refuse to give consent. However, the audit noted significant variation across interviewers. For example, examination of the consent report revealed that 24% of CAPI interviewers had refusal rates of more than 50%. As a result, there is often little sample available to monitor. The DCM responsible for CAPI interviewers in the Edmonton region noted that these reports were provided for informational purposes only and no action is taken to determine why consent rates varied. During interviews it was noted that there have been several instances of interviewers coaching respondents to decline having the interview recorded.

CAPI interviewers are not monitored as frequently as required by their plan. The reason given was because there is little sample available to monitor, and respondents have the option to decline being recorded. Additionally, CAPI monitors stated that providing timely feedback to CAPI interviewers is difficult. Because CAPI interviewers work in the field, the monitor must call and leave a message requesting the interviewer to call back. Many CAPI interviewers are reluctant to call back and avoid receiving feedback because it is perceived negatively. Without effective feedback mechanisms for CAPI interviewers, it is unlikely that risks to the quality of collections resulting from ineffective interviewing skills are adequately addressed.

Validation monitoring consists of telephoning respondents to verify that a Statistics Canada employee has been to their residence and has done an interview. Key data are confirmed and the respondent is questioned about the overall professional impression of the interviewer. The audit tested and confirmed that validation monitoring takes place at the Edmonton regional office, as it was the only one of the three sites visited responsible for CAPI interviewers.

Observation monitoring occurs when a senior interviewer accompanies a CAPI interviewer in the field and observes collection activities. The CAPI Statistical Survey Operations (SSO) employee handbook noted that observational monitoring must be conducted at a minimum of once every two years. Within the Eastern and Western regions, the audit confirmed that observational monitoring is taking place. In the Central region, staff noted that observational monitoring had been discontinued and has not taken place since 2010.

Recommendations:

The Assistant Chief Statistician, Census, Operations and Communications and the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure should ensure that:

• Business Register Training is adapted for SSO staff to ensure quality frame maintenance.

Management Response:

Management agrees with this recommendation.

• The Director of the Collection Planning and Research Division, in collaboration with the Director of the Business Register Division, will ensure the Business Register training for interviewers is adjusted to meet the users' needs.
  
  Deliverables and Timeline:
  • New streamlined course based on needs identified through consultation with CRSB. This was completed in December 2013.
  • First iteration of new course delivered nationally. This was completed in March 2014.
  • New training materials made available to SSO/CRSB. This was completed in May 2014.
  • To ensure all interviewers and Senior Interviewers are properly trained, CRSB and the Regions use an Excel listings of trained or to be trained staff on the BR. This will be an ongoing activity.

Recommendations:

The Assistant Chief Statistician, Census, Operations and Communications field should ensure that:

• The CATI and CAPI interviewer monitoring plan targets are met and that the results of monitoring are examined at the interviewer and survey level to ensure the quality of operations.
• For CAPI operations, the feedback from QCFS monitoring be provided to interviewers in a timely manner, interviewers and respondents are made aware of the importance of monitoring as a quality assurance tool to minimize refusals to monitoring, and there is clarification of the observation monitoring requirements.

Management Response:

Management agrees with this recommendation.

• For CATI, the Assistant Directors/Operations will work with Collection Planning and Research Division to update the monitoring targets, and develop a plan to ensure those targets are met and tracked. Feedback mechanisms to interviewers will be examined as a part of the plan.
  
  Deliverables and Timeline: Updated CATI monitoring targets and plan for assessing the program by March 2015
• For CAPI, Regional Directors will work with Collection Planning and Research Division to develop and cost out a revised monitoring program, similar to the CATI program, and develop an implementation plan. If necessary, an LTP proposal may be developed to fund CAPI monitoring. This plan will set time requirements for feedback and clarify monitoring requirements for CAPI interviewers.
  
  Deliverables and Timeline: CAPI monitoring program proposal and implementation plan will be developed by March 2015.