Evaluation of the Census of Population Program, Post-Field Activities

The report in short

The statistical information produced by the Census of Population Program provides a statistical portrait of Canada and its people. It measures the changes occurring in the demographic, social and economic characteristics of Canadians, and supports a variety of statutory, legislative and regulatory requirements. The information is used by a wide range of organizations, including various levels of government, not-for-profit and private organizations, and academic institutions, as well as by individual Canadians.

The evaluation of the program was conducted by Statistics Canada in accordance with the Treasury Board Secretariat's Policy on Results (2016) and Statistics Canada's Risk-Based Audit and Evaluation Plan (2018/2019 to 2022/2023). It covered 2016 Census post-field activities and related planning activities for the 2021 Census. The main objective of the evaluation was to provide a neutral, evidence-based assessment of the governance, design and delivery, and user-centric approach of census post-field activities in support of decision making, accountability and improvement. Because extensive consultations on census outputs were being planned and undertaken by the program during the evaluation, the relevance of the 2016 Census outputs was excluded from the evaluation. It is expected that the information gathered will help inform the development of the 2021 Census release strategy.

The evaluation methodology consisted of a document review, administrative reviews and key informant interviews with a wide range of Statistics Canada professionals working in census subject-matter areas (SMAs), the Census Subject Matter Secretariat (CSMS), the Census Management Office, the Census Operations Division (COD) and other relevant divisions. The triangulation of these data collection methods was used to arrive at the overall evaluation findings.

Key findings and recommendations

Overall, 2016 Census post-field activities were successfully completed, and multiple teams worked collaboratively to deliver high-quality outputs to Canadians in a timelier manner than in the past. While the vast majority of activities were performed effectively and efficiently, the evaluation found opportunities for improvement.

Governance

For the 2016 Census, an overarching governance structure existed that generally worked effectively and efficiently. Roles and responsibilities were articulated, and post-field activities mostly took place as planned. However, the evaluation did find three areas that could be improved: the institutional review process for the 2016 Census created unexpected burden on staff, pressured timelines and increased risk; there was a lack of clarity around roles and responsibilities for the Dissemination Sub-Project; and the CSMS could have more effectively represented and coordinated SMAs and more effectively managed operational tasks such as scheduling and communications.

The Assistant Chief Statistician (ACS), Census, Regional Services and Operations (Field 7), in collaboration with the ACS, Social, Health and Labour Statistics (Field 8), should ensure that:

Recommendation 1:

With respect to institutional reviews, governance, roles and responsibilities, and processes are reviewed and adapted to ensure that clarity exists and that reviews adhere to policy. This should also include a more effective approval model. In addition, an approach should be developed for earlier approval from senior management on the dissemination strategy.

Recommendation 2:

The roles and responsibilities for the areas (including the Communications and Dissemination Branch, CSMS, COD, and SMAs) involved in the Dissemination Sub-Project are clarified and updated to ensure effective and efficient delivery of the sub-project. Governance should ensure that roles and responsibilities remain clear and are adhered to over time.

Recommendation 3:

The roles and responsibilities of CSMS with respect to SMA representation and coordination are updated and strengthened.

In addition, CSMS processes, tools and strategies used for managing schedules, access privileges and communication flows, and for sharing and adopting best practices, should be reviewed and updated.

Design and delivery

Compared with 2011, the last release from the 2016 Census was approximately 10 months earlier (i.e., all releases completed 18 months after Census Day, versus 28 months). For the 2021 Census, the program is in the process of identifying additional efficiencies within post-field activities through numerous working groups and consultations. For an accelerated release schedule (i.e., all major releases completed in less than 18 months from Census Day), the dissemination strategy and coding activities appear to have the greatest potential for gains.

There were a number of risks identified that could impact the schedule: revised certification structures and processes, potential additional content for the 2021 Census, and capacity in terms of systems and human resources.

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that:

Recommendation 4:

In alignment with Statistics Canada's goal of continually striving to release data in a shorter timeframe, all post-field activities are reviewed to identify potential time-saving measures. In particular, the dissemination strategy and coding activities appear to have the greatest potential. The risks identified in the evaluation should also be taken into account when planning such activities.

User-centric approach

Various mechanisms are used to capture information on census data user needs, such as formal census consultations, ongoing consultations and interactions with regular users carried out by SMAs, and web metrics. Given the breadth and amount of information available, it is a significant challenge to analyze all data at once to gain a holistic view of users and their needs (content, types and formats of products, uses, and timing of releases). The evaluation found no evidence that integrated information is generated to support the development and prioritization of the release strategy.

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that:

Recommendation 5:

A detailed and integrated analysis of user needs takes place. For example, user profiles based on the multiple sources of information already available could be created. The 2021 release strategy should be driven by the findings and conclusions of such an analysis and should also take into consideration the risks identified in the evaluation.

What is covered

The evaluation was conducted in accordance with the Treasury Board Secretariat's Policy on Results (2016) and Statistics Canada's Risk-Based Audit and Evaluation Plan (2018/2019 to 2022/2023). In support of decision making, accountability and improvement, the main objective of the evaluation was to provide a neutral, evidence-based assessment of the governance, design and delivery, and user-centric approach of post-field activities. The evaluation covered the 2016 Census and relevant planning activities for the 2021 Census.

The census

Under the Statistics Act (R.S.C., 1985, c. S-19), Statistics Canada has the legal obligation to conduct a Census of Population every five years. The census provides a statistical portrait of Canada and its people, covering various demographic and social topics. The statistical information supports the measurement of the changes occurring in the demographic, social and economic characteristics of Canadians. It also supports a variety of statutory, legislative and regulatory requirements. The information is used by a wide range of organizations, such as various levels of government, not-for-profit and private organizations, and academic institutions, as well as by individual Canadians. The 2016 Census covered a range of topics: population (including age and sex) and dwelling counts (including type of dwelling); families, households and marital status; language (including language of work); income; immigration and ethnocultural diversity; housing; Aboriginal peoples; education; labour; journey to work; and mobility and migration.

Information is collected through mandatory short-form and long-form questionnaires. The short-form questionnaire is sent to the entire population. The long-form questionnaire, which includes the same questions as the short form plus additional questions on topics such as education and labour, is sent to a sample of the population. Some information is obtained directly from administrative records, such as income from the Canada Revenue Agency and category of immigrants from Immigration, Refugees and Citizenship Canada.

During the census, Statistics Canada conducts a number of activities, such as content determination, field operations (data collection) and post-field operations.^{Footnote 1} Once census data are collected, a number of post-field activities are conducted:

• coding: all written responses on the questionnaires are converted to numerical codes
• edit and imputation (E & I): omissions and inconsistencies in the collected data are detected and corrected
• sampling and weighting: final responses are weighted to represent the Canadian population
• data quality assessment: an evaluation of the data is conducted to ensure the overall quality (assessments take place during different phases from content determination to dissemination)
• dissemination: data products in various formats are disseminated to address user needs.

The Census of Population Program is managed by the Census Management Office (CMO) and involves a large number of functional divisions within Statistics Canada that operate at different levels throughout the cycle. According to the Census of Population Performance Information Profile, the following are the main organizations involved in post-field activities:

• CMO: The CMO plays a key role in the planning and coordination of the program. The main function of the CMO is to lead and support the overall census project. This includes the coordination and integration of plans and budgets, census research, testing, and quality management.
• Subject-matter areas (SMAs): These provide input and expert advice on content, coding of written responses, E & I rules and dissemination. They also perform specific key activities in support of field operations, processing, certification and dissemination. They evaluate and analyze data and provide professional advice and assistance to data users. SMAs are located in the Census Subject Matter, Social and Demographic Statistics Branch (CSMSDSB) and the Education, Labour and Income Statistics Branch (ELISB).
• Census Subject Matter Secretariat (CSMS): CSMS manages the Census Program Subject Matter Sub-Project and coordinates the provision of subject-matter input, identifies and capitalizes on opportunities for collaboration and cooperation within SMAs and with other census teams, and facilitates the development and maintenance of census subject-matter expertise.
• Census Operations Division (COD): COD provides the infrastructure and personnel for field operations, mail-out mail-back (including questionnaire printing), processing, dissemination and other support services.
• Communications and Dissemination Branch (CDB): CDB develops and implements communications strategies that support the program.
• Social Statistics Methods Division (SSMD): SSMD provides methodological services and quality evaluation advice on sampling, data collection methodology, weighting and other technical aspects of the program.

The Strategic Management Committee (SMC),^{Footnote 2} composed of the Chief Statistician (CS) and assistant chief statisticians (ACSs), acts as the senior governance body for the program. The Census Steering Committee (CSC), which reports directly to the SMC, is composed of senior officials participating in the program. It is co-chaired by the ACS, Census, Regional Services and Operations Field (Field 7), and the ACS, Social, Health and Labour Statistics Field (Field 8). The CSC performs a guidance and information-sharing function. The Census Project Team (CPT), which reports directly to the CSC, is composed of sub-project managers and chaired by the Director General (DG) of CMO. It serves as a decision-making and managing body. A number of working groups (WGs) and committees report to the CPT.

Over the 2016 Census cycle, expenditures represented approximately $554.4 million, of which approximately $62.1 million was dedicated to post-field activities.

The evaluation

The scope of the evaluation was defined based on interviews with the main divisions involved in the program's post-field activities. Since extensive consultations on census outputs were planned, developed and implemented by the program during the evaluation, the relevance of the 2016 Census outputs was excluded from the evaluation. Instead, the evaluation focused on the internal governance mechanisms, design and delivery structures, and user-centric approaches to deliver census outputs. The following areas were identified for review in the evaluation:

What we learned

1.1. Governance of census post-field activities

The evaluation assessed the roles and responsibilities of the different groups involved in census post-field activities with respect to horizontal coordination and delivery of the 2016 Census outputs. The evaluation found areas for improvement within the Dissemination Sub-Project, institutional reviews, and CSMS's role as an SMA representative and coordinator.

Matrix management

The 2016 Census Project Charter established the foundation for the project structure, described expectations for project management, and identified the ACS, Field 7, as the executive sponsor for the program and the DG, CMO, as the Census Project Manager. It also identified roles and responsibilities for the SMC, the CSC and the CPT.

The census is operated using a matrix management approach. A number of sub-projects, overseen by the CPT, are planned and implemented to support the overall achievement of the program. Individual charters are developed, and each sub-project is broken down into a number of tasks and sub-tasks. Depending on the nature of the sub-project, managers can be from different functional areas of Statistics Canada. Figure 1 illustrates the approach taken to deliver the census using a matrix management approach.

Description - Figure 1 Census matrix management approach

The figure depicts the theoretical approach taken to deliver the census using a matrix management approach. The figure shows that under the Census Project Team there are a number of distinct sub-projects. Under each of these sub-projects is then a series of tasks which in turn have a number of sub-tasks.

Specifically in the figure, under the Census Project Team box there are two boxes (Sub-project 1.0 and Sub-project 2.0) along with a ‘…’ indicating there are other similar boxes (Sub-project 3.0, Sub-project 4.0 and so on). Under the Sub-project 1.0 box are two task boxes (Task 1.1 and Task 1.2). Under each of these task boxes are Sub-task boxes (Sub-task 1.1.1 and Sub-task 1.1.2). The same structure is found under the Sub-project 2.0 box.

Roles and responsibilities were not always clear or understood with the Dissemination Sub-Project

The evaluation found that roles and responsibilities were established and communicated for the 2016 Census through the Census Project Charter and associated sub-project charters. Within post-field activities, there were four main sub-projects:

• the Processing Sub-Project
• the Dissemination Sub-Project
• the Census Communications Sub-Project
• the Census Program Subject Matter Sub-Project

For the Processing Sub-Project, which included tasks such as coding, sampling and weighting, and E & I, the roles and responsibilities were well understood. Processing was led by COD, with the participation of SMAs and other groups such as CSMS and SSMD. Clear roles and responsibilities were also found for the Census Communications Sub-Project (led by CDB), which provided intelligence communication in support of the dissemination activities. Similarly, roles and responsibilities for the Census Subject Matter Sub-Project (led by CSMS), which included tasks such as content determination and certification, were also clear.

While official release dates were met, uncertainty around roles and responsibilities for the Dissemination Sub-Project had a negative impact on effectiveness and efficiency. According to the sub-project charter, COD was the lead on the Dissemination Sub-Project and was responsible for the overall management of dissemination, including the development and review of some products and decision making. In practice, however, divisions tended to take ownership of (i.e., felt accountable for) their outputs, believing that since these were end products, they belonged to them. This lack of clarity, consistency and application with respect to roles and responsibilities created bottlenecks throughout the process (e.g., what changes can be requested, who can request such changes, who approves the outputs and who negotiates timelines). This was further compounded as there were different leads for the development of each output (e.g., the development of reference material, analytical documents, data tables, videos and infographics was led by different groups or different parts of Statistics Canada).

Institutional reviews created unexpected burden, pressured timelines and increased risk

Overall, while all 2016 official release timelines were met, the execution of institutional reviews impacted the efficiency of the process.

Similar to other products at Statistics Canada, the release of census outputs was to follow an established approval process that included peer reviews followed by institutional reviews. According to Statistics Canada's Policy on Peer and Institutional Review (2016), a peer review is a "technical assessment by recognized professionals, either internally or externally, in the relevant discipline(s) that ensures that analyses and interpretations are methodologically sound, well-supported, consistent with the definitions and other characteristics of data, cognizant of the current research literature as appropriate, respect terminological conventions, and articulate pertinent limitations of the data or the analysis." The policy also defines an institutional review as "an assessment by Statistics Canada senior managers to ensure that interpretive information products, analytical products and methodological products disseminated to the public are free of material which would compromise the Agency's reputation for quality, professionalism, non-partisanship, objectivity and neutrality."

For the 2016 Census, the coordination of peer and institutional reviews was managed by CSMS. SMA director approval was required after each type of review (Figure 2) and SMA DG and ACS reviews were required prior to the output being submitted for CS approval.

In alignment with the policy, there was a common understanding within divisions that technical elements would be addressed during peer reviews and that corporate elements (i.e., Statistics Canada's reputation for quality, professionalism, non-partisanship, objectivity and neutrality) would be addressed during the institutional reviews. Schedules were developed taking into consideration that peer reviews typically take longer than institutional reviews.

The evaluation found that CSMS was effective in reconciling feedback and managing the overall review process, and that the peer reviews worked as planned. The institutional reviews, however, did not function as planned. The nature of the comments provided during institutional reviews was more typical of those received during peer reviews, which had already been completed. In addition, reviewers not initially included in the institutional review board were involved in the process, which increased the amount of feedback to be reviewed and addressed. This created additional burden on staff (e.g., rewriting products), increased risk and pressured timelines. The impact was further compounded by the approval processes as bottlenecks formed. With the exception of a number of products, everything else needed director approval.^{Footnote 3} The evaluation found that the level of expertise of analysts within SMAs could allow for a more decentralized approval structure for some products.

Description - Figure 2 Review and approval process of census outputs

The figure depicts the approval process for products. It begins with a draft product; once approved by the subject matter director, the draft product then goes for Peer review. Once the Peer review is complete, the subject matter director provides approval and the draft product then goes for Institutional review. Once the Institutional review is complete, the subject matter director provides approval and the draft product then goes for final approval by the Chief Statistician. Once approved by the Chief Statistician, the product is considered final.

Director general and assistant chief statistician review take place throughout the review process.

Guidance was sufficient; however, late changes impacted effectiveness

The evaluation found that enough guidance was provided to direct the initial development of census outputs; however, subsequent late changes had a negative impact on schedules and workload (e.g., requests for new analytical perspectives and new dissemination products such as interactive maps). CSMS's role in providing guidance to SMAs on changes was found to be effective and helped improve clarity regarding expectations. Earlier approval of the dissemination strategy would allow for more effective planning and delivery of the Dissemination Sub-Project.

Horizontal coordination of SMAs could be improved

Different horizontal coordination models have been used over time. Prior to the 2006 Census, the function was embedded within COD, where staff coordinated the work of SMAs related to dissemination. For the 2006 Census, in an effort to improve coordination, the function was centralized in Field 8 by creating the Census Subject Matter Program (CSMP), which included subject-matter experts and coordination staff within the same group. For the 2011 Census, the model was restructured and experts returned to their respective divisions while the coordination staff remained within CSMP (renamed CSMS). This same model was used for the 2016 Census.

For the 2016 Census, CSMS's mandate was twofold. First, it acted as a sub-project manager for the Census Program Subject Matter Sub-Project, which included content determination, certification, formal reviews of population and dwelling counts, and product line determination and analysis. Second, it provided SMA representation and coordination services for all the other sub-projects. CSMS was the central body ensuring horizontal coordination for all subject-matter groups involved in the census.

The evaluation found that CSMS effectively fulfilled the first part of its mandate, managing the Census Program Subject Matter Sub-Project.

With respect to the second part of its mandate, while some tasks were performed effectively, the evaluation found opportunities to increase both effectiveness and efficiency. CSMS had a broad range of tasks and responsibilities: representing SMAs on various WGs and committees; negotiating, establishing, finalizing and monitoring schedules for post-field sub-projects when SMAs were involved; serving as a communication conduit between COD and SMAs for some sub-projects (e.g., processing and dissemination); fostering information sharing between SMAs on best practices and lessons learned; and, finally, managing access to certain key files.

CSMS represented SMAs on various WGs and committees, and included them directly as required when more complex issues needed to be addressed. For example, subject-matter experts were directly involved in their respective variables during E & I. The evaluation found that the perspective of SMAs was not always adequately advocated for and that the views of the SMAs were at times not integrated (i.e., SMAs "speaking as one voice") by CSMS. This led to some situations where either decisions were delayed or conflicting information was provided.

With respect to negotiating, establishing, finalizing and monitoring schedules for post-field sub-projects when SMAs were involved, the evaluation found that while CSMS was effective and meticulous in the development of all schedules, the management of the schedules was less than optimal. CSMS resources were divided into teams, matching the different post-field activities of the program (collection and processing, quality assurance, and dissemination). Often, post-field activities ran in parallel with processing, certification and dissemination activities carried out simultaneously by SMAs. At times, SMAs received conflicting or overlapping deadlines from CSMS (e.g., a deadline for certification and another for dissemination within a close timeframe). In addition, not all schedules were updated throughout the cycle because of resource pressures and limitations of the project management tool. The late delivery of schedules and updates to CSMS by other production partners added to the challenges of managing schedules. CSMS could strengthen the integration and coordination of schedules within its teams prior to reaching out to SMAs. Other production partners should also be made aware of the impact of the late delivery of schedules and updates on horizontal coordination.

The communication flow between COD and SMAs through CSMS was found to be effective for some sub-projects (e.g., processing, Census Program subject matter). JIRA, Confluence, shared directories and regular meetings were identified as efficient and effective tools for communication. The communication route through CSMS during the Dissemination Sub-Project was not found to always be effective, timely or value-adding, particularly during high-volume busy periods. As a result, communication sometimes took place directly between groups (rather than through CSMS) to improve efficiency.

In terms of information sharing between SMAs on best practices and lessons learned, although there was evidence that such activities took place (e.g., verification tools), it appeared they were typically driven by SMAs rather than by CSMS. CSMS could take a greater role in fostering discussions and sharing practices to improve effectiveness and efficiency.

Sharing between SMAs during the development of the outputs is important to ensure consistency across the releases. Access permissions for products were managed by CSMS and were found to be too restrictive (i.e., need-to-know approach) by the SMAs. The restrictions limited the capacity of SMAs to access products between them, which posed challenges in adopting consistent approaches for storylines—particularly when releases included cross-topic analyses. For the last releases, the situation was improved as access permissions were widened. CSMS is aware of this issue and stated that it will develop a strategy for the 2021 Census to increase information sharing between SMAs.

Interviewees provided a number of opinions as to why challenges were encountered by CSMS. These included a lack of decision-making power for CSMS, a lack of expertise, and the challenging mandate of having to reconcile priorities, sometimes conflicting, between Field 7 and Field 8. Although areas for improvement were identified for CSMS, it was clear that CSMS played a critical role in the success of the census.

1.2. Design and delivery of census post-field activities

Given that Statistics Canada is continually striving to improve timeliness, the evaluation looked into identifying areas where measures could be implemented to shorten the timeframe for census dissemination. To address this question, the evaluation looked into the initiatives implemented to support the revised dissemination schedule for the 2016 Census, including lessons learned and additional potential areas.

The dissemination schedule for the 2016 Census was shorter compared with the 2011 Census (Table Timelines of the 2011 and 2016 Census releases). Although the first release (i.e., population and dwelling counts) was completed within the same timeframe for both cycles, all the other releases were completed earlier. The last release for the 2016 Census (i.e., education, labour, journey to work, language of work, and mobility and migration) took place 287^{Footnote 4} calendar days earlier than the last release for the 2011 Census (i.e., income and housing).

Changes made to support the 2016 dissemination schedule

The evaluation found that all post-field activities were subject to assessments at the project, sub-project or task levels. These assessments were led by a specific group with the assistance of other groups—for example,

• 2016 Census Coding Evaluation (CSMS)
• 2016 E & I Task Evaluation Report (COD)
• 2016 Census E & I Process (SSMD)
• 2016 Census Overview of Data Releases (CDB)
• 2016 Census Subject Matter Dissemination Evaluation (CSMS)
• 2016 Census Project Close-Out Report (CMO).

A number of initiatives were implemented for most post-field activities to support the shortened dissemination schedule (all releases were completed approximately 18 months from Census Day). Given the complexity and numerous interdependencies that existed among the various activities and across divisions, the specific number of days saved by each initiative could not be determined; however, evidence suggested that the new approach for E & I had the largest impact.

Edit and imputation

During the 2011 Census, E & I modules were processed individually and sequentially—once the processing of a module was deemed successful, the following module was processed, and so on. For the 2016 Census, the entire E & I process was run at once (i.e., straight-through run), including increased parallel processing and overlapping processes to reduce production time.

To be ready for production in 2016, development and testing activities started two years in advance and included the identification of dependencies between all modules. The testing used data from the previous cycle to ensure the robustness of the new approach (i.e., results run through the new E & I process were the same as results from the previous sequential process). To ensure a stable environment for the revised E & I process, the system used for E & I was frozen more than a year in advance to avoid any issues.

The use of JIRA to manage changes and issues was identified as a significant contributor for tracking, actioning and monitoring changes to modules and processes. JIRA simplified work, particularly when switching from one task to another was required (i.e., often processing, certification and dissemination tasks overlap).

Coding

A number of initiatives were implemented that impacted the dissemination schedule. For example, similar records were coded simultaneously to save time. Another example was the reduction of the size of the batches for coding, which was done at the record level instead of at the questionnaire level for faster processing. Although coding initiatives supported the revised dissemination schedule, their impact was limited for a number of reasons. For example, because of differences between variables in terms of complexity, the initiatives were not implemented universally across all SMAs. As a result, while some time saving was observed in some areas, it was not the case in other areas.

Sampling and weighting

With all the changes, methodological support served to ensure quality. There was no specific methodological initiative, however, that had an impact on the revised dissemination schedule.

Data quality assessment

Certification strategies developed by SMAs included a number of tools and approaches to shorten the time required to certify the data. For example, automated tools (e.g., tables), lower levels of geography and secondary data linkages were used. The evaluation found that no common standard tools were used by SMAs for certification, a similar finding as the Audit of Quality Assurance – 2016 Census.^{Footnote 5} Although tools were developed to support the faster delivery of certification in some SMAs, no substantial time savings were gained from this activity.

Dissemination

Measures to develop products (including data tables, the Census Dictionary and other reference material such as reference guides and storylines) earlier were used for the 2016 Census. Automated tables for verification were also developed. A number of factors, however, limited the benefits of early development. First, the reintroduction of the mandatory long-form questionnaire changed the scope available for analysis. Second, policy reorientations changed priorities. For example, early specifications for tables and storylines were developed to provide data on income splitting, but this was subsequently removed. Third, new requirements from senior management were implemented between Census Day and the first release, which had an impact on the analysis and the dissemination tools. For example, a "Canada 150" component was included for all releases to provide Canadians with a historical perspective of the data. Also, new interactive tools not initially planned for were introduced to facilitate access for Canadians. These changes had a positive impact; however, they reduced the benefits from the early development of products.

The evaluation found that while the early development of products did not contribute to large gains, it nonetheless provided extra time during peak periods, particularly when additional perspectives and products for dissemination were requested. Early development had a positive impact in helping to manage releases in parallel because of tighter timelines towards the last releases (i.e., releases 4, 5 and 6).

Opportunities for an accelerated dissemination schedule

The evaluation found that the 2016 Census dissemination schedule could be replicated in 2021 while maintaining the same approach for the release strategy (i.e., number of products and frequency of releases). A number of areas, however—particularly those related to governance (noted previously)—should be addressed to ensure a more effective and efficient cycle.

In terms of an accelerated dissemination schedule (less than 18 months), in addition to addressing governance issues, the release strategy should be revisited. The most significant gains in time could be found by adopting a different model (e.g., fewer products, staggered schedule). To a lesser extent, changes in the coding approach could also result in time savings. Changes to E & I, sampling and weighting, and data quality assessment would have a limited impact.

Revised release strategy

For an accelerated dissemination schedule, the release strategy, particularly the number and type of products, and the frequency of the releases should be revisited.

The approach taken for the last cycle where a variety of products were released on a given date could be rethought. The evaluation noted that users, particularly government-level partners, may have had challenges digesting all the information. Interviewees suggested that a more staggered or phased approach could be adopted: for example, a scenario where a Daily article, combined with a limited number of tables and interactive products, could be released on a given date along with a communiqué about additional products to come in the near future.

The evaluation found that over time, the media appeared to lose some interest in the media lockups (i.e., less media represented, fewer interview requests and less coverage). Internal data showed that the first release had the highest number of journalists in attendance (n=40) and the second release had the lowest (n=23). The number of journalists for the remaining lockups fluctuated between 25 and 30. While the number of interviews given during the lockups remained relatively stable, the number of interviews given on the days after release day decreased over time (33 for the first release and 6 for the last release). Media coverage^{Footnote 6} also declined from the first release (n=525) to the last release (n=238). Although interest seemed to decrease over time, the required amount of preparation remained the same—a significant level of effort for a number of divisions.

Clearer objectives for each type of product (e.g., analytical products versus general public products versus branding products) could also help tailor the release strategy toward multiple objectives. Alternative and flexible products, such as data cubes where users could build their own tables based on their needs, were identified as an approach to consider. The cubes would require less verification since only the master tables require verification.

A number of reviews are currently underway, such as analyses by the Core Tables WG, Process Flow WG, Accelerated Release WG and the Census Dissemination Consultation (2019), which will provide direct input on potential opportunities.

More automated coding activities

The evaluation found that efficiencies could potentially be gained in coding through three main avenues: increased use of machine learning (i.e., artificial intelligence) to supplement the auto-coding approach; use of the corporate tool for coding (i.e., Coding and Correction Environment); and a timelier operationalization of the Codefix system used to code records, fix incorrect codes and verify the coding (certification). The Coding Process WG, co-chaired by COD and SSMD, will evaluate and provide a proof of concept for the introduction of machine learning technology for the 2021 auto-coding processes. There is an expectation that the use of machine learning would reduce the time needed for coding since it would increase the auto-coding rate of different variables. It would also increase overall quality since manual intervention would be reduced.

Risks for the dissemination schedule

Although not initially in scope for the evaluation, four key risks for the 2021 Census were identified that could impact the dissemination schedule: potential new content, new systems for dissemination, revised certification processes and human resource capacity.

The 2021 Census test questionnaires, which will be tested in the coming period, include potential new and revised content compared with the 2016 Census. Although at the time of the evaluation it was too soon to determine what content could be included after testing, new and revised content would have an impact on all post-field activities (i.e., processing, certification and dissemination).

New tools and systems are being examined for use in the 2021 Census. A new set of tabulation tools centred on G-Tab as the tabulation engine was being developed at the time of the evaluation. The program is looking to move to the corporate dissemination platform, the New Dissemination Model. Capacity and functionality are being tested for both systems. Other systems such as SharePoint and Codefix, as well as the SAS grid, were also identified as risks.

The revised certification strategy (still in development at the time of the evaluation) that will be implemented for 2021 could increase the level of complexity required for this task.

Regarding resources, because of the cyclical nature of the census, the level of resources required is variable from year to year. As a result, there are challenges in retaining and renewing expertise from cycle to cycle. Although evidence showed that divisions are implementing measures such as rotating resources and job-shadowing, more discussions could take place within and between divisions to develop human resource strategies and document processes to support future resources (i.e., securing corporate knowledge). In addition, lower budgets toward the end of the cycle limited the capacity for SMAs to invest toward research and development for future cycles to foster innovation.

1.3. Mechanisms to foster a user-centric approach

Since one of the priorities of Statistics Canada's modernization initiative is to strengthen its user-centric approach, the evaluation examined the mechanisms in place to capture information on user needs, as well as the use of such information in the development of the release strategy.

An integrated view of user needs is a gap

The evaluation found that the program has multiple mechanisms in place to capture information on user needs. Consultations take place every cycle to gather information, and a previous evaluation conducted on the census noted that for the 2011 and 2016 cycles, consultations were adequate.

The consultation for the 2021 Census release strategy started January 15, 2019, with an electronic questionnaire being sent to known users. The questionnaire was available until the end of March 2019. In an effort to reach additional users and to build on initiatives implemented to address the 2014 Spring Report of the Auditor General of Canada,^{Footnote 7} banners (i.e., on Statistics Canada's website and on the census website) were developed and users were invited to respond to five short questions, and then to respond to a longer electronic questionnaire. Pamphlets were also distributed at different events to invite census data users to respond to the questionnaire. It is expected that these efforts will increase the representativeness of the users consulted. There were also new topics covered in the consultation since questions were asked about user satisfaction with timelines (e.g., delay between Census Day and all releases, delay between major releases), and preferences for release dates and topics (e.g., number of release dates compared with topics covered in each release) were gathered. It is expected that these perspectives will inform the more strategic development of the 2021 Census release strategy. Following the analysis of consultation data, a series of in-depth interviews will be conducted to refine the findings.

Other mechanisms were also used to capture information on user needs. Besides formal census consultations, SMAs also conducted consultation activities within their own areas of expertise. These consultations did not necessarily focus on the census, but they nonetheless captured relevant information. The regular relationship maintained by SMAs with users, particularly regular users, enabled the SMAs to understand user needs and their evolution through time. In addition, web metrics were used to gauge the extent to which census products were accessed (i.e., number of downloads and number of hits).

Little evidence was found of information being assembled about users in the days following the official releases. Knowledge on the types of users, the products used and the purpose of their use on the day of release or the days following would provide strategic input for the development of the release strategy. COD is currently exploring ways to capture this type of information for the 2021 Census by, for example, including a pop-up window on the website that would ask short questions to users. It is expected that this new information will be used for the 2026 Census. Another potential area being explored is the development of a repository of the most popular requests so that the needs of users throughout the country can be better understood.

Although multiple sources exist to capture information on user needs, the generation of an integrated view remains a gap. It would provide a holistic view of census data users and their needs in terms of content, types and formats of products, and use and timing of releases. It would allow for better prioritization when developing the release strategy. There was no evidence that such integrated information was generated to support the development of the release strategy.

External and internal factors are considered when developing the release strategy

For the 2016 Census, the development of the release strategy (i.e., products and schedule) was initiated by CSMS, in consultation with SMAs, through the Census Release WG. A set of recommendations was developed and presented to COD (which leads the Dissemination Sub-Project) for consideration. The final recommendations were then presented to the CPT. The proposed release strategy then followed Statistics Canada governance structures and received final approval.

The evaluation found that the program took user needs and internal needs into consideration for the release strategy. While user needs were placed at the forefront of the development of the strategy, other factors significantly influenced the development of the release strategy. One of the biggest factors was tradition and history. The number and type of products that were released during the previous cycle served as the starting point. Overall, the number of products has varied over time; however, the complexity of products has increased. For example, the number of data tables has decreased since 2001, but the number of cells in the tables has increased, since a number of them have been merged together. This is due in part to the requirement to develop tables for each data point included in analytical products. The result was greater effort to create and verify the tables.

For the 2021 Census, work was initiated through the 2021 Census Core Tables WG to define a list of tables that must be produced (i.e., tables addressing a large number of user needs, produced across cycles or needed to fulfill legislative requirements). One of the objectives is to allow an early start on table designs. This WG, led by CSMS, includes COD, client services, regional offices and SMAs.

Internal constraints, such as budgets and resources available, as well as timelines and the IT systems available, were taken into consideration when developing the release strategy. Other factors were also considered, including continuity (i.e., stability of statistical information and legislative or policy requirements), accessibility requirements (e.g., the Treasury Board Secretariat's Standard on Web Accessibility), sequence of processing (e.g., education data cannot be released before population or labour data) and approaches taken in other countries.

In determining future directions for census dissemination and releases, there may be something to learn from the experiences and viewpoints of other statistical agencies.

The Chief Executive and National Statistician of the United Kingdom Statistics Authority offered the following assessment: "I fully expect that, in five years' time, what we will be doing will be radically different. More data will be real-time; our services will be digital by default; the quality of our advice and insight will have earned us a seat at the table where the most important decisions are made."^{Footnote 8} This being the case, the Office for National Statistics (ONS) will feature a flexible dissemination system allowing users to create their own datasets by selecting the geography, population base and variables they require, rather than having to wait for ONS to develop tables. The census will also feature the greater use of administrative data to improve supplemental outputs, more easily accessible metadata, and more detailed analysis and insight.^{Footnote 9}

For the 2021 Census, the Australian Bureau of Statistics (ABS) is designing its census for an effective user experience. Release schedules and products are being reviewed to identify unmet user needs, whether products are fit for purpose, ease of use and accessibility, and opportunities to harness emerging dissemination techniques. Users of 2016 Census data can manage and build custom tabulations online by subscribing to "TableBuilder" on the ABS website.

How to improve the program

Governance

For the 2016 Census, teams worked collaboratively and delivered high-quality outputs for Canadians. However, the evaluation found areas for improvement in the governance of post-field activities. In particular, these included expectations with respect to institutional reviews, roles and responsibilities within the Dissemination Sub-Project, and CSMS as it relates to SMA representation and coordination.

The evaluation found that the institutional review process for the 2016 Census did not function as planned. This, combined with the need for repeated approvals and the lack of early approval of guidance from senior management on census dissemination outputs, pressured timelines, created duplication of work (i.e., rewrites of products) and increased burden on staff to meet the 18-month release schedule. The lack of clarity around overall roles and responsibilities for the Dissemination Sub-Project impacted effectiveness and efficiency. These challenges remain for the 2021 Census.

CSMS is vital for the success of the census. In terms of coordination and representation, the evaluation found it could function more efficiently and effectively in some areas. During the 2016 Census, CSMS experienced challenges managing schedules, communicating, and fostering information sharing and adoption of best practices and lessons learned.

Recommendation 1:

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that with respect to institutional reviews, governance, roles and responsibilities, and processes are reviewed and adapted to ensure that clarity exists and that reviews adhere to policy. This should also include a more effective approval model. In addition, an approach should be developed for earlier approval from senior management on the dissemination strategy.

Recommendation 2:

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that the roles and responsibilities for the areas (including the Communications and Dissemination Branch, CSMS, COD, and SMAs) involved in the Dissemination Sub-Project are clarified and updated to ensure effective and efficient delivery of the sub-project. Governance should ensure that roles and responsibilities remain clear and are adhered to over time.

Recommendation 3:

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that the roles and responsibilities of CSMS with respect to SMA representation and coordination are updated and strengthened.

In addition, CSMS processes, tools and strategies used for managing schedules, access privileges and communication flows, and for sharing and adopting best practices, should be reviewed and updated.

Design and delivery

A number of initiatives were implemented for the 2016 Census to support the 18-month release schedule. While initiatives were implemented in the majority of post-field activities, evidence suggested that the measures adopted for the E & I processes were the ones that had the largest impact on timelines.

For the 2021 Census, the program is in the process of identifying additional efficiencies within post-field activities through numerous channels, including the Census Dissemination Consultation, Process Flow WG, Core Tables WG, Accelerated Release WG and Coding Process WG. For an accelerated release schedule (i.e., all major releases completed in less than 18 months from Census Day), the dissemination strategy and coding activities appear to have the greatest potential.

There were a number of risks identified that could impact the schedule: revised certification structures and processes, potential additional content for the 2021 Census, and capacity in terms of systems and human resources.

Recommendation 4:

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that in alignment with Statistics Canada's goal of continually striving to release data in a shorter timeframe, all post-field activities are reviewed to identify potential time-saving measures. In particular, the dissemination strategy and coding activities appear to have the greatest potential. The risks identified in the evaluation should also be taken into account when planning such activities.

User-centric approach

Various mechanisms are used to capture information on census data user needs, such as formal census consultations, ongoing consultations and interactions with regular users carried out by SMAs, and web metrics. The evaluation found that program areas felt they had a good understanding of user needs in their specific domain. At the time of the evaluation, extensive consultations on census outputs were being planned and undertaken by the program. It is expected that the information gathered will inform the development of the 2021 Census release strategy.

Although user needs are taken into consideration during the development of the release strategy, other factors also have an influence: tradition (e.g., producing the same products as the last time) and internal constraints (systems, resources and accessibility). For the 2021 release strategy, WGs such as the Core Tables WG are engaging multiple divisions within Statistics Canada to identify tables that would be required for each cycle (e.g., as per legislative requirements).

Given the breadth and amount of information available, it is a significant challenge to analyze all data at once to gain a holistic view of users and their needs (content, types and formats of products, uses, and timing of releases). The evaluation found no evidence that integrated information is generated to support the development and prioritization of the release strategy.

Recommendation 5:

The ACS, Field 7, in collaboration with the ACS, Field 8, should ensure that a detailed and integrated analysis of user needs takes place. For example, user profiles based on the multiple sources of information already available could be created. The 2021 release strategy should be driven by the findings and conclusions of such an analysis and should also take into consideration the risks identified in the evaluation.

Finally, while the recommendations have been split into the three themes covered by the evaluation (governance, design and delivery, and user-centric approach), it is vital to note that they are closely interconnected. For example, the adoption of an accelerated dissemination schedule must consider factors such as information on user needs, content of the 2021 Census, IT systems and human resource capacity. Because of the interdependency of each of these themes and the underlying recommendations, an integrated view should be adopted when developing the management responses and action plans to address the recommendations.

Management response and action plan

Governance

Recommendation 1:

With respect to institutional reviews, governance, roles and responsibilities, and processes are reviewed and adapted to ensure that clarity exists and that reviews adhere to policy. This should also include a more effective approval model. In addition, an approach should be developed for earlier approval from senior management on the dissemination strategy.

Management response

Management agrees with the recommendation.

A new dissemination strategy will be designed for the 2021 Census Program and presented to the CSC. Related processes for the review of analytical outputs for official releases will be reviewed and adjusted based on lessons learned from 2016 and results of this evaluation.

The strategy to review analytical documents before release will include clear accountabilities and governance processes for the review of analytical outputs. This is in line with Statistics Canada's policies and directives on the review of analytical documents and official release.

CMO will document the new dissemination strategy. This strategy will be approved by the CSC. CSMS will be responsible for developing and documenting the process for the review of analytical documents.

Deliverables and timelines

The ACS, Field 7, in collaboration with the ACS, Field 8, will provide

• a strategy for the review of analytical documents before release (December 2019)
• a final dissemination strategy (March 2020).

Recommendation 2:

The roles and responsibilities for the areas (including the Communications and Dissemination Branch, CSMS, COD, and SMAs) involved in the Dissemination Sub-Project are clarified and updated to ensure effective and efficient delivery of the sub-project. Governance should ensure that roles and responsibilities remain clear and are adhered to over time.

Management response

Management agrees with the recommendation.

In line with the new dissemination strategy, deliverables, production processes, and clear roles and responsibilities will be defined and documented.

Project charters will reflect deliverables in relation to the strategy at the program level and for each sub-project involved (e.g., CSMS, Dissemination, ARGeo, Census Communications).

CMO will document the program-level deliverables, such as the dissemination strategy and the project charter. Individual sub-project leads will document their sub-project charters.

Deliverables and timelines

The ACS, Field 7, in collaboration with the ACS, Field 8, will provide

• project charters (including sub-project charters) (December 2019)
• a final dissemination strategy (March 2020).

Recommendation 3:

The roles and responsibilities of CSMS with respect to SMA representation and coordination are updated and strengthened.

In addition, CSMS processes, tools and strategies used for managing schedules, access privileges and communication flows, and for sharing and adopting best practices, should be reviewed and updated.

Management response

Management agrees with the recommendation.

The processes for conducting the various post-collection operations and for defining and producing the various dissemination outputs involve multiple sub-projects, and SMAs have key roles to play in most of these. Roles and responsibilities will be reviewed for all of these, to streamline work practices where warranted. Related responsibilities for CSMS and SMAs will be clearly defined as agreed upon with SMAs. Where CSMS maintains a coordination role with SMAs, tools and processes will be put in place to automate and coordinate calendars, and to ensure that SMA expertise is optimized as well as coordinated and available when required.

CMO will document the project charter. CSMS will document its individual sub-project charter and produce the calendar.

Deliverables and timelines

The ACS, Field 7, in collaboration with the ACS, Field 8, will provide

• sub-project charters for relevant sub-projects (December 2019)
• a coordinated calendar (March 2020).

Design and delivery

Recommendation 4:

In alignment with Statistics Canada's goal of continually striving to release data in a shorter timeframe, all post-field activities are reviewed to identify potential time-saving measures. In particular, the dissemination strategy and coding activities appear to have the greatest potential. The risks identified in the evaluation should also be taken into account when planning such activities.

Management response

Management agrees with the recommendation.

A new dissemination strategy will be designed for the 2021 Census Program and presented to the CSC. The various processes related to post-field activities will be reviewed accordingly for relevance and timeliness. Risks will be identified and addressed as part of that review.

Relevant outstanding issue documents related to the review of process files in JIRA and major risks entered in the Census Program risk register will be included.

Deliverables and timelines

The ACS, Field 7, in collaboration with the ACS, Field 8, will provide

• relevant outstanding issue documents (December 2019)
• the Census Program risk register (December 2019)
• a final dissemination strategy (March 2020).

User-centric approach

Recommendation 5:

A detailed and integrated analysis of user needs takes place. For example, user profiles based on the multiple sources of information already available could be created. The 2021 release strategy should be driven by the findings and conclusions of such an analysis and should also take into consideration the risks identified in the evaluation.

Management response

Management agrees with the recommendation.

A new dissemination strategy will be designed for the 2021 Census Program that details the process to clarify user needs. It will be presented to the CSC. The strategy will be based on user segmentation and clarification of main user needs by segment. Risks will be identified and addressed as part of that review.

SMAs and CSMS will provide input on user needs based on external subject-matter relationships and networks in addition to the information gathered from user feedback via electronic questionnaire and in-person meetings.

Relevant outstanding issue documents related to the review of process files in JIRA and major risks entered in the Census Program risk register will be included.

CMO will document the dissemination strategy. CPT is responsible for the outstanding issues and risks.

Deliverables and timelines

The ACS, Field 7, in collaboration with the ACS, Field 8, will provide

• relevant outstanding issue documents (December 2019)
• the Census Program risk register (December 2019)
• a dissemination consultation report (December 2019)
• a final dissemination strategy (March 2020).