Evaluation of Statistics Canada's 2021 Census of Population: Innovation Activities

How the report is structured

• The report in short
• Acronyms and abbreviations
• What is covered
• What we learned
  • 1. Performance—2021 Census key innovations
  • 2. Lessons learned and continuous improvement
• How to improve the Census Program
• Management response and action plan
• Appendix A Interview quantification scale
• Appendix B 2021 Census innovation activities

The report in short

The 2021 Census of Population was Canada's 23rd national census. The COVID-19 pandemic emerged in Canada in early 2020 and affected all steps of the 2021 Census process, from data collection to dissemination. In light of this, the Census Program undertook an extensive review of its operations to adjust its plans and procedures to ensure the continuity of census activities while protecting the safety of all Canadians and its employees. This included, among others, focusing on contactless approaches (e.g., electronic questionnaire and telephone follow-up), revising engagement and communication efforts, and introducing remote work. Moreover, changes in content and analysis were incorporated into released products to enhance the understanding of the pandemic's effects on various census topics.

In addition to these adjustments, the Census Program also continued its pursuit of innovation through research and development, as well as the implementation of new and existing innovation projects aimed at maintaining a high level of data quality, supporting data relevance, achieving a high response rate, reducing respondent burden, and improving efficiency and cost-effectiveness. Innovations occurred in all stages of the 2021 Census, from content determination to dissemination.

The evaluation was conducted in accordance with the Treasury Board Policy on Results and Statistics Canada's Risk-based Audit and Evaluation Plan (2023/2024 to 2027/2028). The objective of the evaluation is to provide credible and neutral information on the ongoing performance of the Census of Population and to identify potential areas to consider for the next census cycle. The scope of the evaluation focused on the key innovations from the 2021 Census of Population and the lessons learned from these innovations to inform continuous improvement within the Census Program and the broader agency.

The evaluation methodology consisted of a document review, interviews and an internal survey. Interviews were conducted with representatives from the Census Program, subject-matter areas and other partners within Statistics Canada. The survey was sent to directors. The findings outlined in this report are based on the triangulation of these data collection methods.

Key findings and recommendations

The 2021 Census introduced a spectrum of innovations and process improvements, ranging from small advancements building on past successes to important breakthroughs. With the onset of the COVID-19 pandemic, additional changes were implemented to help mitigate risks. Overall, the innovation activities contributed to the high performance of the 2021 Census and achievements in operational efficiency and effectiveness.

Notwithstanding the significant challenges introduced by COVID-19, the pandemic was an impetus for rapid and breakthrough change within the program. Building on the momentum of COVID-19, leveraging success factors (e.g., buy-in, dedication, strong governance and project management) and learning from challenges (e.g., rigid timelines, last-minute changes, unanticipated complexities) will support the Census Program going forward.

The Census Program has effective debriefing and governance mechanisms to support the identification of best practices and lessons learned at the end of each cycle. Guided by these activities, Census Project Team has a defined vision for 2026 and is already working on improving its processes. Some felt that the post-cycle debriefings could be improved by enhancing communication across project teams and key stakeholders to support awareness of the next steps, and by focusing on a more continuous documentation of lessons learned across the cycle to ensure the timely identification and planning of innovations for the next census. More broadly, staff reported barriers to moving from innovative ideas to execution. The formalization of Census Futures is expected to help the program to overcome some of the barriers to engaging in long-term innovation planning.

Across the agency, there are some mechanisms for knowledge exchange between the Census Program and other areas, both formally and informally. Generally, the Census Program is seen as a driver for innovation within the agency, as investments in new processes and tools can be leveraged for wider benefit. However, representatives within and outside the program agreed that learning across the agency could be improved. This could include exploring structured and dedicated forums for innovation sharing and investigating opportunities such as employee rotations, innovation-focused communities of practice and learning symposiums.

Considering these findings, the following recommendations are proposed:

Recommendation 1

The Assistant Chief Statistician (ACS) of Census, Regional Services and Operations (Field 7) should ensure continued support of the activities of the Census Futures subproject to facilitate the timely advancement of long-term and breakthrough innovation projects.

Recommendation 2

The ACS of Field 7 should ensure that mechanisms for the continuous and ongoing documentation of lessons learned throughout the census cycle are explored to support the timely identification and planning of innovations for the next cycle.

Recommendation 3

The ACS of Field 7 should ensure that internal communication around decisions on innovations and improvements is enhanced so that all stakeholders are aware and interdependencies are understood.

Recommendation 4

The ACS of Field 7, in collaboration with other ACSs, should ensure that mechanisms are explored to further support knowledge exchange between the Census Program and other parts of the agency regarding innovation activities (in both directions).

Acronyms and abbreviations

ACS

Assistant Chief Statistician

BT

Behavioural test

CCPT

Combined Census Project Team

CIIR

Census Innovation and Improvement Register

CIRC

Census Innovation Review Committee

CPT

Census Project Team

CPTP

Census Program Transformation Project

CSC

Census Steering Committee

CSMS

Census Subject Matter Secretariat

CSU

Collection Support Unit

E&I

Edit and imputation

EQ

Electronic questionnaire

IRCC

Immigration, Refugees and Citizenship Canada

IT

Information technology

NRFU

Non-response follow-up

R&D

Research and development

REQM

Research, Evaluation and Quality Management

What is covered

The evaluation was conducted in accordance with the Treasury Board Policy on Results and Statistics Canada's Risk-based Audit and Evaluation Plan (2023/2024 to 2027/2028). The objective of the evaluation is to provide credible and neutral information on the ongoing performance of the Census of Population and to identify potential areas to consider for the next census cycle.

The Census of Population

Statistics Canada has the legal obligation to conduct a Census of Population every five years. The census provides a statistical portrait of Canada and its people, covering various demographic and social topics. The statistical information produced by the Census Program supports the measurement of the changes occurring in the demographic, social and economic characteristics of Canadians. It also supports a variety of statutory, legislative and regulatory requirements. The information is used by a wide range of organizations, such as various levels of government, not-for-profit and private organizations, and academic institutions, as well as by individual Canadians.

Census information is collected through mandatory short-form and long-form questionnaires. The short-form questionnaire, containing a basic set of questions, is distributed to all households, whereas the long-form questionnaire, containing the full set of questions, is distributed to a sample of households. Some information also continued to be collected directly from administrative records, such as tax data for income-related variables from the Canada Revenue Agency, and admission category and applicant type information from Immigration, Refugees and Citizenship Canada (IRCC). In 2021, for the first time, records from IRCC were also used for immigrant status and year of immigration.

2021 Census cycle

The 2021 Census of Population was Canada's 23rd national census. The COVID-19 pandemic emerged in Canada in early 2020 and affected all steps of the 2021 Census process, from data collection to dissemination. In light of this, the Census Program undertook an extensive review of its operations to adjust plans and procedures to ensure the continuity of census activities while protecting the safety of all Canadians and its employees. This included, among others, focusing on contactless approaches (e.g., electronic questionnaire [EQ] and telephone follow-up), revising engagement and communication efforts, and introducing remote work.

The 2021 Census cycle also covered a broader range of topics and included new and revised content, including instruction in the minority official language, a modified question on sex at birth, a new question on Canadian military experience, updates to ethnic or cultural origins, new labour questions addressing underemployment, updated terminology related to Indigenous peoples, and new questions on membership within a Métis organization or Settlement and enrolment under an Inuit land claims agreement. Moreover, changes in content and analysis were incorporated into released products to enhance the understanding of the pandemic's effects on various census topics.

In addition to these adjustments, the Census Program also continued its pursuit of innovation through research and development (R&D), as well as the implementation of new and existing innovation projects aimed at maintaining a high level of data quality, supporting data relevance, achieving a high response rate, reducing respondent burden, and improving efficiency and cost-effectiveness. Innovations occurred in all stages of the 2021 Census, from content determination to dissemination.

Ongoing improvements and innovations

A key strength of the Census Program is its strong attention to critical inquiry and the identification of best practices and areas for improvement in subsequent cycles. At the end of each cycle, the Census Program undertakes a comprehensive review of all program elements to identify lessons learned and areas for improvement. As depicted in Figure 1, this exercise involves a multi-step evaluation, reporting and debriefing process leading to key decisions on innovation priorities for subsequent cycles. The exercise is overseen by the Census Project Team (CPT) and the Census Steering Committee (CSC).

The CSC and its operational subcommittee, the CPT, are cross-divisional and cross-field governance structures that provide overall direction to the census. The CSC reports to the Strategic Management Committee, the senior governance body at Statistics Canada, and is made up of senior officials within the Census Program.

Led by the Director General of the Census Management Office, the CPT reports to the CSC and is a decision-making and managing body for the Census Program. The CPT also comprises subproject managers who are each responsible for different areas and aspects of the Census Program (e.g., Census Subject Matter Secretariat [CSMS]; Research, Evaluation and Quality Management [REQM] Project; Census Futures). Through these debriefing and governance mechanisms, key decisions are made regarding planned innovations for the next cycle or thereafter based on an assessment of factors, including feasibility, priority, scope and risk.

The evaluation

The scope of the evaluation focused on the key innovations from the 2021 Census of Population and the lessons learned from these innovations to inform continuous improvement within the Census Program and the broader agency.

For the purposes of the evaluation, key innovations were defined as those identified following the 2016 Census cycle, developed through business cases by subproject teams and approved for the 2021 Census cycle. Innovations implemented to mitigate the risks related to the COVID-19 pandemic were also considered. Appendix B provides the list of the 2021 Census key innovations included in the scope of this evaluation.

The evaluation issues and questions that were identified for the evaluation are the following:

• 1. Performance of 2021 Census innovations
  • 1.1 To what extent did key innovations of the 2021 Census of Population cycle contribute to its overall performance (efficiency and effectiveness)?
  • 1.2 What factors facilitated and impeded the implementation of key innovations for the 2021 Census of Population cycle?
• 2. Lessons learned and continuous improvement
  • 2.1 To what extent are the identified best practices and lessons learned from key innovations leveraged to support continuous improvement within the Census of Population Program?
  • 2.2 To what extent are the best practices and lessons learned from innovations shared and leveraged between the Census of Population Program and the broader agency

The evaluation was conducted from September 2023 to March 2024.

The data collection methods outlined in Figure 2 were used. The findings outlined in this report are based on the triangulation of these data collection methods.

Limitations were identified and mitigation strategies were employed, as outlined in Table 1.

What we learned

1. Performance—2021 Census key innovations

Summary

Following the 2016 Census, a suite of research and innovation items was identified for consideration and possible implementation for the 2021 Census cycle to support the achievement of census priorities (i.e., efficiency, effectiveness).

The debriefing and strategic planning sessions held at the end of the 2016 Census cycle gave rise to 18 outstanding items (research items), which were action items assigned to a subproject lead for further research or inquiry in response to recommendations or identified areas of interest. Following subsequent reviews, most of these research items advanced to development, alongside the identification and investigation of additional ones, all leading up to the 2021 Census cycle.

These research items culminated into a spectrum of innovations and process improvements introduced in the 2021 Census, ranging from small advancements building on past successes to breakthrough changes. According to the REQM Project, they can be broadly categorized under three types of innovation:

1. Incremental: Leverages existing processes and systems, has a clear benefit and can be implemented rapidly (e.g., wave methodology, mail-out expansion, optimization of the Collection Support Unit [CSU] operators).
2. Strategic: Requires higher effort and supports achievement of program objectives or drivers (e.g., data quality indicators, machine learning).
3. Breakthrough: Requires extraordinary effort, signifies a paradigm shift and is the source of further innovation (e.g., administrative data contingency plan, COVID-19 adjustments).

For the 2021 cycle, the key innovation drivers were to improve efficiency and cost-effectiveness; maintain or improve the timeliness of census releases; reduce respondent burden; maintain coverage, relevance and high data quality; and reduce reliance on paper in favour of electronic media.

Additional changes were introduced to the 2021 Census to address the challenges posed by the COVID-19 pandemic.

At the onset of the COVID-19 pandemic (spring 2020), the Census Program undertook an extensive risk analysis of the anticipated effects of COVID-19, resulting in a single plan with the assumption that the 2021 Census would be conducted in a pre-vaccine context. The plan was approved by the Chief Statistician and focused on minimizing in-person contact and maintaining data quality. The Census Program received additional funding to absorb costs related to COVID-19 (e.g., personal protective equipment, front-line worker salary, printing and shipping costs, information technology [IT] infrastructure). Among others, the COVID-19 pandemic led to the following rapid adjustments:

• Expansion of EQs: Use of EQs was expanded to allow administrators of institutional collective dwellings (e.g., hospitals, nursing homes and residences, long-term care facilities) to complete their information online. Collective dwellings were also enumerated via telephone interview rather than in person.
• Focus on self-response: The 2A-R paper questionnaire was redesigned to enable canvasser and reserve areas to self-respond (i.e., without administration from a canvasser completing in-person enumeration). To reduce in-person canvassing, the mail-out, mail-out with drop-off and list/leave collection modes were used.^{Footnote 1} More resources were deployed for telephone follow-up to reduce the number of in-person visits.
• Introduction of remote work: IT infrastructure was expanded to support remote work, and census offices were used primarily as distribution facilities for field staff. Furthermore, training for staff was delivered virtually via Microsoft Teams and over the phone.
• Administrative data contingency planning: A contingency data plan was developed with a focus on using administrative data to impute a limited number of characteristics of non-responding households after collection and under specific circumstances.^{Footnote 2}

Several of the innovations introduced in the 2021 Census contributed to gains in efficiency and cost-effectiveness of the 2021 cycle.

Several of the 2021 innovations and process improvements contributed to operational efficiency across the cycle and helped the census to manage complexities introduced by COVID-19 (e.g., collection delays, staffing shortages).

Incremental improvements to census collection methodologies, including the 2021 wave methodology and the mail-out expansion, helped the census to achieve its goal of decreasing the unit cost per dwelling.^{Footnote 3} The revised wave methodology also contributed to an overall increase in the Internet self-response rate, from 68% in 2016 to 84% in 2021. This reduced the costs and processing required. The observed increase in online self-response was also likely driven by COVID-19 (i.e., rapid digital transformation), though it was impossible to quantify this effect.

With the large-scale implementation of the mail-out expansion, roughly 1 million former list/leave dwellings were converted to mail-out with drop-off, and this brought the overall level of the mail-out across the country to approximately 90%. A few regional interviewees stated that these changes led to some efficiency gains in field operations: list/leave areas were streamlined, fewer field staff were required for delivery, and the switch to a door-dropper invitation reduced the potential for error within the field.

For post-field activities, machine learning significantly improved coding efficiency, facilitating the coding of 31 questions and approximately 7 million write-in responses using fastText. Several interviewees highlighted machine learning's contribution in reducing the time, staff and costs associated with coding (approximately $4 million saved), while also unexpectedly helping to overcome hiring challenges caused by COVID-19.

Some interviewees also noted that pivoting to a virtual environment because of the COVID-19 pandemic led to operational efficiency, resulting in cost savings and improved collaboration (e.g., savings in travel time and the costs for recruitment and training, the use of platforms like Microsoft Teams). However, a few interviewees noted that the rapid implementation of virtual training via teleconference posed challenges, particularly in equipping field staff with technical knowledge. For the next cycle, interviewees recommended improving training through a hybrid approach (e.g., combining video-based onboarding and immersive in-person activities) to maximize effectiveness.

A few interviewees familiar with Meltwater found the social monitoring tool to be a more efficient approach to monitoring census communications, as it significantly reduced the time required to review a large volume of information.

Innovation activities contributed to the achievement of most key outcomes for the 2021 Census, including maintaining high data quality, supporting data relevance and achieving a high response rate.

Maintaining high data quality

The 2021 Census introduced process improvements and contingency planning to support the achievement of high-quality data.

The 2021 certification process was revised following data quality issues identified in 2016. Program reports and interviewees highlighted that the revised process introduced greater scrutiny, allowed for the identification of quality and comparability issues, and ensured that census products contained relevant information for data users (e.g., pandemic-related effects, limitations, comparability). New virtual consultation activities with external users were also reported to be a valuable opportunity to validate and discuss the data before their release.

The dissemination strategy for data quality indicators was revamped to allow data users to better assess the quality and relevance of the data as they relate to their needs. Although almost all interviewees stated that the uptake and overall utility of the quality indicators are not yet known, most of those familiar with this innovation reported that it was important for improving the transparency of census data quality and aligning it with data standards.

Considering the COVID-19 pandemic, the development and activation of the administrative data contingency plan supported the imputation of 12,000 non-responding households. While this represents a very small proportion of private dwellings (less than 0.1%), a few felt that the contingency plan was a proof of concept for the further use of administrative data to ensure data quality.

Supporting data relevance

A few interviewees familiar with changes to the census content explained that the content consultations conducted before 2021 helped the census to maintain data relevance, align with priority issues, and address identified gaps or informational needs (e.g., new variables for sex at birth and gender, minority language instruction, Métis membership, Canadian military experience). Some concrete examples of identified uptake included the following:

• In the Federal 2SLGBTQI+ Action Plan 2022, census data were cited to support insights on population diversity (i.e., statistics on transgender and non-binary populations).^{Footnote 4}
• New content focused on education in the minority official language responded to recommendations in the House of Commons Standing Committee on Official Languages' 2017 report.^{Footnote 5} Interviewees within subject-matter areas reported that language of instruction data have been used by linguistic organizations, and their relevance is also indicated by recent Supreme Court decisions.

Achieving a high response rate

The 2021 Census achieved its overall national target response rate of 98%, including high provincial response rates,^{Footnote 6} a 97.4% long-form questionnaire total response rate and high levels of self-response. The census communications campaign and incremental improvements to the wave methodology were cited as two innovations that contributed to achievements in the response rate.

The census communications campaign used a mixed-media approach, which focused on promoting Internet response and targeting key audiences. Among the achievements in 2021, the campaign contributed to an approximately 22% increase in the online response rate of harder-to-enumerate audiences. The fine-tuning of the wave methodology was also highlighted by a few as contributing to the high response rate achieved in the census, as evidenced by high levels of self-response and online response.

It was not possible to fully account for the extent to which the COVID-19 pandemic influenced the response rate achieved in the 2021 Census (positively or negatively). Interviewees explained that there were some advantages to operating the census in a pandemic context as individuals were at home and perhaps more readily available to complete their questionnaire; however, the census also experienced difficulty in meeting some of its response targets (as predicted by the wave methodology). This may have been because of factors such as anti-government sentiment (i.e., related to COVID-19 lockdowns) and limited in-person collection. The 2021 Census had a lower response rate for some populations (roughly 85% of Indigenous communities and 70% of homeless shelter residents were enumerated). A few interviewees stated that this illustrated the challenges caused by COVID-19 and indicated that not all COVID-19-related adjustments were successful in mitigating all effects.

Decreasing respondent burden

The results for reducing respondent burden were mixed as the census implemented several content additions, which increased overall questionnaire length; however, some innovations helped to manage added burden. Looking ahead, more consideration of respondent burden is highlighted as a priority for the 2026 cycle.

The census leveraged administrative data through IRCC to introduce two new variables of interest (immigrant status and year of immigration), and this minimized the impact on questionnaire length given other content changes. As well, data from the Canadian Correctional Services Survey were used to replace collection in correctional institutions. Some interviewees expressed that there are opportunities for the census to continue to expand the use of administrative data, which can reduce costs, maintain data quality and avoid added burden (e.g., integration of data from the Canadian Housing Statistics Program).

In the 2021 Census, the ability to complete the census questionnaire online was expanded to include respondents of the 2A-R and collective dwelling questionnaires. A few interviewees indicated that access to the online questionnaire was positively received by remote, northern and Indigenous communities as it responded to complaints in 2016 (although connectivity can be a challenge). While the questionnaire content was the same, features such as automated skip patterns (i.e., skipping inapplicable questions) may have contributed to greater convenience for respondents.

The 2021 wave methodology and the COVID-19 collection adjustments focused on minimizing contact with respondents through a communication strategy that increased the level of self-response before the start of non-response follow-up (NRFU). Follow-up without contact was also implemented for private dwellings, and face-to-face interviews for collective dwellings were removed in 2021 because of COVID-19. The 2021 Census had a self-response rate of 88.1%, and approximately 84% of households completed the questionnaire online.

The 2021 Census communication strategy used behavioural and sociodemographic segmentation to help identify hard-to-enumerate audiences. This supported targeted communications to reach areas that are less predisposed to self-response while accounting for key characteristics in the design and execution of communication activities.

Summary

The COVID-19 pandemic introduced complex and evolving challenges and highlighted underlying factors that made the implementation of innovations successful in the 2021 Census.

The COVID-19 pandemic was highlighted by most interviewees as having a significant influence on innovation activities, both positive and negative. Release dates were firm, even though the census encountered COVID-19-related delays to collection activities, the cancellation of early enumeration, restrictions on travel and in-person activities, and unique and ever-evolving provincial and territorial public health restrictions. Managing these risks required the identification of creative and rapid solutions and highlighted the underlying success factors to the 2021 Census.

Numerous factors were outlined by interviewees as supportive to the implementation and overall success of innovations, especially in the context of COVID-19. The most reported contributors to success were the following:

• Having buy-in from affected stakeholders: Interviewees explained that the census has many interdependencies, and one change can lead to multiple consequences and downstream effects for other subprojects and areas. Having buy-in and flexibility from key stakeholders (e.g., subject-matter areas, subprojects) and strong leadership from senior management in response to the pandemic was important to ensuring that changes were implemented successfully—particularly last-minute changes driven by COVID-19.
• Implementing a formal test: Before each cycle, the census implements formal research and testing (i.e., a "dress rehearsal") at the program and subproject levels to assess feasibility, minimize risk, and verify the performance of new technologies and innovations. The findings from these activities are then presented to the CPT and the Data Quality Review Board for review and approval. Interviewees reported these testing activities as important to increasing confidence and preparation ahead of the 2021 cycle. Given the insights that testing activities can offer, some interviewees stated that they would like to see the testing activities expanded to include innovations and process changes later in the census cycle.
• Strong governance and project management: The census effectively used governance structures, including the CPT and CSMS, to support overall coordination and decision making. A few interviewees highlighted that these structures worked well for discussing issues and staying informed on the progress of key projects. Given the unique context of COVID-19, attendance at CPT meetings was also expanded to allow for some census representatives to observe virtually. This was highlighted as a valuable opportunity for information gathering. The Census Program also used project management tools such as JIRA and the kanban method to manage workflow, coordinate activities, and track change requests and decisions (e.g., COVID-19 planning) leading up to and throughout the 2021 cycle.
• Collaboration and dedication: Most interviewees highlighted that collaboration within the 2021 cycle was strong, and a few felt that collaboration had improved since the 2016 cycle (e.g., between subject-matter areas and methodology). Activities such as regular meetings and virtual platforms (e.g., Microsoft Teams, SharePoint) helped teams to navigate the learning curves of innovations, manage and communicate dependencies, and troubleshoot unforeseen challenges. For example, the development of the administrative data contingency plan was facilitated by strong integration and collaboration between the Census Futures subproject and other census teams. Census staff also emphasized that dedication across all levels of the program (e.g., senior management, operational staff, division partners) was instrumental in the achievement of census objectives against complex changes and unanticipated challenges.

Most interviewees reported that a lesson learned from the pandemic is that the pace of innovation within the census can change in extraordinary circumstances. While COVID-19 was a unique event for the census, leveraging the success factors and building on the momentum generated can help the program continue to advance significant and timely innovations.

Some barriers were encountered in the implementation of innovation activities in the 2021 Census because of COVID-19 and other factors.

Census staff and post-cycle debriefing reports pointed to some challenges in implementing innovations in 2021, which were primarily associated with the census schedule and timeline (i.e., not enough time allocated), unexpected COVID-19 delays, and higher-than-anticipated workload.

• Lack of time within planned schedule: The most consistently reported challenge was managing innovations and new activities against rigid and tight timelines. Content innovations, certification activities and data quality indicators required more time than initially planned, and the existing schedule did not adequately accommodate the time required for process changes in machine learning. This was also exacerbated by COVID-19 delays. Census staff reported that tight timelines had a detrimental and domino effect on teams later in the cycle, as staff were required to process, analyze and disseminate a record amount of data; meet new dissemination deadlines; and adapt to new tools and processes (e.g., certification activities, new systems). This led to high overtime hours and stress among operational staff.
• High volume of work: Some innovations were reported to have added a large volume of work among some teams that may not have been adequately planned and accounted for (and was exacerbated by having insufficient time). This was highlighted by a few staff involved with certification activities (e.g., time required for external consultations, presentation attendance), and with the new data quality indicators (e.g., doubling of work, repetitive and manual tasks, management of cell limits).
• System and technical constraints: Practical challenges related to system and technical capacity hindered some projects. With the integration of question-level data quality indicators (i.e., confidence intervals), a corresponding tripling of data table size created practical limitations to developing tables. The transition to a new tabulation system involved limited integration with other census tools, leading to a duplication of efforts. Similarly, representatives familiar with machine learning described limitations in processing capacity, and system constraints were reported with the generation of more infographics and visuals (e.g., images) as part of The Daily releases.
• Clarity around governance, roles and responsibilities: Some interviewees suggested that having a more defined lead in place at the outset of some projects may be beneficial to ensure activities can proceed efficiently and to avoid delays in decision making and confusion at the operational level (e.g., reported within the administrative data contingency plan, machine learning innovation). Interviewees also reported that key innovations such as the contingency plan were hindered by changing priorities among senior management as factors such as risk, public perception and data quality were balanced. Other subprojects, including edit and imputation (E&I) and dissemination, also noted that roles and responsibilities should be better defined for the 2026 cycle.

Three main factors were identified by interviewees as influencing innovations going forward: more census and non-census alignment of processes, lack of IT resources, and the changing data consumption patterns of users.

Looking ahead to the next census cycle, interviewees highlighted a few factors that may influence what types of innovations are prioritized and a possible risk to their development.

• Increasing census and non-census alignment of processes: In accordance with Statistics Canada's strategic plan (One Plan),^{Footnote 7} some interviewees reported that there are increasing initiatives to align processes and systems between the census and other areas of Statistics Canada, including the New Dissemination Model and other corporate collection and processing systems (discussed further under evaluation question 2.2).
• IT development resources: A few interviewees reported a lack of IT development resources and the inability to effectively recruit IT resources as a threat to 2026 and long-term R&D. As IT resources are often occupied with production and operation tasks, there is limited time for engagement in development activities.
• Changing user consumption patterns: A few interviewees highlighted that the dissemination vehicles used by the census (e.g., The Daily) are "stuck in the print publication world" and have not kept pace with user preferences, such as infographics and other visual outputs. As well, it is important for the census to be aware of and responsive to changing technology, including the use of artificial intelligence to retrieve and synthesize information.

2. Lessons learned and continuous improvement

Summary

The Census Program has established rigorous processes, including evaluation mechanisms and governance, to facilitate the identification and incorporation of best practices and lessons learned.

Following the 2021 cycle, the Census Program launched once again its cyclical evaluation and debriefing process; subproject close-out reports detailing specific activities, objectives and lessons learned were developed and later reviewed by the CPT and the CSC. Two debriefing and strategic planning sessions focused on specific parts of the census cycle were also held, leading to key recommendations on where to prioritize efforts ahead of the 2026 cycle.

Most interviewees within the Census Program agreed that the census evaluation and debriefing activities are useful overall, well structured, and supportive for discussion and rigorous documentation. A vision for 2026 that sets to build on the successes of 2021 and provide a reference tool for staff has also been established.

At the operational level, census staff indicated that they are already working on measures to improve and adapt their processes (e.g., certification, machine learning, content innovations) based on findings from 2021, including the following:

• reviewing the census schedule and plan for 2026 to ensure that enough time is allocated for workflow changes and to help reduce pressure for staff
• finding opportunities to further streamline and automate processes to reduce manual and repetitive tasks (e.g., in certification, verification, E&I, the preparation of data tables)
• reviewing census products and outputs with the highest influence and overall value to ensure innovation efforts are commensurate with user gains
• improving consultation and early planning efforts to ensure that proposed changes are feasible and that the effects are well understood and minimized in advance (e.g., content changes, certification, collection activities, coding).

A few challenges and opportunities were shared by program staff to improve the census evaluation and debriefing process within the program:

• Risk of low engagement and capacity: A few operational staff stated that the census evaluation activities follow an intense period in the cycle, and this may lead to low engagement and capacity to participate in activities (i.e., risking a "checkbox" approach). A few also commented that it can be difficult to reflect on experiences that occurred in earlier stages of the cycle if they are not adequately documented throughout.
• Attention across the cycle: A few interviewees involved in later stages of the cycle (i.e., E&I to major releases) felt that the timing of the debriefing exercise may favour the identification of and investment in innovation activities in earlier stages of the cycle (e.g., collection-related innovations). Staff in later stages of the cycle may also not be adequately consulted on the corresponding impacts of the proposed changes to activities earlier in the cycle.
• Communication efforts: Representatives of enabling divisions recommended that the Census Program improve communication following the debriefing exercise. Some regional office staff reported concerns as recommendations were shared across more than one cycle (e.g., improvement of training modules, changes to data tables) and stated that there was a lack of follow-up on whether and how their recommendations were explored within the program. Similarly, divisional representatives felt that their input on Indigenous-specific collection processes has not always been extensively considered.

Following the identification of innovation areas, some barriers are encountered in advancing the development of innovations, potentially leading to repeated processes and delays in implementation.

Interviewees highlighted that there are obstacles in the ensuing steps of actioning and developing innovations (i.e., moving from idea to execution). The most consistently reported challenge is the lack of time available to engage in development within a five-year cycle. Given immediate production tasks, about half of Census Program staff reported that it is difficult to prioritize innovation planning, particularly for long-term activities requiring complex change. As a result, the program has encountered delays in advancing breakthrough innovations between cycles, such as the introduction of handheld devices to field collection. Having limited time to develop and implement innovations is further hampered by the complexity and scale of the census. As there are numerous interdependencies to account for between activities, one change can have an unintended and chain effect on other processes, requiring considerable logistical planning.

Additional high-level barriers as outlined by interviewees included resistance to change and loss of corporate knowledge. Some census staff highlighted that there can be a reluctance to change processes and systems because of familiarity and their perceived relative effectiveness, despite opportunities afforded by innovations and technology (e.g., automation, artificial intelligence). Staff may also experience concern over effects on their processes and activities. Interviewees explained that staff rotations and a reduction in staff between cycles are common within the program. For example, staff in subject-matter areas may be moved to other program areas after census production. This can lead to missed opportunities to develop and implement lessons learned ahead of the next cycle, particularly if key takeaways are not documented before staff depart and new staff are moved to the program.

Alongside the anticipated role of Census Futures (outlined below), various suggestions were shared by interviewees to overcome innovation barriers. A few program staff who were familiar with change management techniques reported that the Census Program could focus on identifying change agents to socialize change and gain buy-in among key stakeholders. Some felt that the census could be improved by ensuring lessons learned and action items are consistently documented throughout the cycle to maintain a continuous outlook on new opportunities and retain corporate knowledge. For example, this could include a regular "pulse survey" or assigning personnel on each subproject team to regularly note challenges, areas to investigate and best practices that should be scaled. Lastly, some operational staff recommended that the census focus on improving internal communication across subprojects and with key stakeholders to ensure that proposed improvements are well understood and that all implications are assessed early on. For example, this could include engaging in submapping activities to build a common understanding of process changes and ensuring that working groups are consulting all affected teams.

The Census Futures subproject was formalized to enhance long-term planning and the development of major innovations.

A key change made within the Census Program is the formalization of the Census Futures subproject to support modernization efforts. It was originally named the Census Program Transformation Project (CPTP), which was launched in June 2016 with the two-fold mandate of creating the Statistical Register Infrastructure and researching a new "combined census" model that relies on administrative data. Building on its success in advancing research and methodological innovations for collection activities, the CPTP was transformed into Census Futures in March 2023 with the added responsibility of skip-cycle planning and long-term strategy development (i.e., priority items that cannot be managed in the period between cycles). Among areas of focus, the Census Futures subproject is continuing the long-term planning of increased use of administrative data and the replacement of paper forms with handheld devices.^{Footnote 8}

Interviewees familiar with the subproject echoed that Census Futures plays an important role in helping the Census Program to advance priority innovations and overcome some of the known barriers to innovation development. While the project was only recently formalized, a few stated that the effects have already been demonstrated, including advancements in research and logistical considerations on the implementation of handheld devices. Given the required expertise from census operational staff and other key stakeholders in the planning and development of innovation activities (e.g., identifying logistical considerations, dependencies, feasibility and constraints), a few stated that it will be important that Census Futures has sufficient opportunities to integrate and collaborate with relevant staff.

Summary

Knowledge exchange between the Census Program and other parts of the agency occurs through formal and informal mechanisms.

Various forums can support knowledge exchange between the census and other Statistics Canada divisions and programs. Based on survey findings and interviews, structured mechanisms for being kept informed include census governance structures (e.g., CSC, CPT, CSMS), collaboration and service meetings (e.g., field debriefs; meetings with other survey areas, IT, communications, methodology), and participation in committees and working groups. A few representatives outside the Census Program also reported referencing census documents to learn about activities and findings from the previous cycle (e.g., project or program close-out reports).

Alongside structured approaches, approximately half of all interviewees stated that informal knowledge exchange, such as casual discussions with colleagues, was a common approach to learning about innovation activities and best practices in other areas of the agency; however, with the increased reliance on virtual working environments, opportunities for informal conversations are less common. Staff who move to other areas of the agency also help facilitate the cross-pollination of ideas and processes.

Some interviewees within the Census Program reported having touchpoints with other federal departments (e.g., Employment and Social Development Canada, Health Canada) and through international census forums to share best practices. A few census staff felt that they were more engaged in discussions outside the agency than through internal mechanisms.

Cross-agency knowledge exchange has led to some awareness of and uptake in innovation activities.

Sharing census innovations across Statistics Canada

Most interviewees positioned the Census Program as a driver for innovation across the agency. With greater resources available, census investments in processes and systems can be leveraged for overall benefit within Statistics Canada. For example, the introduction of the new Field Management System^{Footnote 9} within the Census Program was later expanded to other program areas, generating cost savings and streamlining processes. Specific examples of sharing best practices and lessons learned from 2021 Census innovations included the following:

• Presentations and workshops on topics such as the use of segmentation in the census communications campaign and the application of machine learning to other survey areas were delivered to other areas of the agency. Discussions with some program areas (e.g., Labour Force Survey) have also arisen to share and harmonize processes in coding.
• Subject-matter areas have harmonized variables and concepts introduced in the census with other survey areas (e.g., COVID-19-related income variables, content innovations).
• Divisional representatives surveyed stated that they considered integrating the lessons learned from administrative data imputation in the 2021 Census and census collection processes (e.g., wave methodology, communication strategies, follow-up operations).
• Methodology areas have collaborated through symposiums and communities of practice, focusing on topics such as machine learning coding and the use of quality indicators.

Leveraging Statistics Canada innovations and best practices within the census

With a greater focus on aligning census and non-census processes, the 2021 Census was marked by the integration of several corporate tools, such as the Common Output Data Repository, TestRail, Power BI, G-code and the New Dissemination Model. A few interviewees within the census noted that while streamlining systems has an overall benefit for the agency (e.g., cost savings, optimization of processes), it can be challenging for the census to adopt corporate systems when they are not developed to fit the parameters of the census at the outset. For example, the transition to the New Dissemination Model and G-code required additional expenditures and led to delays in their adoption as the systems required enhancements to meet the needs of the Census Program.

In addition to the integration of corporate systems within the census, subject-matter representatives reported bringing new ideas and practices to the 2021 Census based on lessons and techniques developed from other survey areas (e.g., surveys conducted more frequently). This included the modernization of data tables and charts and the migration of income linkages.

Most agreed that learning across the agency is impeded by factors such as a lack of awareness, time constraints and attitudinal beliefs.

Most agreed that learning between the Census Program and other parts of the agency could be improved. For example, census staff felt that processes and tools developed within the census (e.g., automation, coding, collection methodologies) could help other program areas to increase efficiencies and overcome challenges such as declining response rates. Similarly, interviewees from other Statistics Canada divisions stated that the census could learn more from surveys that are conducted more frequently. For example, other survey areas could provide insights on respondent behaviour and other contextual factors that could influence the next census. Innovative approaches that are introduced in other survey and program areas can also reduce risk and increase confidence for implementation in the Census Program (i.e., opportunity to "test the waters").

Nevertheless, awareness of innovation activities appears to be low across the agency. Based on the results of the internal survey that was distributed to all Statistics Canada's directors, most respondents reported that they had not integrated, or considered integrating, any innovations, best practices or lessons learned from the 2021 Census, and most indicated that they were not aware of 2021 innovations. The overall low response rate to the internal survey (23%) further suggests that knowledge is low. A few interviewees also described the duplication of processes and systems as evidence of limited awareness (e.g., some software tools have been unknowingly purchased twice), and one interviewee noted that the census has invested in the development of some tools with the intention of broader corporate use, though not all divisions have agreed to use them (instead developing a new tool for their own use).

Several factors were described by interviewees as limiting awareness and the uptake of innovations. Some highlighted that there is an absence of a formalized or strategic approach to sharing innovations across Statistics Canada, such as an agency-wide innovation committee (i.e., a "top-down" approach). Other barriers identified in interviews and the survey include the following:

• Lack of time: Interviewees emphasized that high workloads and immediate priorities within the Census Program can limit the extent to which census staff can readily engage in outside networking. Similarly, representatives from other divisions agreed that time constraints can hinder their ability to participate in forums and incorporate new processes and tools.
• Insufficient knowledge transfer: A few interviewees explained that innovation activities may be shared with staff who have differing levels of technical familiarity with a given subject or approach, and that the Census Program tends to communicate information with census-specific language and terminology. Because of this, there may be less understanding of the possible applications of innovation activities, and the information may not be relayed at the operational level.
• Perceptions and attitudes: Interviewees inside and outside the Census Program agreed that attitudinal factors can greatly influence interest in innovations occurring in other areas of the agency. The census can be perceived by other program areas as a more complex, higher-priority and better-resourced program. Because of this, other programs may feel fatigue and have limited interest in learning about census activities. There can also be a reluctance to abandon existing processes and systems in favour of those used by the census. In the other direction, the census may not embrace all learning opportunities, as they are not perceived as relevant given the complexity and scale of the program.
• Incompatible processes and practical constraints: Other common factors hindering the ability to incorporate census innovations include having a limited workforce, incompatible systems and processes, and budget constraints.

Awareness and information exchange could be improved through a multi-faceted approach.

Interview discussions highlighted that a multi-faceted approach is required to address the barriers to knowledge exchange, as there are practical factors (e.g., time, capacity) and broader challenges (e.g., attitudes, beliefs). Among the suggestions, introducing a more formalized and structured mechanism that supports objective and open dialogue (i.e., avoiding one program overshadowing another) could increase awareness across the agency. A few interviewees noted that an innovation working group committee had previously been in effect and that reinstating such an approach would also align with the broader objectives of the One Plan. For example, a structured forum could support the identification of where census and non-census processes and tools can be further aligned, benefitting the broader agency (i.e., leveraging best practices) while potentially generating cost savings.

Other recommendations that could target staff at the operational and management level were also shared:

• Improve the dissemination of census documents (e.g., census close-out reports, innovation documents, Census Futures reports) across the agency (e.g., at field debriefings, in committees, in forums) to increase awareness of census activities.
• Increase opportunities to participate in focused communities of practice with a mandate of sharing best practices and lessons from innovations (e.g., in collections, methodology, dissemination).
• Reinstate optional knowledge exchange activities, such as "lunch and learns" or other regular presentation forums.
• Encourage opportunities for employee rotation between the Census Program and the rest of Statistics Canada to build awareness of practices and processes across the agency and to transplant knowledge.

How to improve the Census Program

The most cited barrier to advancing innovation within the census is the inability to engage in long-term planning and development activities given the immediate operational priorities within the program. Furthermore, interviewees highlighted the limited time for innovation development within a five-year cycle. The work of the Census Futures subproject could help the program in overcoming these barriers and should continue to be supported.

The timing of the census evaluation and debriefing exercise follows an intense period within the cycle. Interviewees reported that the lessons learned and recommendations identified in earlier stages may be missed in the absence of ongoing documentation or mechanisms for feedback. Given the limited timeframe between cycles, identified opportunities for innovation may already be considered too late for research and implementation. Ensuring mechanisms are in place for ongoing feedback and documentation can help streamline innovation planning and avoid missed opportunities.

Interviewees involved with the census reported that internal communication could be improved. Some subproject teams also stated that communication could be strengthened when innovations and process improvements are being introduced in other areas of the census cycle to avoid unforeseen and downstream effects. Identifying mechanisms that can provide updates on proposed changes and support adequate consultation can improve preparation for the next cycle.

While the Census Program is generally seen as a driver for innovation within the agency, representatives within and outside the program agreed that learning across the agency could be improved. This could include exploring structured and dedicated forums for innovation sharing and investigating opportunities such as employee rotations, innovation-focused communities of practice and learning symposiums.

Recommendation 1

The Assistant Chief Statistician (ACS) of Census, Regional Services and Operations (Field 7) should ensure continued support of the activities of the Census Futures subproject to facilitate the timely advancement of long-term and breakthrough innovation projects.

Recommendation 2

The ACS of Field 7 should ensure that mechanisms for the continuous and ongoing documentation of lessons learned throughout the census cycle are explored to support the timely identification and planning of innovations for the next cycle.

Recommendation 3

The ACS of Field 7 should ensure that internal communication around decisions on innovations and improvements is enhanced so that all stakeholders are aware and interdependencies are understood.

Recommendation 4

The ACS of Field 7, in collaboration with other ACSs, should ensure that mechanisms are explored to further support knowledge exchange between the Census Program and other parts of the agency regarding innovation activities (in both directions).

Management response and action plan

Recommendation 1

The ACS of Field 7 should ensure continued support of the activities of the Census Futures subproject to facilitate the timely advancement of long-term and breakthrough innovation projects.

Management response

Management agrees with the recommendation.

In the 2023/2024 fiscal year, Census Futures was established as a subproject of the Census of Population, with an associated high level of visibility within the project structure of the census.

A new Combined Census Project Team (CCPT), reporting to the CSC, will begin in the first quarter of 2024/2025 to provide governance over the main research activities of the Census Futures subproject and prepare for a 2027 decision about whether to transform the 2031 Census to a hybrid administrative and traditional census.

Other Census Futures research (i.e., priority "breakthrough" innovations) will be governed at CPT and Field 7 Planning Board meetings, starting in the first quarter of 2024/2025.

Deliverables and timelines

• Implement the new CCPT, including its terms of reference, in April 2024.

Recommendation 2

The ACS of Field 7 should ensure that mechanisms for the continuous and ongoing documentation of lessons learned throughout the census cycle are explored to support the timely identification and planning of innovations for the next cycle.

Management response

Management agrees with the recommendation.

In the second quarter of 2024/2025, a new subcommittee of the CPT, the Census Innovation Review Committee (CIRC), will be created to oversee the establishment and ongoing development of the Census Innovation and Improvement Register (CIIR):

• The CIRC will be responsible for updating the register using predetermined criteria (defined and approved by the CPT) to review and classify the lessons learned proposed by subprojects or other entities (e.g., regional offices) and to suggest additional items.
• The CIRC will be responsible for reviewing and classifying the lessons learned as they arise during the cycle.
• The CIRC will report to the CPT on updates and seek approval for new items, movement through some stages and gates, and any significant changes required throughout the cycle.

In the second quarter of 2024/2025, an evergreen CIIR will be established and maintained.

• Each item on the register will include the lessons learned, the approval status, the target cycle of implementation, the type of innovation, etc.
• The register is a new product, not cycle-specific, leveraging existing census and corporate tools.
• The register will be accessible to all census staff and stakeholders at any time.

Deliverables and timelines

• Implement the new CIRC, including its terms of reference, before July 2024.
• Establish the first register before September 2024.

Recommendation 3

The ACS of Field 7 should ensure that internal communication around decisions on innovations and improvements is enhanced so that all stakeholders are aware and interdependencies are understood.

Management response

Management agrees with the recommendation.

The evergreen CIIR will contain the latest information, such as the stage and status, allowing any Statistics Canada staff to consult the register for information on innovations, improvements and interdependencies.

The CIIR will include the following items:

• lessons learned associated with the creation of an item or rationale
• the date the item was added
• CPT or CSC approval status
• the target cycle of implementation (current, next, skip-cycle)
• the subproject leading the elaboration and implementation
• the type of innovation (incremental, strategic, breakthrough)
• the origin of the innovation (census, corporate, etc.)
• the stage (elaboration, development, implementing, implemented)
• the status (standby, ongoing, completed)
• interdependencies
• a link to the innovation's GCdocs main directory
• the date of implementation.

The CIRC will present updates to the CPT and seek approval of new items added to the register.

The CIIR will be presented to the Field 7 Planning Board and the CSC.

Deliverables and timelines

• Present the CIIR to the CPT before October 2024.
• Present the CIIR to the Field 7 Planning Board and the CSC before December 2024.

Recommendation 4

The ACS of Field 7, in collaboration with other ACSs, should ensure that mechanisms are explored to further support knowledge exchange between the Census Program and other parts of the agency regarding innovation activities (in both directions).

Management response

Management agrees with the recommendation.

Divisional innovation champions will be named and will form a Field 7 committee that will develop the Field 7 Innovation and Improvement Register and will report to the Field 7 Planning Board.

The CIIR will be shared and used to facilitate communication with other parts of the agency:

• The register will be shared with the Field 7 Planning Board and will form part of the broader Field 7 Innovation and Improvement Register planned for late 2024/2025.
• This Field 7 register will be shared with other fields.

Linkages between census and corporate innovations will be sought, for instance, through participation by the Director of the Census Operations Division at the Enterprise Architecture Review Board.

Deliverables and timelines

• Implement the Field 7 Innovation and Improvement Committee, including its terms of reference, before December 2024.
• Build the initial Field 7 Innovation and Improvement Register integrating the CIIR before March 2025.

Appendix A Interview quantification scale

Interview responses are quantified and categorized in this report using the scale shown in the table below.

Appendix B 2021 Census innovation activities