Evaluation of Statistics Canada's COVID-19 Data Response
Crowdsourcing and Probability Panel Products and Specific COVID-19 Related Health Statistics
Evaluation Report
October 2021
How the report is structured
- The report in short
- Acronyms and abbreviations
- What is covered
- What we learned
- Design and delivery
- Responsiveness to users' needs
- Moving forward
- How to improve the program
- Management response and action plan
- Appendix A: List products (and surveys) released
- Appendix B: Webtrends
The report in short
Throughout the pandemic, Statistics Canada has responded rapidly and with agility to the data needs of various users by developing many new products relevant to COVID-19 and its impacts on Canadians. While this response was agency-wide, this report focuses on two types of products that were developed in the context of the pandemic: 1) products derived from crowdsourcing and probability panels, and 2) specific COVID-19 related health statistics.
This evaluation was conducted by Statistics Canada in accordance with the Treasury Board Policy on Results and Statistics Canada's Risk-based Audit and Evaluation Plan (2020/2021 to 2024/2025). The objective of the evaluation was to provide a neutral, evidence-based assessment of Statistics Canada's data response to COVID-19. The evaluation aimed at providing valuable information about the timeliness, the relevance and the impact of these products. It also looked at some of the related challenges and opportunities faced by the divisions responsible for these products to inform future direction.
The evaluation methodology consisted of a document review, administrative reviews and key interviews with Statistics Canada professionals working in different divisions responsible for social statistics. Additionally, interviews were conducted with key users and partners external to Statistics Canada. The findings outlined in this report are based on the triangulation of these data collection methods.
Key findings and recommendations
Design and delivery
The evaluation reviewed organizational changes that were implemented to develop COVID-19 health and social statistical products. The evaluation found that Statistics Canada took rapid action by assessing and delivering products that shed light on issues of importance during the pandemic. Delivery of COVID-19 statistical products was facilitated by an organizational shift towards business agility, which included innovative approaches to manage resources and analyze data. The dedication of staff was instrumental in the successful release of those products. However, the sheer volume and push for expediency impacted staff well-being.
Responsiveness to users' needs
To inform Statistics Canada on the effectiveness of its response related to social statistical data needs during COVID-19, the evaluation assessed the extent to which the products released addressed the needs of key users. The majority of users considered the products released to be relevant, useful, timely, accessible and unique for providing a national perspective. Although some challenges were noted, users were also satisfied with the agency's level of engagement and the methodological information provided. Finally, users wanted more granular data, particularly regarding at-risk and marginalized populations.
Moving forward
The evaluation assessed the extent to which there is an ongoing need for COVID-19 social and health statistical products and the factors that must be taken into account going forward to deliver both those products and regular programs. Users indicated an ongoing need for health statistics related to the pandemic as well as for crowdsourcing and probability panel products. As the pandemic evolves, more disaggregated data on the differential impacts of COVID-19 on different populations will be needed. Forward approach needs to balance demands arising from emerging data needs, such as disaggregated data, ongoing pandemic information requirements and ongoing program delivery. Opportunities remain to fully explore the lessons learned from the pandemic.
Recommendation 1:
The Assistant Chief Statistician (ACS), Social, Health and Labour Statistics (Field 8), should ensure that a comprehensive strategy is developed to identify, prioritize and respond to emerging data needs (i.e., disaggregated data, ongoing pandemic information requirements and ongoing program delivery).
The strategy should consider the key enablers identified (i.e., prioritization of demands, human resources, ongoing innovation and data sources, user engagement, and internal collaboration), and include approaches and plans that set out how:
- emerging data needs will be identified (e.g., stakeholder engagement approaches);
- data gaps will be addressed;
- data needs will be prioritized based on intended outcomes and resource capacity; and
- ongoing monitoring of data needs will be carried out.
Recommendation 2:
The ACS, Social, Health and Labour Statistics (Field 8), should ensure that a lessons learned exercise from the COVID-19 pandemic is conducted to identify approaches that could be applied to respond to COVID-19 data needs and improve core activities going forward.
- The lessons learned should be presented to a Tier 1 committee that will provide direction on their potential implementation across the agency, as applicable.
- They should also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
Acronyms and abbreviations
- ACS
- Assistant Chief Statistician
- CCAHS
- Canadian COVID-19 Antibody and Health Survey
- CCHS
- Canadian Community Health Survey
- CHMS
- Canadian Health Measures Survey
- CITF
- COVID-19 Immunity Task Force
- HR
- Human resources
- IT
- Information technology
- LFS
- Labour Force Survey
- NSO
- National statistical office
- PHAC
- Public Health Agency of Canada
- SCMH
- Survey on COVID-19 and Mental Health
What is covered
The evaluation was conducted in accordance with the Treasury Board Policy on Results and Statistics Canada's Integrated Risk-based Audit and Evaluation Plan (2020/2021 to 2024/2025). In support of decision making, accountability and improvement, the objective of the evaluation was to provide a neutral, evidence-based assessment of Statistics Canada's data response to COVID-19.
Throughout the pandemic, Statistics Canada has responded rapidly and with agility to the data needs of various users by developing many new products relevant to COVID-19 and its impacts on Canadians. The evaluation focused on two types of products that were developed in the context of the pandemic: 1) products derived from crowdsourcing and probability panels, and 2) specific COVID-19 related health statistics. The evaluation aims to provide valuable information about the timeliness, the relevance and the impact of these products. It also looked at some of the related challenges and opportunities faced by the divisions responsible for these products to inform future decisions.
Crowdsourcing and probability panel products
To provide Canadians with the timely data and insight they need to respond to the COVID-19 pandemic, Statistics Canada leveraged two non-traditional methods of collecting information efficiently: crowdsourcing and probability panels.
Crowdsourcing involves collecting information from a large community of users. It relies on the principle that individual citizens are experts within their local environments. All Canadians were eligible to participate; there was no sample selection and participation was voluntary. Data collection was conducted exclusively online by participant self-completion of surveys in a crowdsourcing application. Demographic projections of the number of people by province and territory, sex and age group were used to calculate a benchmarking factor for every participant to compensate for over- and underrepresentation. Because crowdsourcing is non-probabilistic, it has potential selection bias. It is not statistically representative of the general population because it reflects only the views of those who responded.
Probability panels (probabilistic web panels) are a pooling tool that uses a sample of former respondents to the Labour Force Survey (LFS) who volunteer to take supplementary Internet surveys over a set period of time. Since the sample is derived from the LFS, results can be presented with statistical validity. While the panel offers the ability to collect data quickly, there is a greater risk of bias because of some aspects of the design and a lower response rate compared with traditional surveys. It excludes a small portion of the population (e.g., those without Internet access) and it cannot reflect all population groups and geographies.
Crowdsourcing and probability panels enabled Statistics Canada to rapidly gather data and produce analysis on how the COVID-19 pandemic was affecting the lives and well-being of different groups of Canadians. Products derived from both methods explored various dimensions of the COVID-19 impacts on Canadians (e.g., education, employment, mental health, child care, discrimination and stigma, substance abuse, community crime, safety and violence during the COVID-19 pandemic).
Specific COVID-19 related health statistics
In addition to its ongoing health statistics program, Statistics Canada worked closely with federal, provincial and territorial governments and organizations to provide specific COVID-19 related health statistics to Canadians. Those included in the scope of the evaluation encompass:
- vital statistics
- COVID-19 epidemiological reports
- products from upcoming surveys.
To meet the need for more timely information on public health, Statistics Canada has been working collaboratively with its health partners since the beginning of the pandemic to release preliminary health statistics related to COVID-19. These include provisional figures on mortality for vital statistics and the preliminary number of confirmed cases via the COVID-19 epidemiological reports. To continue to provide relevant COVID-19 health data to Canadians, Statistics Canada was also working on two surveys at the time of the evaluation: the Canadian COVID-19 Antibody and Health Survey (CCAHS) and the Survey on COVID-19 and Mental Health (SCMH).
The Evaluation
The scope of the evaluation encompassed two types of products that were developed in the context of the pandemic: 1) products derived from crowdsourcing and probability panels, and 2) specific COVID-19 related health statistics. The scope was established in consultation with the divisions involved in these areas.
Although products derived from crowdsourcing and probability panels were labelled in their own category for the purpose of the evaluation, a number of products derived from these approaches targeted health issues and heavily involved the groups responsible for the specific COVID-19-related health statistics as well.
The evaluation was conducted from November 2020 to April 2021 and covered products listed in Appendix A.
The following issues and questions were identified for review.
| Evaluation issues | Evaluation questions |
|---|---|
| Design and delivery | What actions were taken by Statistics Canada in response to COVID-19 in regard to products derived from crowdsourcing and probability panels, and specific COVID-19 related health statistics? |
| Responsiveness to users' needs | To what extent did the products released address the needs of key users?
|
| Moving forward | What factors should be taken into consideration moving forward? |
Guided by a utilization-focused evaluation approach, the following quantitative and qualitative collection methods were used.
Description - Quantitative and qualitative collection methods used
The figure depicts the four collection methods used for the evaluation: international scan, document review, key external interviews, and key internal interviews.
The international scan identified new products released and methods used by other countries in response to COVID-19 (e.g., France, United States, Australia and the United Kingdom).
The document review included a review of Statistics Canada's internal files and documents
The key external interviews included semi-structured interviews with: federal government departments, organizations, provincial and municipal governments, non-profit and academic sectors. There were 31 external interviews conducted.
The key internal interviews included semi-structured interviews with Statistics Canada employees. There were 14 internal interviews conducted.
Three main limitations were identified and mitigation strategies were employed:
| Limitation | Mitigation strategy |
|---|---|
| Because of the large number of users and partners using data, the perspectives gathered through external interviews may not be fully representative. | External interviewees were selected using specific criteria to maximize a strategic reach for the interviews. Different types of organizations that use COVID-19-related data extensively from a wide range of locations across Canada were selected. Evaluators were able to find consistent overall patterns. |
| Interviews have the possibility of self-reported bias, which occurs when individuals who are reporting on their own activities portray themselves in a more positive light. | By seeking information from a maximized circle of stakeholders, evaluators were able to find consistent overall patterns. |
| Limited documentation was available on the projects sampled for the evaluation. | Key staff working on the relevant projects were interviewed and a strategy to gather additional documents during the interview sessions was put in place. Additional interviews were conducted as needed to fill the gaps. |
What we learned
1. Design and delivery
Evaluation question
What actions were taken by Statistics Canada in response to COVID-19 in regard to crowdsourcing and probability panel products and specific COVID-19 related health statistics?
Summary
The evaluation reviewed organizational changes that were implemented to develop COVID-19 health and social statistical products. The evaluation found that Statistics Canada took rapid action by assessing and delivering products that shed light on issues of importance during the pandemic. Delivery of COVID-19 statistical products was facilitated by an organizational shift towards business agility, which included innovative approaches to manage resources and analyze data. The dedication of staff was instrumental in the successful release of these products; however, the sheer volume and push for expediency impacted staff well-being.
In response to COVID-19, Statistics Canada took rapid action to assess the impacts of the pandemic on Canadians by releasing various statistical products and using innovative approaches
The COVID-19 pandemic continues to have significant impacts on Canadians. During this crisis, timely and credible data has been crucial to understanding and managing the health and social effects of the pandemic. In response to COVID-19 data needs, Statistics Canada developed new products and adapted some of their regular products. These aimed at providing more timely and relevant data for monitoring the impacts of the pandemic on Canadians. Products were developed by the Census Subject Matter, Social Insights, Integration and Innovation Branch (Centre for Social Data Insights and Innovation Division and Centre for Social Data Integration and Development Division) and the Health, Justice, Diversity and Populations Branch (Centre for Population Health Data Division and Diversity and Sociocultural Statistics Division) under the Social, Health and Labour Statistics Field. Both branches worked in close collaboration, particularly on health data produced through crowdsourcing and probability panel approaches. Other divisions involved in various support functions (e.g., communications, information technology (IT), methodology, collection) also contributed significantly to the timely delivery of COVID-19 statistical products.
Crowdsourcing and probability panels
As part of Statistics Canada's modernization, crowdsourcing and probability panels have become innovative ways to collect information rapidly and efficiently for statistical purposes. During the early stages of the pandemic, Statistics Canada launched a series of online crowdsourcing and probability panel questionnaires to collect data on how Canadians were reacting. These alternative methods of collecting information enabled delivery of results within three weeks of collection.
Between the summer and fall of 2020, over 70 information products derived from crowdsourcing and probability panels were produced to inform Canadians. The products were disseminated through public use microdata files and communicated through releases in The Daily and other analytical products.
Specific COVID-19 related health statistics
Vital statistics
Vital statistics are gathered through an administrative survey that collects demographic and medical information (cause of death) annually and monthly from all provincial and territorial vital statistics registries on all deaths in Canada. In the context of the pandemic, the duration of data collection was shortened considerably. Since May 2020, Statistics Canada has been releasing provisional figures on excess mortality and causes of death, including deaths attributed to COVID-19. The data released were provisional as they were not based on all deaths that occurred during the reference period because of reporting delays and did not include the Yukon. These provisional death estimates were adjusted to account for incomplete data where possible and were revised monthly as new information became available.
COVID-19 epidemiological reports
As part of its continued commitment to provide Canadians with much-needed information during the pandemic, Statistics Canada partnered with the Public Health Agency of Canada (PHAC) to make preliminary information on the number of COVID-19 cases available to the public. Beginning in March 2020, epidemiological reports on detailed confirmed cases (preliminary data) were released on a weekly basis until July 2020, at which time the frequency was changed to biweekly. Aggregated datasets with preliminary case information across six dimensions were also released on an occasional basis. Given the rapidly evolving nature of the pandemic and the multi-tiered approach to data collection on case reporting, epidemiological reports were considered preliminary. Confirmed case data (laboratory-confirmed cases of COVID-19) were collected by PHAC from various sources, including provincial and territorial health ministries. These data were not considered comprehensive, given a possible time lag between when cases occur and when they are reported. Cases in this database may be revised as new information is captured related to a person's hospitalization status, intensive care unit admittance or death. Data were collected for all units of the target population (entire Canadian population); therefore, no sampling was done.
Upcoming surveys
The CCAHS is a voluntary survey to learn about how the COVID-19 virus impacts overall health, how it spreads and whether Canadians are developing antibodies against it. The content of the survey was developed by Statistics Canada with input from the COVID-19 Immunity Task Force (CITF) and in consultation with Health Canada and PHAC. The CCAHS is a unique survey that will collect information in two parts. The first part is an electronic questionnaire about general health and exposure to COVID-19. The second part is an at-home finger-prick blood test, which is sent to a lab to determine the presence of COVID-19 antibodies. The target population for the survey is people 1 year of age and older (children under 14 will have the questionnaire completed by a parent or guardian) living in the 10 provinces or 3 territorial capitals. Respondents will be contacted via mail and provided with a code to access an online questionnaire and a dried blood spot kit to collect their blood sample.
The SCMH collects data to assess the impacts of COVID-19 on the mental health and well-being of Canadians. The survey includes topics on behaviours and symptoms associated with depression, anxiety and post-traumatic stress disorder, suicide risk, pressure on parents, substance use, household violence, as well as general mental health. Participation is voluntary and the target population is non-institutionalized people aged 18 years or older, living in Canada's 10 provinces and 3 territorial capitals, who are not members of collectives or living on reserves.
Common approaches were used by other national statistical offices to develop products addressing COVID‑19 statistical data needs
In the early months of the pandemic, Statistics Canada faced similar challenges and constraints to those experienced by other national statistical offices (NSOs): lockdowns, deferral of regular activities, surging demands for disaggregated data and rapid statistics on the impacts of COVID-19. Within this context, Statistics Canada's approach to measuring the social impacts of COVID-19 generally aligned with that of other jurisdictions in terms of major themes covered, methodologies utilized and data disaggregation. The evaluation did not find major content gaps with respect to social and health surveys conducted in other jurisdictions. Statistics Canada's crowdsourcing and probability panel products explored very similar topics to panel surveys conducted by NSOs in Australia and the United States, both of which were developed in response to the pandemic, and the United Kingdom's omnibus survey, which was modified to increase frequency from monthly to weekly in response to COVID-19. The level of methodological detail provided by Statistics Canada also aligned with the type of information provided by other NSOs.
The main difference between Statistics Canada's approach and that of other NSOs was the agency's use of crowdsource questionnaires to explore the social impacts of COVID-19 (although France's public health agency employed a similar approach). Relative to other NSOs, Statistics Canada was one of the first to publish data in response to the pandemic.
Delivery of COVID-19 statistical products was facilitated by an organizational shift towards business agility and through the agency's modernization initiatives
Much of the agency's response to the pandemic was rolled up into a special project to ensure expediency and efficiency. As part of this special project, several initiatives that were already under development, such as crowdsourcing and probability panels, were accelerated, adapted and operationalized. To ensure the timely delivery of products related to COVID-19, changes were also made to organizational approaches and internal processes to increase agility.
Processes for crowdsourcing and probability panels were focused on expediency; an iterative approach led to shortened timelines. The processes were not fundamentally modified, but implemented at an accelerated pace with the objective of making improvements after each iteration. Because of the speed of response, there were some instances of data quality issues early on, as well as a need to educate some employees about the limits of these approaches in the beginning. However, these issues were addressed over time.
For COVID-19 related health statistics, process changes accelerated their development, analysis and release. For example, content development and analysis timelines were shortened by implementing quicker turnaround times. Some steps and subprocesses were accelerated and adapted even if the same overall process was followed. For example, editing or approval time was reduced for the SCMH and the CCAHS. For vital statistics, methodological changes were developed in collaboration with the Modern Statistical Methods and Data Science Branch and the Centre for Demography to produce provisional figures.
Changes in governance processes also accelerated decision-making and were generally seen as necessary and positive. Early in the pandemic, most existing oversight bodies halted operations, except the Strategic Management Committee, which continued to ensure alignment with strategic direction. The increased involvement of management in operational activities also presented opportunities to identify and approve process modifications to address challenges or bottlenecks.
In addition, existing and new bodies facilitated decision-making, action and implementation of response. For example, for crowdsourcing and probability panels, the new Social Impacts and New Data Development Working Group facilitated the agency's ability to collaborate and work horizontally, serving as the key body for planning and decision-making. For specific COVID-19 related health statistics, new and existing bodies (e.g., Centre for Social Data Insights and Innovation, CCAHS interdepartmental working group, CITF) enabled prioritization, increased awareness across participating divisions, and provided opportunities to solicit and share feedback from partners.
Statistics Canada's response was also facilitated by innovation projects and organizational culture shifts linked to its modernization agenda, which focused on the pillars of user-centric service delivery, leading edge methods and data integration, statistical capacity building and leadership, sharing and collaboration, and a modern and flexible workplace. For example, prior development of crowdsourcing and probability panel methods as part of the modernization initiative were integral in mounting a timely response. Electronic tools and processes (e.g., previously developed electronic questionnaires, system to collect vital statistics from provinces and territories) also facilitated increased speed of data collection and response. The modernization agenda also provided traction to further shift the organization towards innovation and collaboration which facilitated the agency's COVID-19 data response.
A flexible and dedicated workforce with a shared sense of purpose was instrumental in the successful release of products, though pace of work impacted staff well-being
At the onset of the pandemic in March 2020, Statistics Canada's primary focus was on maintaining its mission-critical programs and addressing data needs related to COVID-19. Within the Social, Health and Labour Statistics Field, virtually all other activities and initiatives (e.g., development of the Canadian Health Measures Survey [CHMS], mental health data strategy, opioid epidemic data acquisition) were put on hold, which provided flexibility in the reorganization of work and reassignment of employees. For instance, resources and expertise were shared across teams and reallocated to areas of priority. For example, staff working in areas that were temporarily put on hold, such as the CHMS, were redirected to work on the CCAHS.
The collective sense of urgency to meet the informational needs of Canadians drove decisions regarding content and timelines. Communication, coordination and collaboration between divisions within the Social, Health and Labour Statistics Field and across the agency were critical success factors that contributed to the timely release of COVID-19 products. Support provided by corporate partners (i.e., Communications Branch, Modern Statistical Methods and Data Science Branch, Collection and Regional Services Branch, Processing and Operations Branch) was also crucial in the production and dissemination of statistical information. Though program areas noted some opportunities to further increase efficiency (e.g., prioritizing demands, adopting leaner processes, improving IT infrastructure and tools), corporate partners were viewed as responsive and helpful in the face of tight timelines. Overall, the evaluation found that the successes and timeliness of the COVID-19 data response were greatly attributable to the dedication, expertise and flexibility of the workforce.
While employees' drive and adaptability were integral to the delivery of data, the increased workloads and tight timelines contributed to lower employee morale in some areas, as well as the delay of projects not related to the pandemic. Employees worked considerable overtime, frequently for a sustained period, to deliver on the COVID-19 products, often with limited available resources. Almost all internal key interviewees reported that the high volume of work and tight timelines increased stress and anxiety or otherwise negatively impacted the mental health and well-being of employees. They also discussed a reduction in work-life balance associated with workloads, coupled with external challenges such as increased family stress during the pandemic (e.g., because of senior or child care responsibilities). Additionally, some pointed out that certain employees with critical areas of expertise were especially overburdened with work and faced detrimental effects to their well-being. Challenges were largely mitigated by employees' hard work, enthusiasm and a shared sense of purpose in meeting the data needs of Canadians during a period of uncertainty. Further, internal interviewees noted that the adverse effects on employee well-being as a result of increased workloads and tight timelines were somewhat mitigated by recognition and appreciation of their hard work, as well as the support provided by their managers.
Finally, the evaluation found that heightened expectations of staff did not significantly lessen over time. Resource allocation to COVID-19 products delayed regular program activities or initiatives outside of mission-critical programs. Many internal interviewees found the impacts on staff well-being to be somewhat manageable early on in the pandemic response, when other work was largely on hold. As other work began to resume by fall 2020, however, continued resource allocation to COVID-19-specific products meant delays to numerous regular programs activities or initiatives, such as work on meeting users' needs for disaggregated data, development of the CHMS, and work on the Mental Health Data Strategy and Northern Data Strategy. There is some indication that these challenges will continue to hinder the response of program areas to emerging data demands going forward.
2. Responsiveness to users' needs
Evaluation question
To what extent did the products released address the needs of key users?
- Engagement
- Relevance and Utility
- Clarity (methodology)
- Timeliness
- Access
Summary
To inform Statistics Canada on the effectiveness of its response related to social statistical data needs during the pandemic, the evaluation assessed the extent to which the released products addressed the needs of key users. The majority of users considered the products released to be relevant, useful, timely, accessible and unique for providing a national perspective. Although some challenges were noted, users were satisfied with the agency's level of engagement and the methodological information provided. Finally, users wanted more granular data, particularly related to at-risk and marginalized populations.
Statistics Canada received an influx of data demands primarily through informal channels; new bodies and mechanisms were used to manage these demands
In the spring of 2020, Statistics Canada faced significant challenges including adapting to telework and new internal processes, while at the same time contending with major shifts and increases in user demands for relevant COVID-19 data. Facing the influx of data demands, time constraints and decentralized mechanisms for gathering users' needs presented challenges at times. Internal interviewees emphasized that, during the early stages of the pandemic, the pressure to produce information fairly quickly left little or no time to proactively reach out to users about their data needs. It was also noted that, had time constraints been less urgent, it would have been advantageous to actively consult a broader base of users.
Nonetheless, the evaluation found that most users had opportunities to communicate their needs to Statistics Canada, particularly through their existing contacts within the agency and ad hoc correspondence (e.g., meetings, emails, phone calls) at all levels and areas of the organization. Well-established networks with health partners (e.g., partnerships in the production of vital statistics) and formal structures (e.g., at the same table, council) were also leveraged to share information on data needs.
Early in the pandemic, the emphasis on informal mechanisms to understand users' needs before and after product releases presented some challenges for information management and prioritization (e.g., survey content had to be revised after users reached out with demands to senior management, at times slowing work). Many of these challenges were mitigated through new bodies and mechanisms to manage users' demands, such as the Social Impacts and New Data Development Working Group. This working group was noted as a key factor in the successful delivery of COVID-19 products. It brought together assistant directors, directors and subject matter experts from across the organization to better identify and prioritize critical data needs about the social impacts of the pandemic, including health statistics that would be filled through crowdsourcing and probability panel methods. It served as a funnel to screen and prioritize users' demands, and communicate needs and decisions to users. This working group enabled processes to prioritize demands, raise awareness across participating divisions, and exploit opportunities to solicit and share feedback from partners.
As Statistics Canada's pandemic response evolved, more structured and proactive mechanisms were used to engage users (e.g., COVID-19 Social Impacts Network, CITF, presentations to users, interdepartmental working groups and committees). In regard to COVID-19 related health statistics, these mechanisms were used alongside ad hoc communication to engage more frequently with partners from provincial and territorial governments, partners at federal departments and experts within academia.
While most users were satisfied with the level of responsiveness and communication from Statistics Canada, some opportunities for improvement were noted
Based on external interviews, the majority of users and partners had the opportunity to share their needs with Statistics Canada and were satisfied with the agency's level of engagement. They were satisfied with the effectiveness of the well-established relationships with Statistics Canada and noted how fast and responsive the agency was to respond to their inquiries. Health partners, specifically, commonly discussed their strong and mutually beneficial relationship with Statistics Canada, the value of its expertise and an increase in the overall level of collaboration during the pandemic; they hoped the high level of responsiveness would continue in the future.
However, some challenges were identified by key users. For example, it was difficult at times to find the right contact or identify the right mechanism to communicate their needs to the agency, especially among those who did not have well-entrenched relationships with Statistics Canada. Additionally, though they acknowledged the time pressures, some users stated that they would have desired a more proactive, formalized and strategic approach to engagement. These users also pointed out that, rather than a one-time conversation, engagement and communication should be actively sought throughout product life-cycle, from content development, to proactive communications around product releases, to seeking feedback from users after product releases. A few of these users suggested that, going forward, an engagement strategy or a dedicated engagement group would be beneficial. Overall, there was a general sense from users that their expectations have been raised for the future.
Both internal and external interviewees highlighted that health information in Canada relies on a complex governance structure involving a number of players and stakeholders, of which Statistics Canada is one. Collaboration at the federal level plays a crucial role in Statistics Canada's production of health data via subject-matter knowledge sharing and funding of some data and surveys. The agency works in partnership with provincial and territorial vital statistics registrars and cancer registries, as well as data providers and users at the federal level (Health Canada and PHAC), provincial level (provincial ministries of health), regional level (health regions) and with many other partners (e.g., Canadian Institute for Health Information). Jurisdictional complexities and a fragmented data environment contribute to challenges in collaboration among Canadian health data partners.
Many interviewees noted that the production of COVID-19 data as a whole highlighted the need for a more coordinated approach in presenting public health data to Canadians. For instance, several federal health partners discussed how they would have benefited from clearer communication about products that were planned or being released by Statistics Canada. Partners suggested that this could prevent a duplication of efforts and give them an opportunity to provide the agency with more of their data needs as well as their subject-matter expertise. Moreover, several partners discussed challenges in clarifying roles and responsibilities between federal health partners in the production of pandemic-related health statistics. Additionally, most partners interviewed perceived missed opportunities for data sharing that would have enabled them to provide insights and receive more advanced notice prior to releases. Some expressed a continued need to improve governance processes around collaboration. For example, the need to address jurisdiction and data-sharing challenges, including those between Statistics Canada and other governmental departments. For instance, many internal and external interviewees discussed challenges pertaining to vital statistics data collected by provinces and territories and shared with federal partners, and the importance of strengthening these partnerships and improving IT infrastructure.
Statistics Canada played a key role in meeting evolving needs—products released were used for policy analysis, modelling, decision-making and the creation of knowledge products
The evaluation found that most users felt that the new products were useful, they filled information gaps, or otherwise strengthened the evidence base from which users could inform their work and make decisions. These products also provided an overall national picture of the impacts of the pandemic and allowed for comparisons across jurisdictions at a high level. Crowdsourcing and probability products were used for briefing purposes, policy research and analysis, program and service planning and decision-making, and to a lesser extent, for creating knowledge products and modelling. Key users of COVID-19 related health statistics also used the information for briefing purposes and for creating knowledge products for other audiences, modelling, and to a lesser extent, program and service planning, decision making and public health surveillance and health system monitoring.
Because the impacts of COVID-19 go beyond the effects of the virus, users wanted data on both the direct and indirect impacts of the pandemic, especially in relation to identity factors (e.g., ethnicity, gender) and intersectionality. Based on interviews, users were interested in products related to various health and social impacts of COVID-19, including vital statistics and mortality, mental health and well-being, impacts on families, impacts on health services and other unintended consequences of COVID-19 response measures (e.g., employment, school closures).
As depicted in Appendix B, webtrends data from the Statistics Canada website indicate that many of the topics that users' needed information on were well aligned with the most popular probability panel and crowdsourcing products. Additionally, these webtrends data also showed a significant increase in demand for vital statistics and mortality data, compared with historical use (these products made up 6 of the top 10 products from April 2020 to March 2021); COVID-19 epidemiological data were also highly viewed relative to other Statistics Canada data products.
Statistics Canada's products were largely considered unique by users; however, they wanted more granular data, particularly regarding at-risk and marginalized populations
The evaluation found that external users readily used Statistics Canada products and noted that they represented the best source of information available, particularly in the early days of the pandemic. While some complementary products were available from other organizations, Statistics Canada's new products served a unique purpose for most users. In some cases, users said new Statistics Canada information was used as a supplement to other sources (e.g., cross-referencing with member feedback, other studies) or incorporated into an index, suggesting relevance and usefulness even when alternatives existed.
Further, the evaluation found that users felt there was minimal duplication between information produced by Statistics Canada and that produced by other organizations. Many used the agency's products because they were the highest quality or the only information available, especially early on. Although health data are produced by a variety of jurisdictions, users noted the uniqueness and value of the national level COVID-19 related health statistics provided by Statistics Canada. Many external users from different types of organizations noted that they use new Statistics Canada information despite any limitations because it was the best source available or there were no alternatives.
However, the evaluation found that there were a number of areas in which external users sought additional data related to COVID-19. Even within the topics covered by Statistics Canada's products, many users identified areas in which they saw data gaps. For example, almost half of key users of COVID-19 related health statistics discussed gaps pertaining to impacts of the pandemic on subgroups (e.g., individuals with chronic conditions, seniors, caregivers). Users of probability panel and crowdsourcing products also highlighted gaps pertaining to the impacts on children and families in regard to learning, parenting, child care and violence at home.
Users frequently identified a desire for greater disaggregation to address their specific information needs, especially to conduct equity analyses as well as to support more local or population-specific program and service planning. Many users discussed the need to have more granular information for areas such as at-risk and marginalized populations (e.g., Indigenous groups), and in particular how COVID-19 was impacting these populations. Some also noted their understanding of the limited coverage on marginalized populations, partly because of the challenges in reaching some populations, particularly for crowdsourcing, which is open to the public but requires Internet access.
Additionally, some users were interested in more data that would track the impacts of the pandemic over time; several users mentioned a need for probability panel and crowdsourcing products to track trends over time and follow up on previous surveys. More generally, some users discussed a need for more data that track changes in social measures and well-being.
Methodologies used by Statistics Canada, as well as their related strengths and limitations, were understood by users; some opportunities for improvement were noted
The evaluation found that a majority of users consulted were aware of the strengths and limitations of crowdsourcing, probability panel and health statistics. Methodological information, including limitations, was documented and published in The Daily with the releases of the products. Users were also kept informed through their engagement with Statistics Canada's employees. Methodological information released by the agency was also included in briefings for decision-makers.
Some opportunities for improvement were noted regarding the communication of methodological information. Users desired more information about limitations in simple language, particularly about the lack of demographic disaggregation. It was also noted that methodological information for COVID-19 related health statistics was less apparent than usual and that labelling crowdsourcing as a "survey" created some confusion at the beginning.
A few users chose not to use crowdsourcing data because of a lack of representation, comparability and disaggregated data (e.g., limited coverage of marginalized populations, caused partly by challenges in reaching some populations with limited Internet access). Despite these limitations, the majority of users were satisfied overall with the crowdsourcing and probability panel methodologies used, given the lack of alternatives to produce similar data in a timely manner.
Based on the experience of using crowdsourcing data during the pandemic, users foresee value for the continued use of this approach. Although they would prefer an increased use of probabilistic samples where possible, many would consider continuing to use crowdsourcing for "pulse checks" and urgent informational needs. To increase the participation of marginalized populations in crowdsourcing, some users suggested a combined approach with a telephone option, as well as partnering with the organizations that serve marginalized populations to extend reach. In addition, users indicated that crowdsourcing could grow with more guidance and controls (e.g., when its use is most appropriate given the limitations). As a majority of users consulted were not able to comment extensively on probability panels, there may be an opportunity to raise awareness about this tool.
Generally, users were aware of Statistics Canada releases and satisfied overall with both the timeliness and the accessibility of data
Timeliness was of particular importance to users, given the need to rapidly adapt and respond to the uncertainty of the evolving landscape during the pandemic. The evaluation found that COVID-19 products were considered timely or at least somewhat timely, though health users wanted even timelier data. Almost all users acknowledged the fast turnaround of results and general timeliness of the agency's response. For health statistics, there was a high need and expectation for timely and frequent (even "real-time") health information. This was partially driven by users benchmarking against other data providers and statistical agencies. However, it is important to note that a key contributing factor beyond Statistics Canada's control was the delay in receiving data from provinces, territories and PHAC for vital statistics and epidemiological reports. Overall, users expressed gratitude for the fast turnaround of products.
Accessibility and product awareness were also important for users and most felt they had a good overall awareness of the data and found it accessible. Most users accessed the new products through the website and encountered no barriers, although several expressed a need for the improvement of website navigability.
Some health partners expressed a need for earlier and enhanced access to unpublished data through data-sharing agreements, especially those involved in the production of the data. These partners wanted improved and earlier communication about what products would be released and when.
3. Moving forward
Evaluation question
What factors should be taken into consideration moving forward?
Summary
The evaluation assessed the extent to which there is an ongoing need for COVID-19 social and health statistical products and the factors that must be taken into account going forward to deliver both those products and regular programs. Users indicated an ongoing need for health statistics related to COVID-19 as well as for crowdsourcing and probability panel products. As the pandemic evolves, more disaggregated data about the differential impacts of COVID-19 on different populations will be needed. The forward approach needs to balance demands arising from emerging data needs, such as disaggregated data, ongoing pandemic information requirements and ongoing program delivery. Opportunities remain to fully explore the lessons learned from COVID-19.
While users foresee a continued need for current COVID-19 products, their focus has started to shift towards data needs to monitor pandemic recovery
Key users indicated an ongoing need for health statistics related to COVID-19, such as vital statistics, and data on topics such as mental health, vaccines, immunity and impacts on health services (e.g., health workforce, wait times). They also foresaw a need for products deriving from crowdsourcing and probability panels as the pandemic evolves. Crowdsourcing in particular could be leveraged to fill urgent social data needs in the absence of probabilistic alternatives.
Key users identified the following emerging topics mainly focusing on some of the longer-term impacts of the pandemic and its recovery:
- greater understanding of health impacts related to COVID-19, including long-haulers
- information on COVID-19 vaccination including uptake, readiness and hesitancy
- longer-term mental health outcomes and impacts
- impacts on health services, including wait times, diagnoses and the health workforce
- information on labour market changes, such as skills in demand and changes in location of work.
In addition, more disaggregated data on differential impacts of COVID-19 on different populations will be needed as the pandemic evolves; other methods, such as longitudinal studies and data linkage, could be leveraged to generate new insights.
Statistics Canada is aware of the current data needs and intends to fill some of these gaps with a number of initiatives (e.g., creation of the SCMH, adapting the Canadian Community Health Survey [CCHS] to the COVID-19 context, planning an omnibus survey, leveraging the Disaggregated Data Action Plan, ongoing discussions with different stakeholders).
Forward approach needs to balance demands arising from emerging data needs, such as disaggregated data, ongoing pandemic information requirements, and ongoing program delivery; several key enablers for success were identified
Statistics Canada's rapid response to COVID-19 underscored its ability to meet new data needs during an unprecedented period. The COVID-19 statistical products released were largely considered relevant, timely and useful. Releases of related health statistics were accelerated and experimental approaches (i.e., crowdsourcing and probability panels) were leveraged to deliver quick results. Overall, users were satisfied with these products given that they provided rapid information during a period of crisis.
To release the COVID-19 products in a timely manner, a significant increase in workload was necessary, which resulted in a sense of fatigue further exacerbated when work resumed for activities and initiatives previously put on hold. This sense of fatigue was also felt by corporate partners, given their essential supporting role in the rollout of the COVID-19 products. In addition to these and regular activities, a number of new initiatives (e.g., creation of the SCMH, adapting the CCHS to the COVID-19 context, planning an omnibus survey, leveraging the Disaggregated Data Action Plan) are at their initial stages which will further add to the workload of divisions, including supporting divisions of corporate partners.
In light of this context, the forward approach must balance demands arising from emerging data needs, such as disaggregated data, ongoing pandemic information requirements, and ongoing program delivery. It should also take into consideration the increased level of effort required by divisions and corporate partners supporting divisions. Prioritization of demands, human resources (HR), ongoing innovation and data sources, user engagement, and internal collaboration were identified as key enablers during internal interviews.
Chart 1: Key enablers to balance demands arising from emerging data needs
Prioritization of demands
As the pandemic evolves, user's data needs will also evolve (e.g., data to monitor recovery) and a robust approach to prioritization that effectively balances arising demands will be needed. The approach to prioritization will also need to consider the capacity of corporate partners (e.g., IT, methodology, collection and communications).
Human resources
HR management will be crucial to maintaining and enhancing staff expertise and skills. Timely delivery of relevant COVID-19 products was, in large part, attributable to the expertise of staff. A robust approach to HR management will be needed to build on new opportunities and approaches, such as access to a Canada-wide talent pool and the use of agency-wide resources for specific COVID-19 data projects. It will also be necessary to continue efforts targeting the well-being of staff and good mental health (e.g., recognition, work-life balance).
Ongoing innovation and data sources
Ongoing innovation and acquisition of new data sources to continue to meet the evolving needs of users will require that business processes and methods are revisited and streamlined to create efficiencies.
Modernization, machine learning and artificial intelligence should be considered to provide data to meet unpredictable and changing needs. The pandemic demonstrated the value of administrative and non-traditional data to increase timeliness and coverage.
User engagement
Key partners will need to be continuously engaged to ensure that they are given the opportunity to be informed during the data production process. It is necessary to ensure that users have opportunities through a variety of mechanisms to share their needs on an ongoing basis, including consultation after a product is released.
Internal collaboration
To build on the agency-wide successes in response to COVID-19, it is necessary to continue to collaborate, work horizontally and share knowledge across the agency to increase efficiencies and reduce duplication of work.
Opportunities remain to fully explore the lessons learned from the pandemic to date
Management has taken steps to leverage some lessons learned from the pandemic to drive their strategic objectives and priorities for the next year—for example, through annual planning exercises at branch levels. However, given that Statistics Canada is continuing to operate in an evolving pandemic, divisions involved in with products related to COVID-19 have not had the opportunity to conduct a full-fledged assessment of lessons learned, aimed at finding approaches to improve core activities and benefit the whole agency. Staff recognized the need to identify such lessons and find approaches to improve core activities to ensure that the organization as a whole builds on its pandemic-related experience to create synergies for the future. The identification of lessons learned related to HR management, supporting tools and resources as well as well-being of employees was identified as critical. Some preliminary key lessons learned in the areas of collaboration, engagement, innovation, process changes, resource allocation and future of work were identified during internal interviews, as follows:
- Continued collaboration and coordination within the agency are important to ensure clear communication, direction, roles and responsibilities.
- More active and ongoing engagement with external partners will be crucial to identifying outstanding and emerging information gaps and needs, and maintaining or establishing new collaborative relationships.
- Innovation supports increased timeliness and responsiveness to needs, but better tools could further increase timeliness while reducing HR needs.
- Process changes that removed lower value steps or allowed for reasonable risks are necessary to help increase timeliness and address resource constraints.
- Teleworking has many benefits, such as increasing employee productivity and facilitating employee attraction and retention.
- Flexible reallocation of resources to areas of priority allows the agency to be nimble in responding to pressing needs; however, considerations must be made for some of the limitations and challenges of this strategy (e.g., finding qualified candidates, competing for same candidates across the agency, time needed to train resources, overtime requirements).
Leveraging these lessons learned would help the organization further improve its COVID-19 data response and core activities. Lessons learned could also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to the pandemic.
How to improve the program
Responsiveness to users' needs
Recommendation 1:
The Assistant Chief Statistician (ACS), Social, Health and Labour Statistics (Field 8), should ensure that a comprehensive strategy is developed to identify, prioritize and respond to emerging data needs (i.e., disaggregated data, ongoing pandemic information requirements and ongoing program delivery).
The strategy should consider the key enablers identified (i.e., prioritization of demands, human resources, ongoing innovation and data sources, user engagement, and internal collaboration) and include approaches and plans that set out how:
- emerging data needs will be identified (e.g., stakeholder engagement approaches);
- data gaps will be addressed;
- data needs will be prioritized based on intended outcomes and resource capacity; and
- ongoing monitoring of data needs will be carried out.
Moving forward
Recommendation 2:
The ACS, Social, Health and Labour Statistics (Field 8), should ensure that a lessons learned exercise from the COVID-19 pandemic is conducted to identify approaches that could be applied to respond to COVID-19 data needs and improve core activities going forward.
- The lessons learned should be presented to a Tier 1 committee that will provide direction on their potential implementation across the agency, as applicable.
- They should also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
Management response and action plan
Recommendation 1:
The Assistant Chief Statistician (ACS), Social, Health and Labour Statistics (Field 8), should ensure that a comprehensive strategy is developed to identify, prioritize and respond to emerging data needs (i.e., disaggregated data, ongoing pandemic information requirements and ongoing program delivery).
The strategy should consider the key enablers identified (i.e., prioritization of demands, human resources, ongoing innovation and data sources, user engagement, and internal collaboration), and include approaches or plans that set out how:
- emerging data needs will be identified (e.g., stakeholder engagement approaches);
- data gaps will be addressed;
- data needs will be prioritized based on intended outcomes and resource capacity; and
- ongoing monitoring of data needs will be carried out.
Management response
Management agrees with the recommendation.
Management will respond by strengthening its existing Field strategic plan and will also address the recommendation through the development of the Disaggregated Data Action Plan, Quality of Life Indicators Framework and the Health Care Access, Experiences and Resulting Outcomes Initiative.
The Field, and each centre of expertise within the Field, has a comprehensive, evergreen strategic plan updated annually for ongoing monitoring of data needs. These plans focus on the Field's four pillars to ensure emerging data needs are identified, addressed and prioritized based on outcomes and resource capacity. These strategies include:
- A strategic engagement plan to ensure meaningful and purposeful outreach and engagement with stakeholders, experts and Canadians to better understand data needs;
- Development of centres of expertise with an HR plan that ensures the centres have a vibrant and engaged subject matter workforce with the ability to prioritize needs and resources and can monitor the evolution of data needs;
- A modernization strategy focused on the development of the new social data integration platform, including integration and onboarding of existing programs to the new platform, and the acquisition of new and alternative sources of data;
- Expansion of "horizontal work" in order to address data gaps with the integration of data from multiple domains and the development of intersectional analyses to arrive at more comprehensive insights.
Deliverables and timelines
Updated Field 8 Strategic Plan (April 30, 2022).
Recommendation 2:
The ACS, Social, Health and Labour Statistics (Field 8), should ensure that a lessons learned exercise from the COVID-19 pandemic is conducted to identify approaches that could be applied to respond to COVID-19 data needs and improve core activities going forward.
- The lessons learned should be presented to a Tier 1 committee that will provide direction on their potential implementation across the agency, as applicable.
- They should also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
Management response
Management agrees with the recommendation.
Management will undertake discussions of lessons learned and documentation of follow-up actions, the results of which will be tabled at the Tier 1 committee and shared at field presentations across the agency.
The results of this exercise will be presented to the Modernization Management Committee to foster a fulsome discussion on these lessons learned and their potential implementation agency-wide.
Field 8 to tour the Field debriefs to present the new social data integration platform, which will include sharing of a use case on the development and implementation of web panels.
Deliverables and timelines
Results of lessons learned exercise presented to the Modernization Management Committee (April 30, 2022).
Field 8 presentations of the new social data integration platform, which will include sharing of a use case on the development and implementation of web panels, through field debriefs (July 31, 2022).
Appendix A – List products (and surveys) released
| Date of release | Source | Products |
|---|---|---|
| Vital statistics | ||
| May 13, 2020 (first release) | Canadian Vital Statistics – Death Database | Data: Adjusted number of deaths, expected number of deaths and estimates of excess mortality, by week Data: Weekly death counts, by age group and sex
|
| COVID-19 epidemiological reports | ||
| March 30, 2020 | PHAC | |
| November 12, 2020 | PHAC |
|
| Upcoming Surveys | ||
| To be determined (2021) Collection period: From November 2, 2020, to March 26, 2021 | Canadian COVID-19 Antibody and Health Survey (CCAHS) This survey asks questions about
|
|
| To be determined (2021) Collection period: September 11, 2020, to December 4, 2020 | Survey on COVID-19 and Mental Health (SCMH) The survey asks questions about
|
|
Appendix B: Webtrends
Description - Figure 1: Crowdsourcing and probability panel products webtrends between April and December 2020
The figure depicts the total page views of crowdsourcing and probability panel analytical products from April to December 2020.
- Economic impact of COVID-19 among visible minority groups: 6,737
- Indigenous people and mental health during the COVID-19 pandemic: 5,179
- Gender differences in mental health during the COVID-19 pandemic: 4,678
- Food insecurity during the COVID-19 pandemic: 10,224
- Mental health of Canadians during the COVID-19 pandemic:,7,263
- Canadians report lower self-perceived mental health during the COVID-19 pandemic: 5,761
Based on total page views, crowdsourcing and probability panel analytical products that were the most viewed reflect the topics that were mentioned by users.
| Rank in 2020/2021 | Title | Page views in 2020/2021 | Page views in 2019/2020 | % increase in popularity |
| 1 | Population estimates, quarterly | 489,857 | 115,815 | 423% |
|---|---|---|---|---|
| 2 | Leading causes of death, total population, by age group | 451,415 | 115,890 | 390% |
| 3 | Population estimates on July 1, by age and sex | 403,248 | 284,442 | 142% |
| 5 | Deaths, by month | 319,175 | 39,173 | 815% |
| 6 | Deaths and mortality rates, by age group | 254,569 | 50,605 | 503% |
Vital statistics made up 6 of the top 10 most viewed data tables from Statistics Canada's website in 2020/2021.
| Rank in 2020/2021 | Title | Page views in 2020/2021 |
| 10 | Detailed confirmed cases of coronavirus disease (COVID-19) (preliminary data), PHAC | 143,148 (data only available for two months: April and May 2020) |
|---|---|---|
| 20 | Detailed preliminary information on confirmed cases of COVID-19 (revised), PHAC | 87,010 |
| 23 | Detailed confirmed cases of coronavirus disease (COVID-19) (preliminary data), Canada | 72,728 (data only available for two months: April and May 2020) |
COVID-19 epidemiological data were within the top 25 most viewed data tables from Statistics Canada's website in 2020/2021.