Evaluation of the Health Statistics Program
(2011/2012 to 2015/2016)

Evaluation Report

February 2017

Executive summary

1. Context
2. Findings: Relevance
   • 2.1 Continued need for the Health Statistics Program
   • 2.2 Alignment with government and key partners' priorities
   • 2.3 Consistency with federal roles and responsibilities
3. Findings: Performance
   • 3.1 Accuracy
   • 3.2 Timeliness
   • 3.3 Accessibility
   • 3.4 Coherence
   • 3.5 Interpretability
   • 3.6 Relevance
   • 3.7 Use of the statistical information
4. Findings: Economy and Efficiency
5. Conclusions and Recommendations
6. Management Response and Action Plan

Executive summary

The Evaluation and Professional Practices Division (EPPD) of the Audit and Evaluation Branch of Statistics Canada conducted an independent assessment of the Health Statistics Program (HSP), including, exclusively, the activities of the Health Statistics Division (HSD). The results provided objective information and recommendations to support program improvement and inform future decisions by Statistics Canada and Parliament with respect to the government's management of resources and programs.

The evaluation was designed and conducted in accordance with the TB Policy on Evaluation (2009)^{Footnote 2} and the associated directive and standards, and fulfills an accountability commitment set out in Statistics Canada's Risk-based Audit and Evaluation Plan (2013/2014 to 2015/2016).

Findings

Overall, the Health Statistics Program (HSP) has generally met the needs and priorities within its budget, and generated the expected outcomes, namely providing access to high-quality, relevant statistical information to support public debate, decision-making and research.

Relevance: Evaluation evidence demonstrates a continued need for the program. As evidence is critical in a knowledge-based society, there is a need for timely, accessible and relevant statistical data on the health of Canadians.

Delivery of the HSP by the federal government is consistent with its roles and responsibilities as it enables global action on social determinants of health to foster social justice and equity in health. Statistics Canada is one of the five federal government organizations that supports the federal government in its health mandate. Moreover, the health and well-being of Canadians have been identified as a priority in every federal budget from 2011 to 2015. The 2011 Speech from the Throne underlined the importance of collaboration between key partners. Evaluation findings confirmed the alignment of HSP activities with the most pressing needs and priorities in HSP areas of responsibility.

Delivery of the HSP by the federal government is consistent with its roles and responsibilities as it enables global action on social determinants of health to foster social justice and equity in health. Statistics Canada is one of the five federal government organizations that supports the federal government in its health mandate. Moreover, the health and well-being of Canadians have been identified as a priority in every federal budget from 2011 to 2015. The 2011 Speech from the Throne underlined the importance of collaboration between key partners. Evaluation findings confirmed the alignment of HSP activities with the most pressing needs and priorities in HSP areas of responsibility.

Performance: The potential impact of the program is directly correlated with the quality and relevance of the statistical information that is available. In this regard, the evaluation reports on each dimension of quality of the statistical information available, from the accuracy of the information to the extent of its use.

• Accuracy: The evaluation findings confirmed that the HSP statistical information is accurate despite challenges related to declining response rates. The exclusion of key demographic groups and geographic regions have been noted as a possible limitation to the generalizability of research and development of appropriate policies and programs focusing on those groups and regions.
• Timeliness: Overall, the HSP encountered certain challenges in meeting release dates, notably with vital statistics, cancer products and CCHS–Nutrition. However, the products for two of the main components, the CCHS–annual and the CHMS, were mostly released on time. The timeliness issue impacted user satisfaction and the responsiveness of the HSP.
• Accessibility: The evaluation findings confirmed that accessibility, in terms of availability of products and means of access, has increased and improved over time, and that products are of interest. There is a general sense that mechanisms for information dissemination are functional—they meet users' basic needs. However, it was suggested that more can or should be done by Statistics Canada to facilitate access. Key informants raised issues related to accessibility, including those associated with confidentiality/privacy clauses within the Statistics Act, data sharing agreements (DSAs), research data centres (RDCs) and Statistics Canada's website.
• Coherence: Although some decisions about changes in surveys were made over time, there are no issues with the coherence of the HSP statistical information. Additional information is provided to support user information needs as they relate to changes and their impact on coherence. HSP management demonstrated commitment in managing coherence. The trade-off between coherence and relevance is managed adequately in the decision-making process.
• Interpretability: The evaluation findings confirmed that the interpretability of the HSP statistical information is deemed adequate, most notably with high quality metadata and documentation. The HSP has provided data users with the required user documentation.
• Data relevance: The analysis demonstrated that the HSP consults with its stakeholders to identify the changing or emerging needs and requirements of users, as well as potential information gaps, and addresses them accordingly. The evaluation found that the HSP as a whole, including its specific survey programs and administrative datasets, is able to provide health statistical information that meets stakeholders' information needs and priorities. However, while continued efforts are deployed by the HSP to meet stakeholders' needs via consultations and reviews, some gaps in alignment with needs were identified by the evaluation. The more frequent and emerging needs expressed were in regard to the concept of big data and determinants of health, lack of geographic data, increased need for administrative data linkage and longitudinal data. Other identified needs were for:
  • research on health equity, social determinants of health, and the broader concept of wellness;
  • data, more data or more frequent data collection on specific topics, including environmental exposure, pharmaceutical use, disabilities, mental health, institutionalization, nutrition (primarily consumption patterns), health needs of refugees, health needs of veterans, aging, birth defects, cancer screening, children's health, and First Nations health;
  • data/research that captures health outcomes;
  • patient-oriented research;
  • larger survey sample sizes, and the inclusion of key demographic groups and regions; and
  • technologies for human measurement (e.g., technologies for measuring chemical exposure in humans, genotyping).

Use of statistical information: Overall, the HSP data are used by all levels of government to fulfill their mandates and responsibilities, by researchers, by international organizations, by non‑governmental organizations, and by Canadians. At Statistics Canada, the HSP data are used as inputs to other statistical and research programs.

Efficiency and Economy: The evaluation's evidence confirms HSP management's commitment and good practices to offer quality outputs that meet overall needs and priorities. However, some issues related to partnership with key federal players and collaboration with provinces/territories as well as a decline in response rates and technological challenges have been identified as key factors that affect efficiency and program outcomes (i.e. timeliness).

Preliminary results of the pilot project launched in 2013/2014, to address the recommendations of the previous CHMS assessment appear promising. The project collects financial information by survey cycle based on processes and activities to improve measurement of efficiency. The results confirmed that the approach will make it possible to analyze costs of one survey cycle against another cycle by survey process phase.

Conclusions and Recommendations

Overall, the HSP has generally met the needs and priorities within budget, and generated the expected outcomes, namely providing access to high-quality, relevant statistical information to support public debate, decision-making and research.

Based on the evaluation findings, there is a recognition that the program must also ensure that the relevance of the statistical information is maintained by focusing on needs analysis versus capacity (surveyed populations, health determinants, collection frequency for certain themes). Lastly, the challenge of declining response rates with respect to accuracy must remain a priority for planning.

The evaluation's findings also show that the HSP encountered challenges in meeting release dates with vital and cancer statistics.

Recommendation 1:

Performance (timeliness)

Within the context of the relationship with provincial and territorial data providers, it is recommended that the Health Statistics Program take measures to improve the timeliness of the vital and cancer statistical information in order to meet target release dates and fulfill its mandate.

Issues were raised with respect to the current relationship with federal partners, particularly: 1) a perceived deterioration of the partnership in recent years, which has come to resemble a client relationship; 2) a lack of involvement and information sharing in the decision-making process; and 3) challenges with respect to data sharing agreements.

Recommendation 2:

Performance and efficiency (partnerships)

It is recommended that the Health Statistics Program work with its federal partners to improve the partnership approach. Particularly, this includes engaging with partners to review the current governance structure with respect to roles and responsibilities, the decision-making process, communications and expectations for data sharing.

1. Context

As Canada's national statistical agency, Statistics Canada plays a fundamental role in generating information on Canada's socioeconomic conditions. Statistical data are used to support government decision-making at all levels by describing changing events, enabling analysis of policies and programs, and monitoring program outcomes. Some data produced by Statistics Canada are also required by legislation.

The Health Statistics Program

The mandate of the Health Statistics Program (HSP) is to provide accurate, timely and relevant information about the health of Canadians. The Health Statistics Division (HSD), in collaboration with Statistics Canada's Health Analysis Division (which is not included in the scope of this evaluation), provides statistical analysis and information about the health of the population, the determinants of health, and the scope and use of Canada's health care resources. This information is used to assist and support health planners and decision-makers at all levels of government, to sustain demographic and epidemiological research, and to report to the Canadian public about their collective health and health care system performance.

HSD works in partnership with provincial and territorial vital statistics registrars and cancer registries, as well as data providers and users at the federal level—Health Canada and the Public Health Agency of Canada (PHAC)—provincial level (provincial ministries of health) and regional level (health regions). The program also works with many other partners, including the Canadian Institute for Health Information (CIHI) to provide statistical information on, and undertake research and analysis on the health status of Canadians and the health care system.

The HSP's activities can be divided into the following three broad areas by key survey and data product^{Footnote 3}:

• Statistical and Administrative Programs
  • Canadian Community Health Survey (CCHS)
  • Canadian Health Measures Survey (CHMS)
  • Vital Statistics
  • Cancer Statistics Program – Canadian Cancer Registry (CCR)
• Products and Services
  • Integration and Analysis
  • Client Services and Dissemination
• Special Projects and Initiatives
  • National Routing System (NRS)
  • Canadian Coroner and Medical Examiner Database (CCMED)
  • Neurological Survey Program
  • Survey of Living with Chronic Diseases in Canada
  • Long Term Care Facilities Survey (LTCFS)
  • Canadian Health Survey on Children and Youth

About the Evaluation

An evaluation was conducted as a neutral assessment of the relevance and performance of the HSP, in accordance with the requirements of the Treasury Board of Canada's Policy on Evaluation (2009).^{Footnote 5} The objective of this evaluation was to assess whether HSP provides quality key statistical information that is being generated efficiently and that responds to user needs.^{Footnote 6}

The evaluation covered the period from April 1, 2011, to March 31, 2016, focusing on HSP's activities and associated custom products and services. A mixed-method approach, integrating quantitative and qualitative evidence, was used with a total of 15 evaluation questions as part of the strategy to assess the core issues related to the relevance and performance of the program.^{Footnote 7} Multiple lines of evidence were employed, including

• document and literature reviews;
• an administrative data review;
• key informant interviews with 58 internal and external stakeholders from the public sector [federal, provincial and municipal], the private sector, non‑governmental organizations and academia;
• a survey of 325 data users who were knowledgeable about the program and its products; and
• a bibliometric and webometric study.

Logical and statistical analysis were used, and findings were correlated to identify trends and patterns. Data from these sources were analyzed and triangulated to develop the conclusions and recommendations.

The evaluation encountered limitations, which were mitigated with the strategies described below.

2. Findings: Relevance

This section summarizes evaluation findings that emerged from the data collection process. It focuses specifically on the relevance of the Health Statistics Program (HSP) by exploring the demonstrable need for the program, its alignment with government priorities, and its consistency with the roles and responsibilities of the federal government.

2.1. Continued need for the Health Statistics Program 

Summary

Health, as defined by the World Health Organization, is “a state of complete physical, mental and social well-being and a resource for everyday life.”^{Footnote 8} In contemporary knowledge-based societies, health has been recognized as a key factor for economic prosperity, a human right, a global public good, and an issue of social justice and equity. Consequently, “governments, businesses, communities and citizens increasingly engage in governance for health.”^{Footnote 9} Since evidence is critical in a knowledge-based society, there is a need for timely, accessible and relevant statistical data on the health of Canadians for measuring and reporting on public interventions, supporting evidence-informed action, designing interventions, and providing accountability.^{Footnote 10} As noted in the Report from the Fourth Consensus Conference on Evaluating Priorities for Canada's Health Indicators, “with respect to health data, the mandate of Statistics Canada is to focus on pan-Canadian population data (as opposed to data collection based on interventions or care delivery). Within this mandate, the objective is to have high-quality data so that the indicators and measures developed from them have strong integrity.”^{Footnote 11}

Furthermore, “the understanding that health is determined as much or more by social, economic, environmental and cultural factors than it is by genetic or medical factors”^{Footnote 14} means that, in order to inform action on reducing the health inequalities, there is a need for health data linked to factors such as income, level of education, occupation, social hierarchy and housing. Moreover, since these factors relate closely to Canadian communities and influence diverse population groups, there is a need for health data at a lower level (small geographies and populations), in order to support activities and interventions at regional levels.

Last, but not least, “to ensure that Canadians continue to have access to high-quality health care, and benefit from effective health policies, the country's health researchers and system innovators need to make effective use of health and health-related data, including administrative health and social data. This need will increase in the future as technology continues to develop and digitized data […] become ever more abundant.”^{Footnote 15}

In the future, the need for health information is expected to increase, as implied by the conclusions and recommendations of a report by the European office of the World Health Organization (WHO)^{Footnote 16} on the new health policy Health 2020. As stated in the report, “evidence is critical in a knowledge-based society. […] Similar to other fields of governance, independent expert bodies, such as federal agencies, commissions, regulators and auditors, are playing increasingly vital roles in providing evidence, watching ethical boundaries, extending accountability and strengthening democratic governance in health, as related to privacy, risk assessment, quality control and health technology and health impact assessment. Their importance increases as one moves to a knowledge-based society with more rapid innovation. The literature also indicates, however, that metrics must be improved, for example, by including both objective and subjective measures, to capture what is happening to most people and to the most disadvantaged people.” Furthermore, the report mentions the future trend for the increased role of health promotion, and consequently, the “joint action of health and non-health sectors, of public and private actors and of citizens for a common interest.”

Health information is needed not only by organizations directly responsible for health and health governance, but also by diverse groups of users, including individual citizens. One of the recommendations of the WHO report, “A commitment to 'the informed citizen' and to citizen participation” explicitly talks about future needs for health information—implementation of new information and communication technologies, open-data initiatives and tracking systems that ensure better public accountability for health in all sectors; digital governance (e-governance) approaches as well as a comprehensive strategy to strengthen health literacy.^{Footnote 17}

The mandate of the HSP to provide accurate, timely and relevant information and analysis about the health of Canadians is relevant to inform various aspects of the health sector for organizations directly responsible for health and health governance, as well as for diverse groups of users, including individual citizens.

2.2. Alignment with government and key partners' priorities

Summary

All federal budgets from 2011 to 2015, as well as the Speech from the Throne in 2011, 2013 and 2015 mention the health and well-being of Canadians as one of the priority areas. In addition, the 2011 Speech from the Throne^{Footnote 18}notes the importance of collaboration between the federal government and the provincial and territorial governments to ensure the sustainability of the health care system and accountability of results.

The HSP is responsible for key sources of information on population health—such as population surveys, administrative programs and health indicators—which support health administrators in the federal, provincial and territorial governments, as well as in health regions. The information is used as a source of input for policy and legislation development by federal, provincial and territorial governments and by various health agencies, including the Public Health Agency of Canada (PHAC), Health Canada (HC), the Canadian Institute for Health Information (CIHI), the Canadian Institutes for Health Research (CIHR), the Canadian Partnership Against Cancer, the Canadian Cancer Society and the Vital Statistics Council. The HSP's role as one of the key national data providers of health information has been confirmed by the recently developed national Health Data Strategy, for which Statistics Canada has been identified as a leader for two of the key national priorities (children's health and the social domain record linkage environment) and as a contributor to filling most of the 19 nationally significant data gaps identified by the strategy.

Moreover, Statistics Canada's HSP has been collaborating with the other federal organizations which provide health information (HC, PHAC, CIHI and CIHR) in determining the priority areas and strategic directions for national health information and indicators. While the availability of, and access to, health data have improved dramatically in recent years, as a result of recent collaborative efforts, a number of challenges identified during the Roadmap Initiative^{Footnote 19} still persist, such as fragmented health data, incomplete data, inaccessible data and data analysis gaps.

The five organizations have collectively looked at health data availability, data gaps and how best to work together to address those gaps (Health Data Strategy). Consequently, in order to respond to needs and priorities, the following initiatives have been identified for the HSP at Statistics Canada. These initiatives align with the 2014/2015 Statistics Canada Corporate Business Plan as well as with the priorities raised by the Health Information Advisory Committee as follows:

• Facilitate access to Vital Statistics data by linking death data to the Discharge Abstract Database (DAD), which will reduce duplication of efforts between CIHI and Statistics Canada.
• Conduct a pilot survey on the health of children and youth in collaboration with Health Canada and the Public Health Agency of Canada.
• Make effective use of the new Social Data Linkage Environment (SDLE) to extend health record linkage to other social statistics domains such as education and justice. This new environment, now in production, will help reduce fragmented health data sources as well as increase the analytical potential of health data in responding to emerging policy needs.

2.3. Consistency with federal roles and responsibilities

Summary

Under the Statistics Act, Statistics Canada is required to “collect, compile, analyse, abstract and publish statistical information relating to the commercial, industrial, financial, social, economic and general activities and condition of the people.”^{Footnote 20} In support of this requirement, the HSP contributes to the federal government's “government affairs” outcome^{Footnote 21} by playing a role in Statistics Canada's two strategic outcomes: “Canadians have access to timely, relevant and quality statistical information on Canada's changing economy and society for informed debate, research and decision-making on social and economic issues,” and, “Specific client needs for high-quality and timely statistical services are met.”^{Footnote 22} To achieve this, the HSP is specifically designed to provide access to quality and relevant statistical information and associated services, which are used to inform public debate, research and decision-making.

The 2008 report of the World Health Organization (WHO) Commission on Social Determinants of Health (CSDH) appealed to national governments worldwide “to lead global action on the social determinants of health in order to promote social justice and health equity among all populations.”^{Footnote 23} Moreover, Pathak et al. (2012) summarized findings from previous research that “the fragmented structure of governance has created formidable barriers even to communication between local, provincial and federal governments” and argued that, to overcome structural and ideological barriers, “there is a need for a comprehensive national agenda to integrate efforts across all sectors and at all levels of governance to address social determinants at a system-wide level in Canada.” Globally, the literature review found evidence for this role, but it was also clearly demonstrated that the involvement of other jurisdictions and organizations (provinces, territories, regional and municipal governments, health-quality councils, and other health organizations) is crucial in addressing a field that cuts across multiple aspects of society—as does health.

In Canada, Statistics Canada and the Canadian Institute for Health Information (CIHI) jointly fulfill the role of producing national health data and indicators related to health. The distinction in the roles is that CIHI is mandated to lead the development and maintenance of comprehensive and integrated health information that enables sound policy and effective health system management,^{Footnote 24} while Statistics Canada (e.g., the HSP) provides information about the health of the Canadian population, the determinants of health, and the use of Canada's health care resources.

Based on the review of international publications, the national governments (e.g., the national statistical institutions) take on the role of producing health statistics. In Canada, this is done by the HSP (this includes, for instance, the core set of health indicators suggested by the WHO that are collected and used globally and nationally). As a comparison, in the United States, these functions are carried out by one centralized organization—the National Center for Health Statistics (NCHS) - which collects information on population health and the health care system.

Key federal partners and advisors were asked to comment on the appropriateness and clarity of the federal government's current role in providing health statistical information through the HSP. Overall, most of these key informants agreed that the role of the HSP is both appropriate and well-defined. However, nearly one‑half of these key informants expressed some confusion about how the roles of the various federal health information organizations, including Statistics Canada's HSP, differed from one another and, in particular, differed from the CIHI.

3.1 Accuracy

Summary

Overall, evaluation findings confirm the accuracy of the HSP statistical information resulting from adequate processes and program efforts and initiatives to overcome challenges. Furthermore, most key informants consider the HSP's statistical information to be highly accurate. For some key informants, this belief stems from their own personal scrutiny of the agency's information products (i.e., their determination that information products are free of significant errors or their perception that errors are corrected quickly when identified), while others cited Statistics Canada's expertise, reputation, attention to detail, or other factors (e.g., high response rates). The high level of interviewees satisfied with the accuracy of the HSP products confirm this trend.

The declining trend in the response rate of the CCHS–annual in the past 15 years (Figure 1) is one illustration of this phenomenon. The program is trying to mitigate the impact of this trend, for example, through outreach and promotion and by using a weighting methodology to adjust and correct non-response bias. More generally, Statistics Canada is considering other options, such as making social surveys mandatory, increasing monetary incentives to participate, contacting respondents by cell phone, and making surveys accessible online. The HSP is part of these efforts: in particular, after the Canadian Community Health Survey redesign, the survey will use improved sampling strategy and one sample frame, and will offer an electronic collection option (in 2017).

Figure 1: Canadian Community Health Survey response rates over time

Source: Statistics Canada (2015), new sampling and collection strategy in the 2015 Canadian Community Health Survey redesign, HMSD seminar June 30, 2015.

3.2 Timeliness

Summary

The HSP experienced some challenges meeting the timeliness targets, although this was also specifically singled out as a dimension of quality that key informants believe has improved in recent years. The two major components (CCHS–annual and CHMS) released their products mostly on time, while the releases of the Vital Statistics Program, Cancer Statistics, and CCHS–Nutrition have been delayed.

The CCHS–annual component releases were mostly timely, with 11 out of 14 releases on time from 2012 to 2015. The three delayed releases were in 2012 (two) and 2013 (one). The HSP innovated with the implementation of the Rapid Response files, which consist of cost-recovery questions collected over two or four months and are released three to six months after collection^{Footnote 29}for timeliness.

With regard to the CCHS focus component, some key informants viewed the frequency of data collection as problematic, particularly in the context of CCHS focus components related to nutrition and mental health that are undertaken several years apart from each other (e.g., mental health was the Focus Content Component for the 2002 and 2012 surveys, while nutrition was studied in the 2004 and 2015 surveys).

The CHMS releases were mostly timely with 10 out of 14 releases on time from 2012 to 2015. The four delayed releases were all in 2015. With the goal to release data as soon as possible, the program's management made a decision to release some analytical products (fact sheets and health report articles) later than their corresponding data.^{Footnote 30}

According to the Program Consultation Feedback Annual Report (2015), stakeholders have significant input into the ways that these surveys are developed and are pleased with this aspect. However, there are three years between the selection of a topic and the release of data and the stakeholders regularly question why everything takes so long.

In terms of vital statistics, births in the database were mostly timely, with 10 out of 15 releases on time from 2012 to 2015. The 5 delayed releases were in 2012 (3) and 2014 (2). In terms of the deaths database, none of the 6 releases from 2012 to 2015 were on time. Vital statistics had one of the highest levels of dissatisfaction in the user survey, at 22% (22/114) of respondents. The interviewees confirmed the same issue.

The timeliness of the vital events data has been identified as a risk in the 2011 Program Performance Report (PPR). The report states that the vital events have been routinely behind schedule due to inefficient data processing practices and the inability of some provinces to provide the data to Statistics Canada in a timely manner. Even though the National Routing System (NRS) has the potential to provide these data in almost real time, not all jurisdictions use the system. The consequences include the possibility of releasing partial (but timely) national data, or releasing the complete data at a later time.

Since 2011, Statistics Canada has been working on negotiating new agreements (including new funding models) with each province/territory to adopt the NRS as the mandatory mechanism for providing input to the agency. The program expects to conclude agreements with all jurisdictions by 2016/2017 instead of by 2012/2013 as initially planned.^{Footnote 31} In 2016/2017, the program is working on implementing the NRS for the territories, as well as developing messages for all provinces about submitting stillbirth data through NRS. Currently, five provinces are sending stillbirth data through the NRS, and work is under way in another jurisdiction.

Since 2012, there have been delays in the release of the death file further to a processing backlog resulting from the transition to the Social Survey Processing Environment (SSPE). The operational plan prescribes improvements of the data collection and data processing practices by working with data providers (provincial/territorial vital statistics registrars) and client service areas (Operations and Integration Division and Processing), and it was expected that by 2016 the program would put the release schedule on track (now scheduled for 2017). The improvement in timeliness of vital statistics (and specifically the death file) is important for Service Canada's requirement to provide more organized information and services to the families of the deceased. Delays in data releases also have an impact on stakeholders such as the Canada Revenue Agency.^{Footnote 32}

In an effort to improve timeliness, the HSP is working with Statistics Canada methodologists to assess the quality of the preprocessed data and whether it can be helpful as a type of flash estimate while the data are being processed. In addition, the U.S. National Center for Health Statistics is moving to quarterly data releases—HSP management is consulting with them to understand how they have overcome timeliness issues when dealing with their data providers (at the state level) to find out if they have any best practices that can be adopted.

None of the four cancer statistics data releases from 2012 to 2015 (one each year) were on time. The lack of timely releases of cancer statistics was reflected in the user survey: 27% (18/67) of respondents were dissatisfied with the timeliness of these releases. The interviewees confirmed the same problem. Several concerns related specifically to the timeliness of vital statistics and the Canadian Cancer Registry (CCR) were identified.

According to the HSP, the Cancer Registry data are high quality, however given the large number of partners and differences in the way the source data are processed and submitted, they do take time to produce. In addition, the unit has done some necessary retooling to processing, and although that will lead to gains in timeliness, the initial implementation took longer than planned and led to significant delays.

Lastly, some key informants mentioned that it would be helpful for Statistics Canada to take action that would reduce uncertainty associated with the timing of data collection and the release of information products—they suggested publishing a release schedule in advance and/or providing advance notice when delays are expected to occur.

3.3 Accessibility

Summary

The HSP's products are available to users in a number of formats. Primary access to these products is via the Statistics Canada website, where users can find such items as Daily articles, Canadian Socio-Economic Information Management System (CANSIM) tables, summary tables, publications and reference documents. HSP products can also be accessed using custom data tabulations, public use microdata files, research data centres (RDCs) (for microdata or aggregate data), Real Time Remote Access (RTRA) and, for some surveys, the Data Liberation Initiative (DLI).

For instance, the CHMS microdata data files can be accessed at the RDCs located in over 20 universities across Canada. CHMS data documentation, including rounded data dictionaries and the Researchers' Guide, is available to academics and researchers affiliated with over 75 university libraries.

The CHMS Biobank repository is available to Canadian researchers who successfully complete a review process. Selected researchers are granted restricted access to conduct tests and studies using anonymous biospecimens. As of 2016, four approved Biobank projects are in progress and one has been completed. The topics of the approved projects are posted on the Statistics Canada website.

Users can also access data from the CCHS–annual component using various methods such as:

Information on health indicators can be accessed through the health indicators main webpage, directly in CANSIM, through the rubric “What's New” on the Statistics Canada website, or through The Daily on release day. Outside the agency, the health indicators can be accessed from the master files at the RDCs, share files available to partners according to respective data-sharing agreements, or from other institutions' publications that use Statistics Canada health indicators to produce and publish estimates (e.g., CIHI). The strategic review of Statistics Canada's health indicators collected performance data from administrative sources, including the Webtrends tool.

Moreover, in the last five years, a number of efforts have been made to improve access to health data and products. For instance, in 2011/2012, the HSP completed several access-related activities as follows:

• Website redesign project: A new interface tool was created to allow health data to be extracted directly from CANSIM and manipulated in real-time by users, and webpage design was optimized to simplify the searching process for users.
• Administrative data project: Activities were undertaken in response to the high demand for data from the administrative programs available to researchers at the RDCs.
• New dissemination vehicles were developed to release health data to the general population. The Health Client Satisfaction Survey and other consultations with users identified a gap: while Health Reports is a highly regarded and popular product for advanced users, the information is not useful for the general public to a great extent. To address the gap, the HSP created a new publication called Health at a Glance, which was to be released periodically starting in 2011/2012.
• Following an update of its release policies, standards and guidelines, the HSP improved the access of key partners (HC, PHAC, Vital Statistics Council for Canada (VSCC), and Council of Canadian Cancer Registry (CCCR)) to advanced health data and product releases.
• New guidelines were developed for access to the CHMS Biobank to facilitate health research based on this resource.

Statistics on access (visiting the website, viewing products, downloading and accessing data) show that there is interest in HSP statistical products. Between 2012/2013 and 2015/2016, 6.9 million pages on health statistics (CANSIM tables, publications and summary tables) were viewed, compared with 64.8 million for Statistics Canada as a whole. Health Profile and Health Fact Sheets were the most popular publications, and interest in Health at a Glance increased throughout the period. It is worth mentioning that Health Profile (and also Health Trends) was discontinued because the New Dissemination Model will not be able to support such a product.

Source: Statistics Canada, administrative databases Webtrends

The English Health Indicators main webpage (including the What's New rubric and the link to new and updated indicators) had over 20,000 views from May 2013 to October 2015, while the French webpage had 6,000 views. The top webpages—with at least 500 views—generated over 63, 000 total views. The Health Indicators publication, overall, was viewed 172,539 times from May 2013 to October 2015. This compares to Health Reports' 1.4 million visits during the same period. Among health-related publications it is sixth, behind Health Reports, Health Profile (discontinued), Health Fact Sheets, Health at a Glance, and Health Trends (discontinued). Of note, the Health Profile, a now discontinued publication that presented most health indicators together for a single geography, was seventh on the overall list with 560,296 views (and ninth in the first seven months of the 2015/2016 fiscal year).

For the same period (2012/2013 to 2015/2016), the number of downloads (of CANSIM tables and publications) for HSP and Statistics Canada products totalled 447,436 and 13.6 million, respectively, with a sizable increase over the period. Of the various publications, Health at a Glance was downloaded most often and saw an increase throughout the period.

Numerous key informants acknowledged the progress the agency is making with respect to innovative approaches to access and dissemination. There is a general sense among key informants that mechanisms for information dissemination are functional, in that they meet users' basic needs. Furthermore, the survey of users confirmed that the level of satisfaction with the different means of access is generally high.

However, accessibility is one of the dimensions of quality that generates the most comments, suggestions and criticism.

This is corroborated by the national Health Data Strategy (HDS),^{Footnote 33} developed by the Health Information Advisory Committee, which concludes that “there is a need to rethink the best approach to make the data accessible to all. Currently access to data is an impediment to the food industry for potential innovation, to academics who cannot access the Research Data Centres outside regular hours and to Canadians for making informed decisions. Changing the Statistics Act is one way to resolve the access to data.” The HDS also suggested that the key federal health information organizations (CIHI, CIHR and Statistics Canada) should facilitate a meeting of researchers on data access issues to identify the barriers and to propose workable solutions to remove them.

Furthermore, many key informants suggested that more can or should be done to facilitate access, ranging from improving existing resources, to increased engagement with and support for data users, as well as increased availability of microdata currently accessible largely through RDCs. It was commonly raised that steps should be taken to improve overall access to microdata—in particular, there appears to be dissatisfaction with the current practice of restricting access to health statistical information to RDCs and a belief that this information should, to the extent possible, be made more widely available.

Challenges

Key informants raised issues related to accessibility, including the following:

Confidentiality/privacy clauses within the Statistics Act

Access to HSP data is hampered by the Statistics Act, and, in particular, by the clauses within the Act that relate to privacy or confidentiality (or, alternatively, an overly cautious interpretation of those clauses by Statistics Canada).

It was suggested that Statistics Canada should recognize that some users of HSP data, such as key federal partners and provincial ministries of health, routinely deal with identifiable data and/or already have governance mechanisms in place with respect to confidentiality, implying that adjustments should be made to the procedures for gaining access to data for such users.

Data-sharing Agreements (DSAs)

Concerns about the processes involved in obtaining access to data through the establishment of DSAs were expressed, including the complexity and rigidity of the process and a perceived lack of focus by the HSP on the needs and priorities of its federal partners.

Research Data Centres (RDCs)

Challenges involved in accessing data through the centres were expressed by RDC users. These challenges include the time, energy and cost required to secure access, which could be greater for individuals who do not personally reside in cities where an RDC is located.

However, a few key informants observed that the challenges associated with accessing data through the RDCs are not only related to time, cost and geography, but also to the processes and requirements that Statistics Canada has in place to protect privacy and confidentiality. As such, accessibility issues with respect to the RDCs are related to this larger issue.

This issue was also raised during program consultation (2015). In terms of access, for instance, all CCHS clients specified that they would like access to the full master file from their office even though they understand the need for the share files and the research data centres (RDC program).

A small number of RDC users also recommended that more be done to keep researchers abreast of developments with the potential to influence their work, such as upcoming data releases or changes to data that are already available.

Statistics Canada's website

Issues were identified with respect to the accessibility of information on Statistics Canada's website—particularly difficulties with website navigation and search capabilities, even for experienced users.

Suggestions pertaining to the website typically related to changes that might improve the general findability of information, such as including additional tiers of subtopics, offering suggestions for related resources, or introducing a service that users can access when they are having difficulties locating information.

Comparison to other data providers

Key federal partners typically view Statistics Canada as comparable to CIHI in data accessibility, although a few described having relatively easier access to CIHI data. However, several key informants compared access to the HSP statistical information to information disseminated by statistical agencies in the United States. In particular, they argued that U.S. health data collected through the National Health and Nutrition Examination Survey (NHANES) is substantially simpler to access than comparable Canadian data, adding that changes to the Statistics Act would likely be required for a similar dissemination approach to be viable in Canada. Among interviewees (primarily users and experts) who work with or have knowledge about other data providers, Statistics Canada is generally viewed as comparable or superior to other data providers in the context of access to health statistical information.

The evaluation found evidence^{Footnote 34} of ongoing initiatives put in place to improve accessibility and respond to some of these concerns, including the following:

• The program is exploring and developing ways (workshops, meetings, conferences, social media) to inform potential users about new data holdings and the various methods to access these holdings, in particular, CHMS developed a specific communication plan, with consideration of two streams of users (a broader audience and specialized academic audience).
• There is increased access to vital statistics and cancer statistics for analysis and research—more files available in the RDCs and RTRAs; new data products based on CCR record linkages with other sources (DAD, radiation therapy); data sharing agreements with the provinces to facilitate access to administrative data for research.
• New approaches to dissemination

  The U.S. survey that is comparable to the CHMS makes their data publicly available so university researchers can access it. That's what researchers would love to see in Canada. I gather that this cannot be done with population health surveys under the Statistics Act.

  
  Source: Interviews

  • Transition to the New Dissemination Model (NDM) of Statistics Canada: The Health Statistics Division Dissemination Renewal Task Force was created with the mandate to review HSD's current dissemination objectives, practices, processes and procedures and propose changes that reflect divisional priorities and new corporate requirements.^{Footnote 36}
  • With the implementation of the NDM, there will be improved accessibility to health indicator information (for example, allowing a view of a specific indicator in a table with pre-selected disaggregates rather than the full list of indicators), and improved accessibility to lower levels of geography (a maps-to-data interface to search within a map for a desired geography location level, resulting in links to any data, analyses, or documentation related to that level of geography). Currently indicators can only be accessed in Canada, province/territory, census metropolitan area, and health region levels.^{Footnote 37}
  • A special working group on dissemination of the results from the CCHS–Nutrition was created to identify dissemination needs and develop a plan to ensure that analysts have the relevant data and documentation.^{Footnote 35}
• Maintain access and availability of high-quality publications (Health at a Glance releasing 8 to 10 articles in 2015/2016; continuing updates to Health Profiles, Health Indicators and Health Trends).^{Footnote 38}

3.4 Coherence

Summary

Regardless of all efforts and thoughtful considerations, there are situations where coherence cannot be maintained, for reasons such as significant and necessary changes of methodology, changes in infrastructure, or content framework. In this regard, the evaluation findings tend to demonstrate that the trade-off between coherence and relevance exists and is managed adequately in the decision-making process. Overall, most of the key informants were satisfied with the coherence of HSP statistical information. Even though some interviewees argued that the agency is sacrificing relevance for coherence and raised the potential implications of methodological changes, others acknowledged the need for change when and where the issues occurred.

One such example is the physical activity module of the CCHS, which has been completely redone. Thus the previous data are no longer comparable and a new trend will be created. With regard to the CCHS, a comparability issue resulting from the recent redesign of the survey was anticipated. In the first release after the redesign for the 2015 reference year (in June 2016), directly comparing the new data with the data released from 2001 to 2014 was not recommended. In such cases, program efforts are directed to providing enough information on the changes and their impact on the comparability/coherence of the data. The survey results confirmed user satisfaction with data coherency—for example, only 7% (16 out of 226) of CCHS–annual component survey respondents were dissatisfied with the coherency, which was 3 % (2 out of 66) and 3% (3 out of 96), respectively, for the CCHS focus components on healthy aging and mental health.

The dissemination plan for the 2015 CCHS–Nutrition survey takes into consideration the concordance and comparison issue with CCHS–Nutrition (2004) for both the foods and the vitamin and mineral supplements given that the coding systems have changed.^{Footnote 39} Moreover, the document review revealed that development of long-term content plans for two major surveys—CCHS (from 2015 to 2022) and CHMS (for Cycles 5 to 8)—helps with the objective to harmonize the content across surveys, as it allows for better coordination of the content, and, consequently, using coordinated definitions, instruments and standards as much as possible. Potential coherence issues are discussed by the Expert Advisory Committees and clarifications are made in order to harmonize the content and avoid duplications.

The coherence of the vital statistics data is facilitated by the legislation governing the collection of vital events information. More specifically, “Provincial and territorial Vital Statistics Acts (or equivalent legislation) render compulsory the registration of all live births, stillbirths, deaths and marriages within their jurisdictions. These acts follow, as closely as possible, a Model Vital Statistics Act that was developed to promote uniformity of legislation and reporting practices among the provinces and territories.”^{Footnote 40} Documents confirm that the Vital Statistics Program is keeping track of changes in provincial legislation so that any data adjustments can be implemented, if necessary.^{Footnote 41}

As reported by the Program Consultation Feedback Report (2015), although the Canadian Cancer Registry (CCR) does a good job of providing pan-Canadian estimates for cancer incidence, treatment of cancer (surgery and radiotherapy) has been identified as a gap for the registry that needs to be filled. Two options are being investigated. The first is record linkage. This would involve bringing existing datasets together with the incidence data on the registry to form a more complete picture of cancer treatment in Canada. The other option is for the provinces to work with care providers to report the data directly to the provinces in a standard way across Canada; the data would then be compiled and added to the Canadian Cancer Registry. Should the record linkage result in insurmountable quality issues, primary collection will be explored. With a national registry, consistency and standardization are key because, in order to compare data across the jurisdictions, all data must refer to the same definitions and standards. Since there are multiple provinces and data providers, these consultations would most likely take a long time to conduct based on the length of time needed to negotiate standards that everyone can work with.

Overall, the 2015/2016 operational plan addresses the coherence of the data through the following actions:

• The Health Survey Content Coherence Committee was established to identify, discuss and resolve content coherence issues among population health surveys (at the questionnaire level). The committee meets monthly and is comprised of representatives from Health Statistics Division and Health Analysis Division.^{Footnote 42}
• Participation in the departmental Harmonized Content Committee to improve coherence in collection and dissemination between surveys for similar concepts.
• Participation in the field working groups that may be formed to ensure consistency across surveys during the transition of collection activities to e-questionnaires and to the Integrated Collection and Operation System (ICOS).
• Completing a comparability study for the VSP Death database, between data coded using the new automated cause of death coding software (IRIS) and the one used from 2000 to 2012.
• Work on the update of age standardization within HDS products, in order to facilitate national and international comparison.

3.5 Interpretability

Summary

The HSP develops and disseminates information about the definitions, methods and various quality characteristics that supplement the health statistics products and help users in the accurate interpretation and analysis of statistical information. Most interviewees were satisfied with the interpretability of statistical information generated through the HSP. In particular, they reported that the metadata and documentation provided by Statistics Canada is of high quality. However, a few interviewees identified issues in locating specific pieces of documentation they viewed as necessary to fully understand the data provided by Statistics Canada (access issue).

The high level of satisfaction with the CCHS–annual user's survey confirmed this trend, with 85% of the 226 interviewees satisfied with the interpretability of the data.

The dissemination plan for the 2015 CCHS–Nutrition suggests that similar practices regarding the release of metadata will be employed—the plan includes activities such as developing the structure of the data users guides; user data training; and the production of supporting materials such as concurrent data user guides (if necessary), a revised set of derived variable specifications, a compendium of tables, and a set of CANSIM table specifications.^{Footnote 43}

As stated in the Program Consultation Feedback Annual Report (2015), the CHMS fact sheets are one of the survey areas where the current products have difficulty meeting needs. According to the program area, given the complexity of the survey, it is difficult to write the fact sheets and make them available and understandable to the general public while, at the same time, still being of value to researchers. Statistics Canada regularly collaborates with experts at Health Canada and the Public Health Agency of Canada in an attempt to bridge these gaps. Moreover, in response to the new Statistics Canada policy (2014) that states each data release in The Daily should provide analytical information, it was noted that in each Daily article, links to CHMS metadata were provided along with the links to the data. More recently, the CHMS communication plan mentions Chat-with-an-Expert sessions (real-time online 1- to 1.5-hour discussions with experts from the program) as part of the support and promotion activities. However, challenges in interpreting results from the CHMS without a full nutrition module and geocoding were identified by federal partners.

The user supporting documentation for the vital and cancer statistics differs from the one related to population surveys as the production process is very different. The Canadian Cancer Registry (CCR) provides details about data elements in the patient records, the tumor characteristics and its diagnosis in tumor records^{Footnote 44} in a document called Canadian Cancer Registry System Guide. Similarly, the VSP supports and updates, on an annual basis,^{Footnote 45} documents such as Data Quality, Concepts and Methodology: Definitions, Vital Statistics Death Database Glossary, which, to a great extent, are consistent with the recommendations of the World Health Organization and the United Nations. More recently, the CCR is developing new metadata products that describe the CCR data holdings in order to provide analytic tools for a varied group of stakeholders and users.^{Footnote 46}

3.6 Relevance^{Footnote 47}

Summary

Engagement and consultation processes

Over the past five years, the HSP has been carrying out multiple activities in order to maintain the relevance of the program to its users, to better address their needs, and to identify data gaps and priorities for all its components.^{Footnote 49}

The evaluation found mixed perceptions of the success and effectiveness of HSP's consultation and engagement processes. While there was overall satisfaction with the consultation process, some stakeholders called for better communication, particularly with respect to the data that are available and the constraints (e.g., budget, survey lengths) to which the HSP is subject. A desire and willingness to be more involved in survey design and data analysis planning, or, at a minimum, to be kept informed about survey content, data analysis plans, and product publication/release schedules was expressed.

Responsiveness to stakeholders' needs

In terms of responsiveness, the evaluation found that, overall, the HSP has been able to meet stakeholders' needs and priorities. The results of the user survey indicate that 138 out of 325 respondents (42%) believe that the HSP completely—or to a great extent— covers the issues of most importance, while 164 out of 325 respondents (50%) indicated it was “to some extent.” The results are slightly different when heavy users or the most knowledgeable respondents of HSP products were asked whether the HSP responds to key emerging issues or trends: 40 out of 203 (20%) indicated “completely or to a great extent”; 135 out of 203 (67%) “to some extent”; and 18 out of 203 (9%) “not at all.” The interviews conducted confirm that the HSP's components are responsive to stakeholders' needs overall. However, some areas in need of improvement in each program component have been identified.

The evaluation found that the CCHS–annual is responsive to stakeholders' needs and priorities. The results of the survey conducted for this evaluation indicate that 203 out of 226 respondents (90%) are satisfied with the products. The CCHS also offers the Rapid Response option (new initiative) to meet emerging and evolving needs for additional content, on a cost-recovery basis. The option allows clients to purchase 3 minutes of questionnaire time for a three-month collection period. Based on the Consultation Feedback Annual Report (2015), all clients have been very satisfied with the Rapid Response option, except for availability of provincial data, where gaps were identified.

Overall, the evaluation found that the CCHS focus is responsive to stakeholders' needs and priorities. For instance, the survey shows a high level of satisfaction for CCHS Healthy Aging (85% satisfied) and for CCHS Mental Health (83% satisfied). However, some gaps, related to time periods between surveys (i.e., mental health and nutrition) and the inclusiveness of geographic areas (CCHS focus content surveys such as mental health and nutrition are not administered in the North) were identified. Moreover, the fact that some subpopulations are not fully represented in the CCHS as well as the wording used in sexual orientation questions were also identified as gaps.

In 2012 the relevance and performance of the CHMS was evaluated as part of the horizontal evaluation of the federal government's Action Plan to Protect the Health of Canadians from Environmental Contaminants. The evaluation report concluded that, although the CHMS is responsive overall to current and emerging content needs, the absence of a coordinated longer-term content plan is seen as a gap. To address the recommendations of the evaluation, the CHMS Program developed an eight-year content plan in 2013.

The results of the user survey indicate that 83 out of 101 respondents (82%) are satisfied with the statistical information available. However, there is one known gap with regional-level data, which is requested by interested data users. Due to cost and logistics, CHMS is conducted at the national level only and, at this point, there is no funding to consider expansion.^{Footnote 50} In addition, comprehensive data on children is not included in the CHMS, which would be valuable for stakeholders.

The Canadian Vital Statistics Program (VSP) remains relevant via consultations with the Vital Statistics Council of Canada. Overall, stakeholders are satisfied with the ongoing work to maintain the relevance and the usefulness of the VSP. The results of the survey conducted for this evaluation indicate that 91 out of 114 respondents (80%) are satisfied with the product. However, a lack of detail has been identified regarding death from drug toxicity—specific drugs are not identified. Furthermore, work must be undertaken to maintain the relevance of the VSP as it pertains to Physician Assisted Death (PAD) and the definition of stillbirth (given the increase in fetal reduction associated with the increase in assisted human reproduction). Another issue is the linkage between the Vital Statistics Program and the Canadian Cancer Registry to ensure that important information, such as survival data, can be analyzed. The HSP is working with stakeholders, such as Health Canada, the Public Health Agency of Canada and the Canadian Institute for Health Information, to ensure that needs are met in these areas.

Although the CCR does a good job of providing pan-Canadian estimates for cancer incidence, cancer treatment (surgery and radiotherapy) has been identified as a gap for the registry and all agree that it needs to be filled. The program is exploring two options to fill these gaps—a record linkage with other existing datasets, or asking care providers to report data to provincial cancer registries in the standard way, which would then be compiled and added to the CCR.

The evaluation found that the health indicators component of the HSP meets the needs and priorities of the stakeholders—97 out of 106 respondents (92%) are satisfied. Consultations take place via the Consensus Conference on Health Indicators (Statistics Canada/CIHI).^{Footnote 51} The fourth Consensus Conference gathered 61 participants across Canada. Most of the 80 indicators identified are relevant and meaningful to the majority of users. For the indicators that are less useful, a review is being conducted to determine whether they should be dropped or modified to increase their relevance.^{Footnote 52}

The evaluation found a high level of satisfaction with Health at a Glance (94% satisfied).

Current/Emerging Needs and Expectations

The survey conducted confirms that certain areas need improvement. Among respondents consulted through the evaluation survey, 143 out of 325 (44%) believe that the HSP should produce additional information. When asked which information should be produced, nearly 40% of the responses indicated the concept of big data (integrated data across social domains) and determinants of health. Lack of geographic data (11%) and longitudinal data (10%) were also identified as needed.

Similarly, the key informants interviewed described a wide range of current and emerging needs and priorities relating to their particular field of expertise or domain of work, including:

• research on health equity, social determinants of health, and the broader concept of wellness;
• data, more data, or more frequent data collection on specific topics, including environmental exposure, pharmaceutical use, disabilities, mental health, institutionalization, nutrition (primarily consumption patterns), health needs of refugees, health needs of veterans, aging, birth defects, cancer screening, children's health, First Nations health;
• data/research that captures health outcomes;
• patient-oriented research;
• larger survey sample sizes; and
• technologies for human measurement (e.g., technologies for measuring chemical exposure in humans and genotyping).

Despite the diversity of key informants' needs and priorities, there were also some notable commonalities. The most frequently cited needs/priorities across all key informant categories were administrative data linkage and increased longitudinal data.

Administrative data linkage

Across all key informant categories, but especially among users and experts, the most commonly identified need was increased administrative data linkage and the related concept of big data.

Longitudinal data

A need for more longitudinal data was identified, primarily by RDC users. These key informants observed that the HSP is focusing on cross-sectional surveys—a trend they have observed throughout Statistics Canada. They emphasized the need for longitudinal data to facilitate the monitoring of health trends and understanding the impact of social determinants of health.

3.7 Use of the statistical information

Summary

This evaluation question explores the extent to which the HSP data are being used to inform debate, research and analysis, as well as decision-making. The scope of issues covered by the program, combined with the fact that all this information is readily available, free of charge from the Statistics Canada website, creates endless possibilities for data use. Consequently, while the evaluation findings shed some light on this matter, it is impossible to capture the full range of potential use of statistical information.

Who uses Health Statistics Program data, for what purpose, and how?

Based on Statistics Canada's administrative databases, document review, literature review, bibliometric and webometric assessments, surveys and interviews, the evaluation identified a variety of uses and users in all areas of activity from the public and private sectors.

The key federal organizations users are HC, PHAC, CIHI, and CIHR. Other federal organizations among the major stakeholders are Employment and Social Development Canada, Immigration, Refugees and Citizenship Canada, Environment and Climate Change Canada, Department of National Defence, Service Canada, Passport Canada, Canada Revenue Agency, and Canada Border Services Agency. At the subnational level, all provincial and territorial governments and municipalities are important users of the HSP's statistics.

Evidence from key informant interviews corroborates the findings drawn from the document review and adds examples of public sector use (federal, provincial, territorial and municipal). At this level, the HSP data are being used to inform the development, performance measurement and evaluation of policies and programs intended to promote the health of Canadians. The HSP data are also used for the formation of guidelines, standards and regulations, particularly at the federal level. As one example collected in the evaluation, analysis of CCHS data identified differences in risk behaviours between two regions that, in turn, stimulated discussion around the adequacy of current regulations.

There is also evidence that HSP products have been used by international organizations. For instance, health indicator publications were used by the Organisation for Economic Co-operation and Development (OECD),^{Footnote 53} United Nations Children's Fund (UNICEF),^{Footnote 54} Pan-American Health Organization (PAHO)^{Footnote 55} and the Commonwealth Fund to release comparative information internationally. HSP information is also used by the World Health Organization to develop the Global Reference List of health indicators.

Evaluation findings from key informant interviews also confirmed that non-governmental organizations use HSP statistics to, for instance, develop products intended for the Canadian public that raise awareness of particular health issues.

Researchers use HSP information in trend studies and analysis, and for in-depth research on various health issues disaggregated by social and economic variables. For example, CCHS data were used by researchers at CIHI and Public Health Ontario for the development of a Life Expectancy Calculator that helps Ontarians understand the effect of certain behaviours on their life expectancy. Moreover, for the period from 2011/2012 to 2014/2015, more than 500 single contracts were granted for RDC access. These contracts were done by an annual average of 520 persons. CCHS–annual and two years combined were, by far, the most-requested files in contracts. Most of the RDC users are students, professors and researchers (39%, 37% and 14%, respectively, in 2014/2015) at universities (92% of the total types of organizations in 2014/2015).

World Wide Web

The visibility of HSP information on the Web was explored as part of the evaluation through a webometric assessment of uptake (hyperlinks to HSP products). Approximately 60,000 hyperlinks to Statistics Canada's website are on the Internet, of which 1,219 (2%) are directed towards HSP products. The hyperlinks directed to HSP products relate to summary tables (489 or 40%); surveys (302 or 25%); CANSIM tables (219 or 18%); publications (143 or 12%); and The Daily (66 or 5%).^{Footnote 56}

The five most referenced products are CCHS–annual (survey); life expectancy at birth, by sex, by province (summary table); health indicator profile, annual estimates, by age group and sex, Canada, provinces, territories, health regions (2013 boundaries and peer groups (CANSIM table); suicides and suicide rate, by sex and by age group (summary table); and CCR (description). See Table 3 below.

HSP-citing web pages appear in various large domains from around the globe. The largest number of citing web pages (587 or 48%) originates from the Canada domain “.ca,” followed by the commercial domain “.com” (376 or 31%); the non-profit organization domain “.org” (162 or 13%); and the network domain “.net” (40 or 3%). Of all 1,219 links to HSP products, 618 come from Canadian websites (.ca; .com; .org; and .net). Of all Canadian links to HSP products, 38% (233) are from private-non-profit organizations' websites; 27% (136) are from private organizations' websites; 23% (143) are from public organizations' websites—75% of them are in the federal government, the majority of which (63%) are from CIHR, HC and PHAC; and 7% (44) are from academic institutions. The most frequently citing organizations are universities: University of Toronto; University of Ottawa; and University of Alberta.

Media

A review of Statistics Canada's administrative records provides evidence of the level of interest in the media. In 2012/2013, HSP products were cited 129 times in the media, 163 times in 2013/2014 and 143 times in 2014/2015. During these three fiscal years, the most cited products were the CCHS and the Death Database.

Impacts of using Health Statistics Program products

The bibliometric assessment of outputs looked at two specific measures related to the scientific impact of HSP products—one that measures the impact of papers published, and another that measures the visibility of journals in which the research papers are published. The impact of the data on published literature is illustrated by the presence and use of HSP statistics in peer-reviewed articles. According to the bibliometric assessment conducted as part of the evaluation, a total of 38,222 citations referring to Statistics Canada in peer-reviewed articles were found between 2000 and 2014,^{Footnote 57} of which 3,970 (10%) refer to HSP releases. A bibliometric uptake assessment of Health Reports (82 003 X) is also provided as context for interpretation purposes. This publication series received about 2,250 references over the period from 2000 to 2014—about 55% of what HSP received overall.

The references to Statistics Canada releases for the periods from 2000 to 2006 and from 2008 to 2014 have grown more than five-fold, while they have grown more than ten-fold for HSP releases for the same time periods. The document types mostly cited for HSP are publications (1,684 or 42%); surveys (1,521 or 38%); CANSIM tables (416 or 10%); The Daily (179 or 5%); and summary tables (170 or 4%). The five most cited products are CHMS (survey); Nutrition: Findings from the CCHS (publication); National Population Health Survey (publication); Health Indicators (publication); and CCHS–annual (survey).

Furthermore, the impact and visibility of publications produced by the HSP seem to be above the world average and even the Canadian average.^{Footnote 58} As illustrated in Figure 2, from 2000 to 2014, the impact of Canadian papers published is 39% higher than the world average, while the impact of Statistics Canada papers published is 32% higher. The impact of Statistics Canada's health sciences papers, which essentially represent HSP products, is 81% higher than the world average (also well above Canadian and Statistics Canada scores). The results for the second measure are also significant. From 2000 to 2014, the visibility of journals in which the Canadian research appears is 25% higher than the world average, while the visibility of Statistics Canada research is 27% higher. The visibility of Statistics Canada's health sciences research, which again represents HSP products, is 44% higher than the world average (also above Canadian and Statistics Canada scores).

Survey respondents were asked if the information produced by the HSP allowed them to understand population health trends and issues better. Of respondents, 93% indicated that the HSP information enabled a better understanding. In fact, the use of HSP products enabled researchers to carry out high-quality research.

Figure 2: Scientific Impact between 2000 and 2014

Appendix 1: References

Bryant, T., D. Raphael, T. Schrecker and R. Labonte. 2011. “Canada: A land of missed opportunity for addressing the social determinants of health.” Health Policy. June. Vol. 101. p. 44–58.

Canadian Institute for Health Information and Statistics Canada. 2015. Rethink, Renew, Retire: Report from the Fourth Consensus Conference on Evaluating Priorities for Canada's Health Indicators. Ottawa.

Canadian Institute for Health Information and Statistics Canada. 2013. Health Indicators 2013. Ottawa.

Commission on Social Determinants of Health. 2008. Closing the Gap in a Generation: Health Equity Through Action on the Social Determinants of Health. Final Report of the Commission on Social Determinants of Health. Geneva: World Health Organization.

Council of Canadian Academies. 2015. Accessing Health and Health-Related Data in Canada.The Expert Panel on Timely Access to Health and Social Data for Health Research and Health System Innovation, Council of Canadian Academies. Ottawa.

Hawe, P., E. Di Ruggiero and E. Cohen. 2012. “Frequently asked questions about population health intervention research.” Canadian Journal of Public Health. Vol. 103, no. 6. p. e468–e471.

Health Information Advisory Committee. 2015. “Appendix 1: Inventory of health data.” Health Data Strategy: Collective Efforts to Address Canadian Health Data Challenges. Internal document.

Library of Parliament. 2008. The Federal Role in Health and Health Care. Publication no. 2008‑58‑E. Ottawa.

Morrison, J., M. Pons‑Vigués, L. Bécares, B. Burström, A. Gandarillas, F. Domínguez‑Berjón, È. Diez, G. Costa, M. Ruiz, H. Pikhart, C. Marinacci, R. Hoffman, P. Santana, C. Borrell and partners from the INEQ‑Cities Project. 2014. “Health inequalities in European cities: Perceptions and beliefs among local policymakers.” BMJ Open. May. Vol. 4, issue 5.

Organisation for Economic Co‑operation and Development (OECD). 2015. Health at a Glance, 2015. How Does Canada Compare? Paris: OECD Publishing.

Pan American Health Organization. 2012. Health in the Americas, 2012 Edition: Country Volume. Canada.

Pathak, S., M.D. Low, L. Franzini and J.M. Swint. 2012. “A review of Canadian policy on social determinants of health.” Review of European Studies. September. Vol. 4, no. 4.

Senate of Canada. 2009. A Healthy, Productive Canada: A Determinant of Health Approach. The Standing Senate Committee on Social Affairs, Science and Technology. Final Report of the Senate Subcommittee on Population Health. Ottawa.

Statistics Act, R.S.C. 1985, c. S-19. Ottawa: Department of Justice.

Statistics Canada. 2017. Statistics Canada's Quality Assurance Framework. Statistics Canada Catalogue no. 12‑586‑X. Ottawa.

Statistics Canada. 2015a. Health Statistics Division. Operational Plan 2015/2016. Internal document. Ottawa.

Statistics Canada. 2015b. Canadian Community Health Survey (CCHS)–Nutrition Dissemination Working Group Charter. Ottawa.

Statistics Canada. 2015c. Health Indicators Strategic Review Report. Ottawa.

Statistics Canada. 2015d. Consultation Feedback Annual Report. Health Statistics. Ottawa.

Statistics Canada. 2015e. Health Statistics Program: Presentation to the National Statistics Council, October 29, 2015. Ottawa.

Statistics Canada. 2014a. 2014-2015 Report on Plans and Priorities. ISSN 2292‑5252. Ottawa.

Statistics Canada. 2014b. Canadian Health Measures Survey (CHMS) Expert Advisory Committee Meeting, September 26, 2014. Internal document. Ottawa.

Statistics Canada. 2014c. Health Statistics Division (HSD) Dissemination Renewal Task Force Terms of Reference. Ottawa.

Statistics Canada. 2014d. Health Surveys Informal Content Coherence Committee Terms of Reference. April 19, 2010. Revised September 22, 2014. Ottawa.

Statistics Canada. 2014e. Evaluation of the LHAD Initiative. Longitudinal Health and Administrative Data Initiative. Ottawa.

Statistics Canada. 2013. Communication Plan for Canadian Health Measures Survey, 2013‑2016. Ottawa.

Statistics Canada. 2012. Data Quality in Statistics Canada's Population Surveys: The Case of the Canadian Community Health Survey. Ottawa.

Statistics Canada. 2011. Program Performance Report: 2008/2009 to 2009/2010. Health Statistics Program. Ottawa.

Statistics Canada. 2009. Statistics Canada Quality Guidelines. 5th edition. Statistics Canada Catalogue no. 12‑539‑X. Ottawa.

Statistics Canada. 2002. Statistics Canada's Quality Assurance Framework. Statistics Canada Catalogue no. 12‑586‑X. Ottawa.

Treasury Board of Canada. 2015. Whole‑of‑Government Framework. Ottawa.

Vital Statistics Council of Canada. 2014. Annual General Meeting 2014. Internal document.

Weiss, D., M. Lillefjell and E. Magnus. 2016. “Facilitators for the development and implementation of health promoting policy and programs: A scoping review at the local community level.” BMC Public Health. Vol. 16, no. 140.

Word Health Organization (WHO) Regional Office for Europe. 2012. Governance for Health in the 21st Century. Prepared by I. Kickbusch and D. Gleicher. Copenhagen, Denmark.

Appendix 2: Program Description

The mandate of the Health Statistics Division (HSD) is to provide accurate, timely and relevant information about the health of Canadians. The HSD, in collaboration with the Health Analysis Division (Analytical Studies Branch), provides statistical analyses and other information about the health of the population, the determinants of health and the scope and utilization of Canada's health care resources. This information is used to assist and support health planners and decision-makers at all levels of government, sustain demographic and epidemiological research, and to report to the Canadian public about their collective health and health care system performance.

HSD works in partnership with provincial and territorial vital statistics registrars and cancer registries, as well as data providers and users at the federal level, namely Health Canada and the Public Health Agency of Canada (PHAC), provincial level (provincial ministries of health), regional level (health regions), and with many other partners, including the Canadian Institute for Health Information (CIHI), to provide statistical information on, and to undertake research and analysis about, the health status of Canadians and the health system.

The Health Statistics Program's activities can be divided into three broad areas:

• statistical and administrative programs
• products and services
• special projects and initiatives.

Table 4: Health Statistics Program key surveys and data products

Appendix 3: Health Statistics Program Logic Model

The program logic model presented in Figure 4 below is based on Statistics Canada's Quality Assurance Framework^{Footnote 64}, which describes the six dimensions for quality management of all statistics data and products, (i.e., accessibility, timeliness, accuracy, interpretability, coherence and relevance). The logic model is also aligned with Statistics Canada's Corporate Management Framework^{Footnote 65}, which illustrates how the agency delivers its strategic outcomes. The six dimensions of quality management are integrated into the framework as enablers in a hierarchical sequence from access, to quality (including accuracy, interpretability and coherence) and then relevance.^{Footnote 66}

Figure 4: Health Statistics Program Logic Model