The Canadian Health Measures Survey 2007/2008 to 2012/2013

4.1.1.1 Continued need

According to documents reviewed, CHMS stems from a need to fill gaps in health information. It was developed to fulfill specific important needs, such as the need for

• national baseline data on the extent of major health concerns (e.g., obesity, hypertension, vitamin and nutrition deficiencies, chronic and infectious diseases, oral health status of Canadians and the level of access to dental care)
• national data on exposure and prevalence levels of environmental chemicals in the population
• children's health information.

The principal objective of CHMS is to collect data on Canadians' health status by direct measures. While Statistics Canada has been collecting health status and related data for many years by other means (i.e., self-reported surveys and administrative data), these data were seen as generally limited in two important ways:

1. Many kinds of data, such as blood pressure and physical fitness, simply cannot be ascertained in an interview; they require direct physical measurement.
2. Health information derived from self-reporting or administrative records may be seriously biased for some variables. For example, one review ^{Footnote 6} shows a consistent reporting bias of even simple measures like height and weight. This bias has the potential to misinform data users. ^{Footnote 7}, ^{Footnote 8}

The need for a direct health measures survey is not unique to Canada. Several countries have a history of conducting surveys that include direct health measures and that have yielded important findings, which validates the need for this type of survey. For example, the U.S. National Health and Nutrition Examination Survey (NHANES), ^{Footnote 12} conducted since the early 1960s, ^{Footnote 13} has provided data to construct standard growth charts for children, thereby allowing doctors and parents to better understand developmental health trajectories. ^{Footnote 14} In the 1960s, NHANES confirmed findings linking high cholesterol and heart disease. It also provided the first evidence that Americans had high lead levels in their blood, which motivated governments to phase out the use of lead as an additive in gasoline and paint. ^{Footnote 15} In Australia, a direct health measures survey conducted from 1999 to 2001 found that for every known case of diabetes, there was one undiagnosed case, and that nearly one million Australians over age 25 have diabetes. ^{Footnote 16} Finland, too, has a legacy of important public health and scientific findings from national direct health measures surveys. ^{Footnote 17,Footnote 18,Footnote 19}

In addition, the findings from the survey of CHMS primary data users also demonstrated and confirmed the current need for CHMS data for many purposes, such as trend analysis, monitoring and tracking changes, providing objective physical measures and lab data, providing a nationally representative sample, research, health impact assessments, diagnoses, prevention, informing policy, and policy development support. All survey respondents confirmed that there was a need to continue to collect and provide CHMS data on an ongoing basis.

4.1.1.2 Inclusion of environmental exposures in the CHMS scope

A growing concern of Canadians about environmental contaminants has been observed in the past decade. For example, a 2006 EKOS Research Associates Inc. survey showed that 89% of Canadians believe the health of their children is being adversely affected by environmental pollution. Nearly one in four Canadians has sought medical treatment for a condition that they believed to be related to the environment.

Two major federal government initiatives, the 2006 Chemicals Management Plan and the 2007 Clean Air Agenda committed to address issues related to environmental contaminants. The need for CHMS data is specifically relevant as the steps already taken by the government concerning these initiatives rely on CHMS to collect environmental contaminant measures. Both industry and non-governmental stakeholders supported these initiatives. They continue to insist that decisions be made on the basis of scientific evidence, and that Canadians have credible information on the impact of chemicals in the environment and the steps that they should take as a result.

A review of CHMS content demonstrates that the survey does address emerging environmental issues. Furthermore, it is evident that children under the age of six are represented in the sample. Table 5 provides an overview of the scope of the environmental exposure variables in the first three cycles of the survey. These data, according to a cost-recovery client, provide a baseline and the huge potential for identifying links between environmental contaminants and health; however, long-term tracking is required to establish such trends (some take up to 20 years to establish) and to address potential implications, interventions, controls, etc. Consequently, at this time such links are not possible to identify.

4.1.1.3 Addressing current and emerging needs

When asked about emerging needs or information gaps that CHMS is expected to address, key informants and several documents identified a number of emerging needs that can be related either to new content themes or to the enhancement of the analytical power through the data. A summary of the most common suggestions collected are presented below.

New or enhanced content

• Data on non-insured components of health (e.g., oral health, hearing, eyesight)
• More comprehensive information on dietary and nutrition intake
• Medication and dietary supplement intake
• Bone health
• Lung function
• Noise exposure, expanded range of environmental exposure (more than 23,000 chemicals that need to be assessed and regulated)
• Genetic testing, genetic links, personalized medicine, genome
• Sleep
• Specific measures (e.g., oral glucose tolerance test, cortisol [stress hormone])

Analytical capacity requirements ^{Footnote 20}

• Record linkages
• Longitudinal data
• Geographic information
• Data on sub-national level (e.g., regional, provincial, specific social groups, specific age groups, vulnerable populations)

Most key informants interviewed and data users surveyed feel that CHMS responds to current needs. As a result of CHMS data, Canada has objective, nationally representative baselines on important health measures, which have the potential to be used by doctors, health professionals ^{Footnote 21} and researchers ^{Footnote 22} as reference for the health characteristics of Canadians.

Further, CHMS is seen by interviewees as being able to address emerging needs, in terms of new content, because:

• It provides an infrastructure that is comprehensive in scope, and a system of collecting direct measures that is diverse and flexible to accommodate the information needs of important health issues that may emerge in the future.
• The content priorities are determined through the federal governance structure, following broad consultations with federal partners and other stakeholders; consequently, CHMS adjusts the survey content to be responsive to the needs indentified by them.

Although most feel that CHMS is responsive to current and emerging content needs, according to interviewees and documents reviewed, there was no content plan ever developed for CHMS to ensure the survey remains relevant and can continue to address emerging content needs.

With respect to the Biobank, ^{Footnote 23} several researchers mentioned its significance, importance and usefulness, highlighting that it is the first Canadian nationally representative repository of biosamples. Internal and external researchers testify on the relevance of the Biobank, and that it provides huge potential for use in future research or surveillance of health issues, when a need arises.

Some analytical limits of CHMS persist because of the sample size. There are many examples where key informants specify that they need larger sample sizes for conducting more complex research and testify that this limitation impacts the use of the data for their own purposes. This aspect is presented in greater detail in the section on performance (validity, reliability and access).

4.1.2.1 Alignment with the government's priorities and Statistics Canada's strategic outcome

Government priorities

Collecting health data has been underscored as a federal priority on several occasions. To recall, the initial funding for the CHMS survey was outlined in the 2003 Federal Budget, where the Government of Canada highlighted the importance of "the availability of accurate and timely information on trends in health status and health system performance as a crucial tool to inform responsive, patient-centered health policy decisions." Moreover, investing in health and safety to provide Canadians with better information on the links between pollution and illness was reaffirmed in the 2008 Federal Budget.

The alignment of CHMS with federal government priorities is ensured by its governance at the strategic level. CHMS is one of several Statistics Canada health information activities under the CPHSP and, as such, its strategic priorities and decisions are determined by the CPHSP tri-partite steering committee representing HC, PHAC and Statistics Canada. By its mandate, the CPHSP committee is responsible for ensuring that federal government health information needs are addressed.

Document reviews revealed evidence that CHMS supports the federal government in fulfilling its legislative obligations, its international commitments and its responsibility to protect the health of Canadians. For example:

• CHMS data inform and support the federal government in its ongoing monitoring, surveillance, risk management and regulating activities, which are required by multiple legislative acts such as the Health Canada Act, the Canadian Environmental Protection Act, the Pest Control Products Act, the Food and Drugs Act, and the Hazardous Products Act.
• CHMS information helps to address the federal government's international commitments and obligations, such as the North American Free Trade Agreement, the Canada–U.S. Great LakesWater Quality Agreement, United Nations negotiations on the Global Treaty for Mercury, the Stockholm Convention on Persistent Organic Pollutants.
• CHMS data inform the creation and updating of many federal government standards and guidelines related to health and environment (such as Guidelines for Canadian Drinking Water and the Federal Tobacco Control Strategy). The survey also provides essential information in support of several broader federal government initiatives, such as the Chemicals Management Plan, the Health Information Roadmap Initiative, and the Clean Air Agenda.

Statistics Canada's strategic outcome

Supporting CHMS and other projects that provide statistical information and analysis about the state of Canadians' health is set out as one of Statistics Canada's activities in the 2011-2012 Report on Plans and Priorities. The CHMS also constitutes one of the programs designed to achieve Statistics Canada's strategic outcome as defined in the agency's 2011 Program Activity Architecture:

Canadians have access to timely, relevant and quality statistical information on Canada's changing economy and society for informed debate, research and decision making on social and economic issues.

In summary, the work of Statistics Canada to conduct CHMS is consistent with the agency's priorities and supports the achievement of its strategic outcome. It also supports the achievement of the Action Plan's objectives by increasing the knowledge base on contaminant levels and potential impacts on health, and consequently contributes to the Action Plan's ultimate outcome (i.e., reducing the health risks to Canadians from harmful environmental contaminants).

4.1.2.2 The federal government's and Statistics Canada's roles in implementing the Canadian Health Measures Survey program

The Constitution Act, 1867 establishes "census and statistics" as an area of federal jurisdiction. Parliament has exercised its responsibility for the census and statistics primarily through the Statistics Act. The act creates Statistics Canada as Canada's national statistical office and establishes its mandate, powers and obligations. Under the act, Statistics Canada must collect, compile, analyze and publish statistical information on the economic, social and general conditions of the country and its people. Under Section 22-c of the Statistics Act, Statistics Canada has the mandate to collect, compile, analyze, abstract and publish statistics on health and welfare.

Key informants indicated that while health is a provincial jurisdiction, there is an important role for the federal government to provide credible, nationally representative survey measures for comparability, consistency, and objectivity. According to them, "there is nobody in a better position than the federal government to create and maintain a statistical program of the magnitude, visibility, and complexity of the CHMS. Internationally, similar programs are also delivered at a federal level."

According to senior management, CHMS was designed in response to a data gap on the general health of the population, based on direct health measures. If CHMS no longer existed, the impact would represent a step backwards in how we understand health in Canada, especially for children aged 3 to 11, as there are currently no data available for this age group. This evaluation finding is reaffirmed by interviewed researchers. Without CHMS, researchers will lose the ability to adjust self-reported data based on objective direct measures. In health policy, a lack of CHMS data may impair decision making and program outcomes, as CHMS provides the clearest picture available of health issues and their determinants for Canadians.

4.2.1.1 Outputs

The status of CHMS activities, as of August 2012, has already been presented in Table 1. As evident, Cycle 1 is fully completed, Cycle 2 is in its final dissemination phase, and Cycle 3 collection is in progress.

Based on administrative data and interviews with program management, the evaluation findings demonstrate that CHMS outputs were produced as planned. Indeed, the Action Plan stated that CHMS should have 5,000 participants in cycles 1 and 2 and 2,500 participants in Cycle 3 (at the time of the evaluation), and that children under the age of 6 should be included in the sample. In terms of measures, the Action Plan required additional environmental measures, including air and water samples.

Table 6 depicts the completion status of the outputs and demonstrates that the targets outlined in the Action Plan were achieved.

In addition, as observed by reviewing the administrative data, biomonitoring data were collected from the 3-to-5 age group, predominantly in cycles 2 and 3. There were 135 biomonitoring measures ^{Footnote 24} taken from children under the age of 6 in Cycle 2, and there are 88 such measures currently being collected from this age group in Cycle 3, compared with only 1 measure taken from this age group in Cycle 1.

Moreover, a range of environmental exposure measures have been collected, including biospecimens (all three cycles) and indoor air samples (cycles 2 and 3). Also, a new source of measures within the environmental exposure theme has been included in Cycle 3: tap water. ^{Footnote 25}

4.2.1.2 Immediate outcome

Reliability and usability

Most of the key informants interviewed stated that CHMS data were reliable and robust. Three-quarters of the survey respondents indicated that they were satisfied or very satisfied with the data's reliability and validity.

Furthermore, evidence from administrative data demonstrates a high response rate for Cycle 1 (88% for the household questionnaire and 85% for the mobile clinic) and for Cycle 2 (90% for the household questionnaire and 82% for the mobile clinic), which is a valid indicator of the quality of the information produced.

Although researchers indicated that they were satisfied with the reliability and usability of CHMS data, they raised the following two concerns: 1) comparability of the data and 2) inability to conduct analysis at the sub-national level.

Concern 1 – Comparability of data

According to interviewees, there are many valid ways to measure health indicators and environmental contaminants, but if they are not measured the same way from one cycle to the next or from one survey to the next, they are difficult to compare for analytical purposes. For example, CHMS measures cadmium (or caffeine) in urine, while NHANES measures it in blood—both are good measures, but are not comparable.

The following possible causes for measuring things differently were mentioned by researchers: 1) laboratory measures are victim to various laboratory methods, measurement approaches, measurement errors and changing measurement protocols and 2) competing interests and money.

Concern 2 – Inability to conduct analysis at the sub-national level

Most researchers mentioned that because of the size of the sample, they were unable to conduct analysis at the sub-national level or more complex analysis on specific subsets of the population (e.g., vulnerable populations, people with rare conditions). This, in turn, limited the reliability and the usability of the data. However, the impact of the limitations is not consistent across content themes (e.g., no issues with physical activity were mentioned).

According to program documentation, CHMS was established to be a national survey and therefore provide national estimates. Conducting analysis at a sub-national level was not in scope for the initiative.

Availability

Evaluation findings revealed that data are not as accessible as they could be. Some of the comments collected are related to the unavailability of microdata files for CHMS on the Internet and the fact that researchers must go to RDCs to use CHMS data files. In terms of RDCs, internal and external researchers raised various concerns, such as a lack of awareness of how to access the data; timelines to get proposals approved (sometimes up to six months); issues with location and times (days and hours) of use; restrictions on material to be brought in and out; and, in some locations, the ability and knowledge of staff to work with the CHMS data files. ^{Footnote 26}

Findings from internal interviews revealed that no issues were identified by CHMS management regarding the availability and accessibility of CHMS data. According to management, the two federal partners have shared files and external researchers have full access to the RDCs. As for the primary data users surveyed, almost three-quarters confirmed this evidence and answered that they were satisfied to very satisfied with awareness on availability. Moreover, CHMS management also stated that a communication strategy will be launched in fall or winter 2013 to make sure that all information about accessing CHMS data and Biobank data is available and visible to the public. Because of the timing, however, it was not possible for the evaluation to validate this statement.

4.2.1.3 Intermediate outcome

According to CHMS administrative data, the general public are interested in Cycle 1 data, as seen in the number of published newspaper articles, TV and radio broadcasts, Internet articles, and website hits, presented in Table 7.

As Table 8 summarizes, the general public has also shown an interest in and awareness of the CHMS products available on the Statistics Canada website.

Finally, Table 9 indicates that external researchers have expressed an interest in Cycle 1 data based on the number of research proposals submitted (64), inquiries to client service areas and requests for custom tabulations.

Although all three tables demonstrate that there is interest in, and awareness of CHMS data, it is difficult to quantify the level of awareness. These numbers, however, can form the baseline against which to measure increased awareness as future CHMS data cycles become available and products are released.

According to all key informants, awareness of data availability exists among researchers to varying extents. It was also noted that the data are still new (Cycle 1 data were released from January 2010 to April 2011) and that, as more publications come out, awareness should increase. However, even with these efforts, interviewees indicated that there may continue to be some issues in increasing awareness because of the 'reach' of the publications and the lack of active promotion of the data to scientific communities.

4.2.1.4 Long-term outcome

Evidence of use

Canadian Health Measures Survey data used within Statistics Canada

Internal researchers highlighted specific areas where CHMS data are used within Statistics Canada. For example, CHMS helps validate other self-reported surveys, such as the Canadian Community Health Survey. It also helps improve micro-simulation models (by increasing the confidence of projections), which are developed by Statistics Canada and used by policy makers for creating or updating policies.

Canadian Health Measures Survey data used in decision making

External researchers provided examples from their area of expertise of where CHMS data have been used. These examples mainly reflect CHMS's contribution to

• determining priorities and pressing issues
• managing risk within HC and PHAC
• making strategic and financial decisions
• planning for the future
• updating guidelines and regulations
• making international comparisons
• informing global negotiations.

In the same vein, almost all primary data users from the evaluation survey had used CHMS data for research purposes. Of them, 50% had used CHMS data for policy analysis, and some of them had used the data for program and service development.

Table 10 depicts the most significant examples to date of CHMS data and research used to influence policies and regulations in four thematic areas: environmental exposure, physical activity, nutrition markers, and oral health.

The selected examples of use are drawn from a very short period (approximately 2.5 years since the data were released). Key informants underlined that "it is a very new survey on the grand scheme, having been around for such a short period of time." Nonetheless, they believed that CHMS's reputation is high and it will have a positive impact greater than any other national survey. They testify that "in various meetings they attended, under various hats, in the physical activity sector, or health sector, the survey comes out and is being mentioned as a useful and respected source of information," that it generates a lot of attention around decision-making tables at HC and PHAC, and that, even though "they might not be in a stage to have a published document per se, [..] they are certainly making decisions based on it."

4.2.1.5 Potential factors that contribute or detract from the achievement of expected outcomes as identified by key informants

Accessibility as a barrier to use

The evaluation findings revealed some issues with access to the data that may lead to adverse effects on their use for research and policy development. Specifically, though most researchers are aware of the data, many are unaware of how to access them. This view is corroborated by external interviewees who believe that the lack of accessibility (or complexity of access) is a barrier to the use of the data.

Documents reviewed revealed that an analytical plan, a strategic document describing the CHMS analytical program and dissemination strategy, was developed in 2005 and updated several times—the last time in 2010. This plan was intended to be a living document that would facilitate the optimal use of CHMS data and define the scope of the analysis to maximize the data's impact and utility. The plan outlines the analytical objectives and the analytical activities related to the survey, including those that support them.

According to external researchers, most of the deliverables that were specified in the plan were completed, but the interviewees were presently unaware of the current analytical plan. They felt that there should be better communication and better collaboration to avoid duplicating research and analysis should different research groups look at the same data.

In the same vein, the evaluation study found that the program does not formally and systematically track all publications that use CHMS data, and interviews with external researchers confirmed that they do not always inform CHMS management about their peer-reviewed publications. This fact contributes to the previously identified weaknesses with regard to coordination and better communication of analytical efforts among researchers and other interested parties. The need for comprehensive, formal and up-to-date information on all studies and publications that are based on CHMS data was clearly identified by several external interviewees as a prerequisite for good collaboration. Researchers would know whether someone has already studied and published results on their potential focus of interest before starting their studies; thus, "they would be able to plan and scope their research better."

According to CHMS management, starting this fall, CHMS will be part of the Health Statistics Division analytical planning process, which is conducted every year. The analytical plan will be produced based on consultations with key stakeholders. As part of this planning exercise, decisions will be made about future articles and shared responsibilities between Statistics Canada, HC, PHAC and external experts.

Other factors

Budgetary restrictions and financial pressure were specified by most of the interviewees as a potential factor that could detract from the achievement of expected outcomes. External researchers (cost-recovery clients) specified cuts to cost-recovery funding, which would mean shorter content and fewer measures.

Program management mentioned that the sustainability of the clinic staff and the knowledge that resides in a few key people could potentially affect the delivery of CHMS. In fact, they are currently developing a new human resources strategy for the mobile clinic staff and are trying to ensure that knowledge transfer occurs. It was also specified during interviews that they are working on a succession planning strategy for mitigating the risks associated with the lack of knowledge transfer; however, no evidence became available within the timeframe of the evaluation study.

One external researcher mentioned a possible "perception danger" as a factor that might affect the survey. He feared that because a trend cannot be observed at the present time, it may be perceived that the measures are not worth the money. He explained that some trends can take a very long period of time (e.g., 20 years) to emerge.

4.2.1.6 Unintended impacts from the implementation of the Canadian Health Measures Survey

There were no significant unintended impacts revealed by this evaluation. Given the short period of time CHMS has been in place, it may be too early to observe unintended outcomes.

According to program management, however, since the survey was modeled after NHANES—a survey that has existed for the last 50 years—there was an opportunity to take into consideration their experience and lessons learned, which, in turn, decreased the likelihood of unintended impacts. According to the internal interviewees, the complexity of the survey and the required extensive and thorough planning process to ensure the controlled environment of operations and quality of data further limits the likelihood of variations and unintended effects occurring and being observed.

In terms of positive impacts, a couple of examples were mentioned by interviewees: 1) some individual medical problems were identified and respondents were notified as soon as possible, which allowed them to seek advice from their doctor in a timely manner; 2) some environmental contaminant data that were collected as supplementary became needed earlier than expected. For example, measures of selenium were included in the metal profile along with other planned measures, because it was easy to include them with the same collection and laboratory efforts. Recently, they became needed for a selenium risk assessment that was not in the plans during collection, as nationally representative data on selenium.

4.2.2.1 Resources in place to support the implementation of the Canadian Health Measures Survey

Evidence shows there was a clear shift from cost recovery to core funding after the 2008 Action Plan, as detailed in charts 3 and 4 below. Cost recovery changed from 90.6% in 2007/2008 to 24.2% in 2011/2012; core funding changed from 9.4% in 2007/2008 to 75.8% in 2011/2012. Many interviewees confirmed the Action Plan's positive impact by stabilizing the funding for CHMS.

4.2.2.2 Possible efficiencies that are being gained as a result of the Action Plan

Internal and external interviewees agree that the CHMS infrastructure provides to HC and PHAC the opportunity to reduce their program costs. They can invest their funding in additional health measures rather than in developing their own parallel infrastructure.

CHMS worked in collaboration with HC's First Nations and Inuit Health Branch (FNIHB) when biomonitoring the First Nations population, which is part of FNIHB's responsibilities under the Action Plan. According to CHMS program management, using comparable methods, procedures, tools, and reference labs created efficiencies for FNIHB at the operational level, but also provided better comparability of the results because the same measures were used in First Nations population as in the general Canadian population. However, program management believes that there is room for even more collaboration between the two initiatives.

Internal and external researchers suggested that a collaborative approach in research provides the best opportunities to combine the strengths of all parties involved, and to increase the knowledge capacity and the analytical power of the studies. Statistics Canada's researchers can leverage their technical capacity, including easy access, manipulation and statistical interpretation of the data. External researchers tend to be the subject-matter experts and thus, they are well positioned to interpret, produce and share results with the scientific community. Through this collaboration, CHMS contributes to the increased analytical capacity in the health domain.

4.2.2.3 Possible duplications, complementarities or alternatives

Evaluation findings revealed that most key informants believed there is no duplication of CHMS with other provincial or international surveys. In fact, right now in Canada there are no other nationally representative direct health measures survey data. While several provincial surveys do exist (e.g., Ontario Health Survey), most of them do not include physical measures or they use self-selected sampling, which, according to program management, could cause bias issues (e.g., only healthy people may choose to participate). Internationally, many countries (approximately 20) are conducting similar programs, and, according to internal management interviews, there are efforts to compare the results of individual countries.

As previously mentioned, evidence has also pointed to the fact that CHMS plays a role in providing complementary data to other kinds of surveys (e.g., self-reported, administrative) collected in Canada, as it helps to validate and adjust them.

Although all key informants believed that CHMS should be funded by the federal government, some external researchers did raise the question of whether the federal government should be conducting all activities versus contracting out some of the work. Alternative models for program delivery were suggested, such as completing some of the CHMS work by non-federal government contracts; however, this alternative may impact respondents' level of trust in CHMS, as well as CHMS's response rate and data quality.