Evaluation of the Census of Population Program
(2009/2010 to 2014/2015)

The Census of Population Program

The CePop Program plans, develops and implements all collection, data processing and dissemination of the periodic decennial and quinquennial censuses of population, Canada's national inventory of key socioeconomic phenomena. The census provides a statistical portrait of Canada and its people. This program is the only reliable source of detailed data for small groups (such as lone-parent families, ethnic groups, industrial and occupational categories, and immigrants) and for areas as small as a city neighbourhood or as large as the country itself.

Prior to 2011 the approach for the CePop program included both a mandatory short-form questionnaire and a mandatory long-form questionnaire. The National Household Survey (NHS) was introduced in 2011 to replace the long-form questionnaire. The survey was administered on a voluntary basis to 30% of households, alongside the 2011 short-form census questionnaire. The NHS was designed to collect the same type of information that was collected by the previous long-form census. In 2011, approximately 2.7 million dwellings responded to the NHS.

Evaluation scope and methodology

The evaluation focused on the CePop and NHS performance from 2009/2010 to 2014/2015. Data for this evaluation were collected from five sources of evidence, including a literature review; a document review; a review of financial and administrative data; interviews with 84 representatives of the public sector, the private sector, non-governmental organizations and academia; and a bibliometric and webometric study. The evaluation strategy relied on qualitative and quantitative evidence. Data from these sources were analyzed and triangulated to develop the findings and generate recommendations based on the conclusions made.

The evaluation looked at the relevance, effectiveness and efficiency of the 2011 CePop Program cycle and the impact of the methodological changes between the 2006 and 2011 cycles. Similarly, the evaluation study examined the program's capacity to achieve its results, given the centralization of internal business processes. This was done by reviewing available information on the planning of the 2016 cycle and information associated with the implementation of best practices and lessons learned from the previous cycles.

It is important to note that the recommendations in this evaluation have been adjusted to reflect the recent contextual change related to the CePop Program, namely the fact that, in November 2015, during the reporting phase of the evaluation, the Government of Canada decided to restore the mandatory long-form census.

Findings

The evaluation findings corroborate Statistics Canada's position on the quality of the statistical information of the 2011 CePop Program. They also confirm findings from other independent sources, such as the 2014 Spring Report of the Auditor General of Canada.

Relevance of the program

The census is the only national source of basic information on each person and each household that is collected at a given time, at regular intervals, and with complete geographic coverage. Such information is essential to supporting the national administration, governance and public policy. The census is crucially important for the integrity of the national statistics system, as it provides a baseline for validating the information collected by Statistics Canada, as well as by other public and private institutions.

Performance of the program—achievement of expected outcomes

In comparison with the 2006 cycle (considered as a baseline), the quality of the census is better in 2011. However, the implementation of the 2011 approach resulted in additional costs, compared with what was planned, and had unfavourable effects on the quality of the statistical information from the NHS.

The quality of the statistical information from the 2011 Census cycle meets the program objectives. However, while the NHS is of high quality for a voluntary social survey, there are concerns about specific dimensions, such as coherence and accuracy, for populations of small domains. The change in methodology has had an impact on the trust of users in the official statistics produced by the CePop Program, even though this impact may not always be justified and may be seen as a result of biased perceptions. The following highlights provide more detail:

• The accuracy of the 2011 Census data is excellent and slightly better than in 2006. The accuracy of the 2011 NHS data is much better than anticipated in 2010 but lower than that of the 2006 long-form census. There is considerable uncertainty among external users about whether any differences between the 2011 NHS and the previous long-form census are the result of real changes in the population or methodological changes.
• The 2011 Census and the NHS data and products were released in a timely manner, with the availability of resources and within the time requirements for processing, quality control and assurance.
• The accessibility of the CePop Program products has improved significantly over time. Since 2012, all standard products generated by the program, including those for the 2011 Census and the NHS, have been released and are accessible to the general public in several different electronic formats, at no cost. The accessibility of data for small geographic areas has been affected by the higher suppression level established for the 2011 NHS statistics and by the fact that the cost of custom services and tabulations (in this context only) is a barrier for some users facing budget restrictions.
• The 2011 Census of Population data are coherent. The program made considerable efforts to assess and reduce the impact of the voluntary nature of the NHS on the comparability of the estimates. However, the change in methodology has made it challenging for users to compare the NHS data with earlier data.
• There are no issues with the interpretability of the 2011 Census data. However, it was acknowledged that for proper interpretation and use of the 2011 NHS data, users must know more about methodology than had previously been the case. Evidence suggests that the program has provided additional guidance to its data users in the form of expanded information.
   
  Nonetheless, for users (especially users of data for small geographic areas and subpopulations, or who have less analytical capacity), it is not always clear which data are reliable and how they may be interpreted. Users also noted the importance of releasing timely technical reports and all necessary metadata.
• The CePopProgram statistical information is responsive to the needs of users overall. In terms of content, no significant information gaps are identified. The NHS is more responsive to information needs at the national and provincial levels, and less responsive to needs related to low geographical levels, rare characteristics and comparability over time for some areas, compared with the 2006 cycle. Therefore, the program's ability to respond to needs varies by type of user. The change in approach also led to disadvantages or costs for users (e.g., in the amount of effort required to understand the changes, to use the data, to conduct validation, or to use replacement data). However, there is no clear evidence that the external users and stakeholders who were consulted implemented new surveys because of unfulfilled needs.
• The CePop Program data are used extensively by all levels of government to fulfill their mandates and responsibilities, by academic researchers, by non-governmental organizations, by the private sector and by Canadians. At Statistics Canada, the census data are used in other surveys for sampling and benchmarking. The evaluation found that the 2011 methodological change had some implications in terms of how the data are being used, and for what purposes.

Performance of the program—efficiency and economy

The late decision to change the approach in the 2011 cycle had a significant impact and led to challenges at several levels for internal and external actors: financial challenges, operational challenges, challenges in achieving desired outcomes (quality), and, finally, challenges to the credibility of the statistical information available. Overall, the late decisions about the approach by the Government of Canada (in 2010 and 2015) had a negative impact on the efficiency of the program.

Despite the exceptional circumstances resulting from the timing of the decision, it is recognized that good management practices led to the successful implementation of the program during the entire 2011 cycle and the first years of the 2016 cycle. Overall, the program is administered efficiently, and its management has demonstrated a commitment to reducing the cost of the program while ensuring its long-term sustainability. The planned costs of the 2011 cycle were reasonable in comparison with previous cycles for producing the planned outputs and achieving the expected outcomes. However, the decision to change the methodology translated into a cost increase for the 2011 cycle. For the implementation of the 2016 cycle, there are significant challenges related to the current partnership with Shared Services Canada (SSC), which may have an impact on the efficiency of the Census Program.

The CePop Program has demonstrated a commitment to exploring alternative means of achieving the intended program results. Potential opportunities may be considered for future cycles, as the Canadian CePop is expected to evolve. The anticipated changes are not isolated Canadian phenomena: most statistical agencies in developed countries are currently exploring and implementing innovative approaches and methodologies (e.g., increasing the use of administrative and other available data). The successful transition to more efficient censuses that produce better-quality data with a reduced cost and burden to respondents requires considerable long-term planning and effort.

Conclusions and recommendations

The quality of the statistical information from the 2011 Census cycle meets the program objectives. However, while the NHS is of high quality for a voluntary social survey, there are concerns about specific dimensions, such as accessibility (data for small areas and subpopulations), coherence and accuracy.

Based on the analysis of the implementation of the 2016 cycle and the recent change of direction—from voluntary to mandatory—it is expected that the quality of the statistical information for the NHS component should improve in the current cycle.

Recommendation #1 – Relevance and performance (accessibility, accuracy and coherence)

It is recommended that the CePop Program detail the measures taken to improve the quality of the statistical information for the dimensions of accessibility (data for small areas and subpopulations), accuracy and coherence to meet user needs in the 2016 cycle.

The change in methodology has had an impact on the trust of users in the official statistics produced by the CePop Program, even though this impact may not always be justified and may be seen as a result of biased perceptions. Consequently, the credibility of the statistical information among users had an impact on the level of use.

Recommendation #2 – Performance (trust and use)

It is recommended that the CePop Program ensure that an effective and proactive communication system is implemented to inform Canadians, particularly direct users of program data, of the orientation of the program and the exact nature (quality) of the data that are produced and that will be produced, to maintain the credibility of the program and its products over time.

Evidence suggests that the program has provided additional guidance to its data users in the form of expanded information. Nonetheless, for users (especially users of data for small geographic areas and subpopulations, or who have less analytical capacity), it is not always clear which data are reliable and how they may be interpreted. Users also noted the importance of releasing timely technical reports and all necessary metadata.

Recommendation #3 – Performance (interpretability)

It is recommended that the CePop Program ensure the availability of timely information and the necessary support to inform and guide the different types of data users, and satisfy multiple needs, given the different approaches in different census cycles.