Evaluation of the Demography, Aboriginal and Other Social Statistics Program
2010-2011 to 2012-2013

Relevance

The DAOSS Program contributes significantly to multiple areas of public interest and society's well-being and plays a strategic role in informing public debate on socioeconomic issues, in supporting social policy development and in guiding public and private decision making. The three focus components respond to needs that are specific to their subject-matter orientation and are relevant to the work of all levels of government and of many national and international groups and initiatives.

Overall, the DAOSS Program is in a good position to maintain its relevance; however, some areas of particular risks, challenges or gaps exist and need to be addressed. The need for consistent data disaggregated at a smaller geographical level (i.e. municipalities, on-reserve communities) or sub-groups level (i.e. Aboriginal groups, visible minorities) appears to be prominent. The improvement of the measures of interprovincial-territorial migration are linked to a growing area of interest of significant impact, hence, the Demography Division's efforts in this direction need to continue. In the scope of the GSS, the need for more analytical products related to historical series, to social trends occurring over time or to emerging social issues has been identified. In the field of statistics on Aboriginal people, the limited information for the on-reserve population is a gap that requires attention. Furthermore, there are concerns regarding the coherence, comparability and representativeness of the available data, which reduce the analytical power of the existing information on Aboriginal people.

The evaluation findings confirmed that the DAOSS Program is in line with the current federal government priorities and commitments as it supports multiple federal initiatives including legal and contractual obligations. It is also in line with the priorities and the role of Statistics Canada—not only because of its clear link to the agency's strategic outcomes, but also because each of the focus components is interrelated with other statistical programs and is a part of the entire national statistical structure. Further, the federal government appears to be in the best position to deliver the program as there is no other organization that has the capacity, the reputation, and the public trust to collect and process statistical information with comparable scope and quality. Other similar activities conducted by parties external to Statistics Canada are considered to be complementary rather than viewed as an alternative to the DAOSS Program.

Performance (efficiency and economy)

The evaluation findings confirm that the DAOSS Program has made significant investments (effort and reorganization) in efficiency-generating projects oriented toward: (1) reducing response burden by increasing the use of alternative data sources; (2) streamlining and consolidating the operations by using common tools^{Footnote 1}, and (3) incorporating new technologies in operations (e.g., e-questionnaire, interactive website, social media). The program is making progress in adopting a results-based culture; however, it is still too early to assess its implementation. Given the limitations of the financial system, it was not possible in this context to assess the cost of the inputs and whether optimization was reached. However, financial figures for the DAOSS Program as a whole provided some evidence that all variances between the planned budget and actual expenditures have not exceeded 4%, which indicates a good management of resources in general.

Effectiveness

Overall, statistical products on population estimates and projections are accessible, although some concerns were expressed with the transition to the new format of CANSIM tables that resulted in inconsistencies and fewer content details in some sets, and that the search engine is not efficient enough to locate some specific information. The evaluation found that the population estimates and projections were released in a timely manner. The program has met the delivery dates and standards, and has improved its responsiveness to the increased demand of more frequent population projections by enhancing its microsimulation activities.

Evaluation findings suggest that the demographic statistical information empowers users by increasing their demography-related knowledge and capacity. A closer look at the interpretability, coherency and accuracy reveals that overall the population estimates and projections are coherent and accurate. The Demography Division has undertaken many initiatives to improve the coherency and accuracy of its data, including the harmonization of the administrative data and statistical products with other Statistics Canada divisions. Currently, the PE&P is the single internal provider of most demographic estimates. The division is also working on initiatives to address challenges around the coherency and accuracy of indicators for internal migration. This is a well-known issue in the domain of demography and is not unique to Canada. In terms of interpretability, findings suggest that there is some room for improvement. Some users are dissatisfied with the public availability and comprehension of up-to-date technical papers, metadata analysis and methodology papers, as well as accessibility through the website interface and availability of written explanations of the concepts in a format and language that a non-statistician could understand.

All lines of evidence confirm that the statistical information on population estimates and projections is widely used by governments, public institutions, academic researchers, private sector, and Canadians. Multiple examples of use were found in documents, and were provided by key informants and survey respondents. The bibliometric and webometric analysis confirmed that the information has been used in the domain of social sciences and humanities, and the natural sciences and engineering for academic research.

Efficiency

The PE&P is currently working on initiatives intended to improve its efficiency, such as the harmonization of the demographic content, improved documentation practices, the incorporation of new technology for dissemination and microsimulation, and the identification of new sources of administrative data. Efforts to increase the use of administrative data must continue. It is worth mentioning that population projections' projects contribute significantly to the revenue of the component (almost 50%), and the Demosim microsimulation activities alone generated 24% of the annual revenue in 2013/2014.

Effectiveness

Overall, the review found that the GSS statistical information is accessible. A wide range of GSS products are available to key stakeholders, the general public and researchers, through a variety of channels (e.g., the web, RDCs or customized services). The major GSS outputs are released in a timely manner, according to the planned schedules and deadlines. Areas of improvement identified are linked to the need to make the website more user-friendly to better acquire the information, and reviewing the frequency of cycles for some themes so that appropriate adjustments could be made in the future with the new GSS redesign.

By providing accurate, interpretable and coherent statistical information, the GSS is seen as a trusted source of social data and products which contribute to the enhancement of the knowledge in the social domain. While the efforts of the program toward achieving better results were acknowledged, there is still room for improvement in terms of interpretability (e.g., providing more methodological papers and explanations around the GSS changes) and coherency (e.g., harmonizing the content through cycles over time to ensure comparability and to allow the establishment of trends over time for some specific variables). Concerns were also expressed with regard to the challenges that the GSS is facing with declining response rates that may have quality implications.

Evaluation evidence provides strong confirmation of the fact that GSS statistical information is widely used, to inform a multitude of diverse activities carried out by clients such as policy-makers, governments, private and public institutions, academic institutions and social researchers, not-for-profit organizations, etc. The GSS information is also used by other statistical surveys, for comparability and validation, and to supplement the information.

Efficiency

Many initiatives were put in place to generate efficiencies and cost savings. For example, the implementation of e-questionnaires for data collection is expected to stabilize or increase response rates and improve the quality of data. However, it is still too early to analyze the real effect of this change on efficiency. The exploration of opportunities for the use of more administrative data sources in lieu of primary data is seen as a potential option for cost saving as well as for improving the quality and coherence of the data. It is also believed that an interdepartmental forum, which oversees horizontal social areas of statistical interest at the strategic level, is a much needed component of the governance structure of social statistics, in order to better identify priorities, avoid duplication, fill information gaps, and create synergies across policy departments.

Effectiveness

Overall, the statistical information on Aboriginal people is accessible. There are some concerns and uncertainties about the accessibility of on-reserve data collected by the First Nations Information Governance Centre. Statistical information on Aboriginal people is being released in a timely manner. Some preferences for more frequent collection and some issues with data being outdated have been expressed.

While the statistical information on Aboriginal people from the Aboriginal Peoples Survey (APS) is of good quality, a number of challenges and potential risks to maintaining the accuracy were identified. To mitigate the situation, the program is working towards increasing the use of administrative data. There are still concerns about the comparability and coherence of off-reserve/on-reserve data, data over time, and the comparability of statistical information on Aboriginal people with general population information. The interpretability of the statistics on Aboriginal people is good because metadata is available and Statistics Canada provides good support and response to custom tabulations and special analyses.

In the public sector, statistics on Aboriginal people are used for planning, policy development, decision making, reporting and funding allocation. The main key users are federal partners. The data also support the activities of National Aboriginal organizations and Aboriginal governments and communities. Some limitations to data use are a result of aggregating the data at a national level, which results in blending the specifics of the different Aboriginal groups, and restricts the analysis of smaller subgroups of the Aboriginal population. Despite some issues linked to the insufficient liaison capacity and the scope of training activities, evidence suggests that these program components are very much needed and useful, and they could create leverage for both Statistics Canada and the Aboriginal communities if their full potential is explored.

Efficiency

Between 61% and 80% of the activities of the Aboriginal Statistics component are cost-recovery funded. Consequently, the efficiency of statistics on Aboriginal people is highly influenced by factors that are beyond its direct control. The elimination of the training program in 2012 limited the opportunities for Statistics Canada to communicate with the Aboriginal communities. This might have an impact (yet unknown) on the quality of the data in the future, and in turn, on the effectiveness of the program. The internal governance mechanisms are solid and are working well; however, potential for improvement regarding coordination with the National Aboriginal Organizations and a need for a better definition of the roles and responsibilities with federal partners have been identified.

Recommendation 1 (DAOSS Program):

To maintain the continued relevance of data products to the needs of its clients, the DAOSS Program should establish a structured process for documenting, analyzing and responding to constructive feedback from the full range of its users. The process should include a systematic tracking and documentation of available information on these users.

Recommendation 2 (DAOSS Program):

The DAOSS Program in collaboration with Dissemination Division should consider actions to strengthen the analytical capacity of its data and products, and to increase their value to the users. To the extent possible:

1. the Social and Aboriginal Statistics Division should look at appropriate alternatives, to increase the availability of disaggregated data for the General Social Survey and the Aboriginal Statistics components;
2. the Social and Aboriginal Statistics Division and the Demography Division should ensure that metadata are comprehensive and interpretable by users with varied statistical proficiency, for the General Social Survey, the Aboriginal Statistics and the Population Estimates and Projections components; and
3. the Dissemination Division should ensure that the implementation of Statistics Canada's New Dissemination Model will provide users with varied statistical proficiency effective search tools and streamlined access to products.

Recommendation 3 (General Social Survey):

The General Social Survey component should consider actions to strengthen its data and products and increase the value of the survey to users. To the extent possible, efforts should be focused on:

1. expansion of the scope and the variety of analytical products, including both products that involve re-packaging and utilization of existing data, and products that address themes of emerging importance and interest to key users;
2. revision of the frequency of the survey themes; and
3. revision of existing fora and interdepartmental consultation mechanisms to identify potential horizontal social statistical interest.

Recommendation 4 (Population Estimates and Projections):

The Demography Division should continue its efforts in providing relevant demographic data particularly by focusing its developmental and research work on the increased use of administrative data to improve the estimations of different components of population change (e.g. interprovincial migration), and by exploring ways to effectively address users needs for population projections.