Relevance

The DAOSS Program contributes significantly to multiple areas of public interest and society's well-being and plays a strategic role in informing public debate on socioeconomic issues, in supporting social policy development and in guiding public and private decision making. The three focus components respond to needs that are specific to their subject-matter orientation and are relevant to the work of all levels of government and of many national and international groups and initiatives.

Overall, the DAOSS Program is in a good position to maintain its relevance; however, some areas of particular risks, challenges or gaps exist and need to be addressed. The need for consistent data disaggregated at a smaller geographical level (i.e. municipalities, on-reserve communities) or sub-groups level (i.e. Aboriginal groups, visible minorities) appears to be prominent. The improvement of the measures of interprovincial-territorial migration are linked to a growing area of interest of significant impact, hence, the Demography Division's efforts in this direction need to continue. In the scope of the GSS, the need for more analytical products related to historical series, to social trends occurring over time or to emerging social issues has been identified. In the field of statistics on Aboriginal people, the limited information for the on-reserve population is a gap that requires attention. Furthermore, there are concerns regarding the coherence, comparability and representativeness of the available data, which reduce the analytical power of the existing information on Aboriginal people.

The evaluation findings confirmed that the DAOSS Program is in line with the current federal government priorities and commitments as it supports multiple federal initiatives including legal and contractual obligations. It is also in line with the priorities and the role of Statistics Canada—not only because of its clear link to the agency's strategic outcomes, but also because each of the focus components is interrelated with other statistical programs and is a part of the entire national statistical structure. Further, the federal government appears to be in the best position to deliver the program as there is no other organization that has the capacity, the reputation, and the public trust to collect and process statistical information with comparable scope and quality. Other similar activities conducted by parties external to Statistics Canada are considered to be complementary rather than viewed as an alternative to the DAOSS Program.

Performance (efficiency and economy)

The evaluation findings confirm that the DAOSS Program has made significant investments (effort and reorganization) in efficiency-generating projects oriented toward: (1) reducing response burden by increasing the use of alternative data sources; (2) streamlining and consolidating the operations by using common tools^{Footnote 1}, and (3) incorporating new technologies in operations (e.g., e-questionnaire, interactive website, social media). The program is making progress in adopting a results-based culture; however, it is still too early to assess its implementation. Given the limitations of the financial system, it was not possible in this context to assess the cost of the inputs and whether optimization was reached. However, financial figures for the DAOSS Program as a whole provided some evidence that all variances between the planned budget and actual expenditures have not exceeded 4%, which indicates a good management of resources in general.

Effectiveness

Overall, statistical products on population estimates and projections are accessible, although some concerns were expressed with the transition to the new format of CANSIM tables that resulted in inconsistencies and fewer content details in some sets, and that the search engine is not efficient enough to locate some specific information. The evaluation found that the population estimates and projections were released in a timely manner. The program has met the delivery dates and standards, and has improved its responsiveness to the increased demand of more frequent population projections by enhancing its microsimulation activities.

Evaluation findings suggest that the demographic statistical information empowers users by increasing their demography-related knowledge and capacity. A closer look at the interpretability, coherency and accuracy reveals that overall the population estimates and projections are coherent and accurate. The Demography Division has undertaken many initiatives to improve the coherency and accuracy of its data, including the harmonization of the administrative data and statistical products with other Statistics Canada divisions. Currently, the PE&P is the single internal provider of most demographic estimates. The division is also working on initiatives to address challenges around the coherency and accuracy of indicators for internal migration. This is a well-known issue in the domain of demography and is not unique to Canada. In terms of interpretability, findings suggest that there is some room for improvement. Some users are dissatisfied with the public availability and comprehension of up-to-date technical papers, metadata analysis and methodology papers, as well as accessibility through the website interface and availability of written explanations of the concepts in a format and language that a non-statistician could understand.

All lines of evidence confirm that the statistical information on population estimates and projections is widely used by governments, public institutions, academic researchers, private sector, and Canadians. Multiple examples of use were found in documents, and were provided by key informants and survey respondents. The bibliometric and webometric analysis confirmed that the information has been used in the domain of social sciences and humanities, and the natural sciences and engineering for academic research.

Efficiency

The PE&P is currently working on initiatives intended to improve its efficiency, such as the harmonization of the demographic content, improved documentation practices, the incorporation of new technology for dissemination and microsimulation, and the identification of new sources of administrative data. Efforts to increase the use of administrative data must continue. It is worth mentioning that population projections' projects contribute significantly to the revenue of the component (almost 50%), and the Demosim microsimulation activities alone generated 24% of the annual revenue in 2013/2014.

Effectiveness

Overall, the review found that the GSS statistical information is accessible. A wide range of GSS products are available to key stakeholders, the general public and researchers, through a variety of channels (e.g., the web, RDCs or customized services). The major GSS outputs are released in a timely manner, according to the planned schedules and deadlines. Areas of improvement identified are linked to the need to make the website more user-friendly to better acquire the information, and reviewing the frequency of cycles for some themes so that appropriate adjustments could be made in the future with the new GSS redesign.

By providing accurate, interpretable and coherent statistical information, the GSS is seen as a trusted source of social data and products which contribute to the enhancement of the knowledge in the social domain. While the efforts of the program toward achieving better results were acknowledged, there is still room for improvement in terms of interpretability (e.g., providing more methodological papers and explanations around the GSS changes) and coherency (e.g., harmonizing the content through cycles over time to ensure comparability and to allow the establishment of trends over time for some specific variables). Concerns were also expressed with regard to the challenges that the GSS is facing with declining response rates that may have quality implications.

Evaluation evidence provides strong confirmation of the fact that GSS statistical information is widely used, to inform a multitude of diverse activities carried out by clients such as policy-makers, governments, private and public institutions, academic institutions and social researchers, not-for-profit organizations, etc. The GSS information is also used by other statistical surveys, for comparability and validation, and to supplement the information.

Efficiency

Many initiatives were put in place to generate efficiencies and cost savings. For example, the implementation of e-questionnaires for data collection is expected to stabilize or increase response rates and improve the quality of data. However, it is still too early to analyze the real effect of this change on efficiency. The exploration of opportunities for the use of more administrative data sources in lieu of primary data is seen as a potential option for cost saving as well as for improving the quality and coherence of the data. It is also believed that an interdepartmental forum, which oversees horizontal social areas of statistical interest at the strategic level, is a much needed component of the governance structure of social statistics, in order to better identify priorities, avoid duplication, fill information gaps, and create synergies across policy departments.

Effectiveness

Overall, the statistical information on Aboriginal people is accessible. There are some concerns and uncertainties about the accessibility of on-reserve data collected by the First Nations Information Governance Centre. Statistical information on Aboriginal people is being released in a timely manner. Some preferences for more frequent collection and some issues with data being outdated have been expressed.

While the statistical information on Aboriginal people from the Aboriginal Peoples Survey (APS) is of good quality, a number of challenges and potential risks to maintaining the accuracy were identified. To mitigate the situation, the program is working towards increasing the use of administrative data. There are still concerns about the comparability and coherence of off-reserve/on-reserve data, data over time, and the comparability of statistical information on Aboriginal people with general population information. The interpretability of the statistics on Aboriginal people is good because metadata is available and Statistics Canada provides good support and response to custom tabulations and special analyses.

In the public sector, statistics on Aboriginal people are used for planning, policy development, decision making, reporting and funding allocation. The main key users are federal partners. The data also support the activities of National Aboriginal organizations and Aboriginal governments and communities. Some limitations to data use are a result of aggregating the data at a national level, which results in blending the specifics of the different Aboriginal groups, and restricts the analysis of smaller subgroups of the Aboriginal population. Despite some issues linked to the insufficient liaison capacity and the scope of training activities, evidence suggests that these program components are very much needed and useful, and they could create leverage for both Statistics Canada and the Aboriginal communities if their full potential is explored.

Efficiency

Between 61% and 80% of the activities of the Aboriginal Statistics component are cost-recovery funded. Consequently, the efficiency of statistics on Aboriginal people is highly influenced by factors that are beyond its direct control. The elimination of the training program in 2012 limited the opportunities for Statistics Canada to communicate with the Aboriginal communities. This might have an impact (yet unknown) on the quality of the data in the future, and in turn, on the effectiveness of the program. The internal governance mechanisms are solid and are working well; however, potential for improvement regarding coordination with the National Aboriginal Organizations and a need for a better definition of the roles and responsibilities with federal partners have been identified.

Recommendation 1 (DAOSS Program):

To maintain the continued relevance of data products to the needs of its clients, the DAOSS Program should establish a structured process for documenting, analyzing and responding to constructive feedback from the full range of its users. The process should include a systematic tracking and documentation of available information on these users.

Recommendation 2 (DAOSS Program):

The DAOSS Program in collaboration with Dissemination Division should consider actions to strengthen the analytical capacity of its data and products, and to increase their value to the users. To the extent possible:

1. the Social and Aboriginal Statistics Division should look at appropriate alternatives, to increase the availability of disaggregated data for the General Social Survey and the Aboriginal Statistics components;
2. the Social and Aboriginal Statistics Division and the Demography Division should ensure that metadata are comprehensive and interpretable by users with varied statistical proficiency, for the General Social Survey, the Aboriginal Statistics and the Population Estimates and Projections components; and
3. the Dissemination Division should ensure that the implementation of Statistics Canada's New Dissemination Model will provide users with varied statistical proficiency effective search tools and streamlined access to products.

Recommendation 3 (General Social Survey):

The General Social Survey component should consider actions to strengthen its data and products and increase the value of the survey to users. To the extent possible, efforts should be focused on:

1. expansion of the scope and the variety of analytical products, including both products that involve re-packaging and utilization of existing data, and products that address themes of emerging importance and interest to key users;
2. revision of the frequency of the survey themes; and
3. revision of existing fora and interdepartmental consultation mechanisms to identify potential horizontal social statistical interest.

Recommendation 4 (Population Estimates and Projections):

The Demography Division should continue its efforts in providing relevant demographic data particularly by focusing its developmental and research work on the increased use of administrative data to improve the estimations of different components of population change (e.g. interprovincial migration), and by exploring ways to effectively address users needs for population projections.

Relevance

Statistical information produced by ISP responds to legislative and policy requirements for information to support, among others, the national macroeconomic accounts, the Consumer Price Index calculations, and calculations of provincial transfers. These are used by federal government and external stakeholders as a basis for research; policy development; design and assessment of programs; and to fulfill national and international commitments. ISP responds to the Government of Canada's needs by providing information on income, pensions, and wealth. Information on what Canadians earn, spend and save is used to inform policy development, and to conduct research on the social and economic conditions of families and individuals. No other organization provides income statistics of the same scope, level of detail, and quality as does Statistics Canada.

Given the importance of income statistics in social and economic policy-making and business decision-making, there will continue to be an interest in conducting research that uses income statistics. Demographic changes in the Canadian population, specifically the aging population will drive interest in certain types of data and analyses, for example, pensions. Economic cycles also create interest in certain types of data, for example, income distribution during recession and post-recession periods. In addition to those topics, there are several areas of future research interest, such as: growing income disparities, debt, and the economic performance of Canadian companies in global markets. All these factors support the conclusion that there is a strong continued need for ISP.

Performance (Achievement of Outcomes)

Overall, accessibility and timeliness are satisfactory among users. There may be opportunities to improve the timeliness of periodic/one-time surveys and accessibility issues related to Personal Income Tax data were noted. It should be noted that in 2013-14, the Longitudinal Administrative Database (LAD) was placed in the Federal Research Data Centre as a pilot project to assess the feasibility of allowing access to RDC researchers. The LAD consists of a 20% longitudinal sample of Canadian tax filers and provides researchers and analysts with a tool for studying the changes in income experienced by individuals and their families. The pilot project is currently being evaluated and the next phase will be to extend the pilot to another RDC.

A strong majority of users are satisfied with the accuracy, coherence and interpretability of the program's statistical information. There are concerns raised by users. These relate to information on changes to methodologies and the loss of longitudinal surveys. These concerns could be addressed through more communication between ISP and certain of its product users.

Income statistics have been used extensively to inform debate, research and decision-making. Users, which include provinces and territories, non-government organizations, policy think tanks, financial institutions, researchers, and academics, use ISP information for research purposes. Examples of research include: income disparities (including gender disparities), poverty analyses, and determinants of personal income.

Performance (Efficiency and Economy)

ISP has increased the economy and efficiency of its operations. A number of initiatives have been undertaken to reduce costs and increase efficiency.

ISP continually investigates new data sources to support the program. Users indicated that these new directions should continue to be examined.

Relevance

Tourism is an important sector in supporting the Canadian economy, and tourism statistics contribute to several important travel market analyses and provide key information on this industry to the National Accounts. In 2010, tourism was responsible for $73.4 billion in revenues and represented approximately 2 percent of Canada's overall gross domestic product. It is a dynamic and complex sector, characterized by multiple modes of transportation, multiple points of entry, many geographical areas covered, many activities, and many users. Tourism data are important to users such as academics and businesses to conduct comparative analyses across Canadian jurisdictions and other travel destinations markets. The relevance of tourism statistics is reinforced by the fact that the TSP receives additional requests, on a cost-recovery basis.

While there are other sources of tourism statistics, their products would not meet all the needs of TSP users and, in particular, TSP's internal Statistics Canada users.

The complexity of tourism and the considerable economic impact of the tourism industry suggest that there will be a continued need for these statistics by a wide range of users, including the Government of Canada.

Performance (Achievement of Outcomes)

Accessibility of tourism data is satisfactory. Timeliness has been an issue, although improvements have been noted in this area. TSP's products are indeed contributing some level of knowledge.

Evaluation findings indicate that TPS has been challenged with data quality issues, primarily due to low return rates to its international travel survey. Funding reductions at one of TSP's primary clients in 2012 led to a significant reduction in the funding it provides to TSP. This revenue reduction led to a 50% reduction in sampling for the domestic tourism survey by TSP. As a consequence, the remaining funding partners for the domestic survey are now dissatisfied, which could lead to further funding withdrawals and further sampling reduction. Under the current resourcing model, if partners withdraw their funding, the TSP will no longer have the resources necessary to collect the national tourism data that it needs for the federal government's and Statistics Canada's purposes.

Performance (Efficiency and Economy)

There is evidence of efficient and economical elements within the program. The program has strived to increase economy by relying on other organizations to collect data. Funding received from cost-recovered activities also helps meet the needs of its internal users by funding the provision of data to internal clients. This approach demonstrates economy. However, it increases the risk to Statistics Canada, since TSP's ability to perform core activities depends on its success at finding external funding sources and meeting the needs of cost-recovery clients.

Relevance

Data from the Labour Statistics Program are important sources of information for assessing the economic health of the nation. LSP data are important for organizations such as the Bank of Canada and programs such as Employment Insurance. These data are used to meet legal requirements and international obligations. LSP outputs are an important contributor to the federal government's statutory obligations and policy framework, and to Canada's ability to conduct economic analyses. Because of the government's and country's need for high quality statistics in these topic areas, LSP's involvement in producing these data is fully consistent with federal roles and responsibilities. There will continue to be a demand for labour statistics on the part of governments, the private sector, academia, the not-for-profit sector, and Canadians interested in the economy. Continued interest in the state of the nation's economic health and the high rate of use of LSP data indicate that LSP's products will continue to make an important contribution.

Performance (Achievement of Outcomes)

LSP is achieving its outcomes. Labour statistics were considered to be accessible and timely. The quality of labour statistics data is high. A large majority of users are satisfied with the accuracy, interpretability and coherence of the data. LSP statistics are used to inform debate, research, and decision-making. Its information has been and continues to be used for purposes as varied as economic monitoring and forecasting, program delivery and monitoring, wage settlement, contract escalation, as well as to inform the general public.

Relevance

ESP provides information related to education, training and learning as well as a number of national indicators and some internationally comparable pan-Canadian education indicators. It also provides information that contributes to other Statistics Canada programs and fulfils international obligations. ESP plays a key role by aggregating and harmonizing information, thereby making inter-jurisdictional and international comparisons possible. Internationally, organization like the OECD and United Nations Statistics Division rely on data produced by ESP. Given the importance of education internationally, the demand for information on education statistics is expected to continue. Also, factors such as low population growth, an aging labour force, and rising demand for skilled workers will likely drive continued research into educational matters, furthering the demand for these statistics.

Performance (Achievement of Outcomes)

Users are satisfied to a large extent with the accessibility and timeliness of Education Statistics Program data. They are also satisfied with the accuracy, interpretability and coherence of the data. ESP data are useful to a range of organizations domestically and internationally. Thousands of pages on the Web and documents in scientific literature have cited ESP products.

Recommendation 1 (Income Statistics Program):

The Income Statistics Program should increase its communication with certain of its user groups, to address concerns related to information on changes to methodologies and quality of some data, and resolve issues to the extent possible. This communication could be achieved using existing committee structures, memoranda of understanding, service-level agreements, and other suitable mechanisms.

Recommendation 2 (Tourism Statistics Program):

1. In light of a low response rate for the International Travel Survey, the program should address issues related to methodologies and collection approaches to the Travel Statistics Program.
2. Considering the relative importance of cost recovery revenues for the Tourism Statistics Program, it is important to ensure that a reduction in cost recovery does not prevent the program from producing the statistics needed by Statistics Canada. LEIT management should develop a contingency plan for action if it were to lose a significant volume of revenue-generating clients. The plan could include steps such as the reallocation of resources and working with internal clients to prioritize the statistical inputs required to support Statistics Canada programs.

Conclusions

Relevance

The MEAP plays a critical role in supporting a number of decisions related to Canada's economic growth. First, it provides the required statistical information for the implementation of a number of legislative requirements, including those related to equalization payments, harmonized tax systems, and to the monitoring of foreign ownership and control of Canadian businesses. It also allows Canada to uphold a number of international reporting commitments to the international community such as the IMF and UN.

Evaluation findings also confirm that the nature of the economic information being gathered, the process by which it is gathered, and the purpose for which it is gathered are all consistent with the roles and responsibilities of the federal government. Moreover, considering the economic crisis that Canada has had to face during the period covered by the evaluation, there was a strong alignment between the activities undertaken through the MEAP and the priorities of the federal government.

What evaluation findings also indicate, however, is that a number of drivers are fundamentally changing the nature of economic activities and economic relationships among countries, which creates expectations that different or new statistical data will be produced to adequately monitor the impact of these new trends. To meet these new information needs, the Macroeconomic Accounts Branch has undertaken a number of initiatives, including the implementation of new international standards. While these efforts are significant, evaluation findings also confirm that a number of information gaps remain and, in order to adequately address them, new activities will need to be undertaken.

Achieving expected outcomes

The MEAP has made significant progress towards its expected outcomes. First, the program has allowed all key stakeholders to readily access statistical information related to many dimensions of Canada's economy. While some metadata information requires updating, statistical users have had access to information supporting the appropriate use of the data. While these statistics are released within the prescribed timeframe, the evaluation has found that there are growing pressures to release this statistical information within a shorter timeframe in order to meet the requirements and expectations of key stakeholders.

Second, the MEAP data has enhanced the level of knowledge of its key users. Evaluation findings point to a high level of satisfaction in relation to the interpretability of the data produced. This is particularly significant considering the increasing complexity of the information provided in light of the implementation of new international standards. A structural characteristic that works in favour of Statistics Canada is its high level of operational integration, which contributes to the coherence of the data produced. In this context, the MEAP has been in a position to produce data that have not required significant revisions. This is an indication of strong reliability, which is an important indicator of data accuracy.

Third, the MEAP data is used in a wide range of settings and for multiple purposes. The evaluation findings confirm that it is primarily in applied settings that MEAP data are used to shape economic, fiscal and monetary policies. While the data is also used for research purposes, current practices are such that the data is often used without being directly or properly cited in academic publications. This creates significant challenges in attempting to adequately measure the extent to which the data is used in academic settings.

Efficiency and economy

The evaluation findings on economy and efficiency, which focuses on the IASD as an illustrative example, found that the Division has been required to maintain data releases and other information products within a reduced budget. The fact that the Division has maintained its schedule of data releases with reduced staff levels creates an apparent lowering of unit cost in the production of the international accounts and statistics. On the surface, it would appear that IASD has become more efficient. However, against these efficiency gains must be set an increased potential for errors and slippages in the schedule, which would negatively affect relevance and reputation. While this risk appears to have been managed because of professional commitment, there is evidence that the information production processes are being progressively stretched.

A closely related issue is that the resource constraints may increasingly place the division in a position of not being able to fulfill its entire mandate. The changing nature of the economy, in particular the globalization of manufacturing and services, means that government and industry decision-makers need new varieties of information to set policy and make investments. In addition to meeting this demand for new information perspectives, the resource constraints also pose another relevancy risk, in that they limit the Division's ability to serve other federal government users who often are prepared to participate in a cost-recovery process. Also, the transition towards the centralization survey data collection activities has proven to be more challenging than anticipated, particularly for smaller divisions such as the IASD with highly-specialized survey activities. As a result, the expected financial relief on the Division did not materialize as initially expected.

Two other important findings emerged in the evaluation. First, the financial administration system in general and the time recording processes in particular do not support the tracking of staff time (costs) through the diverse activities that lead to the range of outputs. Second, while the Division maintains procedures that document the activities needed to support data releases, these are not business process models of the activities that support explicit measurement of activities needed to support the realization of the diverse range of IASD outputs.

Recommendations

Based on all evaluation findings gathered as part of this evaluation, the following recommendations are submitted for consideration by the Macroeconomic Accounts Branch:

Recommendation 1 (performance measurement):

It is recommended that the MEAP proceed with the implementation of a performance measurement system to ensure the availability of timely performance information to support ongoing program management and decision making, demonstrate the achievement of program expected outcomes and support future evaluations (including up-to-date business process models for outputs, list of outputs produced and released and a register of data users).

Recommendation 2 (relevance, efficiency and economy):

It is recommended that IASD, establish/implement the appropriate strategy to enhance its capacity and explore avenues to resume the provision of cost-recovery services to deal with the current relevancy risk, the increased potential for errors and slippages in the schedule and to address some of the unmet needs of users.

Recommendation 3 (performance):

To ensure that data users have a proper understanding of its macroeconomic products, it is recommended that the Macroeconomic Accounts Branch ensure the timely update of all applicable metadata information.