Executive Summary and Management Response and Action plan
November 2014
Executive Summary
This report presents the results of the evaluation of the Macroeconomic Accounts Programs (MEAP). It covers three fiscal years, from 2010-2011 to 2012-2013. The evaluation was undertaken by the Evaluation and Performance Measurement Division.
This Report was approved by the Departmental Evaluation Committee and the Chief Statistician on November 19, 2014.
In accordance with the accountability requirements in the Treasury Board 2009 Policy on Evaluation and its Directive, this report is available to the public and posted on the departmental website in both official languages.
Statistics Canada also shared this report with its program delivery partners and key stakeholders, including the National Statistical Council.
Evaluation scope, purpose and methodology
The evaluation covers all components of the MEAP, with a particular focus on the International Accounts and Statistics Division (IASD). It addresses a number of questions relating to the continued need for the program, its alignment with government priorities, its consistency with federal roles and responsibilities, the achievement of its expected outcomes, and the extent to which it demonstrates efficiency and economy. It was included in the Departmental Risk-based Audit and Evaluation Plan for 2012-2013 to 2016-2017, which was approved by the Departmental Evaluation Committee on March 2013.
In accordance with the Government of Canada's Policy on Evaluation, the purpose of the MEAP evaluation is to provide an evidence-based, neutral assessment of its value for money, and more specifically, its relevance and performance. In addition, the evaluation design has incorporated all components of the Standard on Evaluation for the Government of Canada to ensure quality, neutrality and utility. Finally, the list of questions addressed in this evaluation incorporates all core issues identified in the Government of Canada's Directive on the Evaluation Function.
Information from multiple sources was used to address the evaluation questions, including:
- a document and literature review
- a review of financial and administrative data
- a series of key informant interviews with program representatives, internal and external users of MEAP data, as well as representatives of relevant international organizations
- a survey of data users
- case studies
- a bibliometric and webometric analysis.
The Macroeconomic Accounts Program
The Macroeconomic Accounts Program (MEAP) is expected to provide a comprehensive set of statistics on economic activities occurring in Canada, as well as economic activities between Canada and the rest of the world. At the time of the evaluation, the program included several components falling into two main groupings: the core accounts and a set of additional accounts. Each of these accounts has its own purpose, which is to provide a specific perspective on the nature of the Canadian economy.
The four components of the MEAP's core accounts provide the fundamental statistics relating to all aspects of Canada's core economic activities. These accounts include the Canadian System of National Accounts, the Canadian Input-Output Tables, the Government Finance Statistics and the International Accounts. The additional accounts falling under the MEAP include the productivity accounts, the capital stock program, as well as satellite accounts covering different aspects of the Canadian economy, such as tourism and culture.
The MEAP is expected to support Statistics Canada's two strategic outcomes by ensuring that:
- all Canadians have access to timely, relevant and quality statistical information on Canada's changing economy and society for informed debate, research and decision making on social and economic issues
- specific client needs for high-quality and timely statistical services are met.
The Macroeconomic Accounts Branch is responsible for the overall management of the MEAP. Its work is primarily supported by four divisions:
- International Accounts and Statistics Division (IASD)
- Industry Accounts Division (IAD)
- Public Sector Statistics Division (PSSD)
- National Economic Accounts Division (NEAD).
During the three years covered by the evaluation, the level of resources allocated to the MEAP has fluctuated. The number of budgeted full-time equivalents (FTEs) has varied between 301 and 333, and its total budgeted expenditures have varied between $26 million and $30 million.
Evaluation conclusions and recommendations
The analysis of the information gathered as part of this evaluation resulted in findings and conclusions about the relevance and performance of the MEAP, which led to three recommendations.
Conclusions
Relevance
The MEAP plays a critical role in supporting a number of decisions related to Canada's economic growth. First, it provides the required statistical information for the implementation of a number of legislative requirements, including those related to equalization payments, harmonized tax systems, and to the monitoring of foreign ownership and control of Canadian businesses. It also allows Canada to uphold a number of international reporting commitments to the international community such as the IMF and UN.
Evaluation findings also confirm that the nature of the economic information being gathered, the process by which it is gathered, and the purpose for which it is gathered are all consistent with the roles and responsibilities of the federal government. Moreover, considering the economic crisis that Canada has had to face during the period covered by the evaluation, there was a strong alignment between the activities undertaken through the MEAP and the priorities of the federal government.
What evaluation findings also indicate, however, is that a number of drivers are fundamentally changing the nature of economic activities and economic relationships among countries, which creates expectations that different or new statistical data will be produced to adequately monitor the impact of these new trends. To meet these new information needs, the Macroeconomic Accounts Branch has undertaken a number of initiatives, including the implementation of new international standards. While these efforts are significant, evaluation findings also confirm that a number of information gaps remain and, in order to adequately address them, new activities will need to be undertaken.
Achieving expected outcomes
The MEAP has made significant progress towards its expected outcomes. First, the program has allowed all key stakeholders to readily access statistical information related to many dimensions of Canada's economy. While some metadata information requires updating, statistical users have had access to information supporting the appropriate use of the data. While these statistics are released within the prescribed timeframe, the evaluation has found that there are growing pressures to release this statistical information within a shorter timeframe in order to meet the requirements and expectations of key stakeholders.
Second, the MEAP data has enhanced the level of knowledge of its key users. Evaluation findings point to a high level of satisfaction in relation to the interpretability of the data produced. This is particularly significant considering the increasing complexity of the information provided in light of the implementation of new international standards. A structural characteristic that works in favour of Statistics Canada is its high level of operational integration, which contributes to the coherence of the data produced. In this context, the MEAP has been in a position to produce data that have not required significant revisions. This is an indication of strong reliability, which is an important indicator of data accuracy.
Third, the MEAP data is used in a wide range of settings and for multiple purposes. The evaluation findings confirm that it is primarily in applied settings that MEAP data are used to shape economic, fiscal and monetary policies. While the data is also used for research purposes, current practices are such that the data is often used without being directly or properly cited in academic publications. This creates significant challenges in attempting to adequately measure the extent to which the data is used in academic settings.
Efficiency and economy
The evaluation findings on economy and efficiency, which focuses on the IASD as an illustrative example, found that the Division has been required to maintain data releases and other information products within a reduced budget. The fact that the Division has maintained its schedule of data releases with reduced staff levels creates an apparent lowering of unit cost in the production of the international accounts and statistics. On the surface, it would appear that IASD has become more efficient. However, against these efficiency gains must be set an increased potential for errors and slippages in the schedule, which would negatively affect relevance and reputation. While this risk appears to have been managed because of professional commitment, there is evidence that the information production processes are being progressively stretched.
A closely related issue is that the resource constraints may increasingly place the division in a position of not being able to fulfill its entire mandate. The changing nature of the economy, in particular the globalization of manufacturing and services, means that government and industry decision-makers need new varieties of information to set policy and make investments. In addition to meeting this demand for new information perspectives, the resource constraints also pose another relevancy risk, in that they limit the Division's ability to serve other federal government users who often are prepared to participate in a cost-recovery process. Also, the transition towards the centralization survey data collection activities has proven to be more challenging than anticipated, particularly for smaller divisions such as the IASD with highly-specialized survey activities. As a result, the expected financial relief on the Division did not materialize as initially expected.
Two other important findings emerged in the evaluation. First, the financial administration system in general and the time recording processes in particular do not support the tracking of staff time (costs) through the diverse activities that lead to the range of outputs. Second, while the Division maintains procedures that document the activities needed to support data releases, these are not business process models of the activities that support explicit measurement of activities needed to support the realization of the diverse range of IASD outputs.
Recommendations
Based on all evaluation findings gathered as part of this evaluation, the following recommendations are submitted for consideration by the Macroeconomic Accounts Branch:
Recommendation 1 (performance measurement):
It is recommended that the MEAP proceed with the implementation of a performance measurement system to ensure the availability of timely performance information to support ongoing program management and decision making, demonstrate the achievement of program expected outcomes and support future evaluations (including up-to-date business process models for outputs, list of outputs produced and released and a register of data users).
Recommendation 2 (relevance, efficiency and economy):
It is recommended that IASD, establish/implement the appropriate strategy to enhance its capacity and explore avenues to resume the provision of cost-recovery services to deal with the current relevancy risk, the increased potential for errors and slippages in the schedule and to address some of the unmet needs of users.
Recommendation 3 (performance):
To ensure that data users have a proper understanding of its macroeconomic products, it is recommended that the Macroeconomic Accounts Branch ensure the timely update of all applicable metadata information.
Management Response and Action Plan
Recommendation 1
Performance Measurement
Focus : IASD and MEAB
It is recommended that the MEAP proceed with the implementation of a performance measurement system to ensure the availability of timely performance information to support on-going program management and decision making, demonstrate the achievement of program expected outcomes and support future evaluations (including up-to-date business process models for outputs, list of outputs produced and released and a register of data users).
Statement of Agreement / Disagreement
Management agrees with the proposed recommendation.
Management Response
Management has already started implementing a system to produce regular and timely performance measures to address all outcomes identified in the MEAB logic model. In addition, the corporation has begun to develop standardized performance measures as well. Once these indicators have been developed the MEAB will update its performance management strategy and ensure that the PMS accurately describes the performance measures. In addition to the performance measures IASD and the MEAB will create a catalog of its various outputs and maintain an up to date list of data users which have contacted the agency for MEAB information.
| Timeline | Deliverable(s) | Responsible Party |
|---|---|---|
| April 2015 | Development of the MEAB specific Performance Measures. | Director General – MEA / Manager National Accounts Integration Group |
| April 2015 | Incorporation of corporate performance measures into the MEAB Performance Measurement Strategy. | Corporate Task force on performance measurement. |
| April 2015 | Register of Data users based on individuals who have contacted the branch for MEA related information. | Director General – MEA / Manager National Accounts Integration Group |
Recommendation 2
Relevance, Efficiency and Economy
Focus : IASD and MEAB
It is recommended that the IASD, establish/implement the appropriate strategy to enhance IASD capacity and explore avenues to resume the provision of cost-recovery services to deal with the current relevancy risk, the increased potential for errors and slippages in the schedule and to address some of the unmet needs of users.
Statement of Agreement / Disagreement
Management agrees with the proposed recommendation.
Management Response
Work is already underway to enhance the capacity of the International Account and Statistics Division. Recently the division merged with the International Trade Division. This merger has brought in a great deal of expertise into the division as well as an opportunity to streamline operations and create new data products which will lead to increased capacity to not only deliver the current set of programs but also take on cost recovery work. In addition, the International Trade Division, has a long history of undertaking extensive cost recovery work – this expertise can be shared with other programs in the division. The branch is seeking and has secured some new funding that will ensure the division has increased capacity to deliver its programs and expand its programs where required. Some of the funding has been provided from the Economic Statistics Field while other funding opportunities are being discussed with other departments. Given the cross cutting nature of this work (measuring global production, merchanting, international financial flows) and their impact in other program areas the agency as a whole will need to find ways to increase its capacity to deal with these emerging issues
The division has been engaged in a number of staffing processes over the last fiscal year and will be hiring a number of economists. Finally, the division has been actively participating in the development of the Macroeconomic Accounts training program. This training program will be used to develop the human capital within the division to allow them to more effectively perform their duties.
While there are some unmet needs of users it will be difficult for IASD to move forward on a number of data products until there is international agreement around concepts, methods and the implementation timeframe for these data products. Statistics Canada cannot unilaterally implement changes to its international accounts program until the changes are implemented by our major trading partners (e.g. the US). If these changes are implemented unilaterally international account asymmetries will arise making the data confusion for our users and less relevant. The international accounts program will remain active on the international front to ensure these changes are coordinated among countries.
| Timeline | Deliverable(s) | Responsible Party |
|---|---|---|
| April 2014 | Merger of the International Trade Division and International Accounts and Statistics Division | Assistant Chief Statistician – Economic Statistics / Director General – MEA / Director General – Economy Wide Statistics / Director – International Accounts and Statistics / Director – International Trade Division |
| April 2015 | Secure additional funding for programs such as the Exporter / Importer Register, Foreign Affiliate Statistics and Securities programs to meet emerging user needs. | Director General – MEA / Director – International Accounts and Statistics Division |
| April 2015 | Secure additional funding for programs to address the needs of the international community for data related to financial stability and international linkages (G20 data gaps initiative and SDDS+). | Assistant Chief Statistician – Economic Statistics / Director General – MEA / Director – International Accounts and Statistics / Director – International Trade Division / Director – Financial Planning Division |
| April 2015 | Completion of the Macroeconomic Accounts Training curriculum. | Director General – MEA / Director – International Accounts and Statistics Division |
| April 2015 | Hiring or promoting a number of economists within the division. | Director – International Accounts and Statistics Division |
Recommendation 3
Performance
Focus : MEAB
To ensure that data users have a proper understanding of its macroeconomic products, it is recommended that the Macroeconomic Accounts Branch ensure the timely update of all applicable metadata information.
Statement of Agreement / Disagreement
Management agrees with the proposed recommendation.
Management Response
The Macroeconomic Accounts Branch is developing a meta data model and associated meta database to be used to store SNA meta data, display those meta data to internal and external users and facilitate the loading, editing and auditing of those meta data. The model will be embodied in a relational meta database and is presently a work in progress. Once the model has been completed programs within the MEA will be targeted to populate the database with current meta data information. Once populated the information will be loaded into Statistics Canada Integrated Meta Database (IMDB) and made public via the Statistics Canada website where it can be accessed directly from CANSIM (in association with the actual data). The International Accounts and Statistics programs will be targeted as one of the first programs to supply the updated meta data information.
| Timeline | Deliverable(s) | Responsible Party |
|---|---|---|
| December 2015 | Release of: User Guide to the Macroeconomic Accounts | Director General - MEA |
| April 2015 | Release of updated and detailed BOP metadata in the IMDB. | Director General – MEA / Director – International Accounts and Statistics Division. |
| April 2016 | Release of updated and detailed IIP and other IATD metadata in the IMDB | Director General – MEA / Director – International Accounts and Statistics Division. |
The full report is available upon request. Please contact:
AEB-Professional-Practices@statcan.gc.ca