Evaluation of the Census of Agriculture Program
(2009/2010 to 2013/2014)

Executive summary and Management response and action plan

April 2016

• Executive summary
• Management response and action plan

Acknowledgements

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Departmental Evaluation Committee as well as all interviewees who provided insights and comments crucial to this evaluation.

This report was approved by the Chief Statistician.

In accordance with the accountability requirements in the Treasury Board Policy on Evaluation (2009) and Directive on the Evaluation Function, this report is available to the public and posted on the departmental website in both official languages.

Statistics Canada has also shared this report with its program delivery partners and key stakeholders, including the National Statistics Council.

Prepared by the Evaluation and Performance Measurement Division, Audit and Evaluation Branch. Khaddouj Souaid led the Evaluation Project Team, which included Sonia Ben Amor, Justin Wabgou, Mihaela Tapuc, Sonia Demers and Tony Haddad.

List of acronyms and abbreviations

AAFC

Agriculture and Agri-Food Canada

ASP

Agriculture Statistics Program

BR

Business Register 

CEAG

Census of Agriculture

CePOP

Census of Population

CFIA

Canadian Food Inspection Agency

CRMS

Client Relationship Management System 

FR

Farm Register

EPMD

Evaluation and Performance Measurement Division

NAICS

North American Industry Classification System

NHS

National Household Survey

PMRA

Pest Management Regulatory Agency

RBAEP

Risk-Based Audit and Evaluation Plan

QAF

Quality Assurance Framework

RDAS

Remote Data Access System

SSC

Shared Services Canada

TB

Treasury Board

TBS

Treasury Board Secretariat

FAO

Food and Agriculture Organization of the United Nations

Executive Summary

The Evaluation and Performance Measurement Division (EPMD) of the Audit and Evaluation Branch of Statistics Canada conducted an independent assessment of the Census of Agriculture (CEAG) Program. The results provided objective information and recommendations to support program improvement and inform future decisions by Statistics Canada and Parliament with respect to the government's management of resources and programs.

The evaluation was designed and conducted in accordance with the Treasury Board (TB) Policy on Evaluation (2009) and associated directives and standards, and fulfills an accountability commitment set out in Statistics Canada's Risk-Based Audit and Evaluation Plan (2012/2013 to 2014/2015). The evaluation's objective is to provide credible and neutral information on the ongoing relevance and performance (effectiveness, efficiency and economy) of the program.

This report presents the results of the evaluation of the CEAG. It was presented to the Departmental Evaluation Committee and approved by the Chief Statistician.

The Census of Agriculture Program

The Census of Agriculture is conducted every five years. It collects information on the state of all agricultural operations in Canada to provide Canadians with a comprehensive profile of the agriculture industry and its people. The objectives of the CEAG Program are: (1) to maintain an accurate and complete list of farms and types of farms; (2) to provide comprehensive agriculture data for detailed geographic areas; (3) to provide measurement of rare/emerging commodities; and (4) to provide critical input for the purpose of managing federal and provincial governments' expenditures in support of the agriculture sector. The budget for the 2011 cycle of the CEAG Program was $46.6 million over six years (2008/2009 to 2013/2014).

Evaluation scope and methodology

The evaluation's five-year reference period was from 2009/2010 to 2013/2014; this included the conduct of the 2011 CEAG and the planning phase of the 2016 cycle. The evaluation also examined the performance of cost-recovered customized products and services related to the CEAG. Data collection for this evaluation was completed between February and October 2015. Six lines of evidence, both quantitative and qualitative methods, were used, including document and literature reviews, an administrative data review, key informant interviews^{Footnote 1} with 60 internal and external stakeholders and data users, efficiency and economy analysis, and bibliometric and webometric studies. Data from these sources were analyzed and triangulated to develop the findings and generate recommendations based on the conclusions made.

Findings

Relevance of the program

The evaluation demonstrates a continued need for the CEAG to fulfil and support legislative and regulatory requirements under federal/provincial/municipal acts and regulations. An examination of the program relevance conducted in 2012^{Footnote 2} found that a quinquennial CEAG was necessary in order to meet the wide range of information needs of its various stakeholders and data users. The program is a unique, nationally uniform data source with complete enumeration of all Canadian farms, and meets the policy, program and research needs of various public and private data users. In some instances, the data for small geographic areas are the only source of critical information used for environmental reporting, crisis management, land planning and regulation, and food safety.

The program is aligned with the priorities of the federal government in support of its investments in the agriculture industry and its partnerships with the provinces and territories. Conducting a census of agriculture is consistent with the federal government's roles and responsibilities, as stipulated in the Statistics Act. The program supports Statistics Canada's mandate "to ensure that Canadians have access to a trusted source of statistics on Canada to meet their highest priority needs"^{Footnote 3} and contributes to its strategic objectives.

The evaluation found no evidence of viable alternatives for the collection of data of the same scope, level of detail, and quality to replace the CEAG data.

Performance of the program—achievement of expected outcomes

The 2011 CEAG program was successful in achieving its intended outcomes by providing accessible high-quality statistics and related custom-data products. It met the information needs of its various stakeholders, and its statistical information was widely used, for multiple purposes, such as academic and policy analyses and program design and evaluation. In 2014, the budget allocation for the CEAG was reduced by 9.2% and its scope limited to focus on core activities. The federal government's decision consequently reduced the CEAG outputs. It is anticipated that the discontinuation of analytical products and CEAG–CePOP-linked data for the 2016 cycle will reduce accessibility and use of CEAG data in the future. In the context of continued demand for these products, many stakeholders expressed concerns that the information gaps could lead to the use of less reliable data resources.

The evaluation examined the quality of the CEAG statistics by assessing their timeliness, accuracy, interpretability, coherence and accessibility. Evidence indicates that the 2011 CEAG core products were released according to pre-announced schedules. However, there were delays, ranging from 7 to 17 months, in the release of the analytical publications. These were due mainly to resource constraints. The 2011 CEAG had a 95.9% response rate and demonstrated improvements to Internet uptake (which increased from 5.0% in 2006 to 10.9% in 2011), and decreases in estimated net undercoverage (from 3.4% in 2006 to 1.8% in 2011). The CEAG statistics are also generally coherent with other datasets and validated by other sources (e.g., commodity and farm income and prices programs).

The CEAG continued to respond to existing and emerging data needs while maintaining content stability for data consistency and comparability over time. CEAG data were provided by theme and properly coded; up-to-date definitions and supplementary information were available for appropriate data interpretations. The 2011 CEAG provided public access to five types of statistical products free of charge, in addition to delivering over 500 high-quality cost-recovered custom-data requests, such as customized tabulations for small-area data. Accessibility appears to have been limited by significant delays in the delivery of these requests, as was reported by data users. It is notable that, in 2011/2012, the process of developing the Corporate Open Licence Agreement was taking place in the context of unrestricted access to, and free dissemination of, government data, which was introduced government-wide by the TBS. This, along with resource constraints, resulted in an overall backlog of custom-data requests.

The introduction of the Canadian Socioeconomic Information Management database (CANSIM) in 2012 was a significant improvement with respect to accessibility, as consistently reported by stakeholders. Analysis of data monitoring of online access^{Footnote 4} to CEAG's products shows high levels of activity (4.5 million downloads and 3.5 million page views of products over three years from 2011/2012 to 2014/2015), particularly in the release years. The program has effective governance structures, longstanding partnerships with internal and external stakeholders, and comprehensive consultation processes to solicit and attain feedback from a cross-section of experts and data users (federal–provincial governments, agriculture industry, academia, and the general public). Some unmet or insufficiently met information needs were identified in some areas, such as environmental practices, organic production, and technology use. While these needs are clearly important for the program to adapt to a rapidly evolving sector, it is noted that the specific and detailed information relating to these issues may likely be within the realm of more targeted surveys (e.g., agriculture or environment) and not within the scope of the Census.

The use of the CEAG statistical information to inform public debate, research, and decision making is evidenced by the extent to which it is referenced in policy, research, and program development and evaluation documents reviewed. Evidence indicates that the CEAG products were widely cited in peer-reviewed scientific publications, on the World Wide Web, and in newspaper articles.^{Footnote 5}

Performance of the program—efficiency and economy

Overall, the CEAG program has operated efficiently and economically by achieving incremental cost savings over successive cycles. Efforts at optimization of resource utilization, including reviews and investigation studies, were made between 2011/2012 and 2013/2014 to identify and implement ways of delivering the program more cost-effectively. The CEAG is conducted concurrently with the Census of Population. The two programs share key processes and infrastructure (to ensure accurate coverage and reduce duplication); this generates significant cost savings. Financial records indicate that the CEAG's approved funds were consistently lower than the projected cost estimates in three consecutive CEAG cycles (2006, 2011 and 2016). This resulted in the application of further cost-savings to deliver the program within allocated budgets. While the budget in constant dollars^{Footnote 6} for the 2011 CEAG ($47 million) was similar to that for the 2006 cycle ($46.8 million), the program produced the same number of outputs, as well as an additional analytical publication, while yielding an overall improvement in data quality.

Building on major changes in methodology, processing, and use of technology in the 2006 cycle, the 2011 CEAG included strategic investments of $2.7 million that aimed to achieve further efficiency. They consisted of the CEAG Farm Register and the Census of Population Address Compatibility initiative (to decrease mail-out rates and reduce reliance on human resources) as well as the Multi-Model Response Development Initiative (providing respondents with an option to complete the questionnaire by telephone). These improvements resulted in a higher Internet response rate in 2011. Statistics Canada has set a target Internet response rate of 30% for the 2016 cycle, especially given the reduction in the questionnaire size.^{Footnote 7} The CEAG Program is aligned with transformations in the context of Statistics Canada's Corporate Business Architecture to ensure efficient ways of conducting statistical programs.

It is noted that the program is currently implementing the 2016 CEAG cycle as per the resources level and scope approved by the government in 2014, with a budget of $42.6 million, a 9.2% decrease from the 2011 cycle. Qualitative and quantitative evaluation evidence suggests that further resource decreases would limit the CEAG Program's capacity to produce the same level and quality of statistical information and maintain its overall quality.

Conclusions and recommendations

The evaluation reveals a demonstrable need for the CEAG Program to meet the existing information needs and adapt to emerging data requirements of a rapidly changing industry. The 2011 cycle achieved its intended outcomes efficiently and economically, by providing access to high-quality data and responding to most information needs of data users.

Some unmet or insufficiently met information needs were identified, particularly relating to environmental practices, organic production, technology use and direct marketing. These needs are known to Statistics Canada; the feasibility of meeting some of them was examined and, where appropriate, action was taken. The evaluation concludes that the use of other surveys to bridge the gap is a more appropriate and efficient approach. Leveraging resources to jointly conduct consultations for the CEAG and other agriculture surveys may lead to a more efficient use of resources. Some areas for improvement could enhance access to, and use of, the CEAG data, including the production and dissemination of analytical products for the 2016 CEAG and timely delivery of custom-data products and services.

While the evaluation found evidence of emerging information needs and trends, it also found that the CEAG may not be the most appropriate and efficient tool to address these needs. The evaluation noted as well that the CEAGs stakeholders and data users are also the primary users of the surveys of the Agriculture Statistics Program (ASP), which collect more detailed information on specific agriculture topics. It is important that a close coordination of activities between the two programs be in place, particularly planning and stakeholder consultation, to identify the most appropriate and efficient use of resources for each program.

Recommendation #1 – Performance (efficiency)

It is recommended that the Assistant Chief Statistician of Economic Statistics consider the feasibility of coordinating stakeholder consultation activities of the Census of Agriculture Program with those of the Agriculture Statistics Program. Efficiencies may be found in reducing the duplication of efforts and identifying the most appropriate and effective use of the resources of each program to best meet the information needs of stakeholders and data users, given that the latter are almost identical for the two programs.

The CEAG's analytical products are highly accessed and widely used by a variety of data users. In 2014, the government decided to focus the CEAG on core activities, consequently reducing its budget and scope. The program is currently implementing the 2016 CEAG cycle as approved. It is anticipated that the discontinuation of analytical products and the CePOP-CEAG linked data may cause significant limitations to future accessibility and use of the CEAG data as measured against previous cycles.

Recommendation #2 – Performance (accessibility and use)

Given the important value consulted users attached to analytical products, it is recommended that the Assistant Chief Statistician of Economic Statistics explore options for external sources of funding to support continued production of analytical products for the CEAG in order to maintain accessibility and use of the CEAG data.

While the evaluation evidence indicates that the program delivers high-quality custom-data products and effective client-centered services, it also identified the need for improvements in timeliness in the delivery of these products and services (as reported by data users) and in tracking and documenting performance related to these products and services.

Recommendation #3 – Performance (timeliness)

1. It is recommended that the Assistant Chief Statistician of Economic Statistics ensure that a review of lessons learned internally and externally be carried out to develop strategies/options for consideration in relation to the provision of timely cost-recovery custom-data products and services, and that improvements be made to the tracking and documenting of performance with respect to these services.
2. It is recommended that the Assistant Chief Statistician of Census, Operations and Communications ensure that improvements be made to Statistics Canada's Client Relationship Management System (CRMS) to enable documentation of performance related to timeliness in the delivery of custom-data products and services with respect to all statistical programs.

Management response and action plan