Age-standardized Rates

Rates

Rates are a useful tool for comparing characteristics across different populations, different segments of a population, or the same population over time. One type of rate is a percentage^{Footnote 1}. The number of Canadians who, for example, smoke or are obese is often expressed as a percentage of the population to facilitate comparison across provinces, sexes and age groups.

Crude Rates

When rates are used to examine unusual events, such as certain crimes or the incidence of rare diseases, they are often expressed as the number of people or occurrences per 1,000 or 100,000 individuals in the population. These rates are often referred to as crude rates^{Footnote 2}. As with percentages, these rates take into account the underlying population size.

For example, the data in Table 1, from 2000, show that 62,672 Canadians died of cancer, while, in 2011, 72,476 died. Over this same time period, the Canadian population grew from 30,685,730 in 2000, to 34,342,780 in 2011. When we express this information as a crude rate, we see that the cancer mortality rate was 204.2 deaths per 100,000 persons in 2000, and 211.0 deaths per 100,000 persons in 2011. By using a rate, we are able to quickly and clearly see that over the 11-year period, the rate of death due to cancer has increased.

Table 1: Cancer deaths and population estimates, Canada, 2000 and 2011
Age group	Characteristic	2000	2011
0 to 39 years	Estimate of population	17,068,876	17,191,850
	Number of deaths	1,345	1,004
	Crude rate	7.9	5.8
40 years and over	Estimate of population	13,616,854	17,150,930
	Number of deaths	61,325	71,472
	Crude rate	450.4	416.7
Total all ages	Estimate of population	30,685,730	34,342,780
	Number of deaths	62,672	72,476
	Crude rate	204.2	211.0

However, during the same period, the crude rate for each age group decreased. Is something wrong? The short answer is that the crude rate for the total population, while it accurately represents the incidence of death due to cancer each year, it is not the correct indicator to use to compare the incidence between years.

Age-standardized Rates

Comparing rates between two time periods or two different geographical areas is usually more representative when taking into account differences in the age structure of the two populations. This is particularly true if the characteristic being observed varies by age. This is the case in our example of mortality rates since cancer affects considerably more Canadians in their later years of life than those in their younger years.

Age-standardized rates are often used to make such comparisons, as they account for the differences in the age structure of the populations being compared. In the calculation of the age-standardized rate, either one population is mathematically adjusted to have the same age structure as the other; or both populations are mathematically adjusted to have the same age structure as a third population, called the standard population^{Footnote 3}. In this way, the two groups are given the same age distribution structure so that a more representative picture of the characteristic in question is provided.

In the cancer mortality example, the 2011 Canadian population has a higher proportion of those 40+ than the 2000 population does: almost half (49.9%) of the 2011 population was 40 years of age or older, compared to 44.4% in 2000. Due to the high mortality rate in the 40+ age group, considerably more cancer deaths are observed in 2011. But, it is only by removing the effect of the differing age distributions that we can make conclusions about the relative decreases or increases in mortality over time. The exact calculation of the age-standardized rate for this example is given at the end of the Fact Sheet.

In the interest of simplicity, the example here used only two age groups: often, the characteristic being studied varies considerably across ages and therefore, narrower age categories are required. The age-standardized rates appearing in many CANSIM tables use 20 different age groups, making age-specific comparisons more intricate particularly when many years or provinces are involved.

Calculation of the Age-standardized Rate

The detailed calculation of the age-standardized mortality rate is presented here using the example of deaths due to cancer, and the year 2000 data from Table 1. The rates are standardized to the 1991 population.

To calculate the age-standardized mortality rate (ASMR), we must first calculate the age-specific (mortality) rates for each age group by dividing the number of deaths by the respective population, and then multiplying the resulting number by 100,000:

Age-specific rate, 0 to 39 years: = 1,345 (number of deaths) ÷ 17,068,876 (total population) × 100,000; = 7.9 cancer deaths per 100,000 population

Age-specific rate, 40+ years: = 61,325 (number of deaths) ÷ 13,616,854 (total population) × 100,000; = 450.4 cancer deaths per 100,000 population

We then multiply each of the age-specific rates by the proportion of the 1991 population belonging to the particular age group (called the standard population weight). In 1991, 61.6% of Canadians were under 40 years of age and 38.4% were age 40 or older. The age-standardized rate is obtained by adding the resulting numbers:

ASMR: = (7.9 × 61.6%) + (450.4 × 38.4%); = 4.9 + 173.0; = 177.9 cancer deaths per 100,000 standard population.

In a similar fashion, the 2011 age-specific rates and age-standardized rate are obtained, respectively, as 5.8 (0 to 39 years), 416.7 (40+ years) and 163.6 cancer deaths per 100,000 (standard) population.

Using the data from Table 1, we obtained the:

2000 age-standardized mortality rate of 177.9 deaths per 100,000 standard population, and
2011 age-standardized mortality rate of 163.6 deaths per 100,000 standard population.

Note that both age-specific rates are lower in 2011 than in 2000 and yet the crude mortality rate for 2011 is higher. This is because the 2011 population is older than the 2000 population: almost half (49.9%) of the 2011 population was 40 years of age or older, compared to 44.4% in 2000. Because of the much higher mortality rate in this age group, considerably more cancer deaths are observed in 2011 than in 2000, contributing to a higher crude mortality rate although both age-specific crude rates are lower. It is only by adjusting the two populations to have the same age distribution—in this case, that of the 1991 population—that we can make general and more accurate conclusions about relative decreases or increases in mortality.

Footnotes:

Footnote 1

the number of individuals exhibiting a characteristic or particular behaviour per 100 people

Return to footnote 1 referrer

Footnote 2

by dividing both numbers by their respective population size measures, and then multiplying them by 100,000, to express them as a rate

Return to footnote 2 referrer

Footnote 3

For many of the age-standardized rates appearing in CANSIM, the 1991 Canadian Census of Population is used as the standard population, although a transition to a more recent age structure is being considered.

Return to footnote 3 referrer

Financial Information of Community Colleges and Vocational Schools

For the fiscal year ending in 2014

I. Introduction

The main objective of this survey is to obtain detailed revenue and expenditure data on each college and vocational school in Canada. Coupled with what is already available for the university sector, this gathering of data will provide a complete picture of the financial statistics of postsecondary education as well as vocational training in Canada.

The following notes provide the principles, definitions and guidelines necessary for the completion of the data form. Since it is desirable to obtain figures as comparable as possible from one institution to another, each respondent is requested to:

provide accompanying notes of explanation in the observations and comments section of the submission for figures that do not follow the guidelines;
provide comments on items which are excluded from the data, such as cases where provinces are making contributions to repay debt on behalf of an institution or material gifts received as donated service along with their estimated market value;
estimates should be made whenever possible if income and expenditure figures are not readily available in the required format from the financial records of the institution. When estimates are made they should be indicated with an asterisk (*).

II. Submission

The final deadline for the submission is indicated in the covering letter. The completed questionnaire(s) should be returned in the self-addressed envelope provided.

A copy of the institution's Audited Financial Statements is also requested with your submission. If a copy is not available, please advise Statistics Canada as to the date on which they will be forwarded.

III. Coverage

With the exception of private institutions that only offer courses at the trade and vocational level, the survey covers all private and public non-degree granting institutions that offer educational programs at the postsecondary level and/or at the trade and vocational level. For statistical purposes, institutions are classified as follows:

Colleges/Institutes/Polytechnics

Included in this classification are the colleges of applied arts and technology (CAAT's) in Ontario, general and vocational colleges (CEGEP's) in Quebec, institutes of technology and any other institutions providing education in fields such as paramedical technologies, nursing, agriculture, forestry, nautical sciences, etc.. These institutions offer programs at the postsecondary level, and may offer trade-vocational level programs.
Vocational Schools

This classification includes Community Colleges in Saskatchewan and Vocational Centres in Alberta, government training schools, vocational training centres and any other institution offering programs at the trade-vocational level only.
Training in hospitals

Included in this classification are educational centres located in hospitals, which offer educational or training programs, independently of the community college system, in nursing, radiotherapy, radiography, medical technology, etc..

To ensure full coverage, it is important that each reporting officer indicates on section 2 of the questionnaire the affiliated campuses included in and/or excluded from the submission.

IV. Confidentiality

Statistics Canada is prohibited by law from releasing any information it collects which could identify any person, business or organization, unless consent has been given by the respondent or as permitted by the Statistics Act.

V. Authorization to Release

In order for Statistics Canada to release the information provided an 'Authorization to release' form must be signed. The form provided includes two options for release:

Option A: Authorizes Statistics Canada to release the information at the institution level,

Option B: Authorizes Statistics Canada to release the information in aggregation to the provincial/territorial level.

VI. Principles of Reporting

1. Accrual Concept

For the purpose of this survey, the revenue and expenditure data should be reported on an accrual basis. That is, all revenues and expenditures should be reflected in the period in which they are considered to have been earned and incurred respectively. For example, major adjustments, such as retroactive salary and their related benefit costs, should be reported on that basis.

2. Total Income and Expenditures

All income and expenditures of the institution are to be reported. In this regard particular attention should be paid to the following:

when an institution is provincially governed or consists of a branch of a department, all costs related to the operation, maintenance and administration of the institution are to be reported; the actual funds used to finance those expenditures should be shown as a provincial source of funds;
consultations may be required with the institution's research department to obtain detailed breakdowns of income sources and expenses related to sponsored research;
capital expenditures, as well as related revenues, that are financed by a government Department or Ministry other than the one responsible for the institution must be included in this report; the reporting officer is responsible for obtaining and providing this information;
the figures reported should not include income or expenditures for the purpose of creating or eliminating an appropriation; however, any actual income or expenditure transaction recorded directly in reserve accounts should be included in the figures reported; this also applies to other assets and liability accounts; provisions for replacement of assets are considered to be transfers to reserve or appropriation accounts and should not be reported as expenses;
receipts and expenses relating to special purpose, trust and other funds of the institution should, as well, be included in the report.

3. Ancillary Enterprises

An ancillary enterprise is an entity that exists to furnish goods and services to students, staff or others, and that charges a fee directly related to, although not necessarily equal to, the cost of the goods or services. To reflect properly the full cost of these enterprises, you should report their total gross revenues and total gross expenditures in the appropriate cells in the Schedule 1 and Schedule 2A. In addition, a breakdown by type of ancillary enterprises (bookstores, food services, residences, parking) must be completed on the Supporting Schedule A.

4. Reporting of Income

When reporting the sources of funds in the operating, sponsored research and capital income in Schedule 1, it is important to show the revenues under the headings that correspond to the immediate source of funds for the institution. For example, if an institution offers training courses for which Employment and Social Development Canada (ESDC) purchases seats, then the amount of money paid by ESDC should be shown under "Federal" only if the money is received directly by the institution. If the money is received by a third party (provincial government) and then transferred to the institution, then the direct source of funds is the "Provincial Government".

VII. Definitions

1. Program Cost Groups

This section defines the program cost groups to be used in the reporting of direct instruction expenditures on Schedule 2B of the questionnaire.

The criteria used to define the various program cost groups originates from those used in other surveys conducted by Statistics Canada and also from analysis of different educational systems across Canada. Note that these statistical definitions may not correspond identically to other existing definitions used by other organizations or governments.

a) Postsecondary Programs

This program cost group includes all direct expenditures incurred in providing instruction to students enrolled FULL-TIME or PART-TIME in postsecondary programs offered by Colleges/Institutes (see Section III). These programs are of two kinds: university transfer programs and semi-professional career programs.

i) University transfer programs: University transfer programs require secondary school completion to enter and provide a student with standing equivalent to the first or second year of a university degree program with which one can apply for admission to subsequent senior years at a degree granting institution.
ii) Career programs: These programs usually require high school graduation for admission and have a duration of at least one year. More commonly these programs last two, three or four years. Career programs lead to a certificate or a diploma in technology, business, applied arts, nursing, agriculture, etc., and they prepare a student to enter a career directly upon completion of the program, at a level between that of the university trained professional and the skilled tradesperson.

b) Trade and Vocational Programs

This program cost group includes all direct expenditures incurred in providing instruction (or training) to students (or trainees) enrolled FULL-TIME in vocational programs at the trade level for credit towards a recognized standing of proficiency or certification. Also included are direct expenditures related to students enrolled in academic upgrading programs for entry into a vocational program. Such students normally attend regular day classes in provincial trade schools, trade or industrial divisions of community colleges, adult vocational centres and other similar schools. These programs or courses prepare the student (trainee) for an occupational role below the professional or semi-professional level. A period of less than one year is normally sufficient to complete courses at this level. For less complex occupations, a program may last only a matter of weeks. Completion of grade 9 or 10 is usually required for entrance to these courses.

Included are, for example, pre-employment programs, language, skill or academic upgrading programs, refresher courses, apprenticeship programs, training on the job or training in-industry programs associated with educational institution, nursing assistant, etc..

c) Continuing Education Programs

This program cost group includes all direct expenditures incurred in providing instruction to students enrolled PART-TIME in courses, mostly in the evening, offered under the auspices of subsidiary divisions of schools designated by various names such as Division of Continuing Education, Adult Education Division and so on. Excluded are activities which have no sustained instruction or educational purpose such as recreational activities, presentations in the performing arts, art exhibitions and displays, debates fairs, conferences or conventions of clubs or associations.

Included are, for example, courses such as pre-employment programs, language, skill or academic upgrading programs, refresher, professional development, general interest, etc., which are offered on a PART-TIME basis.

2. Funds

a) Operating

This fund accounts for the cost of credit and non-credit instruction, non-sponsored research, academic support services, administration, plant maintenance and other operating expenses of the institution financed by fees, grants and other operating income. This fund will normally include all revenues and expenses regarding materials, supplies or services that are consumed within the year and which the institution considers to be operating, within the functional operating areas referred to in Section 3 below.

b) Sponsored Research

Sponsored Research is a restricted fund that accounts for income and expenditures for all sponsored research as well as Research and Development (R&D). For an activity to qualify as R&D, there must be an appreciable element of novelty. Income is to be reported following the funds flow approach.

Sponsored Research covers the following activities:

Basic Research is any experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observed facts, without any particular application or use in view;

Applied Research is the original investigation undertaken to acquire new knowledge, and directed primarily towards a specific practical objective;

Experimental Development is systematic work drawing on existing knowledge gained from research and/or practical experience that is directed to producing new materials, products or devices, installing new processes, systems and services, or improving those already installed.

The following activities should not be counted as R&D:

all education and training of personnel; however, research by graduates and postgraduate students should be counted;
scientific and technical information services such as collecting, coding, recording, classifying, analyzing, disseminating, translating, and evaluating, except where conducted solely or primarily for R&D support;
routine testing of materials, components, products, processes, soils, etc.;
maintenance of national standards;
administrative and legal work connected with patents and licenses;
investigations of proposed engineering projects using existing techniques; however feasibility studies on research projects are part of R&D;
policy-related studies at the national, regional and local levels, as well as those of business enterprises in pursuit of economic activity;
routine software development, computer maintenance, quality assurance, routine data collection, and market research;
the many steps other than R&D necessary for the development and marketing of a manufactured product;
the raising, management, and distribution of R&D funds; and
routine investigation and normal application of specialized medical knowledge.

Sponsored Research accounts for the institution's income paid in the form of a contract (legally enforceable arrangements under which the institution, or an individual within the institution, agrees to undertake a research project, using the institution's facilities and/or personnel, for a sponsor that provide funds to meet all or part of the costs of the project) or a grant (unconditional payment for which service is not necessarily expected) from a source external to the institution.

Income sources include government, private industry and donors. Income may also include investment income, if the corresponding expenditures are reported in Sponsored Research.

Expenditures include activity funded from Sponsored Research income and exclude activity funded from the General Operating fund. It also includes the purchase of capital assets, if the corresponding income is reported as Sponsored Research.

c) Capital

The uniform reporting practice in the annual return for capital expenditures is to follow the funds flow approach, rather than to capitalize and amortize. Funds received to acquire capital assets are reported as income in the period in which the funds are received or receivable. Funds used to acquire capital assets are reported as expenditures in the year they take place.

For reporting purposes, capital expenditures are to be reported in the same fund as the corresponding income. Specifically, capital expenditures are only reported in the Capital fund when the corresponding income is reported in the Capital fund.

It is a restricted fund that accounts for resources provided to the institution for capital purposes and not reported in any other fund. Fund income includes grants and related investment income, donations and other resources made available to the institution by external funding sources, such as government and donors, specifically for capital purposes. Fund expenditures include building programs, acquisitions of major equipment and furniture, major renovations and alterations, space rental and buildings, land and land improvements.

Capital expenditures, as well as related revenues, being financed by a Government Department or Ministry other than the one responsible for the institution must be included in this report. The reporting officer should be responsible for obtaining and providing this information.

3. Functions (Schedule 2A)

a) Instruction and non-sponsored research

This includes all direct costs related to credit and non-credit courses, summer courses, extension programs and all other academic functions related to instruction and non-sponsored research such as offices of academic department heads, audio-visual services, laboratories, etc..

b) Library

This includes all the operating costs of the main library as well as the campus libraries, if there are any. All costs of library acquisitions from the Operating fund should be shown under this function.

c) General Administration

This includes costs for activities whose primary function is to provide administrative support for the operation of the institution. It includes the activities of the president's office, vice president, registrar, finance, personnel, public relations, secretariats, etc.. It also includes expenditures on convocations, ceremonies, legal and audit fees, long distance phone calls, the internal portion of debt repayments and costs for computing facilities.

d) Physical Plant

This includes the costs related to physical facilities, such as physical plant offices, maintenance of buildings and grounds, fire insurance, telephone service, security, repairs and furnishing, renovations and alterations, mail delivery service.

e) Student Services

This includes costs for activities whose primary purpose is to assist students in their educational or employment pursuits and which are outside of, but supplemental to, the instruction of academic programs. It includes the costs of: counselling, placement, health services, athletics (not physical education), student accommodation services (not residences), student transportation services, bursaries, scholarships and prizes, student financial aid office, cultural activities, etc..

4. Types of Income

a) Government Grants and Contracts

Lines 1 to 10 include grants from, and contracts with, federal government departments and agencies, provincial/territorial government departments and agencies, and municipal governments.

Government grants provide financial support to institutions and the grants may or may not be restricted.

Government contracts provide financial support to institutions under certain stipulations and conditions, including the provision of a deliverable product, such as a piece of equipment, a service or a report. A contract normally includes provisions for institutions to recover certain indirect or overhead costs, with the contract specifying or documenting the basis for the calculation of the recoverable costs.

Federal

Lines 1 to 6 include all research grants, research contracts, grants and contributions from the Government of Canada and its departments and agencies. Income received from the five major federal government agencies is reported on lines 1 to 5 as applicable.

The line items under "Federal" are as follows:

Line 1: Employment and Social Development Canada (ESDC)
Line 2: Canada Foundation for Innovation (CFI)
CFI income is reported under the Sponsored Research fund.
Line 3: Canadian Institutes of Health Research (CIHR)
Line 4: Natural Sciences and Engineering Research Council of Canada (NSERC)
Line 5: Social Sciences and Humanities Research Council
Line 6: Other federal
Income from all other federal government departments and agencies is reported on this line.

Provincial/Territorial

Lines 7 to 9 include income from provincial government departments and agencies. For example, Provincial/Territorial CFI matching grants, Provincial/Territorial CFI matching income (line 8) from the Ministry responsible for the institution is reported under the Sponsored Research fund.

In the case of a provincially/territorially administered institution, direct provincial funding is to be included here.

Municipal

Examples of income to be reported on this line include grants from urban transit, communication and parking authorities.

b) Fees

This includes all mandatory student fees for credit and non-credit courses (with the exception of residence fees, parking fees and other similar fees which should be reported under 'ancillary enterprises - gross') paid by, or on behalf of all FULL-TIME and PART-TIME students.

All other fees charged to students such as laboratory fees, transcript, late registration, application, athletic fees, etc., are to be reported under the heading 'other'.

Normally, whenever revenues from fees are reported in Schedule 1 under specific program(s), related expenditures should be reported for the corresponding program(s) in Schedule 2B.

Note: Fees that are "flow through", such as student activity fees collected for the students' council, etc., are not to be reported as college revenue.

c) Bequests, Donations, Non-Government Grants

This includes receipts from business, industry, foundations, individuals and religious organizations, as well as the value of services donated by various organizations.

d) Investment Income

This includes income from all investments such as dividends, bonds, mortgages, short-term notes and bank interest. Realized gains (or losses) should also be included if they are treated as income in the operating and/or capital funds.

e) Ancillary Enterprises (gross)

This includes total revenues from all ancillary enterprises such as residence or parking fees, and sales of services and products from bookstores, food services (dining hall, cafeterias and vending machines), publishing, laundry services, etc..

It should also be noted that the reporting officer is asked to report, on Supporting Schedule A, a breakdown of total income for the institution's ancillary enterprises.

f) Borrowings

This includes only those borrowings which are used to finance expenditures when repayment is to be made by the institution. Note that borrowings should be reported on an accrual basis.

g) Miscellaneous

This includes net income from rentals (other than ancillary enterprises), library fines and fines for other similar charges, and any income not reported elsewhere.

h) Interfund Transfers

When income from one fund is used to finance expenditures in another fund, report the amount as an interfund transfer. Total interfund transfers must net to zero.

5. Types of Expenditures

a) Salaries and Wages

Salaries and wages (excluding fringe benefits) as well as payments for leave of absence, shown under the appropriate functions and programs, are to be broken down into the following two categories:

(i) Teachers
Included in this category are salaries and wages paid to full-time and part-time teaching staff.
(ii) Other
This category includes all salaries not reported in part (i) above. Specifically, it includes salaries and wages paid to tutors, monitors, demonstrators, markers, laboratory technicians, maintenance personnel, office and technical staff, research and teaching assistants, etc..

b) Fringe Benefits

This includes the institution's contribution (in respect of all salaries and wages) to pensions, group life insurance, workmen's compensation, unemployment insurance, Canada pension, salary contribution insurance, long term disability insurance and other similar benefits. Also include staff development costs paid for by the institution.

c) Library Acquisitions

This includes all purchases of books, periodicals, audio/visual material and other reference material for the library. Costs of binding may also be included if normally considered part of the acquisition costs.

d) Operational Supplies and Expenses

This includes all expenditures for supplies which are normally consumed in the fiscal year, including postage, teaching supplies, photocopying, publications, long distance telephone charges, repair materials, all supplies to operate laboratories, etc..

e) Utilities

This includes all expenditures for fuel, electricity, water, gas, telephone equipment rental, etc..

f) Furniture and Equipment

This includes all expenses for furniture and equipment, such as laboratory equipment (other than consumables), administrative equipment and furnishings, copying and duplicating equipment, computing equipment maintenance equipment, etc.. Rental and maintenance costs as well as other related operating expenses should be shown under the appropriate operational function. Costs for replacing or acquiring new furniture and equipment should be reported under the capital fund.

g) Scholarships and Other Related Students Support

This includes all payments to students including scholarships, bursaries, prizes, fee remissions, gifts, etc..

h) Fees and Contracted Services

This includes all expenses for services contracted to external agencies (except for renovations, alterations and major repairs). Examples would be cleaning contracts, security services, snow removal, etc.. Also included are fees paid to legal counsellors (including retainers for negotiations of collective contracts), auditors' fees, consultant's fees, etc..

i) Debt Services

This includes all payments made to service debts of the institution such as bank interest, mortgage or debenture interest payments, and related charges. Principal payments on loans, mortgages, debentures or repayable grants should be excluded.

j) Buildings

This includes all capital expenditures which are normally considered part of construction costs, except for furniture and equipment as well as land and site services which are to be reported under their respective item. Costs for space rental, building insurances, taxes, minor renovations and alterations on buildings, and all other related operating expenses should be shown under the Physical Plant operational function. Depreciation is not to be included as an expenditure.

k) Land and Site Services

This includes capital expenditures on acquisitions of and improvements to land such as landscaping, sewers, tunnels, roads, etc.. Capitalized professional fees and planning costs related to this category are also to be included. Rental, maintenance and insurance costs as well as other related operating expenses for this item should be shown under the Physical Plant operational function.

l) Miscellaneous

This is to be used when the institution has an operating or capital expenditure not classified in the other categories.

m) Transfers To/From

This item is used for internal transfers of costs between funds or functions whenever it is not feasible to directly adjust the appropriate expenditure items.

The total internal transfers of costs should net to zero.

n) Ancillary Enterprises (gross)

Includes all gross expenditures incurred in the operating of ancillary enterprises (see section 4 (e) above).

It should be noted that the reporting officer is asked to report, on the Supporting Schedule A, a breakdown of total expenditures for the institution's ancillary enterprises.

VIII. Supporting Schedule A

Additional information is to be provided in this section for the total revenue and expenditures of institutional ancillary enterprises (bookstores, residences, food services and parking).

IX. Suggestions

Statistics Canada would welcome any suggestions made to improve this survey.

Canadian Classification of Institutional Units and Sectors (CCIUS) 2012

Introduction

Preface
Acknowledgements
Background
Conformance of CCIUS 2012 with the international standard
Titles and codes at the subgroup level
Summary of the classification structure

Preface

The 2012 version of the Canadian Classification of Institutional Units and Sectors (CCIUS) was developed as a result of the implementation of international recommendations published in the 2008 System of National Accounts manual (SNA 2008). It replaced Statistics Canada's Classification of Institutional Units by Sector (CIUS) that was based on the 1993 System of National Accounts (SNA 1993).

CCIUS is an important classification that provides consistent and coherent definition for concepts underlying the production of economic statistics used for compiling macroeconomic aggregates published by all the programs in the Macroeconomic Accounts Branch. It provides the classification framework needed by statistical infrastructure and survey programs such as the Statistical register as well as Industrial organization and Finance for maintaining up-to-date survey frames, and collecting data from institutional units. CCIUS also serves as the backbone of the system of national accounts framework by providing the necessary linkage to the sequence of economic accounts from the production account all the way to the national balance sheet account, enhancing data integration, as well as the quality of data and metadata. The classification is used for publishing Canada's economic statistics by institutional sectors as well as in Statistics Canada's international data submissions to international organizations such as the Organization of Economic Cooperation and Development, International Monetary Fund, etc.

CCIUS is the grouping of institutional units that make up the sectors of the entire economy as presented in the SNA 2008 manual. An institutional unit is an economic entity that is capable of owning assets, incurring liabilities, and engaging in economic activities and transactions with other entities. Institutional units are allocated to institutional sectors according to the nature of the economic activity they undertake, namely production, consumption and capital accumulation. The allocation leads to the grouping of similar institutional units within a given institutional sector, as well as to the delineation of sectors that are fundamentally different from each other due to their economic objectives, functions, and behaviour.

The international version of institutional classifications as presented in the 2008 SNA incorporates new areas that were not in the 1993 SNA. Statistics Canada has implemented some of these recommendations. Some differences however, did not allow for the full adoption of the classification as recommended by SNA2008. The following is a summary of some of the major differences. One, in the Canadian System of Macroeconomic Accounts (CSMA), formerly the Canadian System of National Accounts (CSNA), all public financial corporations are separately classified as Financial Government Business Enterprises (FGBEs). This practice pre-dates the 1993 and 2008 SNA. This is also true for public non-financial corporations that are classified separately as Non-financial Government Business Enterprises (NGBEs). Two, for dissemination purposes, there is no breakdown between national and foreign institutional units in the CSMA. In other words, there is no delineation between domestic and foreign controlled units. Three, even though Statistics Canada's Statistical Register maintains extra variables (essentially based on administrative data) in order to flag the Non Profit Institutions (NPI), the CSMA does not provide a breakdown between NPIs and For Profit Institutions (FPIs). Four, there are also differences in the extent of sector detail between the CSMA and SNA 2008. CSMA has more detail on insurance corporations and general government (due to the inclusion of Canadian specificity - Aboriginal general government) and SNA 2008 has more financial subsectors than CSMA, 9 versus 5 respectively.

CCIUS 2012 is closely aligned with the international version at the sector level (two-digit). Starting from the subsector level (three-digit), the CSMA and the SNA2008 begin to diverge where the former has 5 financial subsectors and the later has 9 financial subsectors which had to be collapsed into 5. As a result, the subsector level is more of a harmonized version of CSMA's classification with SNA 2008. At the major group (four-digit) and group (five-digit) levels the CCIUS is almost entirely a reflection of the working level details that are currently used in the CSMA. At the subgroup (six-digit) level those institutional units controlled by government are designated with a classification code that ends with 1, while those controlled by national private and foreign institutional units are designated with codes that end with 2 and 3, respectively. This is simply a designation of codes that is consistent with SNA 2008 and does not necessarily follow the standard classification coding scheme that was applied to the rest of the structure.

CCIUS has been developed as a standard classification, compliant with the Statistical Classification Model of the Generic Statistical Information Model (GSIM).

Acknowledgements

The development of Canadian Classification of Institutional Units and Sectors (CCIUS) 2012 required the time, energy and co-operation of a number of experts from Standard division, the Macroeconomic accounts branch, and other divisions within Statistics Canada.

CCIUS was developed by Berouk Terefe under the supervision of Tony Labillois and Philippe Gagné of the National Accounts Integration Group (NAIG). The work has benefited from expert input and guidance from Alice Born, Director of Standards Division, Johanne Pineau-Crysdale, and Michael Pedersen. James Tebrake, Director General Macroeconomic Accounts Branch, Charles Wright and Michel Pascal have also provided valuable feedback. Serge Aumont provided input on the preliminary version.

CCIUS 2012 is published by Standards Division. Thanks to Brent Hultquist and Niloufar Zanganeh for their technical support.

Background

Statistics Canada's Classification of Institutional Units by Sector (CIUS) that was based on the 1993 System of National Accounts (SNA) has been updated and replaced by the 2012 Canadian Classification of Institutional Units and Sectors (CCIUS) based on SNA 2008.

The classification of institutional units and sectors is the grouping of institutional units that make up the sectors and subsectors of the entire economy as presented in the SNA 2008 manual. An institutional unit is an economic entity that is capable of owning assets, incurring liabilities, and engaging in economic activities and transactions with other entities.

Institutional units are allocated to institutional sectors according to the nature of the economic activity they undertake, namely production, consumption and capital accumulation. The allocation leads to the grouping of similar institutional units within a given institutional sector, as well as to the delineation of sectors that are fundamentally different from each other due to their economic objectives, functions, and behaviour.

According to SNA 2008, the main attributes of institutional units may be described as follows:

An institutional unit is entitled to own goods or assets in its own right; it is therefore able to exchange the ownership of goods or assets in transactions with other institutional units;
It is able to take economic decisions and engage in economic activities for which it can be held directly responsible and accountable by law;
It is able to incur liabilities on its own behalf, to take on other obligations or future commitments and to enter into contracts;
Either a complete set of accounts, including a balance sheet of assets and liabilities, exists for the unit, or it would be possible and meaningful, from an economic viewpoint, to compile a complete set of accounts if they were to be required.

The total economy consists of the entire set of resident institutional units. There are two types of institutional units, namely persons or group of persons, and legal or social entities. Persons or groups of persons make up the households sector. Legal or social entities engaging in economic activities and transactions make up the rest of the sectors in the economy, in the form of corporations (non-financial and financial) including quasi corporations, general government, and non-profit institutions serving households (NPISH). Quasi-corporations are unincorporated enterprises that behave the same way as corporations, maintaining a complete set of accounts, including balance sheets. As such, all resident institutional units are grouped together to form institutional sectors, on the basis of their principal functions, behaviour and objectives, and are allocated to one and only one of the following five institutional sectors: the Non-financial corporations sector, the Financial corporations sector, the General government sector, the Households sector, and the Non-profit institutions serving households sector. These sectors are described in more detail in subsequent sections of this document.

The residence of each institutional unit is the economic territory with which it has the strongest connection, (i.e. its centre of predominant economic interest). As a result, each institutional unit is treated as a resident of a single economic territory. An institutional unit has a centre of predominant economic interest in an economic territory when there exists, within the economic territory, some location, dwelling, place of production, or other premises on which or from which the unit engages and intends to continue engaging, either indefinitely or over a finite but long period of time, in economic activities and transactions on a significant scale. The location does not need to be fixed so long as it remains within the economic territory. The actual or intended location for one year or more is used as an operational definition; while the choice of one year as a specific period is somewhat arbitrary, it is adopted to avoid uncertainty and facilitate international consistency.

The resident institutional units other than households are delineated between market and non-market producers where the former consists of the Financial and Non-financial corporations sectors while the latter consists of the General government sector with government controlled institutional units as well as the Non-profit institutions serving households sector. Market producers are establishments whose output is entirely or mostly market production. That is, market production consisting of output intended for sale at economically significant prices. Non-market producers consist of establishments owned by government units or NPISHs that supply goods or services for free, or at prices that are not economically significant, to households or the community as a whole.

SNA2008 distinguishes between three ownership and control types, public, private, and foreign control which are all briefly described below. Readers are referred to the SNA 2008 manual for a more detailed description of each.

Under public control, a corporation is a public corporation if a government unit, another public corporation, or some combination of government units and public corporations controls it, where control is defined as the ability to determine the general corporate policy of the corporation. The expression "general corporate policy" as used here is understood in a broad sense to mean the key financial and operating policies relating to the corporation's strategic objectives as a market producer. For entities that are identified as public-private partnership, because of their fluid nature, there is no prescriptive rule or criteria used for determining control and ownership. Instead, the provision of each public-private partnership is evaluated on a case-by-case basis in order to determine the unit that has control over the entity.

Under private control, a single institutional unit owning more than half of the shares, or equity, of a corporation is able to control its policy and operations by outvoting all other shareholders, if necessary. Similarly, a small, organized group of shareholders whose combined ownership of shares exceeds 50 per cent of the total is able to control a corporation by acting in concert.

Under foreign control, in general, a non-resident unit controls a resident corporation if the non-resident unit owns more than 50 per cent of the equity of the corporation. Branches of non-resident corporations are by their nature always under foreign control. However, control may also be possible with a holding of less than half the equity if the non-resident unit can exercise such powers as exercised under government control, for example via the control of the board or other governing body, control of the appointment and removal of key personnel, or control of key committees of the corporations and so on.

SNA 2008 also distinguishes between Non-profit institutions and For-profit institutions. Non-profit institutions (NPIs) are legal or social entities created for the purpose of producing goods and services but whose status does not permit them to be a source of income, profit or other financial gain for the units that SNA 2008 establishes, control or finance them. Within the Financial and Non-financial corporations sectors, units that are not NPIs are referred to as for-profit institutions, or FPIs. An NPI is not prohibited from making a profit; it is simply prohibited from distributing any profit it makes to its owners. Thus NPIs within the Financial and Non-financial corporations sectors are market producers just as the FPIs are. For more detail on NPIs and FPIs, readers are referred to SNA 2008.

Conformance of CCIUS 2012 with the international standard

Some differences did not allow for the full adoption of the classification as recommended by SNA2008. The following is a summary of some of the major differences.

In the Canadian System of Macroeconomic Accounts (CSMA), formerly the Canadian System of National Accounts (CSNA), all public financial corporations are separately classified as Financial Government Business Enterprises (FGBEs). This practice pre-dates the 1993 and 2008 SNA. This is also true for public non-financial corporations that are classified separately as Non-financial Government Business Enterprises (NGBEs).
For dissemination purposes, there is no breakdown between national and foreign institutional units in the CSMA. In other words, there is no delineation between domestic and foreign controlled units.
Even though Statistics Canada's Business Register maintains extra variables (essentially based on administrative data) in order to flag the NPIs, the CSMA does not provide a breakdown between NPIs and FPIs.
There are also differences in the extent of sector detail between the CSMA and SNA 2008.
1. CSMA has more detail on insurance corporations and general government (due to the inclusion of Canadian specificity - Aboriginal general government).
2. SNA 2008 has more financial subsectors than CSMA, 9 versus 5 respectively.

At the subgroup (six-digit) level, detailed coding by type of control has been developed and made available in case any user division in the CSMA decides to make use of it at some point, provided that the information by type of control becomes available. However, as mentioned above, since the control information is not readily available in the CSMA at this time, data on institutional units cannot be delineated by control type. This issue however, becomes irrelevant when it comes to the General government and Households sectors since all the units within the General government sector are under public control while all units within the Households sector are under national private control.

A more detailed discussion related to differences in conformance with the international version is presented below under each of the respective sectors and subsectors.

Non-financial corporations

Unlike the international version, in the CSMA Non-financial corporations are not disaggregated into non-profit institutions (NPIs) and for-profit institutions (FPIs) due to constraints related to operations and dissemination. As well, in the CSMA non-financial corporations that are government business enterprises (GBEs) are classified separately. In CCIUS 2012 they are presented as Public non-financial corporations. The subgroup level may also be further disaggregated by general government subsectors to show the level of government that has controlling authority over a given institutional unit. According to the international version, these GBEs would be classified under the subgroup National public non-financial corporations

Government business enterprises are usually established by governments through an Act of Parliament or Legislature. In some instances, an entity will become a GBE through the government's takeover of a private corporation, by expropriation, by purchasing a controlling portion of voting shares, or by other means. Certain entities may also be classified as government business enterprises in accordance with international convention.

Financial corporations

As opposed to the 9 subsectors identified in the international standard, the Canadian version aggregates the institutional units of the Financial corporations sector into 5 subsectors. Three of these 5 subsectors (of the international version), Money market funds, Non- money market funds, and Captive financial institutions and money lenders subsectors are all imbedded in one subsector of CSMA 2012, Other financial intermediaries (except insurance corporations and pension funds). Two more subsectors from the international version, Insurance corporations and Pension funds are both rolled up into one subsector in the CSMA 2012, Insurance corporations and pension funds.

In the CSMA, financial corporations that are government business enterprises (GBEs) are classified separately. In CCIUS 2012 they are presented as Public financial corporations. The subgroup level may also be further disaggregated by general government subsectors to show the level of government that has controlling authority over a given institutional unit. According to the international version, these GBEs would be classified under the subgroup National public financial corporations.

Other financial intermediaries except insurance corporations and pension funds

In the CSMA, this subsector explicitly includes money market funds, and Non- money market funds. In SNA 2008, each of these is treated as an independent subsector of the financial corporation sector. The subsector also implicitly includes institutional units of Captive financial institutions and Money lenders. This CSMA subsector amalgamates these subsectors that are otherwise treated as three independent subsectors in the international version.

Financial auxiliaries

In the CSMA, unlike in SNA 2008, all financial auxiliaries are classified as investment dealers and are embedded in Other financial intermediaries.

Insurance corporations and pension funds

In the CSMA, this subsector amalgamates Insurance corporations and Pension funds that are otherwise treated as two independent subsectors in the international version, SNA2008.

General government

According to SNA 2008, the sector consists of three levels of governments and social security funds as its subsectors. The three levels of governments are Central government, State government, and Local government. In the CSMA, these three levels of governments are alternatively known as Federal, Provincial and territorial, and Local government respectively. In addition to the three levels of government, the CSMA includes the Aboriginal general government subsector as another stand-alone level of government. The subsector represents First nations and other Aboriginal government institutional units.

There are two types of sectoring schemes presented in SNA 2008 for the General government sector. The first method subdivides the sector into Central government, State government, Local government, and Social security funds, where the subsectors include NPIs but exclude social security funds at that level of government. The second method subdivides the sector into Central government, State government, and Local government where it is understood that each of the subsectors include both NPIs and social security funds at that level of government. The CSMA follows the first of the two sectoring schemes.

In the CSMA, in addition to those entities identified as institutional units within the General government sector, there are also entities identified as sub-institutional units. These are referred to as Autonomous general government organizations and Non-autonomous general government organizations, respectively. Autonomous general government organizations are classified as institutional units that are empowered to operate independently from their parent government. They have their own employees and may be organized as Crown corporations, boards, commissions or agencies. Non-autonomous general government organizations are sub-institutional units that cannot function independently from their parent government. They operate within a government ministry or department. They do not have separate books of account; rather their activities are part of the ministry's or the department's financial transactions.

Households

There is more than one method of subdividing the Households sector into its subsectors depending on the type of analysis or policymaking for which it is used. The method applied by CSMA is subsectoring according to income. The other alternative methods are subsectoring according to characteristics of a reference person and subsectoring according to household size and location. All three methods are fully described in the SNA 2008 manual.

According to SNA 2008, Household unincorporated market enterprises are created for the purpose of producing goods or services for sale or barter on the market. Although in general households are unlike corporations in that they undertake final consumption, they may also engage in production like corporations. Unincorporated enterprises of partnerships with many partners, such as some large legal, accounting or architectural firms, that are likely to behave like corporations, are treated as quasi-corporations assuming complete sets of accounts are available for the partnerships. Unincorporated enterprises that are based on limited liability partnerships are effectively separate legal entities and are treated as corporations. The unincorporated enterprise can only be treated as a corporation if it is possible to separate all its assets, including financial assets down to the level of cash, into those that belong to the household in its capacity as a consumer from those belonging to the household in its capacity as a producer.

Non-profit institutions serving households

This sector consists of non-profit institutions that are engaged in providing goods and services to households for free or at prices that are not economically significant. Non-profit Institutions (NPIs) are allocated to the Financial or Non-financial corporations sectors when they are engaged in market production and to the General government sector if they are engaged in non-market production but subject to government control. An institutional unit classified to NPISHs is controlled neither by corporations nor by general government. In line with SNA 2008, the CSMA has incorporated NPISH as one of the institutional sectors in the accounts.

Rest of the world

The Rest of the world consists of all non-resident institutional units that enter into transactions with resident units, or have other economic links with resident units. The rest of the world includes certain institutional units that may be physically located within the geographic boundary of a country; for example, foreign enclaves such as embassies, consulates or military bases, and international organizations including central banks of currency unions.

According to SNA 2008, the central bank of a currency union is treated as a special kind of international organization. The members of the international organization of which the central bank is part are the governments or the national central banks of the countries in the currency union. The central bank is treated as being non-resident in any of the member countries of the currency union but is resident in the currency area as a whole. A more detailed discussion on the treatment of currency and economic unions can be found in appendix 3 of the sixth edition of the Balance of Payments and International Investment Position Manual (BPM6). Currently, Canada is not a member of any currency union and no central bank of a currency union exists in Canada.

Titles and codes at the subgroup level

In line with SNA 2008 in the CSMA, institutional units controlled by the various levels of government in Canada are classified as publicly controlled, while those controlled by Canadian private sector institutional units are classified into the national private control category. All subgroups within the general government and households sectors are exclusively under public and national private control, respectively. As a result, in order to avoid redundancy, the titles at the subgroup level within the two sectors, are not referred to as national public and national private.

In order to clearly distinguish those domestically controlled national and private institutional units from the foreign controlled, in the CSMA, the domestically controlled units are referred to as national public and national private, respectively. For institutional units controlled by entities residing outside Canada, the foreign controlled classification is applied.

At the subgroup (six-digit) level those institutional units controlled by government are designated with a classification code that ends with 1, while those controlled by national private and foreign institutional units are designated with codes that end with 2 and 3, respectively. This is simply a designation of codes that is consistent with SNA 2008 and does not necessarily follow the standard classification coding scheme that was applied to the rest of the structure.

Summary of the classification structure

CCIUS 2012 consists of 6 sectors including the Rest of the world, 19 subsectors, 38 major groups, 44 groups, and 62 subgroups. The following summary table shows the counts at each level of the structure.

CIUS 2012 consists of 6 sectors including the Rest of the world, 19 subsectors, 38 major groups, 44 groups, and 62 subgroups. The following summary table shows the counts at each level of the structure.
Economy	Code	Sector	Subsector	Major group	Group	Subgroup	Total number of classes
Total economy	S11	Non-financial corporations	2	2	2	3	9
	S12	Financial corporations	6	15	18	34	73
	S13	General government	5	11	14	14	44
	S14	Households	4	8	8	8	28
	S15	Non-profit institutions serving households	1	1	1	2	5
Rest of the world	S20	Rest of the world	1	1	1	1	4
2	6	Total	19	38	44	62	171

Report a problem or mistake on this page

Date modified:: 2023-01-10

Institutional units and sectors

Canadian Classification of Institutional Units and Sectors, 2012 is developed based on the international version published in the System of National Account, 2008 (2008 SNA). The 2008 SNA is the latest international standard for compiling national accounts statistics.

Canada

Canadian Classification of Institutional Units and Sectors (CCIUS)

International links

System of National Accounts, 2008 (2008 SNA) (PDF) (United Nations Statistics Division)

Integrated Business Statistics Program (IBSP)

Reporting Guide

This guide is designed to assist you as you complete the 2014 Annual Survey of Research and Development in Canadian Industry. If you need more information, please call the Statistics Canada Help Line at the number below.

Help Line: 1-800-972-9692

Your answers are confidential.

Statistics Canada is prohibited by law from releasing any information it collects which could identify any person, business, or organization, unless consent has been given by the respondent or as permitted by the Statistics Act.

Statistics Canada will use information from this survey for statistical purposes.

NOTE:

If this business performs in-house research and development (R&D) and outsources R&D, complete all questions.
If this business performs in-house research and development (R&D) and does not outsource R&D, complete question 1-5, 8-19.
If this business outsources research and development (R&D) and does not perform in-house R&D, complete questions 1-3, 5-7, 12, 16-19.
If this business does not perform in-house research and development (R&D) and does not outsource R&D, complete questions 1-3, 5, 12, 16, 17 and 19.

Difference between Scientific Research and Experimental Development (SR&ED) tax incentive program and this survey

Include the following in this survey:

capital R&D expenditures
R&D expenditures in the social sciences and humanities
payments for R&D performed by organizations outside Canada

For this survey

‘In-house R&D’ refers to

Expenditures within Canada for R&D performed within this business by:

employees (permanent, temporary or casual)
self-employed individuals or contractors who are working on-site on this business's R&D projects

‘Outsourced R&D’ refers to

Payments made within or outside Canada to other organizations, companies or individuals to fund R&D performance:

contracts
grants
fellowships

Reporting period information

Here are some examples of common fiscal periods that fall within the targeted dates:

May 1, 2013 to April 30, 2014
June 1, 2013 to June 30, 2014
August 1, 2013 to July 31, 2014
October 1, 2013 to September 30, 2014
December 1, 2013 to November 30, 2014
January 1, 2014 to December 31, 2014
February 1, 2014 to January 31, 2015
March 1, 2014 to February 28, 2015
April 1, 2014 to March 31, 2015

Here are other examples of fiscal periods that fall within the required dates:

September 18, 2013 to September 15, 2014 (e.g., floating year-end)
June 1, 2014 to December 31, 2014 (e.g., a newly opened business)

Definitions and Concepts

Research and development (R&D) comprise creative and systematic work undertaken in order to increase the stock of knowledge – including knowledge of humankind, culture and society – and to devise new applications of available knowledge

Research is original investigation undertaken on a systematic basis to gain new knowledge.

Development is systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products or devices, to installing new processes, systems and services, or to improving substantially those already produced or installed.

Activities included and excluded from R&D

Inclusions

Prototypes

Include design, construction and operation of prototypes provided that the primary objective is to make further improvements or to undertake technical testing. Exclude if the prototype is for commercial purposes.

Pilot plants

Include construction and operation of pilot plants provided that the primary objective is to make further improvement or to undertake technical testing. Exclude if the pilot plant is intended to be operated for commercial purposes.

New computer software or significant improvements/modifications to existing computer software

Includes technological or scientific advances in theoretical computer sciences; operating systems e.g. improvement in interface management, developing new operating system of converting an existing operating system to a significantly different hardware environment; programming languages; and applications if a significant technological change occurs.

Contracts

Include all contracts which require R&D. For contracts which include other work, report only the R&D costs.

Research work in the social sciences

Include if projects are employing new or significantly different modelling techniques or developing new formulae, analyzing data not previously available or applying new research techniques.

Exclusions

Routine analysis in the social sciences including policy-related studies, management studies and efficiency studies

Exclude analytical projects of a routine nature, with established methodologies, principles and models of the related social sciences to bear on a particular problem (e.g. commentary on the probable economic effects of a change in the tax structure, using existing economic data; use of standard techniques in applied psychology to select and classify industrial and military personnel, students, etc., and to test children with reading or other disabilities), are not R&D.

Consumer surveys, advertising, market research

Exclude projects of a routine nature, with established methodologies intended for commercialization of the results of R&D are excluded.

Routine quality control and testing

Exclude projects of a routine nature, with established methodologies not intended to create new knowledge are not R&D and are excluded even if carried out by personnel normally engaged in R&D.

Pre-production activities such as demonstration of commercial viability, tooling up, trial production, trouble shooting

Exclude although R&D may be required as a result of these steps, these activities are excluded from R&D.

Prospecting, exploratory drilling, development of mines, oil or gas wells

Include only if for R&D projects concerned with new equipment or techniques in these activities, such as in-situ and tertiary recovery research.

Engineering

Exclude engineering unless it is in direct support of R&D.

Design and drawing

Exclude design and drawing unless it is in direct support of R&D.

Patent and license work

Exclude all administrative and legal work connected with patents and licenses.

Cosmetic modifications or style changes to existing products

Exclude where no significant technical improvement or modification to the existing products.

General purpose or routine data collection

Exclude projects of a routine nature, with established methodologies intended for on-going monitoring of an activity.

Routine computer programming, systems maintenance or software application

Exclude projects of a routine nature, with established methodologies intended to support on-going operations.

Routine mathematical or statistical analysis or operations analysis

Exclude projects of a routine nature, with established methodologies intended for on-going monitoring of an activity.

Activities associated with standards compliance

Exclude projects of a routine nature, with established methodologies intended to support standards compliance.

Specialized routine medical care such as routine pathology services

Exclude projects of a routine nature, with established methodologies intended for on-going monitoring of an activity.

In-house R&D expenditures within Canada (Q4)

In-house research and development expenditures are composed of current in-house research and development (R&D) expenditures and capital in-house R&D expenditures.

Current in house R&D expenditures

Wages and salaries of permanent, temporary and casual R&D employees
Include: fringe benefits

Fringe benefits of employees engaged in R&D activities. Fringe benefits include bonus payments, holiday or vacation pay, pension fund contributions, other social security payments, payroll taxes, etc.
Services to support R&D
Include: services of self-employed individuals or contractors who are working on-site on this business’s R&D projects.
Exclude: contracted out or granted expenditures to other organizations to perform R&D.

Payments to on-site R&D consultants and contractors working under the direct control of your business; indirect services purchased to support in-house R&D such as security, storage, repair, maintenance and use of buildings and equipment; computer services, software licensing fees and dissemination of R&D findings
R&D materials

Utilities: water, fuel, gas and electricity; materials for creation of prototypes, reference materials (books, journals, etc.); subscriptions to libraries and data bases, memberships to scientific societies, etc.; cost of outsourced small R&D prototypes or R&D models; materials for laboratories (chemicals, animals, etc.); all other R&D-related materials
All other current costs

Administrative and overhead costs (e.g., office, post and telecommunications, internet, insurance), prorated if necessary to allow for non-R&D activities within the business

Capital in-house R&D expenditures

Capital in-house R&D expenditures are the annual gross amount paid for the acquisition of fixed assets that are used repeatedly, or continuously in the performance of research and development (R&D) for more than one year. They should be reported in full for the period when they occurred. Exclude capital depreciation.

Software
Exclude: capital depreciation

Applications and systems software (original, custom and off-the-shelf software), supporting documentation and other software-related acquisitions
Land
Exclude: capital depreciation

Land acquired for R&D including testing grounds, sites for laboratories and pilot plants
Buildings and structures
Exclude: capital depreciation

Buildings and structures (constructed or purchased) for research and development (R&D) activities or that have undergone major leasehold improvements (modifications, renovations and repairs) for R&D activities
Equipment, machinery and all other
Exclude: capital depreciation

Major equipment, machinery and instruments, including embedded software, acquired for research and development (R&D) activities

Outsourced (contracted out or granted) R&D expenditures (Q6)

Payments made through contracts, grants and fellowships to another organization, company or individual to purchase R&D activities.

Include: contracted out expenditures for research and development (R&D), funding or grants provided to other organizations to perform R&D.

Exclude: expenditures for services of self-employed individuals or contractors who are working on-site on this business's R&D.

Companies

All incorporated for profit businesses and government business enterprises providing products in the market at market rates.
Private non-profit organizations
Voluntary health organizations, private philanthropic foundations, associations and societies and research institutes; they are not for profit organizations that serve the public interest by supporting activities related to public welfare (such as health, education, the environment).
Industrial research institutes or associations

Non-profit organizations that serve the business enterprise sector frequently consisting of their membership. Industrial non-profit organizations include non-profit industrial research institutes.
Hospitals
Universities
Federal government departments and agencies

All federal government ministries, departments and agencies. It excludes federal government business enterprises providing products in the market.
Provincial government departments and agencies

All provincial government ministries, departments and agencies. It excludes provincial government business enterprises providing products in the market.
Provincial research organizations

Organizations created under provincial or territorial law which conduct or facilitate research on behalf of the province or territory.
Other

Individuals, non-university educational institutions, foreign governments

Sources of funds for in-house R&D expenditures in 2014 (Q9)

Include: Canadian and foreign sources

Exclude: payments for outsourced (contracted out or granted) R&D which should be reported in question 6 on Outsourced (contracted out or granted) R&D; capital depreciation.

Funds from this business

Amount contributed by this unit to R&D performed within Canada (include interest payments and other income, land, buildings and structures, equipment and machinery (capital expenditures) purchased for R&D). Include amounts eligible for income tax purposes, e.g., Scientific Research and Experimental Development (SR&ED) program, other amounts spent for projects not claimed through SR&ED, and funds for land, buildings, machinery and equipment (capital expenditures) purchased for R&D.
Funds from parent, affiliated and subsidiary companies

Amount received from parent, affiliated and subsidiary businesses used to perform R&D within Canada (include amounts eligible for income tax purposes, e.g., Scientific Research and Experimental Development (SR&ED) program, other amounts spent for projects not claimed through SR&ED, and funds for land, buildings, machinery and equipment (capital expenditures) purchased for R&D).
Federal grants

Include: R&D grants or R&D portion only of other grants

Funds from the federal government in support of R&D activities not connected to a specific contractual deliverable.
Federal contracts

Include: R&D contracts or R&D portion only of other contracts

Funds from the federal government in support of R&D activities connected to a specific contractual deliverable.
R&D contract work for other companies

Funds received from other companies to perform R&D on their behalf.
Other sources

Funds received from all other sources not previously classified.

In-house R&D expenditures by fields of research and development in 2014 (Q10)

Exclude: capital depreciation and payments for outsourced (contracted out or granted) R&D which should be reported in question 6 on Outsourced (contracted out or granted) R&D.

Natural and formal sciences

Mathematics, physical sciences, chemical sciences, earth and related environmental sciences, biological sciences, other natural sciences.

Exclude: computer sciences, information sciences and bioinformatics (to be reported at lines s and t)

Mathematics

Pure mathematics, applied mathematics, statistics and probability.
Physical Sciences

Atomic, molecular and chemical physics, interaction with radiation, magnetic resonances, condensed matter physics, solid state physics and superconductivity, particles and fields physics, nuclear physics, fluids and plasma physics (including surface physics), optics (including laser optics and quantum optics), acoustics, astronomy (including astrophysics, space science).
Chemical sciences

Organic chemistry, inorganic and nuclear chemistry, physical chemistry, polymer science and plastics, electrochemistry (dry cells, batteries, fuel cells, metal corrosion, electrolysis), colloid chemistry, analytical chemistry.
Earth and related environmental sciences

Geosciences, geophysics, mineralogy and palaeontology, geochemistry and geophysics, physical geography, geology and volcanology, environmental sciences, meteorology, atmospheric sciences and climatic research, oceanography, hydrology and water resources.
Biological sciences

Cell biology, microbiology and virology, biochemistry, molecular biology and biochemical research, mycology, biophysics, genetics and heredity (medical genetics under medical biotechnology), reproductive biology (medical aspects under medical biotechnology), developmental biology, plant sciences and botany, zoology, ornithology, entomology and behavioural sciences biology, marine biology, freshwater biology and limnology, ecology and biodiversity conservation, biology (theoretical, thermal, cryobiology, biological rhythm), evolutionary biology.
Other natural sciences

Engineering and Technology

Civil engineering, electrical engineering, electronic engineering and communications technology, mechanical engineering, chemical engineering, materials engineering, medical engineering, environmental engineering, environmental biotechnology, industrial biotechnology, nanotechnology, other engineering and technologies.

Exclude: software engineering and technology (to be reported at line r)

Civil engineering

Civil engineering, architecture engineering, municipal and structural engineering, transport engineering.
Electrical engineering, electronic engineering and communications technology

Electrical and electronic engineering, robotics and automatic control, micro-electronics, semiconductors, automation and control systems, communication engineering and systems, telecommunications, computer hardware and architecture.
Mechanical engineering

Mechanical engineering, applied mechanics, thermodynamics, aerospace engineering, nuclear-related engineering (nuclear physics under Physical sciences), acoustical engineering, reliability analysis and non-destructive testing, automotive and transportation engineering and manufacturing, tooling, machinery and equipment engineering and manufacturing, heating, ventilation and air conditioning engineering and manufacturing.
Chemical engineering

Chemical engineering (plants, products), chemical process engineering.
Materials engineering

Materials engineering and metallurgy, ceramics, coating and films (including packaging and printing), plastics, rubber and composites (including laminates and reinforced plastics), paper and wood and textiles, construction materials (organic and inorganic).
Medical Engineering

Medical and biomedical engineering, medical laboratory technology (excluding biomaterials which should be reported under industrial biotechnology).
Environmental engineering

Environmental and geological engineering, petroleum engineering (fuel, oils), energy and fuels, remote sensing, mining and mineral processing, marine engineering, sea vessels and ocean engineering.
Environmental biotechnology

Environmental biotechnology, bioremediation, diagnostic biotechnologies in environmental management (DNA chips and bio-sensing devices).
Industrial biotechnology

Industrial biotechnology, bioprocessing technologies, biocatalysis and fermentation bioproducts (products that are manufactured using biological material as feedstock), biomaterials (bioplastics, biofuels, bioderived bulk and fine chemicals, bio-derived materials).
Nanotechnology

Nano-materials (production and properties), nano-processes (applications on nano-scale).
Other engineering and technologies

Food and beverages, oenology, other engineering and technologies.

Software-related sciences and technology

Software engineering and technology, computer sciences, information technology and bioinformatics.

Software engineering and technology

Computer software engineering, computer software technology, and other related computer software engineering and technologies.
Computer sciences

Computer science, artificial intelligence, cryptography, and other related computer sciences.
Information technology and bioinformatics

Information technology, informatics, bioinformatics, biomathematics, and other related information technologies.

Medical and health sciences

Basic medicine, clinical medicine, health sciences, medical biotechnology, other medical sciences.

Basic medicine

Anatomy and morphology (plant science under biological science), human genetics, immunology, neurosciences, pharmacology and pharmacy and medicinal chemistry, toxicology, physiology and cytology, pathology.
Clinical medicine

Andrology, obstetrics and gynaecology, paediatrics, cardiac and cardiovascular systems, haematology, anaesthesiology, orthopaedics, radiology and nuclear medicine, dentistry, oral surgery and medicine, dermatology, venereal diseases and allergy, rheumatology, endocrinology and metabolism and gastroenterology, urology and nephrology, and oncology.
Health sciences

Health care sciences and nursing, nutrition and dietetics, parasitology, infectious diseases and epidemiology, occupational health.
Medical biotechnology

Health-related biotechnology, technologies involving the manipulation of cells, tissues, organs or the whole organism, technologies involving identifying the functioning of DNA, proteins and enzymes, pharmacogenomics, gene-based therapeutics, biomaterials (related to medical implants, devices, sensors).
Other medical sciences

Forensic science, other medical sciences.

Agricultural Sciences

Agriculture, forestry and fisheries sciences, animal and dairy sciences, veterinary sciences, agricultural biotechnology, other agricultural sciences.

Agriculture, forestry and fisheries sciences

Agriculture, forestry, fisheries and aquaculture, soil science, horticulture, viticulture, agronomy, plant breeding and plant protection.
Animal and dairy sciences

Animal and dairy science, animal husbandry.
Veterinary sciences

Veterinary science (all).
Agricultural biotechnology

Agricultural biotechnology and food biotechnology, genetically modified (GM) organism technology and livestock cloning, diagnostics (DNA chips and biosensing devices), biomass feedstock production technologies and biopharming.
Other agricultural sciences

Social sciences and humanities

Psychology, educational sciences, economics and business, other social sciences, humanities.

Psychology

Cognitive psychology and psycholinguistics, experimental psychology, psychometrics and quantitative psychology, and other fields of psychology.
Educational sciences

Education, training and other related educational sciences.
Economics and business

Micro-economics, macro-economics, econometrics, labour economics, financial economics and all other related fields of economics and business.
Other social sciences

Anthropology (social and cultural) and ethnology, demography, geography (human, economic and social), planning (town, city and country), management, organization and methods (excluding market research unless new methods/techniques are developed), law, linguistics, political sciences, sociology, miscellaneous social sciences and interdisciplinary, and methodological and historical science and technology activities relating to subjects in this group.
Humanities

History (history, prehistory and history, together with auxiliary historical disciplines such as archaeology, numismatics, palaeography, genealogy, etc.), languages and literature (ancient and modern), other humanities (philosophy (including the history of science and technology)), arts (history of art, art criticism, painting, sculpture, musicology, dramatic art excluding artistic “research” of any kind), religion, theology, other fields and subjects pertaining to the humanities, and methodological, historical and other science and technology activities relating to the subjects in this group.

In house R&D expenditures by nature of R&D activity in 2014 (Q11)

R&D is performed in the natural sciences, engineering, social sciences and humanities. There are three types of R&D activities: basic research, applied research and experimental development.

Basic research

Experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observable facts, without any particular application or use in view.
Applied research

Also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily towards a specific practical aim or objective.
Experimental development

Systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products or devices, to installing new processes, systems and services, or to improving substantially those already produced or installed.

In-house R&D personnel in 2014 (Q13 to Q15)

Include:

permanent, temporary and casual R&D employees
independent on-site R&D consultants and contractors working in your organization’s offices, laboratories, or other facilities
employees engaged in R&D-related support activities

Researchers and research managers

Scientists, social scientists, engineers and researchers
Include: software developers and programmers

Create new knowledge, products, processes, methods and systems. They include software developers and programmers. They may be certified by provincial educational authorities, provincial or national scientific or engineering associations (Include software developers and programmers).
Senior research managers

Plan or manage R&D projects and programs. They may be certified by provincial educational authorities, provincial or national scientific or engineering associations.

R&D technical, administrative and support staff

Technicians, technologists and research assistants
Include: software technicians

Assist scientists, engineers and researchers in R&D activities, e.g., laboratory technicians, chemical technicians, draftspersons, research assistants and software technicians. They may be certified by provincial educational authorities, provincial or national scientific or engineering associations.
Other R&D technical, administrative and support staff

Include administrative assistants, accountants, bookkeepers, equipment storage, maintenance managers and facilities operators engaged in administration, clerical or other activities in support of R&D activities. Also included are machinists and electricians engaged in the construction of prototypes.

Other R&D occupations

On-site R&D consultants and contractors

Are individuals hired 1) to perform project-based work or to provide goods at a fixed or ascertained price or within a certain time or 2) to provide advice or services in a specialized field for a fee and, in both cases, work at the location specified and controlled by the contracting organization.

Full-time equivalent (FTE)

R&D may be carried out by persons who work solely on R&D projects or by persons who devote only part of their time to R&D, and the balance to other activities such as testing, quality control and production engineering. To arrive at the total effort devoted to R&D in terms of personnel, it is necessary to estimate the full-time equivalent of these persons working only part-time in R&D.

FTE (full-time equivalent) = Number of persons who work solely on R&D projects + the time of persons working only part of their time on R&D.

Example calculation: If out of four scientists engaged in R&D work, one works solely on R&D projects and the remaining three devote only one quarter of their working time to R&D, then: FTE = 1 + 1/4 + 1/4 + 1/4 = 1.75 scientists.

Technology or technical assistant payments in 2014 (Q16 and Q17)

Definitions (equivalent to the Canadian Intellectual Property Office http://www.ic.gc.ca/eic/site/cipointernet-internetopic.nsf/eng/wr00143.html)

Patents

Government grant giving the right to exclude others from making, using or selling an invention.
Copyright

Provides protection for literary, artistic, dramatic or musical works (including computer programs), and three other subject matter known as: performance, sound recording, and communication signal.
Trademark

Word, symbol or design (or any combination of these features) used to distinguish the wares and services of one person or organization from those of others in the marketplace.
Industrial design

Visual features of shape, configuration, pattern or ornament (or any combination of these features), applied to a finished article of manufacture.
Integrated circuit topography

Three-dimensional arrangement of the electronic circuits in integrated circuit products or layout designs.
Original software

Consist of computer programs and descriptive materials for both systems and applications. Original software can be created in-house or outsourced and includes packaged software with customization.
Packaged or off-the-shelf software

Purchased for use by your organization and excludes customized software.
Databases

Consist of files of data organized to permit effective access and use of the data.
Other

Technical assistance, industrial processes and know-how.

Energy-related R&D by area of technology (Q18)

1. Fossil Fuels

Crude oils and natural gas exploration, crude oils and natural gas production, oil sands and heavy crude oils surface and sub-surface production and separation of the bitumen, tailings management, refining, processing and upgrading, coal production, separation and processing, transportation of fossil fuels.

Crude oils and natural gas exploration

Development of advanced exploration methods (geophysical, geochemical, seismic, magnetic) for on-shore and off-shore prospecting.
Crude oil and natural gas production (including enhanced recovery) and storage

On-shore and off-shore deep drilling equipment and techniques for conventional oil and gas, secondary and tertiary recovery of oil and gas, hydro fracturing techniques, processing and cleaning of raw product, storage on remote platforms (e.g., Arctic, off-shore), safety aspects of offshore platforms.
Oil sands and heavy crude oils surface and sub-surface production and separation of the bitumen, tailings management

Surface and in-situ production (e.g., SAGD); tailings management.
Refining, processing and upgrading

Processing of natural gas to pipeline specifications, and refining of conventional crude oils to refined petroleum products (RPPs), and the upgrading of bitumen and heavy oils either to synthetic crude oil or to RPPs. Upgrading may be done at an oil sands plant, regional merchant upgraders or integrated into a refinery producing RPPs.
Coal production, separation and processing

Coal, lignite and peat exploration, deposit evaluation techniques, mining techniques, separation techniques, coking and blending, other processing such as coal to liquids, underground (in-situ) gasification.
Transportation of fossil fuels

Transport of gaseous, liquid and solid hydrocarbons via pipelines (land and submarine) and their network evaluation; safety aspects of LNG transport and storage.

2. Renewable energy resources

Solar photovoltaics (PV), solar thermal-power and high-temperature applications, solar heating and cooling, wind energy, bio-energy – biomass production, bio-energy – biomass conversion to fuels, bio-energy – biomass conversion to heat and electricity, and other bio-energy, small hydro (less than 10 MW), large hydro (greater than or equal to 10 MW), other renewable energy.

Solar photovoltaics (PV)

Solar cell development, PV-module development, PV-inverter development, building-integrated PV-modules, PV-system development, other.
Solar thermal-power and high-temperature applications

Solar chemistry, concentrating collector development, solar thermal power plants, high-temperature applications for heat and power.
Solar heating and cooling

Daylighting, passive and active solar heating and cooling, collector development, hot water preparation, combined-space heating, solar architecture, solar drying, solar-assisted ventilation, swimming pool heating, low-temperature process heating, other.
Wind energy

Technology development, such as blades, turbines, converters structures, system integration, other.
Bio-energy – Biomass production/supply and transport

Improvement of energy crops, research on bio-energy production potential and associated land-use effects, supply and transport of bio-solids, bio-liquids, biogas and bio-derived energy products (e.g., ethanol, biodiesel), compacting and baling, other.
Bio-energy – Biomass conversion to fuels

Conventional bio-fuels, cellulosic-derived alcohols, biomass gas-to-liquids, other energy-related products and by-products.
Bio-energy – Biomass conversion to heat and electricity

Bio-based heat, electricity and combined heat and power (CHP), exclude multi-firing with fossil fuels.
Other bio-energy

Recycling and the use of municipal, industrial and agricultural waste as energy not covered elsewhere.
Small-Hydro – (less than 10 MW)

Plants with capacity below 10 MW.
Large-Hydro – (greater than or equal to 10 MW)

Plants with capacity of 10 MW and above.
Other renewable energy

Hot dry rock, hydro-thermal, geothermal heat applications (including agriculture), tidal power, wave energy, ocean current power, ocean thermal power, other.

3. Nuclear fission and fusion

Materials exploration, mining and preparation, tailings management, nuclear reactors, other fission, fusion.

Nuclear materials exploration, mining and preparation, tailings management

Development of advanced exploration methods (geophysical, geochemical) for prospecting, ore surface and in-situ production, uranium and thorium extraction and conversion, enrichment, handling of tailings and remediation.
Nuclear reactors

Nuclear reactors of all types and related system components.
Other fission

Nuclear safety, environmental protection (emission reduction or avoidance), radiation protection and decommissioning of power plants and related nuclear fuel cycle installations, nuclear waste treatment, disposal and storage, fissile material recycling, fissile materials control, transport of radioactive materials.
Fusion

All types (e.g., magnetic confinement, laser applications).

4. Electric Power

Generation in utility sector, combined heat and power in industry and in buildings, electricity transmission, distribution and storage of electricity.

Electric power generation in utility sector

Conventional and non-conventional technology (e.g., pulverised coal, fluidised bed, gasification-combined cycle, supercritical), re-powering, retrofitting, life extensions and upgrading of power plants, generators and components, super-conductivity, magneto hydrodynamic, dry cooling towers, co-firing (e.g., with biomass), air and thermal pollution reduction or avoidance, flue gas cleanup (excluding CO2 removal), CHP (combined heat and power) not covered elsewhere.
Electric power - combined heat and power in industry, buildings

Industrial applications, small scale applications for buildings.
Electricity transmission, distribution and storage

Solid state power electronics, load management and control systems, network problems, super-conducting cables, AC and DC high voltage cables, HVDC transmission, other transmission and distribution related to integrating distributed and intermittent generating sources into networks, all storage (e.g., batteries, hydro reservoirs, fly wheels), other.

5. Hydrogen and fuel cells

Hydrogen production for process applications, hydrogen production for transportation applications, hydrogen transport and storage, other hydrogen, fuel cells, both stationary and mobile.

Hydrogen production for process applications
Hydrogen production for transportation applications
Hydrogen transport and storage
Other hydrogen

End uses (e.g., combustion), other infrastructure and systems R&D (refuelling stations).
Stationary fuel cells

Electricity generation, other stationary end-use.
Mobile fuel cells

Portable applications.

6. Energy efficiency

Industry, residential and commercial, transportation, other energy efficiency.

Energy efficiency applications for industry

Reduction of energy consumption through improved use of energy and/or reduction or avoidance of air and other emissions related to the use of energy in industrial systems and processes (excluding bio-energy-related) through the development of new techniques, new processes and new equipment, other.
Energy efficiency for residential, institutional and commercial sectors

Space heating and cooling, ventilation and lighting control systems other than solar technologies, low energy housing design and performance other than solar technologies, new insulation and building materials, thermal performance of buildings, domestic appliances, other.
Energy efficiency for transportation

Analysis and optimisation of energy consumption in the transport sector, efficiency improvements in light-duty vehicles, heavy-duty vehicles, non-road vehicles, public transport systems, engine-fuel optimisation, use of alternative fuels (liquid and gaseous, other than hydrogen), fuel additives, diesel engines, Stirling motors, electric cars, hybrid cars, includes air emission reduction, other.
Other energy efficiency

Waste heat utilisation (heat maps, process integration, total energy systems, low temperature thermodynamic cycles), district heating, heat pump development, reduction of energy consumption in the agricultural sector.

7. Other energy-related technologies

Carbon capture, transportation and storage for fossil fuel production and processing, electric power generation, industry in end-use sector, energy systems analysis, all other energy-related technologies.

Carbon capture, transportation and storage related to fossil fuel production and processing
Carbon capture, transportation and storage related to electric power production
Carbon capture, transportation and storage related to industry in end use sector

Include: Industry in the end use sector, such as steel production, manufacturing, etc.
Exclude Fossil fuel production and processing and electric power production
Energy system analysis System analysis related to energy R&D not covered elsewhere, sociological, economical and environmental impact of energy which are not specifically related to one technology area listed in the sections above.
All other energy-related technologies

Energy technology information dissemination, studies not related to a specific technology area listed above.

Integrated Business Statistics Program (IBSP)

Reporting Guide

This guide is designed to assist you as you complete the 2014 Annual Survey of Research and Development in Canadian Industry – Industrial Non-profit Organizations. If you need more information, please call the Statistics Canada Help Line at the number below.

Help Line: 1-800-972-9692

Your answers are confidential.

Statistics Canada will use information from this survey for statistical purposes.

NOTE:

If this organization performs in-house research and development (R&D) and outsources R&D, complete all questions.
If this organization performs in-house research and development (R&D) and does not outsource R&D, complete question 1-6, 9-20.
If this organization outsources research and development (R&D) and does not perform in-house R&D, complete questions 1-4,6-8, 13, 17-20.
If this organization does not perform in-house research and development (R&D) and does not outsource R&D, complete questions 1-4, 6, 13, 17-18 and 20.

For this survey

‘In-house R&D’ refers to
Expenditures within Canada for R&D performed within this organization by:

employees (permanent, temporary or casual)
self-employed individuals or contractors who are working on-site on this organization's R&D projects

’Outsourced R&D’ refers to
Payments made within or outside Canada to other organizations, companies or individuals to fund R&D performance:

contracts
grants
fellowships

Reporting period information

Here are some examples of common fiscal periods that fall within the targeted dates:

May 1, 2013 to April 30, 2014
June 1, 2013 to June 30, 2014
August 1, 2013 to July 31, 2014
October 1, 2013 to September 30, 2014
December 1, 2013 to November 30, 2014
January 1, 2014 to December 31, 2014
February 1, 2014 to January 31, 2015
March 1, 2014 to February 28, 2015
April 1, 2014 to March 31, 2015

Here are other examples of fiscal periods that fall within the required dates:

September 18, 2013 to September 15, 2014 (e.g., floating year-end)
June 1, 2014 to December 31, 2014 (e.g., a newly opened business)

Definitions and Concepts

Research is original investigation undertaken on a systematic basis to gain new knowledge.

Activities included and excluded from R&D

Inclusions

Prototypes

Contracts

Include all contracts which require R&D. For contracts which include other work, report only the R&D costs.

Research work in the social sciences

Include if projects are employing new or significantly different modelling techniques or developing new formulae, analyzing data not previously available or applying new research techniques.

Exclusions

Routine analysis in the social sciences including policy-related studies, management studies and efficiency studies

Consumer surveys, advertising, market research

Exclude projects of a routine nature, with established methodologies intended for commercialization of the results of R&D are excluded.

Routine quality control and testing

Exclude projects of a routine nature, with established methodologies not intended to create new knowledge are not R&D and are excluded even if carried out by personnel normally engaged in R&D.

Pre-production activities such as demonstration of commercial viability, tooling up, trial production, trouble shooting

Exclude although R&D may be required as a result of these steps, these activities are excluded from R&D.

Prospecting, exploratory drilling, development of mines, oil or gas wells

Include only for R&D projects concerned with new equipment or techniques in these activities, such as in-situ and tertiary recovery research.

Engineering

Exclude engineering unless it is in direct support of R&D.

Design and drawing

Exclude design and drawing unless it is in direct support of R&D.

Patent and license work

Exclude all administrative and legal work connected with patents and licenses.

Cosmetic modifications or style changes to existing products

Exclude where no significant technical improvement or modification to the existing products.

General purpose or routine data collection

Exclude projects of a routine nature, with established methodologies intended for on-going monitoring of an activity.

Routine computer programming, systems maintenance or software application

Exclude projects of a routine nature, with established methodologies intended to support on-going operations.

Routine mathematical or statistical analysis or operations analysis

Exclude projects of a routine nature, with established methodologies intended for on-going monitoring of an activity.

Activities associated with standards compliance

Exclude projects of a routine nature, with established methodologies intended to support standards compliance.

Specialized routine medical care such as routine pathology services

Exclude projects of a routine nature, with established methodologies intended for on-going monitoring of an activity.

In-house R&D expenditures within Canada (Q5)

In-house research and development expenditures are composed of current in-house research and development (R&D) expenditures and capital in-house R&D expenditures.

Current in house R&D expenditures

Wages and salaries of permanent, temporary and casual R&D employees Include: fringe benefits

Fringe benefits of employees engaged in R&D activities. Fringe benefits include bonus payments, holiday or vacation pay, pension fund contributions, other social security payments, payroll taxes, etc.
Services to support R&D

Include: services of self-employed individuals or contractors who are working on-site on this organizations R&D projects.
Exclude: contracted out or granted expenditures to other organizations to perform R&D.

Payments to on-site R&D consultants and contractors working under the direct control of your organization; indirect services purchased to support in-house R&D such as security, storage, repair, maintenance and use of buildings and equipment; computer services, software licensing fees and dissemination of R&D findings
R&D materials

Utilities: water, fuel, gas and electricity; materials for creation of prototypes, reference materials (books, journals, etc.); subscriptions to libraries and data bases, memberships to scientific societies, etc.; cost of outsourced small R&D prototypes or R&D models; materials for laboratories (chemicals, animals, etc.); all other R&D-related materials
All other current costs

Administrative and overhead costs (e.g., office, post and telecommunications, internet, insurance), prorated if necessary to allow for non-R&D activities within the company or organization

Capital in-house R&D expenditures

Software
Exclude: capital depreciationApplications and systems software (original, custom and off-the-shelf software), supporting documentation and other software-related acquisitions
Land
Exclude: capital depreciationLand acquired for R&D including testing grounds, sites for laboratories and pilot plants
Buildings and structures
Exclude: capital depreciationBuildings and structures (constructed or purchased) for research and development (R&D) activities or that have undergone major leasehold improvements (modifications, renovations and repairs) for R&D activities
Equipment, machinery and all other
Exclude: capital depreciation

Major equipment, machinery and instruments, including embedded software, acquired for research and development (R&D) activities

Outsourced (contracted out or granted) R&D expenditures (Q7)

Payments made through contracts, grants and fellowships to another organization, company or individual to purchase R&D activities.

Include: contracted out expenditures for research and development (R&D), funding or grants provided to other organizations to perform R&D.

Exclude: expenditures for services of self-employed individuals or contractors who are working on-site on this organization’s R&D.

Companies

All incorporated for profit businesses and government business enterprises providing products in the market at market rates.
Private non-profit organizations

Voluntary health organizations, private philanthropic foundations, associations and societies and research institutes; they are not for profit organizations that serve the public interest by supporting activities related to public welfare (such as health, education, the environment).
Industrial research institutes or associations

Non-profit organizations that serve the business enterprise sector frequently consisting of their membership. Industrial non-profit organizations include non-profit industrial research institutes.
Hospitals
Universities
Federal government departments and agencies

All federal government ministries, departments and agencies. It excludes federal government business enterprises providing products in the market.
Provincial government departments and agencies

All provincial government ministries, departments and agencies. It excludes provincial government business enterprises providing products in the market.
Provincial research organizations

Organizations created under provincial or territorial law which conduct or facilitate research on behalf of the province or territory.
Other

Individuals, non-university educational institutions, foreign governments

Sources of funds for in-house R&D expenditures in 2014 (Q10)

Include: Canadian and foreign sources

Exclude: payments for outsourced (contracted out or granted) R&D which should be reported in question 7 on Outsourced (contracted out or granted) R&D; capital depreciation.

Funds from this organization

Amount contributed by this unit to R&D performed within Canada (include interest payments and other income, land, buildings and structures, equipment and machinery (capital expenditures) purchased for R&D).
Funds from member companies or affiliates

Amount received from member organizations and affiliated organizations used to perform R&D within Canada (include annual fees and sustaining grants, land, buildings and structures, equipment and machinery (capital expenditures) purchased for R&D).
Federal grants

Include: R&D grants or R&D portion only of other grants

Funds from the federal government in support of R&D activities not connected to a specific contractual deliverable.
Federal contracts

Include: R&D contracts or R&D portion only of other contracts

Funds from the federal government in support of R&D activities connected to a specific contractual deliverable.
R&D contract work for other companies

Funds received from other companies to perform R&D on their behalf.
Other sources

Funds received from all other sources not previously classified.

In-house R&D expenditures by fields of research and development in 2014 (Q11)

Exclude: payments for outsourced (contracted out or granted) R&D which should be reported in question 7 on Outsourced (contracted out or granted) R&D; capital depreciation.

Natural and formal sciences

Mathematics, physical sciences, chemical sciences, earth and related environmental sciences, biological sciences, other natural sciences.

Exclude: computer sciences, information sciences and bioinformatics (to be reported at line r, s and t)

Mathematics

Pure mathematics, applied mathematics, statistics and probability.
Physical Sciences

Atomic, molecular and chemical physics, interaction with radiation, magnetic resonances, condensed matter physics, solid state physics and superconductivity, particles and fields physics, nuclear physics, fluids and plasma physics (including surface physics), optics (including laser optics and quantum optics), acoustics, astronomy (including astrophysics, space science).
Chemical sciences

Organic chemistry, inorganic and nuclear chemistry, physical chemistry, polymer science and plastics, electrochemistry (dry cells, batteries, fuel cells, metal corrosion, electrolysis), colloid chemistry, analytical chemistry.
Earth and related environmental sciences

Geosciences, geophysics, mineralogy and palaeontology, geochemistry and geophysics, physical geography, geology and volcanology, environmental sciences, meteorology, atmospheric sciences and climatic research, oceanography, hydrology and water resources.
Biological sciences

Cell biology, microbiology and virology, biochemistry, molecular biology and biochemical research, mycology, biophysics, genetics and heredity (medical genetics under medical biotechnology), reproductive biology (medical aspects under medical biotechnology), developmental biology, plant sciences and botany, zoology, ornithology, entomology and behavioural sciences biology, marine biology, freshwater biology and limnology, ecology and biodiversity conservation, biology (theoretical, thermal, cryobiology, biological rhythm), evolutionary biology.
Other natural sciences

Engineering and Technology

Civil engineering

Civil engineering, architecture engineering, municipal and structural engineering, transport engineering.
Electrical engineering, electronic engineering and communications technology

Electrical and electronic engineering, robotics and automatic control, micro-electronics, semiconductors, automation and control systems, communication engineering and systems, telecommunications, computer hardware and architecture.
Mechanical engineering

Mechanical engineering, Applied mechanics, Thermodynamics, Aerospace engineering, Nuclear-related engineering (nuclear physics under Physical sciences), Acoustical engineering, Reliability analysis and non-destructive testing, Automotive and transportation engineering and manufacturing, Tooling, machinery and equipment engineering and manufacturing, Heating, ventilation and air conditioning engineering and manufacturing.
Chemical engineering

Chemical engineering (plants, products), chemical process engineering.
Materials engineering

Materials engineering and metallurgy, ceramics, coating and films (including packaging and printing), plastics, rubber and composites (including laminates and reinforced plastics), paper and wood and textiles, construction materials (organic and inorganic).
Medical Engineering

Medical and biomedical engineering, medical laboratory technology (excluding biomaterials which should be reported under industrial biotechnology).
Environmental engineering

Environmental and geological engineering, petroleum engineering (fuel, oils), energy and fuels, remote sensing, mining and mineral processing, marine engineering, sea vessels and ocean engineering.
Environmental biotechnology

Environmental biotechnology, bioremediation, diagnostic biotechnologies in environmental management (DNA chips and bio-sensing devices).
Industrial biotechnology

Industrial biotechnology, bioprocessing technologies, biocatalysis and fermentation bioproducts (products that are manufactured using biological material as feedstock), biomaterials (bioplastics, biofuels, bioderived bulk and fine chemicals, bio-derived materials).
Nanotechnology

Nano-materials (production and properties), nano-processes (applications on nano-scale).
Other engineering and technologies

Food and beverages, oenology, other engineering and technologies.

Software-related sciences and technology

Software engineering and technology, computer sciences, information technology and bioinformatics.

Software engineering and technology

Computer software engineering, computer software technology, and other related computer software engineering and technologies.
Computer sciences

Computer science, artificial intelligence, cryptography, and other related computer sciences
Information technology and bioinformatics

Information technology, informatics, bioinformatics, biomathematics, and other related information technologies.

Medical and health sciences

Basic medicine, clinical medicine, health sciences, medical biotechnology, other medical sciences.

Basic medicine

Anatomy and morphology (plant science under biological science), human genetics, immunology, neurosciences, pharmacology and pharmacy and medicinal chemistry, toxicology, physiology and cytology, pathology.
Clinical medicine

Andrology, obstetrics and gynaecology, paediatrics, cardiac and cardiovascular systems, haematology, anaesthesiology, orthopaedics, radiology and nuclear medicine, dentistry, oral surgery and medicine, dermatology, venereal diseases and allergy, rheumatology, endocrinology and metabolism and gastroenterology, urology and nephrology, and oncology.
Health sciences

Health care sciences and nursing, nutrition and dietetics, parasitology, infectious diseases and epidemiology, occupational health.
Medical biotechnology

Health-related biotechnology, technologies involving the manipulation of cells, tissues, organs or the whole organism, technologies involving identifying the functioning of DNA, proteins and enzymes, pharmacogenomics, gene-based therapeutics, biomaterials (related to medical implants, devices, sensors).
Other medical sciences

Forensic science, other medical sciences.

Agricultural Sciences

Agriculture, forestry and fisheries sciences, animal and dairy sciences, veterinary sciences, agricultural biotechnology, other agricultural sciences.

Agriculture, forestry and fisheries sciences

Agriculture, forestry, fisheries and aquaculture, soil science, horticulture, viticulture, agronomy, plant breeding and plant protection.
Animal and dairy sciences

Animal and dairy science, animal husbandry.
Veterinary sciences

Veterinary science (all).
Agricultural biotechnology

Agricultural biotechnology and food biotechnology, genetically modified (GM) organism technology and livestock cloning, diagnostics (DNA chips and biosensing devices), biomass feedstock production technologies and biopharming.
Other agricultural sciences

Social sciences and humanities

Psychology, educational sciences, economics and business, other social sciences, humanities.

Psychology

Cognitive psychology and psycholinguistics, experimental psychology, psychometrics and quantitative psychology, and other fields of psychology.
Educational sciences

Education, training and other related educational sciences.
Economics and business

Micro-economics, macro-economics, econometrics, labour economics, financial economics and all other related fields of economics and business.
Other social sciences

Anthropology (social and cultural) and ethnology, demography, geography (human, economic and social), planning (town, city and country), management, organisation and methods (excluding market research unless new methods/techniques are developed), law, linguistics, political sciences, sociology, miscellaneous social sciences and interdisciplinary, and methodological and historical science and technology activities relating to subjects in this group.
Humanities

History (history, prehistory and history, together with auxiliary historical disciplines such as archaeology, numismatics, palaeography, genealogy, etc.), languages and literature (ancient and modern), other humanities (philosophy (including the history of science and technology)), arts (history of art, art criticism, painting, sculpture, musicology, dramatic art excluding artistic “research” of any kind), religion, theology, other fields and subjects pertaining to the humanities, and methodological, historical and other science and technology activities relating to the subjects in this group.

In house R&D expenditures by nature of R&D activity in 2014 (Q12)

R&D is performed in the natural sciences, engineering, social sciences and humanities. There are three types of R&D activities: basic research, applied research and experimental development.

Basic research

Experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observable facts, without any particular application or use in view.
Applied research

Also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily towards a specific practical aim or objective.
Experimental development

Systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products or devices, to installing new processes, systems and services, or to improving substantially those already produced or installed.

In-house R&D personnel in 2014 (Q14)

Include:

permanent, temporary and casual R&D employees
independent on-site R&D consultants and contractors working in your organization’s offices, laboratories, or other facilities
employees engaged in R&D-related support activities.

Researchers and research managers

Scientists, social scientists, engineers and researchers
Include: software developers and programmers

Create new knowledge, products, processes, methods and systems. They include software developers and programmers. They may be certified by provincial educational authorities, provincial or national scientific or engineering associations (Include software developers and programmers).
Senior research managers

Plan or manage R&D projects and programs. They may be certified by provincial educational authorities, provincial or national scientific or engineering associations.

R&D technical, administrative and support staff

Technicians, technologists and research assistants
Include: software technicians

Assist scientists, engineers and researchers in R&D activities, e.g., laboratory technicians, chemical technicians, draftspersons, research assistants and software technicians. They may be certified by provincial educational authorities, provincial or national scientific or engineering associations.
Other R&D technical, administrative and support staff

Include administrative assistants, accountants, bookkeepers, equipment storage, maintenance managers and facilities operators engaged in administration, clerical or other activities in support of R&D activities. Also included are machinists and electricians engaged in the construction of prototypes.

Other R&D occupations

On-site R&D consultants and contractors

Are individuals hired 1) to perform project-based work or to provide goods at a fixed or ascertained price or within a certain time or 2) to provide advice or services in a specialized field for a fee and, in both cases, work at the location specified and controlled by the contracting organization.

Full-time equivalent (FTE)

Technology or technical assistant payments in 2014 (Q17 and Q18)

Definitions (equivalent to the Canadian Intellectual Property Office http://www.ic.gc.ca/eic/site/cipointernet-internetopic.nsf/eng/wr00143.html)

Patents

Government grant giving the right to exclude others from making, using or selling an invention.
Copyright

Provides protection for literary, artistic, dramatic or musical works (including computer programs), and three other subject matter known as: performance, sound recording, and communication signal.
Trademark

Word, symbol or design (or any combination of these features) used to distinguish the wares and services of one person or organization from those of others in the marketplace.
Industrial design

Visual features of shape, configuration, pattern or ornament (or any combination of these features), applied to a finished article of manufacture.
Integrated circuit topography

Three-dimensional arrangement of the electronic circuits in integrated circuit products or layout designs.
Original software

Consist of computer programs and descriptive materials for both systems and applications. Original software can be created in-house or outsourced and includes packaged software with customization.
Packaged or off-the-shelf software

Purchased for use by your organization and excludes customized software.
Databases

Consist of files of data organized to permit effective access and use of the data.
Other

Technical assistance, industrial processes and know-how.

Energy-related R&D by area of technology (Q19)

1. Fossil Fuels

Crude oils and natural gas exploration

Development of advanced exploration methods (geophysical, geochemical, seismic, magnetic) for on-shore and off-shore prospecting.
Crude oil and natural gas production (including enhanced recovery) and storage

On-shore and off-shore deep drilling equipment and techniques for conventional oil and gas, secondary and tertiary recovery of oil and gas, hydro fracturing techniques, processing and cleaning of raw product, storage on remote platforms (e.g., Arctic, off-shore), safety aspects of offshore platforms.
Oil sands and heavy crude oils surface and sub-surface production and separation of the bitumen, tailings management

Surface and in-situ production (e.g., SAGD); tailings management.
Refining, processing and upgrading

Processing of natural gas to pipeline specifications, and refining of conventional crude oils to refined petroleum products (RPPs), and the upgrading of bitumen and heavy oils either to synthetic crude oil or to RPPs. Upgrading may be done at an oil sands plant, regional merchant upgraders or integrated into a refinery producing RPPs.
Coal production, separation and processing

Coal, lignite and peat exploration, deposit evaluation techniques, mining techniques, separation techniques, coking and blending, other processing such as coal to liquids, underground (in-situ) gasification.
Transportation of fossil fuels

Transport of gaseous, liquid and solid hydrocarbons via pipelines (land and submarine) and their network evaluation; safety aspects of LNG transport and storage.

2. Renewable energy resources

Solar photovoltaics (PV)

Solar cell development, PV-module development, PV-inverter development, building-integrated PV-modules, PV-system development, other.
Solar thermal-power and high-temperature applications

Solar chemistry, concentrating collector development, solar thermal power plants, high-temperature applications for heat and power.
Solar heating and cooling

Daylighting, passive and active solar heating and cooling, collector development, hot water preparation, combined-space heating, solar architecture, solar drying, solar-assisted ventilation, swimming pool heating, low-temperature process heating, other.
Wind energy

Technology development, such as blades, turbines, converters structures, system integration, other.
Bio-energy – Biomass production/supply and transport

Improvement of energy crops, research on bio-energy production potential and associated land-use effects, supply and transport of bio-solids, bio-liquids, biogas and bio-derived energy products (e.g., ethanol, biodiesel), compacting and baling, other.
Bio-energy – Biomass conversion to fuels

Conventional bio-fuels, cellulosic-derived alcohols, biomass gas-to-liquids, other energy-related products and by-products.
Bio-energy – Biomass conversion to heat and electricity

Bio-based heat, electricity and combined heat and power (CHP), exclude multi-firing with fossil fuels.
Other bio-energy

Recycling and the use of municipal, industrial and agricultural waste as energy not covered elsewhere.
Small-Hydro – (less than 10 MW)

Plants with capacity below 10 MW.
Large-Hydro – (greater than or equal to 10 MW)

Plants with capacity of 10 MW and above.
Other renewable energy

Hot dry rock, hydro-thermal, geothermal heat applications (including agriculture), tidal power, wave energy, ocean current power, ocean thermal power, other.

3. Nuclear fission and fusion

Materials exploration, mining and preparation, tailings management, nuclear reactors, other fission, fusion.

Nuclear materials exploration, mining and preparation, tailings management

Development of advanced exploration methods (geophysical, geochemical) for prospecting, ore surface and in-situ production, uranium and thorium extraction and conversion, enrichment, handling of tailings and remediation.
Nuclear reactors

Nuclear reactors of all types and related system components.
Other fission

Nuclear safety, environmental protection (emission reduction or avoidance), radiation protection and decommissioning of power plants and related nuclear fuel cycle installations, nuclear waste treatment, disposal and storage, fissile material recycling, fissile materials control, transport of radioactive materials.
Fusion

All types (e.g., magnetic confinement, laser applications).

4. Electric Power

Generation in utility sector, combined heat and power in industry and in buildings, electricity transmission, distribution and storage of electricity.

Electric power generation in utility sector

Conventional and non-conventional technology (e.g., pulverised coal, fluidised bed, gasification-combined cycle, supercritical), re-powering, retrofitting, life extensions and upgrading of power plants, generators and components, super-conductivity, magneto hydrodynamic, dry cooling towers, co-firing (e.g., with biomass), air and thermal pollution reduction or avoidance, flue gas cleanup (excluding CO2 removal), CHP (combined heat and power) not covered elsewhere.
Electric power - combined heat and power in industry, buildings

Industrial applications, small scale applications for buildings.
Electricity transmission, distribution and storage

Solid state power electronics, load management and control systems, network problems, super-conducting cables, AC and DC high voltage cables, HVDC transmission, other transmission and distribution related to integrating distributed and intermittent generating sources into networks, all storage (e.g., batteries, hydro reservoirs, fly wheels), other.

Hydrogen and fuel cells

Hydrogen production for process applications, hydrogen production for transportation applications, hydrogen transport and storage, other hydrogen, fuel cells, both stationary and mobile.

Hydrogen production for process applications
Hydrogen production for transportation applications
Hydrogen transport and storage
Other hydrogen
End uses (e.g., combustion), other infrastructure and systems R&D (refuelling stations).
Stationary fuel cells
Electricity generation, other stationary end-use.
Mobile fuel cellsPortable applications.

6. Energy efficiency

Industry, residential and commercial, transportation, other energy efficiency.

Energy efficiency applications for industry

Reduction of energy consumption through improved use of energy and/or reduction or avoidance of air and other emissions related to the use of energy in industrial systems and processes (excluding bio-energy-related) through the development of new techniques, new processes and new equipment, other.
Energy efficiency for residential, institutional and commercial sectors

Space heating and cooling, ventilation and lighting control systems other than solar technologies, low energy housing design and performance other than solar technologies, new insulation and building materials, thermal performance of buildings, domestic appliances, other.
Energy efficiency for transportation

Analysis and optimisation of energy consumption in the transport sector, efficiency improvements in light-duty vehicles, heavy-duty vehicles, non-road vehicles, public transport systems, engine-fuel optimisation, use of alternative fuels (liquid and gaseous, other than hydrogen), fuel additives, diesel engines, Stirling motors, electric cars, hybrid cars, includes air emission reduction, other.
Other energy efficiency

Waste heat utilisation (heat maps, process integration, total energy systems, low temperature thermodynamic cycles), district heating, heat pump development, reduction of energy consumption in the agricultural sector.

7. Other energy-related technologies

Carbon capture, transportation and storage related to fossil fuel production and processing
Carbon capture, transportation and storage related to electric power production
Carbon capture, transportation and storage related to industry in end use sector
Include: Industry in the end use sector, such as steel production, manufacturing, etc.

Exclude Fossil fuel production and processing and electric power production
Energy system analysis

System analysis related to energy R&D not covered elsewhere, sociological, economical and environmental impact of energy which are not specifically related to one technology area listed in the sections above.
All other energy-related technologies

Energy technology information dissemination, studies not related to a specific technology area listed above.

Audit of Project Management

May 26, 2015
Project Number: 80590-84

Executive summary
Introduction
- Background
- Audit objectives
- Scope
- Approach and methodology
- Authority
Findings, recommendations and management response
- Governance and roles and responsibilities
- Risk and issue management and project interdependencies
- Project baselines including costs and efficiencies estimates
- Project monitoring and outcomes
Appendices
- Appendix A: Audit criteria
- Appendix B: Acronyms and initialisms

Executive summary

Statistics Canada has a Departmental Project Management Framework (DPMF) in place that identifies the agency's standard project management processes, tools and templates. The DPMF was fully implemented in March 2012 and is mandatory for all projects valued at or above $150,000. It is also highly recommended for projects valued at less than $150,000. This is a key guidance tool for project management within the agency, as it provides context on the project life cycle and the project management deliverables required at these stages.

The Departmental Project Management Office (DPMO) is responsible for supporting the agency's project and portfolio management functions by serving as a resource for project managers and other stakeholders; offering standardized processes, practices and tools across the agency; and supporting professional development for project managers.

The objectives of the audit were to provide the Chief Statistician and the Departmental Audit Committee with assurance that

the agency has an effective and adequate project management framework to ensure a systematic approach to managing its projects and associated interdependencies in compliance with the Treasury Board Policy on the Management of Projects
projects are managed in compliance with the DPMF and its related tools, and are monitored against established milestones and expected results.

Key findings

Statistics Canada has a history of successfully delivering projects, and it is assessing ways of improving its business practices. The DPMF, developed by the DPMO, communicates expectations as well as the agency's strategic direction for project management. It outlines the authorities, roles and responsibilities of the DPMO, of key roles within project management teams, of oversight committees and of other key stakeholders.

The DPMO supports the agency's project and portfolio management functions, as well as its governance committees, by serving as a resource; offering standardized processes, practices and tools; and conducting project portfolio management. The audit revealed inconsistencies in governance practices. Combined with project managers who have limited experience or expertise in project management, this results in the DPMO spending a significant amount of time following up with project managers on missing deliverables rather than providing strategic project management support to the agency.

The DPMF recognizes the importance of risk and issue management as part of strong project management, but it currently provides limited guidance regarding the processes for identifying, assessing, communicating and monitoring project risks and interdependencies within the project life cycle. Additionally, the roles and responsibilities of key project team members and the oversight committees with respect to risk management and monitoring project interdependencies have not been clearly defined within the DPMF.

The DPMO has developed and introduced the Change, Issue, Risk Management Tool (CIRMT) to help document and monitor project risks and issues, but its use has not been made mandatory for all projects. Further, at the time of the audit, a project interdependency tool was being piloted only for Corporate Business Architecture projects.

The DPMF currently outlines the requirement for documenting project baselines, including estimated costs and planned efficiencies, at various points in the project life cycle. It also provides templates to be completed by the project team. The assumptions made in determining the cost and efficiency baselines are not consistently being formally documented.

The audit team was unable to assess the reliability and completeness of these baselines or to determine whether all project stakeholders were consulted in determining or reviewing the cost and efficiency assumptions. Amendments made to a project's scope, schedule or cost and efficiency baselines are not consistently being formally documented as expected, which affects the transparency of project amendments—i.e., the number of amendments made to a project and its overall final cost or expected efficiencies.

As outlined in the DPMF, the monthly executive project dashboards are to be prepared and reviewed according to the established governance structure for each project. As the audit team identified weaknesses in the reliability and completeness of the reporting being made through a sample of dashboards, the DPMO is revising its Executive Project Dashboard template to make a project's status more transparent.

Limited evidence is maintained to demonstrate that project oversight committees are consistently reviewing and approving the monthly executive project dashboards in a timely manner.

Expected business outcomes are being documented in the business case; however, in some instances, these outcomes are not sufficiently developed to provide an understanding of the benefits of the project to the agency, nor are they measurable. Without clearly defined measurable outcomes, it is difficult to assess whether a project has achieved what it was expected to.

From the sample of projects tested that had reached completion/close-out, the audit team noted that project managers are not consistently completing a full assessment of the outcomes against the original or revised targets.

Conclusion

The agency, through the DPMO, has developed and implemented a project management framework aligned with the requirements outlined in the Treasury Board Policy on the Management of Projects and encourages a systematic approach to the management of its projects. The establishment of this framework was essential given the significant and complex transformational projects being carried out within the agency.

Greater clarity of roles and responsibilities, guidance for all stakeholders, and mandatory use of CIRMT in the areas of risk management are required to strengthen the ability of governance bodies and key participants to effectively identify, assess, communicate and monitor project risks and issues and interdependencies. These actions will ensure that projects are managed efficiently and effectively and will lead to greater compliance with the DPMF. As well, they will help ensure that project baselines, including cost and efficiency estimates, are established with sound stewardship; that the status of projects is more effectively monitored; and that reporting against expected results is improved.

Conformance with professional standards

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which include the Institute of Internal Auditors International Standards for the Professional Practice of Internal Auditing.

Adequate and appropriate audit procedures were followed and evidence gathered to support the accuracy of the findings and conclusions in this report and to provide an audit level of assurance. The findings and conclusions are based on a comparison of conditions, as they existed at the time, against pre-established audit criteria. The findings and conclusions are applicable to the entity examined for the scope and time period covered by the audit.

Patrice Prud'homme
Chief Audit Executive

Introduction

Background

As Canada's national statistical office, Statistics Canada has a mandate to serve the statistical needs of all levels of government, Canadian institutions and Canadians. Providing high-quality data that support core economic, environmental and social statistics programs is key to determining how Statistics Canada allocates its funding. In its Departmental Investment Plan, Statistics Canada presented an investment of $512 million on projects (all from within existing reference levels) over the years 2011/2012 to 2015/2016 to meet its operational requirements. The largest project was the 2011 Census of Population and the 2011 National Household Survey, estimated at $360 million.

In April 2010, Statistics Canada created the Departmental Project Management Office (DPMO) to provide leadership, training and support in developing common project management processes and tools to be used by all projects to improve the timely delivery of projects within cost, within scope and to required quality standards.

In April 2011, the DPMO launched the Departmental Project Management Framework (DPMF) and implemented the Treasury Board Policy on the Management of Projects. The framework is a set of standard project management processes, templates and tools used throughout a project's life cycle to initiate, plan, execute, control and close a project.

All of Statistics Canada's projects with total costs of $150,000 and above are required to comply with the DPMF, although it is also recommended for projects valued at less than $150,000. Statistics Canada has based its DPMF on international industry standards, including the Project Management Body of Knowledge Guide, published by the Project Management Institute.

Over the last few years, over 1,000 participants have attended training and information sessions to gradually raise awareness of the DPMF's use and importance.

Statistics Canada's project approval processes are integrated through the DPMF and are aligned with its Integrated Strategic Planning Process (ISPP). The ISPP includes project management governance, performance monitoring and risk management. The ISPP follows an annual calendar of events with clearly identified governance, elements, activities and expected outputs throughout each of the planning phases.

There are six stages within the DPMF: (1) idea generation; (2) project assessment; (3) project initiation; (4) project planning; (5) project execution; and (6) project close-out. When the project is completed, a post-launch review is conducted. The project management deliverables and requirements are defined in the DPMF depending on the Project Complexity and Risk Assessment level of the specific project and its current stage. At each of these stages, key documents are expected to be developed and submitted to the relevant governance bodies for review and gating approval.

Managing projects effectively is key to ensuring that Statistics Canada is providing value-for-money and demonstrating sound stewardship in program delivery. Specifically, a comprehensive approach to managing projects that is integrated across the department and is appropriate to the level of project risk and complexity will enhance the likelihood of realizing project outcomes.

Audit objectives

The objectives of the audit were to assure the Chief Statistician and the Departmental Audit Committee that

the agency has an effective and adequate project management framework to ensure a systematic approach to managing its projects and associated interdependencies in compliance with the Treasury Board Policy on the Management of Projects
projects are managed in compliance with the DPMF and its related tools, and are monitored against established milestones and expected results.

Scope

The scope of this audit included examining the adequacy and effectiveness of the governance and monitoring practices that support clear accountability, oversight, scrutiny and challenge functions. It also included examining the effectiveness of risk management processes for identifying the key risks facing projects, including emerging risks, as well as for developing and monitoring risk management strategies.

Lastly, the scope included examining the operational processes and controls that enable the consistent application of a common project management framework as well as the tools, training and information management practices that support informed project management decisions.

The scope of the audit included assessing project management activities from April 1, 2011, to March 31, 2014.

Approach and methodology

The audit work consisted of examining documents; reviewing relevant procedures, guidelines and frameworks related to project management processes; and interviewing key senior management representatives, a sample of project managers, representatives of the DPMO and a sample of oversight committee members. Additionally, the field work consisted of reviewing a sample of project files to assess the consistency of project management activities and compliance with the DPMF.

The selection of a sample of 12 projects was based on a judgmental sampling approach that considered the size, number of adjustments and complexity of each project. The following table presents the breakdown of the sample:

Projects sampled for the audit of project management, by project category
	Population of projects ^{Table 1 - Footnote 1}	Sample size
Footnotes Footnote 1 For CBA projects, this includes all projects that commenced on or after April 1, 2011 (date of required compliance with the DPMF); for all other projects, it is the number of projects that commenced on or after April 1, 2012 (date of required compliance with the DPMF). Return to footnote 1 referrer Footnote 2 Significant non-CBA projects are those which are complex, and have interdependencies and an impact on the agency. Return to footnote 2 referrer Footnote 3 Non significant non CBA projects are those which do not have interdependencies. Return to footnote 3 referrer
Project category	94	12
Corporate Business Architecture (CBA)	30	5
Significant non-CBA ^{Table 1 - Footnote 2}	17	3
Non-significant non-CBA ^{Table 1 - Footnote 3}	47	4

This audit was conducted in accordance with the Standards for the Professional Practice of Internal Auditing of the Institute of Internal Auditors and the Treasury Board Policy on Internal Audit.

Authority

The audit was conducted under the authority of the approved Statistics Canada integrated Risk-based Audit and Evaluation Plan 2014–2019.

Findings, recommendations and management response

Governance and roles and responsibilities

As Statistics Canada continues to carry out a significant number of large-scale projects as part of its business architecture transformation, it is critical that project management expectations throughout the agency be communicated effectively and consistently. The Departmental Project Management Framework (DPMF), developed by the Departmental Project Management Office (DPMO), is the mechanism used to communicate these expectations as well as the agency's strategic direction for project management. The DPMF further outlines the authorities, roles and responsibilities of the DPMO, of key roles within project management teams, of oversight committees and of other key stakeholders.

The main responsibilities of the DPMO, as defined in the DPMF, are to support the agency's project and portfolio management functions, as well as governance committees, by serving as a resource; offering standardized processes, practices and tools; and conducting project portfolio management. As a result of inconsistencies in the governance practices for past and ongoing projects and given that many project managers have limited experience or expertise in project management, the DPMO continues to spend a significant amount of its time following up with project managers on missing project management deliverables and saving them in the DPMO's repository rather than providing strategic project management support to the agency.

Statistics Canada's strategic direction for project management should be documented and adequately communicated by establishing a framework that clearly defines authorities, responsibilities and accountabilities with respect to initiating, planning, executing and closing-out projects. Additionally, the DPMO should provide employees with the necessary tools, resources, information and training to discharge their responsibilities and achieve expected results.

Strategic direction for project management as well as key roles and responsibilities have been clearly defined and documented in the DPMF.

Strategic direction for project management within the agency has been formally documented and communicated through the DPMF guidelines. Not only does the DPMF define project management for the agency, it has also established a set of standardized project management processes, templates and tools to guide project management activities across the agency. Further, the DPMF has defined the project management deliverables required for each stage of a project: (1) idea generation; (2) project assessment; (3) project initiation; (4) project planning; (5) project execution; and (6) project close-out.

The audit noted that the DPMF has clearly defined the roles, responsibilities and authorities of the project team, including those of the project manager, executives and business sponsors. It has also defined the responsibilities of key oversight committees, including the project steering committees, the Corporate Business Architecture (CBA) Management Committee (CBAMC), the Systems Architecture Review Board and the field planning boards (FPBs).

The audit team did, however, identify opportunities to clarify the roles, responsibilities and authorities of oversight committees, project managers and other stakeholders in the areas of risk management; identifying, assessing and communicating project interdependencies; and project costing and forecasting. These opportunities are presented in more detail in the sections "Risk and issue management and project interdependencies" and "Project baselines including costs and efficiencies estimates" in this report.

A complete list of ongoing projects, including their total approved budget and expenditures, was not maintained by the DPMO.

The audit noted that the DPMO recently created a database to track all ongoing initiatives and projects across Statistics Canada. Given the limitations of the Budget and Revenue Management System, the organization's budget system, and the Financial Reporting System, financial information can be generated for no more than a single fiscal year at a time.

Therefore, to obtain the total approved budget and incurred expenditures for a project, multiple reports need to be generated for the analysis. At the time of the audit, the total approved budget and expenditures for each project were not being maintained by the DPMO.

Project governance and oversight is consistent for CBA projects; however, practices varied for non-CBA projects.

Although the DPMF has outlined the agency's expectations regarding the level of governance and oversight to be provided for projects according to their level of complexity, the audit team noted inconsistencies in the governance and oversight practices established for non-CBA projects.

Specifically, while the audit team noted that steering committees were created for CBA projects and that FPBs consistently oversaw CBA projects, the level of oversight provided by project steering committees and FPBs varied for non-CBA projects. For example, although the DPMF requires that all projects establish a steering committee, the audit team identified three projects (out of seven projects tested) that were not overseen by a steering committee but rather were overseen solely by the project sponsor.

Additionally, for stages 4 to 6 of the project management life cycle, the audit noted that the roles and responsibilities of the FPBs varied according to the type of project and its field, including inconsistencies in the level of documentation maintained as evidence of decision making and gating for these stages, specifically for non-CBA projects.

The audit noted that the senior official accountable for the DPMF is not part of key management governance committees. Including the assistant chief statistician responsible for this function in those committees would ensure that considerations regarding the DPMF are regularly included in deliberations, provide for a more comprehensive and robust review function and increase the rigour of the project management challenge function.

Without the establishment of appropriate governance and oversight for all projects, including non-CBA projects, there is an increased risk that project management deliverables will not be challenged and approved at the appropriate level, resulting in agency resources being used inefficiently.

Expectations for project managers' experience and certification have not been established.

The DPMF does not currently establish any minimum requirements regarding the level of experience or certification for project managers. Project managers are generally assigned on the basis of availability and their specialized expertise in the project topic rather than their experience or expertise in project management. As such, most project managers interviewed indicated having little to no previous project management experience, and only 1 project manager (out of 12 interviewed) held any related certification. In some cases, these individuals were managing large, complex projects.

Without a minimum level of project management experience (or certification) held by those managing complex projects on behalf of the agency, there is an increased risk that project managers may not efficiently and effectively manage their projects in accordance with the expectations outlined in the DPMF.

Project management training has been developed for key roles, but it has not been made mandatory.

A series of training programs have been developed and made available to project managers and executives. Specifically for project managers is a three-day training session on basic project management skills and a one-day in-house training session on the requirements of the DPMF, the gating process and how to develop the project management deliverables. The DPMO has further developed individual training sessions for key project management tools, such as the Change, Issue, Risk Management Tool (CIRMT) and the Executive Project Dashboard.

The audit noted that the project managers for the projects tested generally had not completed the project management training sessions offered to them, as these are not currently mandatory. Project managers' lack of training, as well as the fact that there is no minimum project management experience or certification requirement for project managers based on the complexity of projects, has resulted in inconsistencies in the level of understanding and use of the DPMF and its related tools. When the sampled projects were tested, this was demonstrated by the inconsistent use of the project management tools by project managers and the level of sophistication of certain project management deliverables.

Additionally, a training course designed to introduce executives to the DPMF and their roles and responsibilities, entitled Departmental Project Management Framework for Executives, was developed in 2012. In addition to other topics, this course covers the following areas: interpreting and approving project dashboards, issues and risk management, why governance is key in project management, the DPMF stages, and the gating process. Because this course was introduced as a pilot, it is not currently being offered to executives and, similar to the training offered to project managers, has not been made mandatory for executives who typically have oversight responsibilities relative to projects.

Recommendation

The assistant chief statistician of Corporate Services (and chief financial officer) should ensure the following:

Governance expectations outlined in the DPMF—including the composition of committees, the role and responsibilities of the DPMO in projects, and the requirement to set up a project steering committee on all projects—are reiterated to senior management and project sponsors to ensure consistency across the agency.
Criteria are established regarding minimum experience or certification in project management according to the size, risk, complexity and type of project established and documented in the DPMF.
The basic project management skills and DPMF training courses are made mandatory for all project managers.
The training course developed for executives on the DPMF is made a permanent course and, as possible, is embedded within the training offered to new executives.

Management response

Management agrees with the recommendation.

Following the time period for this March 2014 audit, additional activities occurred to clarify roles and responsibilities related to project management. In 2014/2015, the roles of the FPBs, the Senior Management Review Board and CBAMC were revised and communicated to the governance committees and the project managers. Further actions will be considered to address the recommendations:

The director of Corporate and Financial Planning Division (CFPD) will update the governance expectations to clarify the composition of governance committees, the role and responsibilities of the DPMO in projects, and the requirement to set up a project steering committee based on size and scope, during the review of the DPMF by March 2016. Following the review, this clarification will be communicated to project managers and executives.
The director of CFPD will ensure that criteria regarding recommended minimums for project management experience or certification are established and documented during the review of the DPMF in 2015/2016.

Deliverables and timeline: Update guidance and communication of governance expectations in the DPMF by March 2016.

Since 2012/2013, over 1,000 participants have attended project management training courses, such as those on the Executive Project Dashboard and CIRMT. Large and complex projects were initially focused on to ensure that the project management tools were adopted consistently. Attention will now move to the other projects. These training sessions were essential in providing guidance to project managers. The following action will be addressed shortly:

The director of CFPD will recommend to the Learning and Development Committee that mandatory training be offered to all project managers and new executives. Pending discussions surrounding the transfer of training to the Canada School of Public Service may affect this action.

Deliverables and timeline: Obtain approval from the Learning and Development Committee to provide mandatory project management training by September 2015.

Risk and issue management and project interdependencies

The Departmental Project Management Framework (DPMF) recognizes the importance of risk and issue management as part of strong project management, but it currently provides limited guidance on the processes for identifying, assessing, communicating and monitoring project risks and interdependencies within the project life cycle. Additionally, the roles and responsibilities of key project team members and oversight committees with respect to managing risks and issues and monitoring project interdependencies have not been clearly defined within the DPMF.

The Departmental Project Management Office (DPMO) has developed and introduced the Change, Issue, Risk Management Tool (CIRMT) to help document and monitor project risks and issues, but its use has not been made mandatory for all projects. Further, at the time of the audit, a project interdependency tool was being piloted only for Corporate Business Architecture (CBA) projects.

The audit team believes that the lack of guidance and standardization of tools has led to the inconsistent management and documentation of project risks and interdependencies, as well as little evidence that project risks and interdependencies are being actively monitored.

The DPMF should require and provide the tools so that project risks and interdependencies, both internal and external to Statistics Canada, can be continually identified, assessed, monitored and reported.

The DPMF provides limited guidance regarding managing risks and monitoring project interdependency within the project life cycle.

The DPMF currently provides high-level guidance on the risk and issue management process; however, the audit noted that it does not detail how to embed the expected practices for risk management throughout the stages of the project's life cycle. Additionally, it was noted that there is no guidance in the DPMF regarding how to identify, assess, communicate and monitor project interdependencies.

Although the DPMF identifies the roles and responsibilities of project team members in other project management activities, it outlines the roles and responsibilities of the project manager only in key risk-management activities. The roles of other key project team members and the role of oversight committees regarding risk management within the context of a project have not been formally documented.

Similarly, the DPMF identifies the role of the CBA Secretariat only in relation to monitoring project interdependencies for CBA projects, and does not define the roles, responsibilities and accountabilities of other key project stakeholders, including those of the project manager, in identifying, communicating and monitoring project interdependencies.

As a result of this lack of guidance, key project team members and oversight committees may not have a consistent understanding of their roles in effectively and quickly identifying, assessing, communicating and monitoring project risks, issues and interdependencies. Consequently, the audit team noted inconsistencies in the level of documentation maintained to support identifying and monitoring project issues, risks and interdependencies.

Specifically, the audit identified 6 projects (out of 12) with limited evidence of risk management activities, including identifying relevant project-specific risks. Further, the audit identified four projects for which there was only limited or no evidence that project interdependencies were considered in the project planning and in dealing with project issues and risks.

CIRMT has been created; however, its use has not been made mandatory.

In 2013, the DPMO introduced CIRMT as the corporate risk and issue management tracking tool for documenting issues, risks and changes to a project's scope, schedule, costs and efficiency estimates. The use of a corporate risk and issue management tracking tool forces a level of discipline in identifying and assessing a project's issues and risks. As per the DPMO, all CBA and key non-CBA projects were to be migrated to the new tool by March 31, 2013; however, the use of the tool by all projects has not been made mandatory.

Through a review of CIRMT for the sample of 12 projects selected, it was determined that only 7 projects were using the tool and that only 4 of these 7 projects were using it consistently. The other five projects noted using other tools, such as Excel spreadsheets or Access databases, to log and monitor issues, risks and changes. However, the level of documentation maintained to support project changes as well as issue and risk management activities varied for the five projects.

Although these projects maintained an alternate tool, little or no documentation supporting project amendments and little or no evidence that risk and issue management activities were conducted was available. Without a consistently applied tool that forces a level of discipline when identifying and assessing risks and issues, it is possible the analysis and supporting documentation relative to risk and issue management will be insufficient.

An interdependency tool for tracking and monitoring project interdependencies is currently being piloted for CBA projects.

An interdependency tool has been developed by the DPMO to document interdependencies that exist between projects. This tool was in its pilot stage at the time of the audit and is currently limited to CBA projects.

The tool is maintained by the DPMO, but CBA project managers are asked to review their interdependencies monthly and communicate any changes to the DPMO in a timely manner. Although the DPMO continues to assess the tool as a result of the pilot, project management best practices indicate that a tool for managing project interdependencies should be maintained for all projects.

Recommendations

The assistant chief statistician of Corporate Services (and chief financial officer) should ensure that the DPMF is reviewed and updated to include

the roles of key project team members and oversight committees with respect to risk and issue and project interdependency management
the mandatory use of CIRMT and a corporate-level interdependency tool to track details of projects' risks, issues and interdependencies.

Management response

Management agrees with the recommendation.

In 2014/2015, the DPMO completed a pilot to monitor and report on project interdependencies. The result of this pilot was shared with various governance committees. Its implementation in 2015/2016 will address this recommendation. In addition, the following actions will be addressed:

The director of Corporate and Financial Planning Division (CFPD) will ensure that roles and responsibilities for project managers and oversight committees with respect to project interdependency management will be added during the review of the DPMF by March 2016. In addition, an interdependency tool will be launched to track interdependencies between projects.

Deliverables and timeline: Update guidance in the DPMF by March 2016 and implement a corporate interdependency tool by October 2015.
The director of CFPD will ensure that all projects use CIRMT to track and manage changes, issues and risks once the new Executive Project Dashboard is launched in 2015/2016.

Deliverables and timeline: Make use of CIRMT mandatory by June 2015.

Project baselines including costs and efficiencies estimates

The Departmental Project Management Framework (DPMF) currently outlines the requirement for documenting project baselines, including estimated costs and planned efficiencies, at various points in the project life cycle, and provides templates to be completed by the project team. The audit team identified opportunities to strengthen current cost-management practices.

The assumptions made in determining the costs and efficiency baselines are not consistently being formally documented. The audit team was unable to assess the reliability and completeness of these baselines or to verify that all of the projects' stakeholders were consulted in determining or reviewing the cost and efficiency assumptions.

The audit further revealed that amendments made to a project's scope, schedule or cost and efficiency baselines are not consistently being formally documented as expected. This affects the transparency of project amendments, i.e., the number of amendments made to a project and its overall final cost or expected efficiencies.

Reliable project baselines should be clearly defined through consultation with all relevant stakeholders, and any amendments should be documented and approved as per the established governance structure.

Improvements to the documentation of assumptions underpinning project financial cost and efficiency estimates are necessary.

The audit noted that project managers are generally documenting estimated project costs, as well as any planned efficiencies for the project, in both the Business Case Costing template and the Project Plan, both of which are subject to the project management gating process. In most cases, however, the audit team was unable to obtain documentation outlining the assumptions made in determining the cost and efficiency baselines, as well as any ongoing maintenance costs associated with the initiative. Without these assumptions, the audit team was unable to assess the relevancy and completeness of the assumptions and, as a result, the reliability and completeness of the project costs and efficiencies.

Additionally, since project managers indicated that the assumptions and risks underpinning the cost and efficiency baselines were not discussed formally with relevant stakeholders, the audit team could not confirm that all relevant project stakeholders were consulted in determining or reviewing the cost and efficiency assumptions, such as any system architecture needs.

Project management best practices dictate that not only should project assumptions be formally documented, reviewed and approved to show sound stewardship, but also that project managers should be required to conduct a sensitivity analysis on the costing assumptions (i.e., provide scenarios on costing results should specific assumptions change). This analysis would help project managers, project sponsors and the oversight committees understand what would happen to the project estimates if the assumptions became unreliable or changed over the course of the project.

Sensitivity analysis involves changing the assumptions in a cost-efficiency calculation to see the impact on the project's finances. This analysis helps stakeholders make better-informed investment decisions.

Inconsistencies were noted in the documentation maintained to support scope, schedule and cost and efficiency amendments.

Although there is an expectation that all amendments to the scope, schedule and cost and efficiency baselines will be documented in both the Change, Issue, Risk Management Tool (CIRMT) and Appendix A of the Project Plan, the audit team noted difficulties reconciling the adjustments being made with the projects' baselines. In the absence of a formal audit trail supporting updates made to the scope, schedule or cost and efficiency baselines, the audit team was forced to review email correspondence, review other project management deliverables, and hold discussions with project managers to identify project amendments.

Even through these additional procedures, the audit team was unable to reconcile the cost and efficiency baselines reported in the executive project dashboards and the Project Outcome Report with approved adjustments for 4 of the 12 projects selected for review.

With the Departmental Project Management Office's (DPMO) proposed revisions to the Executive Project Dashboard (discussed in more detail below), project managers will be required to document their proposed amendments to the scope, schedule and cost and efficiency baselines in CIRMT, which will then be populated directly in the Executive Project Dashboard. The Executive Project Dashboard will also track and report on the number of amendments made to the project and will require the project manager to attach evidence of the approval for all amendments directly to the dashboard.

The DPMF provides limited guidance on the roles, responsibilities and accountabilities of key project team members and oversight committees in cost-management activities.

The DPMF outlines the gating process for the Business Case Costing template and the Project Plan, both of which present cost and efficiency estimates. The DPMF does not currently outline expectations regarding the roles, responsibilities and accountabilities of each project stakeholder—including the project manager, the financial management analyst, the project sponsors and the oversight committees—in contributing to or reviewing and approving the cost and efficiency estimates and their underlying assumptions. As such, the audit revealed inconsistencies in the level of engagement of these key stakeholders in developing and reviewing the cost and efficiency estimates, including the financial management analyst's level of engagement.

Recommendations

The assistant chief statistician of Corporate Services (and chief financial officer) should ensure that

the DPMF and its related templates and tools are reviewed and updated to reflect cost-management expectations, including
- the documentation of all assumptions considered in developing the cost baselines, including any potential ongoing maintenance costs associated with the project
- formally documenting the consultations carried out with relevant project stakeholders in developing the cost and efficiency estimates and associated assumptions
- the roles and responsibilities of each project team member and oversight committee in contributing to and reviewing and approving cost and efficiency estimates and assumptions.

Management response

Management agrees with the recommendation.

The director of Corporate and Financial Planning Division (CFPD) will include a new checklist to be used by all projects to ensure that key assumptions are sufficiently documented at the beginning of a project and updated during the roll-out of the project. The checklist will require the project manager to obtain approval from key stakeholders on assumptions. The checklist will include detailed planning, costing and information-technology related questions. This checklist will amalgamate assumptions from various documents and replace existing documents.

Deliverables and timeline: Implement new checklist that documents detailed assumptions for all projects by March 2016.
The director of CFPD will ensure that the roles and responsibilities of each project team member and oversight committee in contributing to and reviewing and approving cost and efficiency estimates and assumptions will be added during the review of the DPMF by March 2016.

Deliverables and timeline: Update guidance in the DPMF by March 2016.

Project monitoring and outcomes

As outlined in the Departmental Project Management Framework (DPMF), the monthly executive project dashboards are to be prepared and reviewed as per the established governance structure for each project. As the audit team identified weaknesses in the reliability and completeness of the reporting being done through a sample of dashboards, the Departmental Project Management Office (DPMO) is revising its Executive Project Dashboard template to make a project's status more transparent.

The audit also noted that limited evidence is maintained to demonstrate that project oversight committees are consistently reviewing and approving the monthly executive project dashboards in a timely manner.

Expected business outcomes are being documented in the business case; however, in some instances, these outcomes are not sufficiently developed to provide an understanding of the benefits of the project to the agency, nor are they measurable. Without clearly defined measurable outcomes, as well as an evaluation approach to their measurement and expected targets, it is difficult to assess whether a project has achieved what it was expected to.

The established oversight bodies should adequately monitor projects, and evidence that they reviewed and approved the monthly executive project dashboards should be maintained. Additionally, relevant project outcomes should be formally established and projects should be assessed against expected results.

The Executive Project Dashboard template is currently being updated to improve its reliability and completeness to ensure effective project oversight.

The monthly Executive Project Dashboard is the main tool used for reporting on a project's status through its established governance structure, including disclosing any emerging risks and issues. The dashboard contains five indicators for monitoring purposes: scope, schedule, cost, risk and issues. At the time of the audit, the Executive Project Dashboard provided solely in-year reporting, fed by both the Financial Reporting System and the project manager. An assessment against each of the five indicators is required (red, yellow and green) and is rolled-up to derive the health of the project's overall status.

The DPMO has revised the dashboard template to make the information being presented in the dashboards more reliable and complete. The revised template—extracted directly from the Financial Reporting System—will present the variance of the project's in-year actual costs vs. budget per quarter and will include a graph demonstrating the same variance for all fiscal years.

This improvement will allow governance committees to review the total costs incurred on the project to date vs. the approved budget, instead of only in-year costs. Additionally, the risks, issues and project amendments presented in the dashboard will be sourced directly from the Change, Issue, Risk Management Tool (CIRMT). This will help ensure the reporting of risks, issues and project amendments are complete. The revised Executive Project Dashboard template is expected to be available for use by all project managers starting April 2015.

The audit team believes that the proposed amendments to the Executive Project Dashboard—specifically that the risks, issues and project amendments will be linked to CIRMT and the financial information will be presented for all fiscal years—may address some of the inconsistencies noted through a review of the dashboards for the 12 projects selected for testing.

Specifically, for a number of projects, the audit team noted difficulties in reconciling the financial information presented in the dashboards with the approved budgets and the risks and issues within CIRMT. However, with the revised template, since the data reported will be sourced directly from the corporate project management tools, the content will be more reliable, ensuring that the project's status is transparent.

Further, the new template may also help ensure that project managers are keeping the content in the project management tools up-to-date, as the governance committee will review them through the Executive Project Dashboard.

Governance committees noted exercising their project oversight responsibilities; however, limited documentation exists to confirm these critical activities.

There are a number of committees tasked with overseeing projects, including the individual project steering committee and the field planning boards (FPBs), as well as the Corporate Business Architecture (CBA) Management Committee (CBAMC) for CBA projects and other significant or complex projects as requested by senior management. The audit noted that the committees are generally exercising their monitoring responsibilities; however, this is being done informally.

Specifically, interviews revealed that the monthly executive project dashboards are generally presented to and discussed by the project steering committees and FPBs. For seven projects out of nine (three projects did not prepare any executive project dashboards), the audit team was unable to obtain evidence that these discussions took place because meeting minutes, action items or resulting decisions were not being consistently documented.

Similarly, limited evidence is maintained to demonstrate that CBAMC reviewed and approved the executive portfolio summary reports and periodic risk reports being prepared by the DPMO for CBA projects and other significant projects.

Expected business outcomes are difficult to assess and measure.

The audit team noted that project managers are documenting expected business outcomes in the project's business case. For 3 out of the 12 projects, the business outcomes were not sufficiently developed to provide an understanding of the benefits of the project to Statistics Canada, nor were they measurable. Without clearly defined and measurable outcomes, as well as an evaluation approach to their measurement and expected targets, it is difficult to assess whether a project has achieved what it was expected to.

The Project Close-Out Report is the mechanism used to communicate the results of the project against its intended benefits. If the expected business outcomes are not achieved, the project manager is provided with an opportunity to explain any deviations within this report. Of the 12 projects selected for review, 4 projects were completed. Of these completed projects, the audit team noted that a full assessment of results against all expected business outcomes (as originally documented within the business case or revised in other project documentation) was not available for two projects.

The Outcomes Realization Report is part of the post-launch review within the project life cycle and complements the Project Close-Out Report, as it allows business outcomes to continue to be measured and reported on beyond the life of the project. This report is especially important because it allows the measurement of outcomes, which it may not be feasible to measure in the short-term.

Even though, as noted above, an evaluation against all expected business outcomes as part of the project close-out was not available for two projects, an Outcome Realization Report was not completed. The audit further noted that project managers generally were not aware of the requirement to complete this report, nor did they understand its purpose.

Recommendations

The assistant chief statistician of Corporate Services (and chief financial officer) should ensure that the DPMF is reviewed and updated to include the following requirements:

The monthly executive project dashboards be formally reviewed and approved as per the established project governance structure, and evidence of their review and approval be maintained.
Project managers, project sponsors and oversight committees review the expected business outcomes to ensure that they provide a comprehensive explanation of the benefits of the project and that they are measurable before the business case is approved. Additionally, during the project planning stage, a timeline for measuring the business outcomes as well as what outcomes will be measured as part of the Project Close-Out Report and which will be reported using the Outcomes Realization Report (medium- and long-term impacts) should be established.

Management response

Management agrees with the recommendation.

The director of Corporate and Financial Planning Division (CFPD) will ensure that evidence of approval for all monthly executive project dashboards will be maintained by the DPMO and that monthly compliance reports will be submitted to the Corporate Planning Committee.

Deliverables and timeline: Implement monthly compliance reports for the Corporate Planning Committee by September 2015.
The director of CFPD will ensure that roles and responsibilities for project managers and oversight committees for reviewing expected business outcomes will be further documented during the review of the DPMF by March 2016.

Deliverables and timeline: Update guidance in the DPMF by March 2016.

Appendices

Appendix A: Audit criteria

Appendix A: Audit criteria
Control objective, core controls and criteria	Sub-criteria	Policy instrument
1) The agency has an effective and adequate project management framework to ensure a systematic approach to managing its projects and associated interdependencies in compliance with the Treasury Board (TB) Policy on the Management of Projects.
Governance and strategic directions 1.1 Statistics Canada has strategic direction in place for project management, including the establishment of a framework that is documented and adequately communicated. (G3 & G4).	1.1.1 Strategic direction by the Departmental Project Management Office (DPMO) is documented and communicated to all relevant stakeholders. 1.1.2 A departmental management framework exists, along with associated guidance and processes, which complies with the TB Policy on the Management of Projects and reflects best practices.	Statistics Act Policy on the Management of Projects (TB) Integrated Strategic Planning Process Departmental Project Management Framework (DPMF) Guidelines (Statistics Canada)
Accountability and stewardship 1.2 Authorities, responsibilities and accountabilities with respect to initiating, planning, executing and closing-out projects have been established and have been formalized and communicated. (AC-1; AC-2; AC-3; AC-4 & ST-13)	1.2.1 Employees' duties and control responsibilities as they relate to project management are clearly defined, documented and communicated to the relevant stakeholders. 1.2.2 Authority, responsibility and accountability are exercised as intended. 1.2.3 Authority is formally delegated, and delegated authority is aligned with individuals' responsibilities. Where applicable, incompatible functions are not combined.	Statistics Act Financial Administration Act Policy on the Management of Projects(TB) Integrated Strategic Planning Process DPMF Guidelines (Statistics Canada)
Risk management 1.3 The DPMF requires and provides the tools for identifying, assessing, monitoring and reporting project risks and interdependencies, both internal and external to Statistics Canada. (RM-3 & RM-6)	1.3.1 Formal processes and guidelines exist and are applied to identify project risks more easily. 1.3.2 Risk information is regularly presented to and discussed at established management and oversight committees and embedded in key performance reports. 1.3.3 Processes have been established to identify and communicate internal and external project interdependencies, such as deliverables, milestones and resource requirements.	Statistics Act Policy on the Management of Projects(TB) Integrated Strategic Planning Process DPMF Guidelines (Statistics Canada) Risk Management Policy (TB)
People 1.4 The DPMO provides employees with the necessary tools, resources, information and training to discharge their responsibilities and achieve expected results. (PPL-4)	1.4.1 A suitable training and development plan exists for stakeholders involved in the project management process. 1.4.2 Employees have access to sufficient tools, such as software, equipment, work methodologies, and standard operating procedures. 1.4.3 An information-sharing process exists to support the efficient and targeted dissemination of relevant and reliable information to the appropriate stakeholders.	Statistics Act Policy on the Management of Projects(TB) Integrated Strategic Planning Process DPMF Guidelines (Statistics Canada)
2) Projects are managed in compliance with the DPMF and its related tools, and are monitored against established milestones and expected results.
Policy and programs, and stewardship 2.1 Applicable DPMF project management deliverables are adequately completed as per established timelines, and approved by individuals with the appropriate delegated authority. (PP-4, ST-15 & ST-18)	2.1.1 The project management deliverables for initiating, planning, executing and closing-out projects are comprehensive, clear and well documented, and are supported by substantiated information, as per established timelines. 2.1.2 All project management deliverables are formally reviewed and approved in a timely manner, as per the established governance structure. 2.1.3 As required, project management deliverables are submitted to TB for review and approval.	DPMF Guidelines (Statistics Canada)
Results and performance, stewardship, and people and risk management 2.2 The project baselines have been clearly defined through consultation with all relevant stakeholders. (RP-2, ST-1, ST-2, ST-3, ST-15, ST-18, PP-4 and RM-2)	2.2.1 Project baselines are developed before projects are implemented. 2.2.2 Evidence exists demonstrating that interdependencies and risks were considered when developing project assumptions. 2.2.3 Relevant stakeholders are involved in developing the project baseline, including Finance and Information Technology (as appropriate).	DPMF Guidelines (Statistics Canada)
Stewardship, risk management, and policy and programs 2.3 Project risks, issues and changes are being escalated as per the established governance structure, and the impacts of project interdependencies are considered. (ST-15, ST-18, RM-1, RM-2, RM-5, RM-6 & PP-4)	2.3.1 An established process exists for escalating risks, issues and changes for review and approval as per the approved governance structure. 2.3.2 Evidence of decision making by the appropriate authority level is maintained. 2.3.3 An established process exists for identifying, assessing, communicating and monitoring project interdependencies, including interdependencies involving external parties.	DPMF Guidelines (Statistics Canada)
Results and performance, stewardship, policy and programs, governance and strategic direction, and risk management 2.4 Effective and timely project monitoring against the established baseline is conducted, and evidence is maintained to support decision making. (RP-2, ST-1, ST-2, ST-3, PP-3, ST-15, ST-18, PP-4, G-6, RM-2 & RP-3)	2.4.1 Monthly status reports and dashboards are being completed by project managers and reviewed as per the established governance structure in a timely manner. 2.4.2 The status reports and dashboards provide sufficient information to monitor progress as per the approved project baseline (e.g., milestones and costs).	DPMF Guidelines (Statistics Canada)
Results and performance, stewardship, and policy and programs 2.5 Relevant project outcomes have been formally established, and projects are assessed against expected results, including their completion on time and on budget. (RP-2, ST-1, ST-2, ST-3, ST-15, ST-18 & PP-4)	2.5.1 Relevant project outcomes (i.e., expected results) are established and approved in the planning stages of the project. 2.5.2 Evidence exists that project outcomes have been assessed in a timely manner against the approved expected results, including expected efficiencies (either by completing the Project Close-Out Report or, as required, through additional monitoring). 2.5.3 The Project Close-Out Report includes an assessment of the project's completion on time and on budget as per the approved original baselines.	DPMF Guidelines (Statistics Canada)

Appendix B: Acronyms and initialisms

Appendix B: Acronyms and initialisms
Acronym	Description
CBA	Corporate Business Architecture
CBAMC	Corporate Business Architecture Management Committee
CFPD	Corporate and Financial Planning Division
CIRMT	Change, Issue, Risk Management Tool
DPMF	Departmental Project Management Framework
DPMO	Departmental Project Management Office
FPB	Field Planning Board
ISPP	Integrated Strategic Planning Process
TB	Treasury Board

Census of Population Program — Public Use Microdata File

Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Consultation objectives

From December 2015 to January 2016, Statistics Canada will be consulting with key users of the Census of Population Program's Public Use Microdata File (PUMF) to obtain feedback on its content, variables, geography and sampling methods. This input will help to ensure that the product continues to meet user needs.

Consultation method

Key data users will be invited to participate in the 2016 Census of Population Program PUMF Consultation. An email questionnaire will be used to obtain feedback.

How to get involved

Individuals who wish to obtain more information or to take part in this consultation may contact Statistics Canada by sending an email to statcan.censusconsultation-consultationrecensement.statcan@canada.ca.

Please note that Statistics Canada selects participants for each consultation to ensure feedback is obtained from a representative sample of the target population for the study. Not all users will be asked to participate in a given consultation.

Statistics Canada is committed to respecting the privacy of consultation participants. All personal information created, held or collected by the Agency is protected by the Privacy Act. For more information on Statistics Canada's privacy policies, please consult the Privacy notice.

Results

Results from this consultation will be published online when available.

Report a problem or mistake on this page

Date modified:: 2023-01-09

Integrated Crop Yield Modelling Using Remote Sensing, Agroclimatic Data and Survey Data

1. Introduction

This report provides the background, general methods and results of a project undertaken to investigate the use of remote sensing, agroclimatic and survey data to model reliable crop yield estimates as a preliminary estimate to the November Farm Survey estimates, an occasion of the Crop Reporting Series at Statistics Canada. These estimates are made available before the September Farm Survey estimates are released. The work was completed by the Remote Sensing and Geospatial Analysis Section, Agriculture Division, and by the Business Survey Methods Division at Statistics Canada in collaboration with Agriculture and Agri-Food Canada (AAFC).

2. General methodology for crop yield modelling

A methodology for modelling crop yield was developed and tested on the crops that are typically published at the provincial and national levels by the September Farm Survey, as shown in Table 1. The five provinces listed account for approximately 98% of the agricultural land in Canada, across a diverse range of climate zones and soil types. The crops that account for approximately 85% of the revenue for the 19 crops listed are referred to as the seven major crops.

Table 1. Crops typically published in the results of the September Farm Survey, by province
Table summary
This table displays the results of Table 1. Crops typically published in the results of the September Farm Survey. The information is grouped by Crop type (appearing as row headers), Province (appearing as column headers).
Crop type	Province
Crop type	Quebec	Ontario	Manitoba	Saskatchewan	Alberta
7 major crops
Barley	X	X	X	X	X
Canola	X	X	X	X	X
Corn for grain	X	X	X
Durum wheat				X	X
Oats	X	X	X	X	X
Soybeans	X	X	X
Spring wheat	X	X	X	X	X
12 additional crops
Canary seed				X
Chickpeas				X	X
Coloured beans		X	X
Fall rye		X	X	X	X
Field peas			X	X	X
Flaxseed			X	X	X
Lentils				X
Mixed grains	X	X	X	X	X
Mustard seed				X	X
Sunflower seed			X
White beans		X	X
Winter wheat	X	X	X	X	X
Note: Fodder corn is typically published in the September Farm survey. However, it was not modelled due to a lack of July Farm Survey yield estimates.

The goal of the model was to produce a preliminary estimate of the expected harvest yield of the crops in late summer using information from existing data sources.

3. Data sources used in the model

The modelling methodology used three data sources: 1) the coarse resolution satellite data used as part of Statistics Canada's Crop Condition Assessment Program; 2) Statistics Canada's Crop Reporting Series data, and 3) agroclimatic data for the agricultural regions of Canada.

3.1 Normalized Difference Vegetation Index

Since 1987, Statistics Canada has monitored crop conditions across Canada and the northern United States using the Advanced Very High Resolution Radiometer (AVHRR) sensor aboard the National Oceanic and Atmospheric Administration (NOAA) series of satellites. This series of satellites produces a daily image of the entire Earth's surface at a spatial resolution of one kilometre. A spectral vegetation index, the Normalized Difference Vegetation Index (NDVI), was used as a surrogate for photosynthetic potential. NDVI is the normalized ratio of the Near-Infrared (NIR) to Red (R) reflectance (NDVI = (ρNIR − ρR)/(ρNIR + ρR)) and varies from −1 to 1, with values close to one indicating high vegetation content and values close to zero indicating no vegetation over bare ground. Materials such as water which absorb more radiation in the NIR than visible wavelengths, have a negative NDVI.

The NDVI data were processed on a continuous basis throughout the agricultural growing season (April to October) for the entire land mass of Canada. Statistics Canada has a time series of NDVI data from 1987 to present, which includes years of severe drought and record production. The daily NDVI images were processed into seven-day composites as described by Latifovic et al. (2005) and the methodology was further refined by Statistics Canada to minimize or eliminate NDVI errors introduced by the presence of clouds (Bédard 2010).

Cropland NDVI statistics by census agricultural region (CAR) were computed and stored in a relational database for each weekly NDVI composite. Only NDVI picture elements, or pixels, that geographically coincide with an agriculture land cover database produced by AAFC as part of an annual crop inventory were extracted to generate the mean NDVI value for cropland within each of the CARs. The agriculture land cover file and the associated metadata file produced by AAFC were accessible at www.geobase.ca/geobase/en/data/landcover/index.html.

After the mean NDVI values were computed they were imported as one of the input variable databases to the crop models as three-week moving averages from week 18 to 36 (May to August).

3.2 Survey area and yield data

Statistics Canada's Field Crop Reporting Series surveys were another dataset used in the model. These surveys obtain information on grains and other field crops stored on farms (March, July, September and December Farm Surveys), seeded area (March, June, July, September and November Farm Surveys), harvested area, expected yield and production of field crops (July, September and November Farm Surveys). These data provide accurate and timely estimates of seeding intentions, seeded and harvested area, production, yield and farm stocks of the principal field crops in Canada at the provincial level.

The survey produces results only when the crop is relatively abundant. If the crop is abundant in a province, the yields are available at a lower geographic level (usually corresponding to the CARs). If there is a crop but it is not abundant, survey data are available at the province level only. Some crops are absent or largely absent in a province and do not have survey data available.

For abundant crops, CAR level crop yield estimates from the July and November Farm Surveys from 1987 to present were used as input variables for the models while yield estimates from the September Farm Survey and the November Farm Survey were used to verify the accuracy of the yield model results. For less abundant crops, the survey data were compiled at the provincial level.

3.3 Agroclimatic indexes

The climate data collected during the growing season were the third data source used for modelling crop yields. The station-based daily temperature and precipitation data provided by Environment Canada and other partner institutions were used to generate the climate-based predictors (Chipanshi et. al. 2015).

Average values of the indexes at all stations within the cropland extent of a specific CAR were used to represent the mean agroclimate of that CAR. If a CAR lacked input climate data, stations from neighbouring CARs were used.

To form a manageable array of potential crop yield predictors, AAFC aggregated the daily agroclimatic indexes into monthly sums and means for the months of May to August. Their standard deviations (Std) over the month were also calculated and included in the modelling methodology (Newlands et al. 2014; Chipanshi et al. 2015). The Std value shows how the daily index varies over the one-month period. The larger the Std, the higher the variability of the parameter in that month.

4. Modelling survey yields

The model was selected by first reviewing the existing models, then by assessing the models available in SAS. Modelling was done at the smallest geographic level for which historical survey data were available. Only the five main crop-producing provinces (Quebec, Ontario, Manitoba, Saskatchewan and Alberta) were modelled.

4.1 Review of existing models

A model must be created for each CAR (or for each province in the case of less abundant crops). Each region has 28 years of data (1987 to 2014) and 80 explanatory variables. For its preliminary evaluation, Statistics Canada used a stepwise multiple linear regression, and showed that the optimal number of explanatory variables to be selected for modelling was five (Bédard and Reichert 2013).

An approach used by AAFC is based on the Bayesian and non-Bayesian methods at different steps (Chipanshi et al. 2015). The variable selection step uses a non-Bayesian approach by the least-angle robust regression algorithm, while cross-validating and keeping the variables that minimize the median of absolute errors. Yields are then estimated using a Bayesian approach.

The Bayesian approach is used to estimate yields at the beginning of the season, when data for the current year are not all available, which will not be the case at Statistics Canada, where estimates are done near the end of the growing season. As shown in Section 5, the methodology used by Statistics Canada generates results that are similar to the AAFC approach (identified as the least-angle robust model).

4.2 An alternative approach to SAS stepwise modelling

It is important to take outliers into account when selecting the explanatory variables (Khan et al. 2007) and performing estimation, and therefore to use robust modelling methods when possible. The objective was to find a robust alternative for both selecting the variables and estimating the yields. It was found that there was no robust selection procedure in the SAS software used at StatCan. An alternative was to use non-robust algorithms at the selection step and then to estimate the model in a robust way. The LASSO (Least Absolute Shrinkage and Selection Operator) method was selected from the five variable selection algorithms available in SAS. The MM method was chosen from the robust regression methods available in SAS, since it processes outliers at both the model and explanatory variable levels (Copt et al. 2006).

In the rest of the document the method retained by StatCan will be referred to as the LASSO robust model and the method used by StatCan for preliminary evaluations will be referred to as the stepwise non-robust model.

4.3 Aggregating model yield estimates to the provincial and national levels

The yield model estimates are created at the CAR level for the majority of crops. The CAR level estimates are weighted based on seeded area and aggregated to produce a provincial estimate. For certain crops that are less common in a province, the model estimates are only created at the provincial level. A similar weighting approach was used to produce a national estimate from provincial estimates.

4.4 Model evaluation method

The November Farm Survey estimates are considered the most accurate estimate of yield for a given year, due to the fact that the data are collected after the majority of harvesting is completed and the sample size is the largest of all six of the survey occasions. The results of the September Farm Survey can be considered a preliminary estimate of the November results. Therefore, the goal of the modelled yield is not to replicate the results of the September Farm Survey but rather to obtain a sufficiently accurate yield estimate in advance of the November survey results. Unless otherwise indicated, the analysis in the following sections was based on the November yield estimate as the benchmark for comparison.

The relative difference (presented as a percentage) between the yield estimate of a given method (i.e., September Farm Survey or the model) and the November survey yield estimate was the measure of quality. A negative relative difference indicated that the given yield estimate was smaller than the November survey estimate, while a positive relative difference indicated that the given yield estimate was larger than the November survey estimate.

$Relative difference = 100 * \frac{Given estimate - November survey estimate}{November survey estimate}$

Many of the summary tables were shown in terms of the absolute relative difference to demonstrate the magnitude of the difference between two estimates and did not take into account the direction of the difference. These absolute relative differences were summarized in terms of the median, 75th percentile, 90th percentile and maximum value calculated over the range of years for which estimates were compared.

5. Results of the model evaluations

Two studies were undertaken to evaluate the quality of the models. The first compared the LASSO robust model with the stepwise non-robust model while the second compared the LASSO robust model approach to the least-angle robust model.

5.1 Comparing results between the LASSO robust model and stepwise non-robust model

The LASSO robust model results were compared with results from the stepwise non-robust model by comparing the relative differences with the November survey yields. Results were generated for the seven major crops from 1987 to 2014 inclusive at the national level.

In general, the LASSO robust model, when compared with the November survey yields, produced results with smaller absolute relative differences than the stepwise non-robust model (Table 2). It was retained for the second study.

Table 2. Median, 75th and 90th percentiles of absolute relative difference of the LASSO robust and the stepwise non-robust models with November survey yields at the national level for 1987 to 2014 for seven major crops
Table summary
This table displays the results of Table 2. Median. The information is grouped by Crop (appearing as row headers), Median, 75 percentile and 90 percentile (appearing as column headers).
Crop	Median		75th percentile		90th percentile
Crop	LASSO robust (%)	Stepwise non-robust (%)	LASSO robust (%)	Stepwise non-robust (%)	LASSO robust (%)	Stepwise non-robust (%)
Barley	3.9	3.3	5.0	6.0	7.7	8.2
Canola	6.4	5.0	10.7	8.0	14.6	15.0
Corn for grain	4.3	6.6	6.2	8.5	8.8	11.6
Durum wheat	3.8	4.9	6.6	8.0	10.3	10.2
Oats	3.8	7.0	5.8	12.5	8.3	18.6
Soybeans	4.3	7.2	10.0	12.4	16.9	19.0
Spring wheat	3.1	4.0	7.0	6.2	9.1	8.8

5.2 Comparing results of the LASSO robust model with the least-angle robust models

For the yield model estimates, the LASSO robust model using SAS was compared with the least-angle robust model using R statistical language software. Statistics Canada, in collaboration with AAFC, determined that the LASSO robust model produced comparable results to those produced by the least-angle robust model. Table 3 indicates that the median absolute differences in yield over 28 years at the national level between the two models were all close to 1% for six of the seven major crops analyzed, and at 2.4% for soybeans.

Table 3. Median absolute difference between yields from the LASSO robust model and the least-angle robust model, national level, for seven major crops
Table summary
This table displays the results of Table 3. Median absolute difference between yields from the LASSO robust model and the least-angle robust model. The information is grouped by Crop (appearing as row headers), Median absolute difference (%) (appearing as column headers).
Crop	Median absolute difference (%)
Barley	0.9
Canola	1.0
Corn for grain	1.4
Durum wheat	1.3
Oats	0.9
Soybeans	2.4
Spring wheat	0.9

Statistics Canada made the decision to adopt the SAS LASSO robust model not only because it produced similar results to the least-angle robust model, but also because SAS is the standard programming tool used at the agency.

6. Comparisons of modelled yields with September survey yield results

The yield estimates produced by the SAS LASSO robust model were compared with the September survey yield in terms of relative differences from the November survey yields. Multiple comparisons were completed to evaluate how the modelled and survey yields performed at national and provincial levels over the long term (1987 to 2014), in a year with normal conditions (2014), and in a year of record production (2013).

6.1 Comparing absolute relative differences with November survey yields at the national level (1987 to 2014)

The series of graphs in Figure 2 show the relative difference of both the September survey estimates and the LASSO robust modelled estimates with the November survey yields, at the national level, for the seven major crops separately from 1987 to 2014.

As can be seen by comparing these seven graphs, there is no consistent pattern when the estimates of the two methods are compared. Neither method is consistently closer to the November survey estimates for any crop. For soybeans and corn for grain, the two methods follow a similar pattern of estimates over the 28 years with regard to how the estimates change from year to year. However, this pattern is not present for the other crops. Additionally, for any given year one method does not consistently perform better for all crops. In general, both methods have comparable relative differences from the November survey estimates. However, the modelled estimates tend to have larger relative differences in cases where an extreme relative difference is observed (e.g., the maximum and minimum relative differences are larger).

One pattern that can be seen is that the September survey results tend to be low when compared with the November survey results (below the x-axis) more often than the model results.

Figure 1. Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops.

Figure 1a Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Barley — Description of Figure 1a – Barley
The title of the graph is "Figure 1a Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Barley."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -15 and ends at 20 with ticks every 5 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -5.939484472 occurring in 2009.
The maximum value is 8.413556905 occurring in 2012.
The title of series 2 is "LASSO robust."
The minimum value is -11.011339104 occurring in 2013.
The maximum value is 17.817838787 occurring in 2012.

Data table for figure 1a – Barley
September survey LASSO robust
1987 -0.657222747 -0.007022293
1988 -2.657886652 -4.127229315
1989 2.529410207 4.062629525
1990 -3.133665940 -4.458154940
1991 1.299204128 2.319516397
1992 -5.120392359 1.216856405
1993 -0.294141830 7.700899896
1994 0.916298603 6.317220039
1995 -0.391672364 -2.258416911
1996 -0.907261465 2.087273987
1997 0.175122672 2.362656427
1998 0.018781177 1.442208318
1999 -1.370081370 -1.408978232
2000 -0.491287422 0.837053541
2001 4.474420578 4.042157977
2002 -2.366187815 5.868762080
2003 -2.940716615 -6.349190164
2004 -4.172236773 -2.400237686
2005 -2.358809299 -0.381758120
2006 -1.552892957 -2.887912767
2007 6.138332474 4.678941003
2008 -3.338165964 -4.117890774
2009 -5.939484472 -7.629567818
2010 3.186219847 3.800574703
2011 2.397051497 4.345646613
2012 8.413556905 17.817838787
2013 -5.007044967 -11.011339104
2014 -0.488764940 -0.992805729

Data table for figure 1a – Barley
	September survey	LASSO robust
1987	-0.657222747	-0.007022293
1988	-2.657886652	-4.127229315
1989	2.529410207	4.062629525
1990	-3.133665940	-4.458154940
1991	1.299204128	2.319516397
1992	-5.120392359	1.216856405
1993	-0.294141830	7.700899896
1994	0.916298603	6.317220039
1995	-0.391672364	-2.258416911
1996	-0.907261465	2.087273987
1997	0.175122672	2.362656427
1998	0.018781177	1.442208318
1999	-1.370081370	-1.408978232
2000	-0.491287422	0.837053541
2001	4.474420578	4.042157977
2002	-2.366187815	5.868762080
2003	-2.940716615	-6.349190164
2004	-4.172236773	-2.400237686
2005	-2.358809299	-0.381758120
2006	-1.552892957	-2.887912767
2007	6.138332474	4.678941003
2008	-3.338165964	-4.117890774
2009	-5.939484472	-7.629567818
2010	3.186219847	3.800574703
2011	2.397051497	4.345646613
2012	8.413556905	17.817838787
2013	-5.007044967	-11.011339104
2014	-0.488764940	-0.992805729

Figure 1b Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Canola — Description of Figure 1b – Canola
The title of the graph is "Figure 1b Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Canola."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -20 and ends at 30 with ticks every 5 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November Survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -17.43219711 occurring in 2002.
The maximum value is 3.504175875 occurring in 1993.
The title of series 2 is "LASSO robust."
The minimum value is -15.94819082 occurring in 2009.
The maximum value is 26.13127244 occurring in 2012.

Data table for figure 1b – Canola
September survey LASSO robust
1987 -1.815302973 -6.188729659
1988 1.778937896 -1.669973545
1989 -2.300620056 4.279096287
1990 -4.372016853 -3.18435676
1991 -5.488248924 -4.309587165
1992 -10.02046016 -6.872016226
1993 3.504175875 14.56962413
1994 1.936704224 14.64306917
1995 1.740564375 11.82582959
1996 -6.636743059 6.610025337
1997 -4.365309845 3.593502921
1998 -2.775068109 2.101951787
1999 -1.43914773 0.433288302
2000 -3.573216957 2.995683426
2001 -5.914594898 2.96223475
2002 -17.43219711 -12.96349565
2003 -6.399001273 -3.013788807
2004 -10.45617437 9.488752483
2005 -10.40464573 -8.26563344
2006 -5.563327755 -7.397672441
2007 0.549075653 13.22887041
2008 -11.73160576 -8.780009327
2009 -16.30524691 -15.94819082
2010 -11.40594687 -3.10402625
2011 -8.075060256 0.319817821
2012 0.339588606 26.13127244
2013 -7.660756337 -10.37879558
2014 -6.45174235 -0.86129483

Data table for figure 1b – Canola
	September survey	LASSO robust
1987	-1.815302973	-6.188729659
1988	1.778937896	-1.669973545
1989	-2.300620056	4.279096287
1990	-4.372016853	-3.18435676
1991	-5.488248924	-4.309587165
1992	-10.02046016	-6.872016226
1993	3.504175875	14.56962413
1994	1.936704224	14.64306917
1995	1.740564375	11.82582959
1996	-6.636743059	6.610025337
1997	-4.365309845	3.593502921
1998	-2.775068109	2.101951787
1999	-1.43914773	0.433288302
2000	-3.573216957	2.995683426
2001	-5.914594898	2.96223475
2002	-17.43219711	-12.96349565
2003	-6.399001273	-3.013788807
2004	-10.45617437	9.488752483
2005	-10.40464573	-8.26563344
2006	-5.563327755	-7.397672441
2007	0.549075653	13.22887041
2008	-11.73160576	-8.780009327
2009	-16.30524691	-15.94819082
2010	-11.40594687	-3.10402625
2011	-8.075060256	0.319817821
2012	0.339588606	26.13127244
2013	-7.660756337	-10.37879558
2014	-6.45174235	-0.86129483

Figure 1c Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Corn for grain — Description of Figure 1c – Corn for grain
The title of the graph is "Figure 1c Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Corn for grain."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -15 and ends at 25 with ticks every 5 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November Survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -12.75834026 occurring in 1998.
The maximum value is 16.09179465 occurring in 1992.
The title of series 2 is "LASSO robust."
The minimum value is -6.566274509 occurring in 1998.
The maximum value is 22.61219224 occurring in 1992.

Data table for figure 1c – Corn for grain
September survey LASSO robust
1987 -8.671828406 -4.679741077
1988 0.511716452 8.63994787
1989 -6.554459361 -3.207685593
1990 -1.549871899 5.050367212
1991 -6.610483949 -0.488994877
1992 16.09179465 22.61219224
1993 1.442430665 8.982220259
1994 -8.160223396 -1.34540209
1995 -2.958741373 6.121643726
1996 -3.424570346 3.447814591
1997 -0.997445365 7.049871548
1998 -12.75834026 -6.566274509
1999 -7.757630794 4.318137453
2000 10.47455392 20.81074797
2001 -5.707984033 5.404491238
2002 -3.819436821 8.651942832
2003 -5.538361517 1.15264252
2004 -8.316167059 -4.068773034
2005 -10.11529083 -3.050867421
2006 -5.565687396 1.816670428
2007 -9.079971325 -3.1113458
2008 -8.229783352 -0.092455981
2009 -1.535975046 4.284288215
2010 -6.585778363 3.483000082
2011 -5.888662677 -0.13813026
2012 -11.30510916 -1.757662926
2013 -6.213861053 5.479065959
2014 -1.214337689 5.172479903

Data table for figure 1c – Corn for grain
	September survey	LASSO robust
1987	-8.671828406	-4.679741077
1988	0.511716452	8.63994787
1989	-6.554459361	-3.207685593
1990	-1.549871899	5.050367212
1991	-6.610483949	-0.488994877
1992	16.09179465	22.61219224
1993	1.442430665	8.982220259
1994	-8.160223396	-1.34540209
1995	-2.958741373	6.121643726
1996	-3.424570346	3.447814591
1997	-0.997445365	7.049871548
1998	-12.75834026	-6.566274509
1999	-7.757630794	4.318137453
2000	10.47455392	20.81074797
2001	-5.707984033	5.404491238
2002	-3.819436821	8.651942832
2003	-5.538361517	1.15264252
2004	-8.316167059	-4.068773034
2005	-10.11529083	-3.050867421
2006	-5.565687396	1.816670428
2007	-9.079971325	-3.1113458
2008	-8.229783352	-0.092455981
2009	-1.535975046	4.284288215
2010	-6.585778363	3.483000082
2011	-5.888662677	-0.13813026
2012	-11.30510916	-1.757662926
2013	-6.213861053	5.479065959
2014	-1.214337689	5.172479903

Figure 1d Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Durum wheat — Description of Figure 1d – Durum wheat
The title of the graph is "Figure 1d Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Durum wheat."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -20 and ends at 20 with ticks every 5 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -13.0334 occurring in 2013.
The maximum value is 0.615541 occurring in 1987.
The title of series 2 is "LASSO robust."
The minimum value is -14.111 occurring in 2013.
The maximum value is 15.92199 occurring in 2002.

Data table for figure 1d – Durum wheat
September survey LASSO robust
1987 0.615541 -4.69621
1988 -0.01519 9.62181
1989 -2.83925 -2.17119
1990 -4.34551 -7.09696
1991 -5.39904 4.238379
1992 -6.55259 -0.17011
1993 -6.64972 6.766385
1994 -1.7052 1.652586
1995 -3.79919 1.212121
1996 -5.9446 0.057774
1997 -2.07523 1.59485
1998 -2.03351 3.432569
1999 -6.36269 -0.54122
2000 -2.2188 -2.06485
2001 -4.28179 11.6726
2002 -5.14184 15.92199
2003 -6.06771 0.039046
2004 -4.08001 -2.92988
2005 -6.95849 -6.58277
2006 -5.10945 -4.80423
2007 -3.12297 4.640448
2008 -8.20229 -0.93006
2009 -5.7484 -9.72166
2010 -1.36524 6.218933
2011 -5.26638 0.198891
2012 -2.76639 6.045082
2013 -13.0334 -14.111
2014 -7.64554 3.170732

Data table for figure 1d – Durum wheat
	September survey	LASSO robust
1987	0.615541	-4.69621
1988	-0.01519	9.62181
1989	-2.83925	-2.17119
1990	-4.34551	-7.09696
1991	-5.39904	4.238379
1992	-6.55259	-0.17011
1993	-6.64972	6.766385
1994	-1.7052	1.652586
1995	-3.79919	1.212121
1996	-5.9446	0.057774
1997	-2.07523	1.59485
1998	-2.03351	3.432569
1999	-6.36269	-0.54122
2000	-2.2188	-2.06485
2001	-4.28179	11.6726
2002	-5.14184	15.92199
2003	-6.06771	0.039046
2004	-4.08001	-2.92988
2005	-6.95849	-6.58277
2006	-5.10945	-4.80423
2007	-3.12297	4.640448
2008	-8.20229	-0.93006
2009	-5.7484	-9.72166
2010	-1.36524	6.218933
2011	-5.26638	0.198891
2012	-2.76639	6.045082
2013	-13.0334	-14.111
2014	-7.64554	3.170732

Figure 1e Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Oats — Description of Figure 1e – Oats
The title of the graph is "Figure 1e Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Oats."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -20 and ends at 20 with ticks every 5 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November Survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -12.44373178 occurring in 2004.
The maximum value is 14.80583742 occurring in 1991.
The title of series 2 is "LASSO robust."
The minimum value is -14.24123762 occurring in 2013.
The maximum value is 14.07863242 occurring in 1991.

Data table for figure 1e – Oats
September survey LASSO robust
1987 -3.573792868 -7.941096757
1988 -4.311855078 -8.99623586
1989 2.834808882 5.77805305
1990 -0.200096332 0.128144902
1991 14.80583742 14.07863242
1992 -3.830921889 0.262725956
1993 -5.259953851 -1.06464003
1994 -3.724484437 4.085965127
1995 -0.857117285 2.47688672
1996 -0.674024533 3.70589263
1997 -0.970886767 3.574453216
1998 -1.291998877 1.236853561
1999 -0.431501943 -1.088683761
2000 -1.769690927 0.561912394
2001 -1.444885132 5.634352269
2002 -4.164273879 1.671227146
2003 -3.744451945 -6.402085232
2004 -12.44373178 -4.455809506
2005 -4.616530553 1.206004176
2006 0.621992368 -3.561077406
2007 4.618098905 4.968902194
2008 -8.281906582 -5.930176325
2009 -5.556229055 -7.365405089
2010 1.193311686 2.27612373
2011 -0.534612999 -0.431212488
2012 1.80506142 3.931244589
2013 -10.84346125 -14.24123762
2014 -4.712562644 -3.816582526

Data table for figure 1e – Oats
	September survey	LASSO robust
1987	-3.573792868	-7.941096757
1988	-4.311855078	-8.99623586
1989	2.834808882	5.77805305
1990	-0.200096332	0.128144902
1991	14.80583742	14.07863242
1992	-3.830921889	0.262725956
1993	-5.259953851	-1.06464003
1994	-3.724484437	4.085965127
1995	-0.857117285	2.47688672
1996	-0.674024533	3.70589263
1997	-0.970886767	3.574453216
1998	-1.291998877	1.236853561
1999	-0.431501943	-1.088683761
2000	-1.769690927	0.561912394
2001	-1.444885132	5.634352269
2002	-4.164273879	1.671227146
2003	-3.744451945	-6.402085232
2004	-12.44373178	-4.455809506
2005	-4.616530553	1.206004176
2006	0.621992368	-3.561077406
2007	4.618098905	4.968902194
2008	-8.281906582	-5.930176325
2009	-5.556229055	-7.365405089
2010	1.193311686	2.27612373
2011	-0.534612999	-0.431212488
2012	1.80506142	3.931244589
2013	-10.84346125	-14.24123762
2014	-4.712562644	-3.816582526

Figure 1f Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Soybeans — Description of Figure 1f – Soybeans
The title of the graph is "Figure 1f Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Soybeans."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -30 and ends at 50 with ticks every 10 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November Survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -17.65518992 occurring in 2012.
The maximum value is 31.47390995 occurring in 2001.
The title of series 2 is "LASSO robust."
The minimum value is -19.27526423 occurring in 1991.
The maximum value is 39.38092709 occurring in 2001.

Data table for Figure 1f – Soybeans
September survey LASSO robust
1987 -10.79142516 -15.92750624
1988 -2.616742538 9.972394633
1989 -4.771484673 3.688444446
1990 -4.200357936 -4.22515471
1991 -12.43500468 -19.27526423
1992 -0.804350519 -4.928637944
1993 -2.549697433 -1.920283667
1994 -2.809249346 -2.598524354
1995 -6.177835618 -3.23361798
1996 -1.40003895 -0.121077377
1997 -1.839609341 2.212303851
1998 -6.398276333 -4.715261948
1999 -3.014151773 0.565451678
2000 2.140796854 8.182956786
2001 31.47390995 39.38092709
2002 1.223961552 13.15859139
2003 18.71359907 24.46603172
2004 -5.482255251 -6.532680591
2005 -4.336856311 -0.0865886
2006 -7.237417347 -4.396426948
2007 3.681452865 9.950566762
2008 -3.873945782 -1.479388137
2009 1.676338539 3.145303769
2010 -6.801313083 -4.234246692
2011 -7.314676856 -4.208300513
2012 -17.65518992 -13.53097055
2013 -5.593830313 -0.426438233
2014 -0.569432156 6.173212476

Data table for Figure 1f – Soybeans
	September survey	LASSO robust
1987	-10.79142516	-15.92750624
1988	-2.616742538	9.972394633
1989	-4.771484673	3.688444446
1990	-4.200357936	-4.22515471
1991	-12.43500468	-19.27526423
1992	-0.804350519	-4.928637944
1993	-2.549697433	-1.920283667
1994	-2.809249346	-2.598524354
1995	-6.177835618	-3.23361798
1996	-1.40003895	-0.121077377
1997	-1.839609341	2.212303851
1998	-6.398276333	-4.715261948
1999	-3.014151773	0.565451678
2000	2.140796854	8.182956786
2001	31.47390995	39.38092709
2002	1.223961552	13.15859139
2003	18.71359907	24.46603172
2004	-5.482255251	-6.532680591
2005	-4.336856311	-0.0865886
2006	-7.237417347	-4.396426948
2007	3.681452865	9.950566762
2008	-3.873945782	-1.479388137
2009	1.676338539	3.145303769
2010	-6.801313083	-4.234246692
2011	-7.314676856	-4.208300513
2012	-17.65518992	-13.53097055
2013	-5.593830313	-0.426438233
2014	-0.569432156	6.173212476

Figure 1g Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops - Spring wheat — Description of Figure 1g – Spring wheat
The title of the graph is "Figure 1g Relative difference from November survey yields at the national level, 1987 to 2014, seven major crops – Spring wheat."
This is a line chart.
There are in total 28 categories in the horizontal axis. The vertical axis starts at -20 and ends at 20 with ticks every 5 points.
There are 2 series in this graph.
The vertical axis is "Relative difference from November Survey (%)."
The units of the horizontal axis are years from 1987 to 2014.
The title of series 1 is "September survey."
The minimum value is -12.60585523 occurring in 2009.
The maximum value is 4.116861766 occurring in 2007.
The title of series 2 is "LASSO robust."
The minimum value is -16.83277596 occurring in 2013.
The maximum value is 13.79836794 occurring in 1993.

Data table for figure 1g – Spring wheat
September survey LASSO robust
1987 0.438755149 -4.608113816
1988 -2.365294418 5.269373623
1989 -3.17182972 -0.719100407
1990 -2.986274922 -7.855321603
1991 -1.79163059 2.460371889
1992 -5.252555347 1.875678904
1993 1.968034242 13.79836794
1994 0.394758501 7.797438953
1995 -4.834423881 -0.901504508
1996 -5.858033463 -0.776666628
1997 -2.32497577 2.286423417
1998 -2.56565658 3.806477535
1999 -2.686920413 -0.99416514
2000 -3.903990529 -1.270051203
2001 -3.26484453 5.131900989
2002 -5.267697933 1.519039148
2003 -7.304765897 -7.552162541
2004 -7.344837843 -0.304415452
2005 -4.283750519 0.060340143
2006 -4.395798171 -3.008719214
2007 4.116861766 6.86110741
2008 -6.213653015 -6.852896882
2009 -12.60585523 -12.10758414
2010 -3.055186387 1.574718883
2011 -4.980634997 0.878794172
2012 -0.734908381 7.577868219
2013 -10.28038135 -16.83277596
2014 -4.97077838 -3.141680431

Data table for figure 1g – Spring wheat
	September survey	LASSO robust
1987	0.438755149	-4.608113816
1988	-2.365294418	5.269373623
1989	-3.17182972	-0.719100407
1990	-2.986274922	-7.855321603
1991	-1.79163059	2.460371889
1992	-5.252555347	1.875678904
1993	1.968034242	13.79836794
1994	0.394758501	7.797438953
1995	-4.834423881	-0.901504508
1996	-5.858033463	-0.776666628
1997	-2.32497577	2.286423417
1998	-2.56565658	3.806477535
1999	-2.686920413	-0.99416514
2000	-3.903990529	-1.270051203
2001	-3.26484453	5.131900989
2002	-5.267697933	1.519039148
2003	-7.304765897	-7.552162541
2004	-7.344837843	-0.304415452
2005	-4.283750519	0.060340143
2006	-4.395798171	-3.008719214
2007	4.116861766	6.86110741
2008	-6.213653015	-6.852896882
2009	-12.60585523	-12.10758414
2010	-3.055186387	1.574718883
2011	-4.980634997	0.878794172
2012	-0.734908381	7.577868219
2013	-10.28038135	-16.83277596
2014	-4.97077838	-3.141680431

Table 4 summarizes the graphical information. At the national level, the median absolute relative differences from the November Farm Survey yields for the seven major crops modelled (barley, canola, corn for grain, durum wheat, oats, soybean, and spring wheat) were very similar to those from the September Farm Survey for the period from 1987 to 2014. In both cases, the median absolute relative difference was 4.1%. The median absolute relative difference results were comparable for some of the 12 additional crops, although larger relative differences were seen for crops that have a limited amount of historical data available. For the 12 additional crops, the overall median absolute relative difference of the modelled estimates (4.4%) was similar to the modelled median of the seven major crops but the overall median absolute relative difference of the September survey for the 12 crops (3.0%) was much lower than its median for the seven major crops.

In general, when larger relative differences were observed, the model's relative differences tended to be larger than those of the September survey. The maximum national absolute relative difference from the November Farm Survey yields for the 19 crops modelled was 39.4%, compared with 31.5% for the September Farm Survey.

Table 4. Median, 75th and 90th percentile and maximum of the absolute relative difference between the September survey yields and the LASSO robust modelled yields and the November survey yields at the national level for 19 crops
Table summary
This table displays the results of Table 4. Median. The information is grouped by Crop (appearing as row headers), Median, 75th percentile, 90th percentile, Maximum and Years of Nov historical data (appearing as column headers).
Crop	Median		75th percentile		90th percentile		Maximum		Years of November historical data
Crop	LASSO robust (%)	Sept. survey (%)	LASSO robust (%)	Sept. survey (%)	LASSO robust (%)	Sept. survey (%)	LASSO robust (%)	Sept. survey (%)	Years of November historical data
Barley	3.9	2.4	5.0	3.5	7.7	5.4	17.8	8.4	28
Canola	6.4	5.5	10.7	8.6	14.6	11.5	26.1	17.4	28
Corn for grain	4.3	6.4	6.2	8.4	8.8	10.7	22.6	16.1	28
Durum wheat	3.8	4.7	6.6	6.1	10.3	7.2	15.9	13.0	28
Oats	3.8	3.6	5.8	4.6	8.3	9.1	14.2	14.8	28
Soybeans	4.3	4.3	10.0	6.9	16.9	14.0	39.4	31.5	28
Spring wheat	3.1	4.0	7.0	5.3	9.1	7.3	16.8	12.6	28
Canary seed	7.2	5.6	14.3	13.3	19.2	19.9	20.6	27.6	16
Chickpeas	5.4	8.3	12.9	13.7	22.0	17.3	22.8	23.9	10
Coloured beans	7.9	5.4	11.9	6.8	13.5	11.2	13.9	16.9	7
Fall rye	4.5	2.6	7.6	4.4	10.0	8.3	27.7	10.4	27
Field peas	4.0	2.3	6.1	5.1	11.6	7.2	21.7	19.7	28
Flax	6.0	4.1	10.6	6.8	14.4	9.0	29.6	12.3	28
Lentils	2.8	3.2	6.9	5.0	12.3	6.6	15.4	11.7	22
Mixed grains	2.4	1.7	4.0	2.8	5.9	5.5	11.4	9.7	28
Mustard seed	3.4	4.6	8.8	8.3	13.6	11.4	21.3	13.5	11
Sunflower	15.9	7.7	25.5	16.7	29.9	22.6	35.5	31.1	10
White beans	11.9	5.0	12.9	6.1	15.4	7.3	19.1	8.8	7
Winter wheat	2.2	1.0	4.3	2.1	7.6	3.7	16.1	12.2	28
Overall (7 major crops)	4.1	4.1	6.9	6.4	13.1	10.3	39.4	31.5
Overall (12 additional crops)	4.4	3.0	8.3	6.0	14.2	11.0	35.5	31.1
Overall (all crops)	4.2	3.6	7.6	6.2	13.7	10.5	39.4	31.5

6.2 Comparing absolute relative differences with November survey yields at the provincial level (1987 to 2014)

Similar comparisons were done at the provincial level. For each crop, only provinces that had at least 10% of the national total area for the crop were included in the summary statistics. The median absolute relative difference from the November Farm Survey yields for the seven major crops modelled was 5.1%, compared with 4.4% for the September Farm Survey; the maximum absolute relative differences were 44.5% and 35.5%, respectively (Table 5). For the 12 additional crops the median absolute relative difference at the provincial level for the modelled estimates was 5.6%, compared with 3.7% for the September Farm Survey. Significantly larger overall maximums of 112.2% for the model and 79.3% for the September survey were observed.

Table 5. Median, 75th and 90th percentile and maximum of the absolute relative difference between the September survey yields and the LASSO robust modelled yields and the November survey yields at the provincial level for 19 crops
Table summary
This table displays the results of Table 5. Median. The information is grouped by Crop (appearing as row headers), Median, 75th percentile, 90th percentile and Maximum (appearing as column headers).
Crop	Median		75th percentile		90th percentile		Maximum
Crop	LASSO robust (%)	Sept. survey (%)	LASSO robust (%)	Sept. survey (%)	LASSO robust (%)	Sept. survey (%)	LASSO robust (%)	Sept. survey (%)
Barley	3.6	3.0	7.5	5.0	10.8	8.0	26.4	12.3
Canola	6.0	6.2	13.4	9.9	18.3	12.6	38.7	21.8
Corn for grain	5.5	5.8	7.7	8.9	9.7	12.5	33.1	26.1
Durum wheat	4.6	4.6	7.6	6.3	10.2	8.2	24.5	14.7
Oats	5.0	3.5	8.4	6.9	13.5	10.4	27.8	22.4
Soybeans	7.1	4.6	10.3	8.9	19.5	14.5	43.1	35.5
Spring wheat	5.0	4.2	8.7	6.5	14.0	9.7	44.5	15.9
Canary seed	7.2	5.6	14.3	13.3	19.2	19.9	20.6	27.6
Chickpeas	5.4	8.3	12.9	13.7	22.0	17.3	22.8	23.9
Coloured beans	8.8	7.3	17.2	13.7	24.0	17.8	112.2	21.1
Fall rye	5.9	4.3	13.2	10.5	21.8	14.5	74.8	79.3
Field peas	5.5	3.6	7.6	7.6	13.6	9.9	34.3	42.3
Flax	7.1	5.2	9.7	8.0	14.3	10.1	40.6	15.4
Lentils	2.8	3.2	6.9	5.0	12.3	6.6	15.4	11.7
Mixed grains	2.9	0.5	5.1	3.5	7.5	6.0	11.2	10.6
Mustard seed	8.1	4.7	14.5	11.4	20.8	17.0	33.8	23.5
Sunflower	15.9	7.7	25.5	16.7	29.9	22.6	35.5	31.1
White beans	12.6	5.9	18.8	8.9	21.0	12.3	31.9	18.6
Winter wheat	4.6	1.8	11.7	5.5	17.7	13.9	43.9	39.9
Overall (7 major crops)	5.1	4.4	8.9	7.4	14.7	11.5	44.5	35.5
Overall (12 additional crops)	5.6	3.7	11.6	8.1	18.5	14.1	112.2	79.3
Overall (All crops)	5.3	4.1	9.8	7.7	16.4	12.4	112.2	79.3

6.3 Comparing relative differences with November survey yields at the national level, 2014

There is nothing unique about the year 2014 in terms of the growing conditions throughout the year or the amount of each crop harvested. It is presented as a "typical" year. In 2014, at the national level, four of the seven major crops modelled and four of the 12 additional crops had smaller relative differences than the September Farm Survey when compared with the November Farm Survey results (Table 6).

Table 6. 2014 national yield estimates for the LASSO robust model and survey (September and November), with relative differences with the November survey estimates for 19 crops
Table summary
This table displays the results of Table 6. 2014 national yield estimates for the LASSO robust model and survey (September and November). The information is grouped by Crop (appearing as row headers), November survey, LASSO robust and September survey (appearing as column headers).
Crop	November survey	LASSO robust		September survey
Crop	Yield^{Table 6 note 1}^{Table 6 note 2}	Yield^{Table 6 note 1}^{Table 6 note 2}	Relative difference (%)	Yield^{Table 6 note 1}^{Table 6 note 2}	Relative difference (%)
Barley	62.4^{Table 6 note 1}	61.8^{Table 6 note 1}	-1.0	62.1^{Table 6 note 1}	-0.5
Canola	34.4^{Table 6 note 1}	34.1^{Table 6 note 1}	-0.9	32.2^{Table 6 note 1}	-6.4
Corn for grain	149.2^{Table 6 note 1}	156.9^{Table 6 note 1}	5.2	147.4^{Table 6 note 1}	-1.2
Durum wheat	41.0^{Table 6 note 1}	42.3^{Table 6 note 1}	3.2	37.9^{Table 6 note 1}	-7.6
Oats	84.1^{Table 6 note 1}	80.9^{Table 6 note 1}	-3.8	80.1^{Table 6 note 1}	-4.8
Soybeans	41.2^{Table 6 note 1}	43.8^{Table 6 note 1}	6.3	41.0^{Table 6 note 1}	-0.5
Spring wheat	45.8^{Table 6 note 1}	44.3^{Table 6 note 1}	-3.1	43.5^{Table 6 note 1}	-5.0
Canary seed	1038.8^{Table 6 note 2}	1034.8^{Table 6 note 2}	-0.4	1074.0^{Table 6 note 2}	3.4
Chickpeas	1770.6^{Table 6 note 2}	1833.1^{Table 6 note 2}	3.5	1780.0^{Table 6 note 2}	0.5
Coloured beans	20.3^{Table 6 note 1}	17.6^{Table 6 note 1}	-13.3	19.3^{Table 6 note 1}	-5.0
Fall rye	38.2^{Table 6 note 1}	38.0^{Table 6 note 1}	-0.5	36.2^{Table 6 note 1}	-5.2
Field peas	34.9^{Table 6 note 1}	36.8^{Table 6 note 1}	5.4	35.0^{Table 6 note 1}	0.4
Flax	22.1^{Table 6 note 1}	24.0^{Table 6 note 1}	8.6	24.1^{Table 6 note 1}	9.0
Lentils	1373.1^{Table 6 note 2}	1421.5^{Table 6 note 2}	3.5	1324.0^{Table 6 note 2}	-3.6
Mixed grains	66.4^{Table 6 note 1}	58.8^{Table 6 note 1}	-11.4	64.8^{Table 6 note 1}	-2.5
Mustard Seed	908.7^{Table 6 note 2}	954.9^{Table 6 note 2}	5.1	883.0^{Table 6 note 2}	-2.8
Sunflower	1775.2^{Table 6 note 2}	1330.0^{Table 6 note 2}	-25.1	1737.0^{Table 6 note 2}	-2.2
White beans	20.4^{Table 6 note 1}	16.5^{Table 6 note 1}	-19.1	19.4^{Table 6 note 1}	-5.0
Winter wheat	64.5^{Table 6 note 1}	67.2^{Table 6 note 1}	4.2	63.1^{Table 6 note 1}	-2.2
Table 6 note 1 Bushels per acre Return to the first table 6 note 1 referrer Table 6 note 2 Pounds per acre Return to the first table 6 note 2 referrer

6.4 Comparing relative differences with November survey yields at the national level, 2013 (a year of record production)

In 2013 —a year of record production for most crops— the model had smaller relative differences than the September Farm Survey compared with the November Farm Survey for two of the seven major crops analyzed and three of the nine additional crops for which comparable 2013 data were available (Table 7).

Table 7. 2013 national yield estimates for the model and survey (September and November), with relative differences with the November survey estimates for 19 crops
Table summary
This table displays the results of Table 7. 2013 national yield estimates for the model and survey (September and November). The information is grouped by Crop (appearing as row headers), November survey, LASSO robust and September survey (appearing as column headers).
Crop	November survey	LASSO robust		September survey
Crop	Yield^{Table 7 note 1}^{Table 7 note 2}	Yield^{Table 7 note 1}^{Table 7 note 2}	Relative difference (%)	Yield^{Table 7 note 1}^{Table 7 note 2}	Relative difference (%)
Barley	72.0^{Table 7 note 1}	64.0^{Table 7 note 1}	-11.0	68.4^{Table 7 note 1}	-5.0
Canola	40.0^{Table 7 note 1}	35.9^{Table 7 note 1}	-10.4	36.9^{Table 7 note 1}	-7.7
Corn for grain	146.9^{Table 7 note 1}	154.9^{Table 7 note 1}	5.5	137.7^{Table 7 note 1}	-6.2
Durum wheat	48.4^{Table 7 note 1}	41.6^{Table 7 note 1}	-14.1	42.1^{Table 7 note 1}	-13.0
Oats	92.8^{Table 7 note 1}	79.6^{Table 7 note 1}	-14.2	82.7^{Table 7 note 1}	-10.8
Soybeans	43.2^{Table 7 note 1}	43.0^{Table 7 note 1}	-0.4	40.7^{Table 7 note 1}	-5.6
Spring wheat	52.9^{Table 7 note 1}	44.0^{Table 7 note 1}	-16.8	47.5^{Table 7 note 1}	-10.3
Canary seed	1395.1^{Table 7 note 2}	1108.3^{Table 7 note 2}	-20.6	1103.0^{Table 7 note 2}	-20.9
Chickpeas	2093.1^{Table 7 note 2}	1616.5^{Table 7 note 2}	-22.8	1799.0^{Table 7 note 2}	-14.1
Coloured beans	--	--	--	--	--
Fall rye	--	--	--	--	--
Field peas	43.7^{Table 7 note 1}	38.3^{Table 7 note 1}	-12.4	43.0^{Table 7 note 1}	-1.6
Flax	27.6^{Table 7 note 1}	23.7^{Table 7 note 1}	-14.1	26.5^{Table 7 note 1}	-3.8
Lentils	1816.4^{Table 7 note 2}	1536.7^{Table 7 note 2}	-15.4	1604.0^{Table 7 note 2}	-11.7
Mixed grains	61.7^{Table 7 note 1}	58.6^{Table 7 note 1}	-5.0	62.2^{Table 7 note 1}	0.8
Mustard seed	950.1^{Table 7 note 2}	934.2^{Table 7 note 2}	-1.7	1013.0^{Table 7 note 2}	6.6
Sunflower	1660.5^{Table 7 note 2}	1368.0^{Table 7 note 2}	-17.6	1619.0^{Table 7 note 2}	-2.5
White beans	--	--	--	--	--
Winter wheat	63.1^{Table 7 note 1}	58.1^{Table 7 note 1}	-7.9	55.4^{Table 7 note 1}	-12.2
Table 7 note 1 Bushels per acre Return to the first table 7 note 1 referrer Table 7 note 2 Pounds per acre Return to the first table 7 note 2 referrer

7. Publishing provincial and national yield estimates

Modelled yield estimates are produced for crops at the provincial and national levels. A set of rules were established to determine which modelled yields are of an acceptable level of quality to publish. These rules are based both on data availability and the coefficient of variation (CV) calculated for each estimate at the provincial level. These rules are applied to each crop.

7.1 Publication rules for modelled yields

A minimum of 12 years of historical survey yield data for both November and July must be available as well as June survey area estimates and July survey yield estimates for the current year. If these conditions are not met, then a modelled yield estimate will not be produced for that region.

A second rule was established: the provincial estimate for a crop will not be published if the total cultivated area from suppressed regions (based on the previous set of conditions) exceeds 10% of the provincial area for the crop. Similarly, if provincial estimates for a crop were not published, the national-level estimate (of the five provinces considered) will not be published if the total cultivated area for the suppressed provinces exceeds 10% of the national area.

In cases where the estimates for certain provinces were suppressed due to quality, but a national level estimate was still produced, only provincial estimates that were of an acceptable level of quality were used.

Finally, if the CV of the provincial or national estimate from the model was greater than 10%, the estimate was not published at that level. Model based CVs are calculated differently than those for survey estimates, and different thresholds are used to determine quality than those used in the Field Crop Reporting Series.

7.2 Publication simulation for 2014

The rules listed in the preceding subsection were applied during a simulation of the production of modelled yields for 2014. Table 8 shows which crops produced publishable results for each province and at the national level, as well as the percentage of the crop area from regions that were suppressed. The results for 2015 and the years to come may be different from this simulation, given that the application of the publication rules will be repeated each year.

Table 8. Crops with publishable yields during the 2014 simulation at the provincial and national levels
Table summary
This table displays the results of Table 8. Crops with publishable yields during the 2014 simulation at the provincial and national levels. The information is grouped by Region Crop (appearing as row headers), Quebec, Ontario, Manitoba, Saskatchewan, Alberta and National (appearing as column headers).
Region Crop	Quebec		Ontario		Manitoba		Saskatchewan		Alberta		National
Region Crop	Published	Supp. (%)	Published	Supp. (%)	Published	Supp. (%)	Published	Supp. (%)	Published	Supp. (%)	Published	Supp. (%)
Barley	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0
Canola	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0
Canary seed	Absent	N/A	Absent	N/A	Absent	N/A	Yes	0.8	Absent	N/A	Yes	0
Chickpeas	Absent	N/A	Absent	N/A	Absent	N/A	No	100	Absent	N/A	No	100
Dry coloured beans	Absent	N/A	No	100	No	100	Absent	N/A	No	100	No	100
Corn for grain	Yes	0.5	Yes	0	Yes	0	Absent	N/A	No	100	Yes	1.1
Durum wheat	Absent	N/A	Absent	N/A	Absent	N/A	Yes	0	Yes	0.5	Yes	0
Fall rye	Absent	N/A	Yes	0	Yes	6.3	Yes	0	Yes	0	Yes	0
Dry peas	Absent	N/A	Absent	N/A	Yes	0	Yes	0	Yes	0	Yes	0
Flaxseed	Absent	N/A	Absent	N/A	Yes	0	Yes	0	Yes	5	Yes	0
Lentils	Absent	N/A	Absent	N/A	Absent	N/A	Yes	0	Absent	N/A	0	0
Mustard seed	Absent	N/A	Absent	N/A	Absent	N/A	Yes	0	No	100	No	27.1
Mixed grains	Yes	0	Yes	0	Absent	N/A	Absent	N/A	No	100	Yes	8.2
Oats	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0
Soybeans	Yes	1.3	Yes	0	Yes	0	Absent	N/A	Absent	N/A	Yes	0
Spring wheat	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0	Yes	0
Sunflower seeds	Absent	N/A	Absent	N/A	No	100	Absent	N/A	Absent	N/A	No	100
Dry white beans	Absent	N/A	No	100	No	100	Absent	N/A	Absent	N/A	No	100
Winter wheat	Yes	0	Yes	0	Yes	2.9	Yes	0	Yes	0	Yes	0
Number of crops published	8	N/A	9	N/A	10	N/A	12	N/A	9	N/A	13	N/A
Note: Supp (%): Percentage of the area for which modelled yields were suppressed. Absent: indicates that the crop is absent or largely absent in this province. N/A: means Not Applicable.

8. Summary

The estimates produced by the SAS LASSO robust model were comparable to those produced by the September survey in terms of relative difference from the November survey estimates for the seven major crops and many of the 12 additional crops published in September. On rare occasions, both the model and the September survey produced extreme relative differences from the November survey estimates, but not necessarily for the same crops/years. These extreme relative differences tended to be larger for the model than for the September survey.

Larger relative differences were observed in the model estimates for crops that have a limited amount of historical data available. Estimates derived from models that were constructed with only a limited number of data points were at risk of being statistically unreliable. Statistics Canada has established three criteria based on the availability of the input data, as well as quality indicators that must be met to ensure the statistical integrity of the estimates and to determine which of the modelled crop yields were of acceptable quality to be published at provincial and national levels. For each year, the yield model estimates for each crop would be evaluated to determine whether their quality is sufficient for publication.

In 2015, modelled yield estimates for crops deemed to have a sufficient level of quality were published as a preliminary estimate to the September Farm Survey estimates. In the longer term, survey managers must determine what is an acceptable level of risk for the published September estimates, and whether the risk of the larger relative differences produced by the model estimates in extreme cases is worth the benefits of eventually replacing the September survey occasion.

9. References

AAFC (circa 2000). http://www.geobase.ca/geobase/en/data/landcover/index.html

Baier, W., Boisvert, J.B., Dyer, J.A., 2000. The versatile soil moisture budget (VSMB) reference manual [computer software], ECORC contribution no. 1553. In:Agriculture and Agri-Food Canada. Eastern Cereal and Oilseed Research Centre, Ottawa, ON, Canada, pp. A1–D4.

Bédard, F. and Reichert, G., 2013. Integrated Crop Yield and Production Forecasting using Remote Sensing and Agri-Climatic data. Analytical Projects Initiatives final report. Remote Sensing and Geospatial Analysis, Agriculture Division, Statistics Canada

Chipanshi, A., Zhang, Y., Kouadio, L., Newlands, N., Davidson, A., Hill, H., Warren, R., Qian, B., Daneshfar, B., Bedard, F. and Reichert, G., 2015. Evaluation of the Integrated Canadian Crop Yield Forecaster (ICCYF) Model for In-season Prediction of Crop Yield across the Canadian Agricultural Landscape. Agricultural and Forest Meteorology, 206:137-150. DOI: http://dx.doi.org/10.1016/j.agrformet.2015.03.007

Copt, S., and Heritier, S., 2006. Robust MM-Estimation and Inference in Mixed Linear Models.Cahiers du département d'économétrie, Faculté des sciences économiques et sociales, Université de Genève

Khan, J. A., Aelst, S. V., and Zamar, R. H., 2007. Robust Model Selection Based on Least Angle Regression. Journal of the American Statistical Association, Vol. 102, No 480, pp. 1289-1299

Latifovic, R., Trishchenko, A.P., Chen J., Park W.B., Khlopenkov, K.V., Fernandes, R., Pouliot, D., Ungureanu, C., Luo, Y., Wang, S., Davidson, A., Cihlar, J., 2005. Generating historical AVHRR 1 km baseline satellite data records over Canada suitable for climate change studies. Canadian Journal of Remote Sensing, vol. 31, no 5, pp 324-346.

Newlands, N.K., Zamar, D., Kouadio, L., Zhang, Y., Chipanshi, A., Potgieter, A., Toure, S., Hill, H.S.J., 2014. An integrated model for improved seasonal forecasting of agricultural crop yield under environmental uncertainty. Front. Environ. Sci. 2, 17, http://dx.doi.org/10.3389/fenvs.2014.00017.

*Reference documents are available in English only

Subscribe to

Date modified:: 2024-11-30