Evaluation of the Manufacturing, Wholesale Trade, Retail and Service Industries Program

The report in short

This report presents the findings of the evaluation of the Manufacturing, Wholesale Trade, Retail and Service Industries Program (MWTRSIP). The evaluation was conducted by Statistics Canada's Evaluation Division and covers the period of 2012/2013 to 2016/2017 and, to some extent, 2017/2018. The evaluation examined the relevance and performance of the program in accordance with the Treasury Board Secretariat's Policy on Results (2016). The evaluation also studied specific areas of interest to the program in support of decision making, accountability and improvement.

The MWTRSIP is responsible for the provision of statistical information and advice on distributive trades, manufacturing, and service industries to governments, private organizations and institutions. The program aims to provide Canadians and the public and private sectors with access to quality statistical information and associated support services to inform public debate, research and decision making.

Key findings

The MWTRSIP is well aligned with federal roles and responsibilities, government priorities, and Statistics Canada's objectives. MWTRSIP activities support government priorities on economic growth and innovation, and the program provides key data for the calculation of gross domestic product (GDP), an indicator used to assess the health of the economy.

There is a clear continued need for the MWTRSIP. The program's statistical information is used for a variety of needs and by a wide range of users. The program is considered a trusted source of information and, while some complementary sources of information exist, no duplicate source was identified. Consultations with external users help the program stay connected, which contributes to meeting users' current and emerging needs—although participants in these consultations felt that more could be done to keep them informed. Gaps were identified during the evaluation, including scope, access, frequency, level of detail and timeliness; several initiatives were underway to fill some of them.

The evaluation found that MWTRSIP statistical information was generally considered accessible, accurate, coherent and interpretable, although there were ways to improve in each area. Most survey respondents and interviewees were quite satisfied with the quality of their customized data and the responsiveness of support service areas once they were able to reach the appropriate contact.

Timeliness is the dimension of quality where some issues were raised. While there was high interest from users to have information earlier, especially for the monthly retail survey, most were not willing to trade timeliness for accuracy—they preferred the status quo. The evaluation found there may be opportunities to improve timeliness by reducing the accompanying analytical information with each release, so long as information on coherence and methodological changes is provided. Finally, there may be opportunities to improve timeliness through the release process.

Overall, evidence indicated that the management of financial resources was challenging because of changes in program operations and resources during the scope of the evaluation. During the period, there were a number of measures implemented to cut costs and increase operational efficiency. The adoption of common tools will bring benefits and efficiency in the long term, but the transition period had an impact on the capacity to maintain quality and on employee workloads. Generally, there appears to be sound management and monitoring in place, although there are opportunities to further integrate activities and provide more guidance for staff.

Recommendations

Recommendation 1: Data gaps

The evaluation found that the program maintains effective relationships with its users and stakeholders and, as a result, has a good understanding of the gaps in the program. Comprehensive consultations took place during the evaluation period, and the program used the information gathered to develop plans to address as many gaps as possible within their resource allocation.

It is recommended that plans address the following key gaps:

• E-commerce and the new economy—an agency-wide approach is required, given that the subject spans several programs. While efforts are currently underway in several areas, there is a lack of coordination, roles and responsibilities are not clear, and senior management engagement needs to increase.
• Greater level of detail—over time, there has been a loss of detail for users. The MWTRSIP should continue working with key partners and stakeholders (internal and external) to explore avenues to address this.
• Data on services—while services make up a large share of the economy, statistical information is rather limited. It is recommended that the MWTRSIP explore opportunities to increase the frequency of its surveys and to increase the scope of the services covered.

In exploring these gaps, the program should continue to use a strong user-centric approach.

Recommendation 2: Timeliness

There were mixed results regarding the level of satisfaction with the timeliness of releases (82% of survey respondents and 55% of interviewees were satisfied with the current release times). When presented with different scenarios trading off timeliness for accuracy, users were generally unwilling to give up accuracy (with more than half preferring the status quo). Some users noted they would be willing to accept a loss in quality for more timely information during their peak seasons (e.g., retail data around Christmas).

It is recommended that the MWTRSIP

• review its process for releases to ensure they are as timely as possible, taking into consideration other key objectives such as relevance and accuracy; this includes exploring the possibility of reducing contextual analysis to improve timeliness, and targeting and coordinating the setting of release dates
• undertake a comprehensive consultation of its key users prior to taking any steps to improve timeliness that would impact accuracy.

Recommendation 3: Priority setting

The MWTRSIP is currently trying to address gaps/needs identified during previous consultations, and respond to the modernization agenda and other agency initiatives. These activities are in addition to its regularly mandated program.

It is recommended that, during this transitional period, MWTRSIP management provide greater clarification on priorities to employees.

Recommendation 4: Outreach and client services

Given the rapid evolution taking place in the program, and the broad, diverse range of users, it is vital that the MWTRSIP maintain strong relationships with users and actively promote its statistical information. During the evaluation, evidence was found of ongoing awareness-raising activities, but little evidence was found of a strategic element such as a written plan.

Client services are a key contact point with users. During discussions on access and cost-recovery services with interviewees, it was noted by several that client services could be improved. More specifically, it was noted that the level and quality of responsiveness were inconsistent (e.g., being less responsive during peak periods).

It is recommended that the MWTRSIP

• develop and implement a comprehensive outreach plan based on information from consultations, staff and other relevant partners (internal and external); the plan should include mechanisms for ongoing communication with users to inform them of program developments and to raise awareness
• prepare a strategy to strengthen client service delivery.

1. What is covered

This evaluation was conducted in accordance with the Treasury Board Secretariat's Policy on Results (2016) and Statistics Canada's Risk-Based Audit and Evaluation Plan (2017/2018 to 2021/2022). The main objective of the evaluation was to provide a neutral, evidence-based assessment of the relevance and performance of the Manufacturing, Wholesale Trade, Retail and Service Industries Program (MWTRSIP) to support decision making, accountability and improvement. The evaluation covered activities undertaken from 2012/2013 to 2016/2017 and, to some extent, 2017/2018. This was the first time the program had been evaluated.

The program

The mandate of the MWTRSIP is to provide trusted, relevant and comprehensive information on the manufacturing, wholesale trade, retail trade and service sectors to inform public debate on economic issues; support economic policy development, implementation and evaluation; and guide business decision making. Total MWTRSIP expenditures over the five-year period (2012/2013 to 2016/2017) were approximately $127 million.

The Manufacturing and Wholesale Trade Division (MWTD) and the Retail and Service Industries Division (RSID) are the two divisions responsible for compiling, analyzing, and disseminating a broad range of comprehensive and integrated statistics that depict the production, sales, financial operations and product types for the manufacturing, retail, wholesale and service sectors in Canada.

The evaluation

Following discussions with the MWTRSIP and senior management, the following seven areas were identified for review in the evaluation:

Relevance and responsiveness to needs

• 1. Alignment with federal roles and responsibilities, government priorities, and objectives
• 2. Continued need for the program
• 3. Responsiveness to user needs

Performance: Effectiveness

• 4. Quality—access, accuracy, coherence and interpretability
• 5. Quality—timeliness

Performance: Efficiency

• 6. Utilization of resources
• 7. Efficiency

Guided by a utilization-focused evaluation approach, the following quantitative and qualitative collection methods were used:^{Footnote 1}

The program also requested that the following four topics undergo greater examination:

• balance between timeliness and accuracy
• increased frequency of the Retail Commodity Survey
• quicker release of the program's key annual surveys (Annual Survey of Manufacturing and Logging Industries, Annual Retail Trade Survey, Annual Wholesale Trade Survey)
• reduction of contextual analysis to expedite release.

2. What we learned

2.1. Alignment with federal roles and responsibilities, government priorities, and objectives

As stipulated in the Statistics Act, Statistics Canada is required "to collect, compile, analyze, abstract and publish statistical information relating to the commercial, industrial, financial, social, economic and general activities and condition of the people."^{Footnote 2} The act also assigns Statistics Canada the responsibility "to promote and develop integrated social and economic statistics pertaining to the whole of Canada and to each of the provinces thereof and to coordinate plans for the integration of those statistics."^{Footnote 3} Topics such as manufacturing, wholesale and retail trade are specifically listed in the act.^{Footnote 4}

Data from the MWTRSIP are vital for the measurement of gross domestic product (GDP). The program includes 15 of 24 sectors as classified in the North American Industry Classification System (NAICS). As a significant contributor to the measurement of GDP, the MWTRSIP also supports the constitutional commitment of the federal government and Parliament to provide equalization payments to the provinces and territories.^{Footnote 5} The annual calculations for the equalization payments are made in accordance with the Federal-Provincial Fiscal Arrangements Act, which includes GDP.^{Footnote 6} The MWTRSIP's statistical information is also used to determine harmonized sales tax transfers in accordance with the Comprehensive Integrated Tax Coordination Agreements between the federal government and the provinces.

Reflecting the relative importance of the industries covered under the program, the government introduced a number of targeted measures through budgets to support its growth between 2012 and 2017. For example, Speeches from the Throne, budget documents and mandate letters have all identified growth in manufacturing and exports as a key priority. The MWTRSIP also supports the government by producing and disseminating statistical information required under a number of international agreements such as the International Monetary Fund's Special Data Dissemination Standard.

2.2. Continued need for the program

The evaluation found evidence that the MWTRSIP was the main source of information for manufacturing, wholesale trade, retail and service industries—all important indicators of the economic performance of the country. The business community, federal government and provincial governments were the main users of MWTRSIP products.

Business community representatives noted that they primarily used MWTRSIP data to conduct economic analysis of trends and industry developments, specifically to understand and interpret how manufacturing, wholesale trade, retail and service industries are faring in the context of the Canadian economy as a whole. Another key use for MWTRSIP data was for econometric modelling and forecasting. For example, the data were used to predict the impact of economic policy changes on corresponding sectors of the economy and to forecast sales for different holiday seasons. Finally, some business community representatives (including those who represented consulting firms and business associations) used MWTRSIP data to develop reports and publications for clients.

Similar to business community users, federal and provincial government stakeholders also used MWTRSIP data for economic forecasting. Provincial government representatives reported they used MWTRSIP data to develop provincial economic accounts estimates, report on provincial GDP estimates and estimate provincial household expenditures.

The evaluation also found that secondary distributors were repackaging program CANSIM data into more user-friendly formats for clients; academic institutions were using MWTRSIP data for research purposes and to contribute to their own publications; and, finally, media were requesting and using the information to support specific research and articles.

When web trends were examined, the Government of Canada was the most frequent user of MWTRSIP products, with Innovation, Science and Economic Development Canada being the most frequent user. Monthly retail trade data were accessed the most frequently, followed by monthly manufacturing data and then monthly wholesale data. Respondents and interviewees reported that MWTRSIP data were used frequently on a daily basis.

2.3. Responsiveness to user needs

Interviewees mentioned that Statistics Canada can play an important role in helping users define emerging trends and in clarifying how information should be gathered and classified. NAICS and other classification systems (e.g., North American Product Classification System) were seen as barriers that were not keeping pace with the rapidly evolving new economy. Internal documentation indicated that the Standards Division within Statistics Canada is currently working on this issue and that discussions are ongoing with trilateral partners to better reflect the digital economy in the next revision of NAICS in 2022.

Overall, the evaluation team found that MWTRSIP representatives had a good understanding of the current and emerging needs and that several initiatives were underway to fill gaps identified by the evaluation participants.

Level of detail

The main gap identified by key informants was the level of detail available. Close to 60% of survey respondents reported dissatisfaction with the level of detail, and many interviewees shared the same view. Areas of improvement that were suggested included greater disaggregation at the five- and six-digit NAICS levels; more retail and manufacturing data for lower levels of geography; and more detailed analysis at the regional, sub-regional, and sub-metropolitan levels.

Evaluation participants also raised concerns regarding data suppressions for confidentiality reasons for certain regions. While key informants understood the rationale for the data suppressions, they wondered whether Statistics Canada could suppress less data.

Several initiatives were put in place by the program following consultations in 2015 for manufacturing and wholesale, and in 2016 for RSID. These included the following:

• More detailed constant dollar estimates for three mission-critical surveys were released in July 2018 for wholesale, and later for the others.
• More local data: starting in June 2017, three CMAs were included in the monthly retail survey that covered 30% of the total retail sales in Canada (five new CMAs were added in January 2019). For manufacturing, 11 CMAs were released in November 2016, and additional data for 12 CMAs will be released in 2018/2019.
• More detailed industry-level seasonal adjustments were added in the Monthly Wholesale Trade Survey starting August 2017.
• The G-Confid software^{Footnote 7} was implemented and the specs were reviewed to ensure the appropriate level of protection for confidential cells while minimizing the loss of information.

Internal users indicated that, to get more detailed information for users, businesses would face an increased survey response burden (e.g., survey length, capacity of businesses to collect information). The MWTRSIP understands this trade-off and strives to find the appropriate balance.

E-commerce and the new economy

Another gap, and the most common trend identified by respondents, was the rise in, and increasing economic importance of, e-commerce. Interviewees indicated that a greater emphasis on e-commerce was needed. They pointed to items such as the development of e-commerce series, e-commerce data categorized by product category and type of store (e.g., e-groceries), separation of shipping costs from true e-commerce costs, and the extent to which retailers are participating in the digital economy. It was noted that improved e-commerce data would allow retailers to better understand the market (and degree of market saturation), compete for shares of retail sales, and prioritize merchandise for online versus in-person sales.

Another important gap identified was the extent to which Canadians are purchasing goods from international online retailers, particularly American retailers. This gap makes it difficult to understand the impact of e-commerce on the Canadian economy. One respondent indicated, "We are almost blind [to] what is going on."

Program representatives were aware of the gap; however, they noted that options are limited and that all countries are struggling with these issues. At the time of the evaluation, the program was taking steps to increase the number of outputs related to e-commerce, with some expected to be released in 2018. It was noted that options were limited for data on household transactions (peer to peer) and on issues around international online retailers. External interviewees generally acknowledged that Statistics Canada recognizes the importance of the issue and has improved the availability of e-commerce data over the past couple of years.^{Footnote 8}

Defining e-commerce also presents a challenge. Some interviewees pointed out that different e-commerce definitions between Statistics Canada and other organizations can lead to vastly different estimates and forecasts related to the proportion of sales made online versus in stores. Furthermore, as mentioned in an internal document, there remains a need for a common definition among survey programs since the word "digital" can be used as a synonym for "Internet/online," which causes conceptual problems with product classification and, de facto, how the sale of a digital product is reported.

Many informants indicated that the boundaries between the sectors and industries of the economy are increasingly blurred. For instance, all supply chain intermediaries can now sell directly to end consumers via e-commerce, services are being bundled with manufactured outputs, and businesses are diversifying their products and services, which makes them harder to differentiate and classify. For example, some grocery stores now serve food and beverages in much the same way as a restaurant ("grocerants"), and some also sell clothing.

Program representatives noted that nebulous boundaries between sectors have always existed, though they have become more prominent with the growth of the service industry and the use of digitalization. Efforts are being made by the MWTRSIP to compare data across the different sectors (manufacturing, wholesale and retail) and to continuously review the Business Register to ensure businesses are classified accurately. However, despite these efforts, limitations still remain. Some program representatives suggested developing multidisciplinary studies by industry and creating online related tables to integrate the three sectors, but this would require resources.

Another trend mentioned was how the expansion of digitalization creates new products and services, such as online marketplaces (e.g., eBay and Amazon), digital publishing, data streaming (e.g., Netflix), and products and services generated by households (e.g., Uber, Airbnb, vloggers). Several factors—such as the statistical framework and tools available to measure the number of household transactions (peer to peer), the organizational structure of businesses that own e-platforms, and Canadian legislation—are hindering the ability to fully assess the sharing economy. It was generally agreed that digital commerce is a gap that needs to be addressed. Statistics Canada is undertaking many initiatives to fill this gap, including a rapid response survey of Canadian households on their use of digital platforms and technologies, and the development of a new digital economy economic account. The Investment, Science and Technology Division (ISTD) is also planning to launch a Canadian Internet Use Survey that will cover some aspects.

Overall, it appeared that better cohesion was required on this issue within Statistics Canada. When the evaluation team discussed concerns about e-commerce and the new economy with program representatives, no clear response arose. While all agreed that it is an important gap and referred to several ad hoc actions and initiatives taken so far, it appeared that there was no clear vision or coordinated approach. This could be explained by the fact that Statistics Canada's economic statistics divisions are primarily organized by industry, so there is no natural home for the subject. In 2016, the ISTD, identified by many as the internal expert on e-commerce, put in place the Subject-Matter Committee on Digital Economy Statistics to improve the exchange of information between the divisions currently working on the digital economy. It was noted that, since the committee's inception, only six meetings had taken place, and attendance was low. It was agreed by many participants that more involvement and commitment from senior management were needed to create more synergies, improve awareness among the different divisions and reinforce divisional accountability to ensure a collaborative effort on digital commerce.^{Footnote 9}

Other trends and needs

Among other trends, informants pointed to changes in the food service industry (third-party food delivery options and higher-quality fast food operations, such as Freshii), growth of the "circular" economy involving the resale and reuse of goods, the impact of the legalization of cannabis, and the impact of fluctuating gasoline prices on retail sales data. The need to capture the impact of the political context, including NAFTA negotiations and other trade agreements, on manufacturing sales was also identified by interviewees.

Some interviewees identified the changing use of retail space as an emerging trend. Physical retailing is no longer as predictable as it once was; large retailers are vacating retail space, and traditional retail space is increasingly being used for services. This phenomenon increases the need for greater detail on commercial real estate vacancy rates, including data on the number of stores available by geographical location, price and sales per square foot.

Finally, key informants mentioned technological changes as an important trend. Data will be needed on labour, employment and productivity. It was also noted that there is a lack of understanding of how these technologies are currently being used, who is using them,^{Footnote 10} and their impact on the economy (e.g., blockchain technologies and cryptocurrencies).

Services

Virtually all users of services data thought improvements were required. Data users universally reported that, although services make up a large share of the economy, statistical information is rather limited. Several specific gaps were identified:

• information on industries such as personal care services (e.g., eyeglass services, hair salons, massage parlours, fitness centres, gyms) and health care services
• more information (historic and current) on intra- and inter-industry trade in services
• more regional and sub-regional disaggregation (e.g., to help meet targets set in provincial growth plans), metropolitan data (tourism), and more industry categories and subcategories (e.g., a subcategory within restaurants that would cover fast casual; clarification between engineering, architecture and other firms)
• greater disaggregation by firm size/type (e.g., according to restaurant type—whether or not it is part of a chain)
• financial and other statistics for the annual food services and drinking places survey (e.g., number of years in business, sales per seat, square footage of the restaurant, number of meals served, franchise affiliation, average bill per customer, number of liquor licences).

Improvements related to access, frequency and timeliness were also raised. In terms of access, many use The Daily as an entry point. They felt that more frequent releases in The Daily, and a dedicated web page with service industry information, would be useful. Grouping service data together and high-profile releases would also make the information more visible to potential users.

The frequency of the data was also viewed as being inadequate, with biannual releases being desired. Internal users indicated that more frequent data releases would provide better quality inputs for estimates such as GDP. The evaluation found that Statistics Canada had produced Quarterly Services Indicators in the past, but no one could speak to the quality of these products. One possible alternative proposed was to release an integrated product on services, similar to what is released in the United States.

Consultations and outreach

The evaluation found that MWTRSIP representatives generally stayed connected through conferences; formal and informal discussions with key users (i.e., provincial, federal, cost-recovery clients and main associations) and counterparts in the United States; business visits; and interdepartmental working groups and committees led by Statistics Canada or other departments, such as Canadian Heritage on culture.

A review of documents and internal interviews showed that initiatives like the manufacturing and wholesale environmental scan in 2015, the retail expert panel in 2016 and the cultural round table in 2018 were the formats that were most successful in identifying needs. As noted in the previous sections, most of the initiatives undertaken by the program to fill the gaps were the result of these larger consultations. Although considered the best formats by some program representatives, these types of consultations are most appropriate when the users and associations are well known. These formats are less effective when consulting with users who are not represented by major industry associations (e.g., service industries).

Some interviewees indicated that the purpose, expectations and outcomes of consultations should have been clearer. For example, post-consultation debriefs could have been conducted. Finally, some respondents noted that consultations were an opportunity to raise awareness of products, especially those that were not available on the website.

All internal informants noted they have excellent collaboration and effective bilateral communication with the MWTRSIP. The majority of the external stakeholders interviewed indicated a high level of satisfaction with the consultation process and believed their concerns were being taken seriously. Those who participated in the consultations agreed that a wide range of stakeholders were brought together and that discussions were facilitated effectively. However, they did comment that more representatives could have been invited, such as retailers, as opposed to those who monitor retail. According to the survey, 19% of the respondents who had never participated in a consultation would be willing to participate and collaborate with Statistics Canada.

It was also suggested that Statistics Canada regularly reach out to users to ensure they are aware of available data (i.e., new products and services). The evaluation found little evidence of strategic formal outreach plans/approaches;^{Footnote 11} instead, activities appeared to be tied to travel/conference plans. The only exception found was a communication plan in Wholesale Trade from a few years ago. Based on this plan, Wholesale Trade did a one-day public question-and-answer session. There were several other activities that took place during the period, including a Talking Stats session, a Data Story, a chat with an expert and several company visits. Limited activities took place afterward because of other priorities. In addition, there was little evidence of program representatives working with regional offices and the Statistical Information Service to inform them of their products.

2.4. Quality—access, accuracy, coherence and interpretability

Overall, the evidence indicated that MWTRSIP statistical information is considered accessible, accurate, coherent and interpretable. However, there were suggestions on how it could be improved.

Access

Website

Statistical information on the website is generally considered accessible once users are familiar with the website. Issues with the website included poor navigation, difficulty finding specific information, technical problems with the website and with downloading information, and challenges around the manipulation of CANSIM tables.^{Footnote 12} Two specific suggestions were the publication of release dates for non-monthly surveys and the inclusion of information on the availability of microdata files. On a more positive note, some key informants acknowledged improvements to the website over the past few years.

Client services

Overall, most survey respondents and interviewees were quite satisfied with the quality of their customized data and the responsiveness of support service areas once they were able to reach the appropriate contact. Some frequent users noted that the level of service offered by different parts of the agency is inconsistent and that, at times, they experienced long turnaround times.

Some frequent users of customized products believed that the availability of information depended on whom they spoke to in the agency. They cited examples where requests were first rejected and then approved. Some interviewees also noted a declining level of program knowledge and of knowledge of their past requests. For these reasons, some interviewers avoided using the client service units (and the main contact line) and instead searched for contacts directly in the subject-matter areas.

Program representatives indicated that, depending on the timing of a request, capacity could be an issue. For example, responding to requests during major releases or to requests that require a significant amount of resources can be challenging. This was confirmed by external clients who noted that responsiveness was uneven at times, with some requests being delayed or not fulfilled.

In general, the main reasons for not fulfilling requests, according to program representatives, were concerns over quality and/or confidentiality.

The evaluation found that proactive preparation for releases may help manage some requests. For example, in MWTD, because there are many requests for detailed data, the dissemination group develops tables in advance that take into consideration accuracy and confidentiality. These tables allow the dissemination group to skip the analysis needed to determine whether or not the statistical information can be shared. This increases responsiveness since clients no longer need to go through a pre-assessment stage.

Finally, some key informants and program representatives indicated that cost was also a barrier for customized products and cost-recovered surveys. Over the last few years, the program noted the loss of some cost-recovery surveys. Confidentiality and long turnaround times were also cited as reasons.

Accuracy

The vast majority of survey respondents reported they were either satisfied or very satisfied with the accuracy of the data (88.5%). This level of satisfaction was also confirmed during interviews, where the majority indicated that accuracy was high or at least suitable for their needs. A review of accuracy indicators from 2015 to 2017 found that the program adequately met its targets (Table 1).

In addition, the majority of survey respondents (75.4%) and interviewees stated that the existing revision rates for monthly mission-critical surveys were acceptable. Interviewees believed the revisions were minor and, in most cases, did not impact the story.

While there was overall satisfaction with accuracy, there were some concerns raised about the variability of estimates at lower levels of detail. Some informants (external and internal) had questions about the reliability of annual estimates since the implementation of the Integrated Business Statistics Program (IBSP) in 2013. The evaluation found that, while the accuracy level generally remained within targets during the implementation, there was some decline. The imputation of data for establishments was also questioned by internal users, especially for small entities. During discussions with some program representatives, they confirmed that additional work is now required to ensure accuracy and that they no longer release information at the five- to six-digit levels because of concerns with accuracy (e.g., Annual Survey of Manufacturing and Logging [ASML]).

Coherence and interpretability

Although they were generally satisfied with the coherence of the data, key informants were nonetheless dissatisfied with how the break in series, which occurred when surveys were transferred to the IBSP, was handled. They indicated that, to date, no information has been made available to determine the detailed magnitude of the impact.^{Footnote 14}

Both internal and external users noted that there were challenges when comparing monthly estimates with annual estimates (such as between the Monthly Survey of Manufacturing [MSM] and the ASML) and when examining sectors across surveys (i.e., manufacturing vs. wholesale vs. retail). Program representatives explained that reconciliations between monthly and annual surveys are problematic since different methodologies are used and annual periods may differ between businesses (e.g., some may use calendar years while others use fiscal years or different periods). Furthermore, annual and monthly surveys serve different purposes—annual for structural aspects and monthly for trends/indicators. In terms of comparability with external data, some interviewees felt that the categorizations and breakdowns used by Statistics Canada were not always consistent with the way retailers categorized commodities or how other research firms reported data.

In terms of interpretability, several areas of improvement were mentioned. Some key informants indicated that definitions accompanying data can be difficult to understand, especially for e-commerce. A need for more and better documentation was identified (e.g., to help better interpret the accuracy of the estimates, information on who is being sampled and how this is achieved, descriptions of survey developments, seasonal adjustments, and how data should be transformed). A review of the Integrated Metadatabase based on a sample of surveys indicated that important information such as response rates and accuracy values either was not systematically updated or was simply missing, especially for 2013 and 2014. Program representatives noted that providing coefficients of variation required several days of work and resources and that they preferred to update the information periodically.

Finally, aggregated variables such as "other expenditures" or "miscellaneous subsector" hindered users from getting a complete picture. For instance, important variations in the wholesale sector are sometimes caused by the "miscellaneous subsector."^{Footnote 15}

2.5. Quality—timeliness

In the evaluation, the quality dimension of timeliness was explored more in depth. More specifically, respondents and interviewees were first asked whether they wanted more timely data, and then they were asked how their responses would be different if there was an impact on accuracy.

Timeliness

In terms of levels of satisfaction with the current release timelines, the evaluation found a difference between survey respondents/internal users and interviewees. Among survey respondents, 82%^{Footnote 16} indicated that they were satisfied with the current release times, and internal users within the agency also expressed a high level of satisfaction with release times for monthly surveys. The percentage was lower, however, for interviewees (roughly 55%). Interviewees had issues that extended to all domains (i.e., manufacturing, retail trade, wholesale, and the service industry) and covered both annual and monthly surveys.

When asked about the timeliness of annual surveys, the very few interviewees^{Footnote 17} who used these products stated that it would be useful to have the estimates a maximum of 12 months after the reference period, as opposed to 18 months, and it was acknowledged that there had been a great improvement with the release of the ASML. Among survey respondents, 52%, especially provincial and municipal users, indicated that shortening the annual releases to 12 months would have a positive impact on their analysis and economic monitoring. There were also indications that the current release times for some culture surveys were too long (as the frequency of some surveys was 24 months) and that this impacted the relevance of the information.

Survey respondents and interviewees were specifically asked about increasing the frequency of the Retail Commodity Survey from quarterly to monthly. While most viewed it positively, they noted that it was not a high priority for them.

Program data on the release times for annual surveys suggest there were challenges in meeting release targets. In 2013 and 2014, 91% and 53%, respectively, of the annual statistical products did not meet Statistics Canada's internal release standard of 18 months. Starting in 2015, improvements were found, and this trend has since continued. Evidence from internal interviews confirmed that there were challenges as a result of the implementation of the IBSP, electronic questionnaires and the 2012 NAICS from 2013 to 2015. Consistent with these findings, internal interviews found that, starting in 2015, implementation issues began to be resolved, and there has since been a gradual return to meeting release times and even beating them in some cases. In the case of the ASML, additional funding from Global Affairs Canada helped to shorten the release time to 12 months.

Figure 1 - Number of lag days after reference period (on average) for monthly surveys

An examination of administrative data revealed that the release times for two monthly mission-critical surveys (i.e., MSM and MWTS) were gradually becoming longer (Figure 1). Specifically, the average release time for the MSM was 45.25 days in 2015/2016 and is anticipated to be 46.17 days for 2018/2019. Similar results were observed for the average release times for the MWTS. In 2015/2016, the number of days to release was 48.75 days and is projected to be 51.17 days in 2018/2019. For monthly retail trade, the trend has been roughly flat. On the other hand, lag days improved for surveys such as the Cement Survey^{Footnote 18} and the Monthly Survey of Food Services and Drinking Places.

External interviewees indicated that they would like to have earlier release times for the three mission-critical surveys, especially wholesale and retail information. Having this information earlier would allow users to more expeditiously examine current trends and seasonal impacts to make important decisions and forecasts for their businesses. For example, retailers would like to have information sooner on sales leading up to Christmas to better plan their purchases and inventories.

Accuracy versus timeliness

Overall, it appears that although users and stakeholders would like data sooner, they were generally not willing to give up much accuracy to achieve it.

When survey respondents were asked, "how important is the timeliness of the statistical information for your work compared to its accuracy?" only 14.3% stated that timeliness was more important. For interviewees, roughly half stated that no trade-off between accuracy and timeliness was acceptable to them, while the remainder indicated they would be willing to accept only a small loss in accuracy.

To explore this trade-off further, four scenarios were presented:

For survey respondents, 54% wanted the status quo (Scenario D), followed by Scenario B (18.6%), Scenario A (15.3%) and finally Scenario C (11.9%).

For interviewees, their preferences were roughly the same—approximately half chose the status quo, followed by Scenario B, then Scenario A and finally Scenario C. Notably, only one of the key informants preferred Scenario C. It was interviewees from financial institutions who were more likely to indicate they wanted earlier release times at the expense of some accuracy. They indicated that they would like to have the releases anywhere from 15 to 30 days after the reference period to produce GDP estimates. They did acknowledge, however, that although they wanted the data sooner, they would not want very large revisions.

Two factors may explain these results. Several respondents stated that the proposed scenario timelines ranging from a few days to a week and half would have no impact on their work; for them, the trade-off was not worth it. Another factor could be related to the structure of the Canadian economy. As mentioned in a C.D. Howe Institute report, "Because Canada's economy has less global importance and is dependent on trends in the US economy, users of Canadian data are more willing to trade less timeliness for more accurate data."^{Footnote 19}

When asked to prioritize the types of data they would want earlier, key informants and survey respondents chose retail information first, followed by manufacturing data and then wholesale data. In terms of the type of statistics, sales data were the most preferred. Impacts of having data earlier included the ability to report, publish and provide more timely analysis to clients, especially in the area of retail food sales in grocery and department stores. Interestingly, over 70% of survey respondents indicated they do not need any specific monthly data released earlier.

Other potential opportunities to increase timeliness

The evaluation found other potential opportunities to increase timeliness without necessarily impacting accuracy. First, there are opportunities to improve timeliness by providing less contextual analysis with the data release. Among survey respondents, 68% indicated their willingness to sacrifice contextual analysis for more timely data, largely because they develop their own contextual analysis. However, they specified that explanations around accuracy and methodological changes should remain.

Another option is to set more targeted release dates. Evidence indicates that it is up to the unit responsible for a release to decide the release date and that there is minimal communication between units when establishing release dates. Coordination with the other monthly release dates could be improved to ensure that information is timelier for financial organizations who indicated they wanted data prior to the monthly GDP release.

Collaboration between program representatives and stakeholders could also improve timeliness. A best practice shared by program representatives is the collaboration between key associations, businesses, service areas and monthly food survey representatives to increase the timeliness of the survey by a week. They worked closely for six months to determine the appropriate balance between the revision rate, timeliness and response rate of the survey.

2.6. Utilization of resources

The evaluation assessed the extent to which the MWTRSIP, including the subject-matter areas and the dedicated internal enablers, used their resources as planned. Evidence indicated that the management of financial resources was challenging because of changes in program operations and resources during the scope of the evaluation.

During the evaluation period, the program went through several reorganizations. The most significant occurred in early 2014 when

• the Manufacturing and Wholesale Trade programs were merged into one new division, the Manufacturing and Wholesale Trade Division (MWTD)
• the Service Industries Division and the Retail Trade Program were merged into one new division, the Retail and Service Industries Division (RSID)
• the Distributive Trades Division was disbanded.

To ensure consistency over the evaluation period, the financial information was allocated at the activity element level.

Program spending over the five-year period of the evaluation was approximately $96.8 million (excluding operation and maintenance).^{Footnote 20} From fiscal year 2012/2013 to 2016/2017, financial information shows that expenditures for activities related to MWTD were continuously over budget (Figure 2). For RSID, financial figures indicated that from 2012/2013 to 2014/2015, the expenditures were below budget, and that program expenditures began exceeding budget starting in 2015/2016, mainly because of an increase in salary expenditures (Figure 3).^{Footnote 21}

Figure 2 - Manufacturing and wholesale trade program portion—current budget vs. expenditures, from 2012/2013 to 2016/2017^{Footnote 22}

Figure 3 - Retail and service program portion—current budget vs. expenditures, from 2012/2013 to 2016/2017^{Footnote 23}

During the period covered by the evaluation, the budgets allocated to services areas, such as collection, significantly exceeded expenditures reported by them; this gap had shrunk by the end of the evaluation period.^{Footnote 24}

Finally, financial figures indicated that from 2012/2013 to 2016/2017, revenue from cost recovery increased by 450%, mainly because of the additional funding from Global Affairs Canada, while the number of projects declined.

2.7. Efficiency

Governance and management

Findings indicated that appropriate governance mechanisms were in place and that there was effective collaboration and coordination with internal users and service areas to ensure sound management and monitoring of key strategic and operational issues, data discussions and information sharing, and implementation of projects. These governance mechanisms included weekly senior-level and chief-level meetings, data discussion groups, steering committees, working groups, and bilateral divisional management meetings.

At the operational level, the reorganization of the divisions and the relocation of employees to the same floor helped facilitate day-to-day exchanges among employees, especially within MWTD. The merger of the Wholesale unit and the Manufacturing unit into the same division led to improved communications, problem-solving capacity (solving complex data issues and reporting issues) and expertise. In comparison, evidence indicated that there was less coordination within RSID since there was a less natural connection between retail and services. Between the two divisions, coordination is also limited because of the nature of the programs and the work.

In terms of management of the program, a number of representatives indicated the need for more guidance. Staff were being asked to maintain their regular activities, work on initiatives identified during past consultations to fill data and information gaps, and explore new projects to support Statistics Canada's modernization initiatives. According to interviews, staff were often asked to perform all these tasks using the same level of resources. To meet these demands, many program representatives indicated that they needed clearer direction from management (including senior management) on where to put their efforts.

A few representatives indicated that the solution to managing multiple priorities resides in strategic thinking within the division and across the program. They felt that knowledge sharing and discussions about where to gain efficiencies were limited. There were opportunities to share lessons learned and best practices, to use common standards across the divisions for similar survey processes (e.g., monthly surveys), to review processes and eliminate low value-added activities, to review the organization of work, and to examine survey sequencing to facilitate the sharing of employees across surveys and divisions.

Concerted efforts between divisions can result in significant added value to users. A good example of this was the coordinated work done in response to the Fort McMurray wildfires in 2016. The three mission-critical survey teams joined together to create an electronic questionnaire (EQ) module that allowed for ad hoc questions to measure the impact of specific events or phenomena on the economy. Another example was the work taking place to streamline the two-phased sampling of the annual surveys to expedite the pre-collection phase and deliver results faster.

Common tools and processes

Statistics Canada implemented major projects starting in 2009 in an effort to be more efficient, robust and responsive to user needs. The implementation of the IBSP was aimed at improving data quality by standardizing and improving sampling quality and timeliness while reducing response burden and ongoing costs to realize efficiency.^{Footnote 25} Another opportunity for efficiency for the agency was the move of all business surveys into the Business Collection Portal environment. This included the adoption of an electronic questionnaire—the primary mode of collection. Efficiency came from reducing the number of different software platforms, implementing new processes and tools, and increasing the proportion of electronic responses.

As mentioned previously, although the adoption of common tools will yield long-term benefits and efficiency, the transition had an impact on the program's capacity to maintain the quality of its outputs and on employee work-life balance (increased stress, frustration and overtime).

Many program representatives also indicated frustration with the lengthy transition process. They felt that the EQ system was less efficient because of a heavy bureaucratic process and that it was less effective. More specifically, it is now difficult to change survey elements and, when changes do take place, many errors occur and extensive revisions are required. This has translated into more time and effort being spent to ensure that changes are correct and that there are no errors. It was also noted that changes are costly and have created frustration with cost-recovered survey clients. Better communication and coordination between the groups would be helpful.

3. How to improve

SWOC analysis

Strengths

• Statistical information does not duplicate any other source
• Program representatives have a good understanding of the current and emerging needs
• Consultation mechanisms are adequate to identify user needs and opportunities
• Good quality and trust
• Recognized expertise and professionalism
• Use of common tools to standardize processes

Weaknesses

• Lack of cohesion and vision around e-commerce and new economy (agency-wide)
• Scope and frequency of releases on services information do not respond to user needs
• Gaps in outreach activities and service delivery
• Management of competing priorities (clarification and allocation of resources) and conflicting resources

Opportunities

• New economy and cross-functional information increase opportunities for new products and collaboration
• Openness of main stakeholders and businesses to collaborate and share information
• Increase in collaboration within and across divisions to innovate and gain efficiencies

Challenges

• Satisfying diverse needs from a wide range of stakeholders in a timely manner
• Lack of program understanding among users
• Measurement of rapidly evolving domains such as the new economy
• Lack of agility of processes such as EQ
• Need for more disaggregated information (geographical and five- or six-digit NAICS level)

Recommendations

Recommendation 1: Data gaps

The evaluation found that the program maintains effective relationships with its users and stakeholders and, as a result, has a good understanding of the gaps in the program. Comprehensive consultations took place during the evaluation period, and the program used the information gathered to develop plans to address as many gaps as possible within their resource allocation.

It is recommended that plans address the following key gaps:

• E-commerce and the new economy—an agency-wide approach is required, given that the subject spans several programs. While efforts are currently underway in several areas, there is a lack of coordination, roles and responsibilities are not clear, and senior management engagement needs to increase.
• Greater level of detail—over time, there has been a loss of detail for users. The MWTRSIP should continue working with key partners and stakeholders (internal and external) to explore avenues to address this.
• Data on services—while services make up a large share of the economy, statistical information is rather limited. It is recommended that the MWTRSIP explore opportunities to increase the frequency of its surveys and to increase the scope of the services covered.

In exploring these gaps, the program should continue to use a strong user-centric approach.

Recommendation 2: Timeliness

There were mixed results regarding the level of satisfaction with the timeliness of releases (82% of survey respondents and 55% of interviewees were satisfied with the current release times). When presented with different scenarios trading off timeliness for accuracy, users were generally unwilling to give up accuracy (with more than half preferring the status quo). Some users noted they would be willing to accept a loss in quality for more timely information during their peak seasons (e.g., retail data around Christmas).

It is recommended that the MWTRSIP

• review its process for releases to ensure they are as timely as possible, taking into consideration other key objectives such as relevance and accuracy; this includes exploring the possibility of reducing contextual analysis to improve timeliness and targeting and coordinating the setting of release dates
• undertake a comprehensive consultation of its key users prior to taking any steps to improve timeliness that would impact accuracy.

Recommendation 3: Priority setting

The MWTRSIP is currently trying to address gaps/needs identified during previous consultations, and respond to the modernization agenda and other agency initiatives. These activities are in addition to its regularly mandated program.

It is recommended that during this transitional period, MWTRSIP management provide greater clarification on priorities to employees.

Recommendation 4: Outreach and client services

Given the rapid evolution taking place in the program, and the broad, diverse range of users, it is vital that the MWTRSIP maintain strong relationships with users and actively promote its statistical information. During the evaluation, evidence was found of ongoing awareness-raising activities, but little evidence was found of a strategic element such as a written plan.

Client services are a key contact point with users. During discussions on access and cost-recovery services with interviewees, it was noted by several that client services could be improved. More specifically, it was noted that the level and quality of responsiveness were inconsistent (e.g., being less responsive during peak periods).

It is recommended that the MWTRSIP

• develop and implement a comprehensive outreach plan based on information from consultations, staff and other relevant partners (internal and external); the plan should include mechanisms for ongoing communication with users to inform them of program developments and to raise awareness
• prepare a strategy to strengthen client service delivery.

4. Management response and action plan

Appendix 1: Program description

The Manufacturing, Wholesale Trade, Retail and Service Industries Program (MWTRSIP) plans, directs, coordinates and controls the provision of statistical information and advice on distributive trades, manufacturing, and service industries to governments, private organizations and institutions. The program supplies the Canadian System of Macroeconomic Accounts with data required by the Federal-Provincial Fiscal Arrangements Regulations, and the Special Data Dissemination Standard with the International Monetary Fund. Data are required as part of Canada's participation in the North American Free Trade Agreement under the snapback provision, and under the Bank Act, section 427. The program also satisfies the information requirements of federal government departments, provincial governments, international organizations, industry associations, the academic community and the public at large.

The Manufacturing and Wholesale Trade Division (MWTD) and the Retail and Service Industries Division (RSID) are the two divisions responsible for compiling, analyzing and disseminating the broad range of comprehensive and integrated statistics that depict the production, sales, financial operations and product types for the manufacturing, retail and wholesale sectors, and a number of service sectors in Canada. Together, they represent 15 of the 24 sectors as classified in the North American Industry Classification System (NAICS) and account for a significant part of Canada's gross domestic product (GDP)^{Footnote 26} and employment.

MWTD and RSID function as separate operational and financial entities, each with a unique set of outputs and each with its own organizational structure.

Manufacturing

The manufacturing and logging sectors comprise establishments that are primarily engaged in the chemical, mechanical or physical transformation of materials or substances into new products (NAICS groupings 31-33, manufacturing). These products may be finished, in the sense that they are ready to be used or consumed, or semi-finished, in the sense of becoming an input for an establishment to use in further manufacturing. Related activities—such as assembling component parts of manufactured goods; blending materials; and finishing manufactured products by dyeing, heat-treating, plating and similar operations—are also treated as manufacturing activities.

MWTD disseminates information via monthly and annual Common Output Data Repository tables,^{Footnote 27} online publications and special tables for clients. The wide range of indicators of economic activity for the manufacturing program includes

• monthly estimates of sales, inventories, and the value of unfilled orders and new orders for 28 manufacturing industry aggregates, provincially and nationally
• monthly estimates of production and destination of shipments of asphalt roofing, sawmill products, and production and disposition of tobacco products^{Footnote 28}
• annual indicators of manufacturing revenue, employment, salaries and wages, cost of materials and supplies used, cost of energy and water utilities, inventories, and information about commodities produced and used in the production process (e.g., supplement to ASML).

Wholesale and retail

The wholesale trade sector provides the link between producers and retailers of goods in both the domestic and the global economies. Many wholesalers are organized to sell merchandise in large quantities to retail business and institutional clients. However, some wholesalers—in particular those that supply non-consumer capital goods—sell merchandise in single units to final users. This sector comprises two main types of wholesalers: merchant wholesalers that sell goods on their own account, and brokers that arrange sales and purchases for others generally for a commission or fee.

The retailing process is the final step in the distribution chain of merchandise. The retail trade sector comprises establishments primarily engaged in retailing merchandise, generally without transformation, and rendering services incidental to the sale of merchandise. Retailers are organized to sell merchandise in small quantities to the general public, which then becomes a primary indicator of consumer demand. The sector comprises two main types of retailers, store and non-store. The store retailers operate fixed point-of-sale locations, located and designed to attract a high volume of walk-in customers. The non-store retailers reach customers and market merchandise with methods such as the broadcasting of infomercials, the broadcasting and publishing of direct-response advertising, the publishing of traditional and electronic catalogues, door-to-door solicitation, in-home demonstration, the temporary displaying of merchandise (stalls), and distribution by vending machines.

MWTD is responsible for producing statistics on a monthly and annual basis for the NAICS grouping 41 (wholesale trade) by industry and geographical detail. Similar information is produced for the NAICS grouping 44-45 (retail trade) by RSID. Together, the two programs disseminate Common Output Data Repository tables and special tabulations for clients on

• annual information on the operating revenues, expenditures and inventory of wholesalers and retailers
• monthly estimates of sales, inventories and inventories/sales ratio for wholesalers
• monthly estimates of sales for retailers
• the monthly commodity breakdown of national retail sales for a panel of about 80 large retail enterprises
• the quarterly commodity breakdown of retail sales by geographical detail
• sales of new motor vehicles in units and in dollars by vehicle type, origin of manufacture and region.

Services

The service sector is characterized by establishments primarily engaged in activities where human capital is the major input. These establishments make available the knowledge and skills of their employees to create added value as opposed to producing tangible goods. Statistics covering only a portion of the service sector are in scope for RSID as other divisions produce service (education, transportation, tourism, etc.) indicators as well. The indicators produced by RSID cover portions of the following service sectors on an annual basis:

• real estate and rental and leasing (53)
• professional, scientific and technical services (54)
• information and cultural industries (51)
• administrative and support, waste management and remediation services (56)
• accommodation and food services (72)
• other services (except public administration) (81)
• management of companies and enterprises (55)
• arts, entertainment and recreation (71).

The output of the program consists of a series of annual indicators by industry and province for various types of revenues and expenses, inventories and exports.

Aside from financial data, the program disseminates a broad range of information on specific characteristics of the different service industries, such as distribution of sales by type of client (engineering services), number of screens (motion picture theatres), advertising and circulation revenue (periodical publishers), etc.

The program also disseminates a monthly indicator of the value of restaurant, caterer and drinking place sales by province and territory and by industry.

Internal processes

Information for manufacturing, wholesale, retail and service estimates is compiled by using a combination of survey and administrative data. The annual and monthly surveys collect information mostly from large and medium-sized establishments; tax information from T1, T2 and GST files is used to estimate production activity for small units.

Over the last few years, as a result of a number of corporate initiatives under the umbrella of the Corporate Business Architecture, there has been a concerted effort to streamline and standardize the infrastructure, methods and concepts in scope for these programs. As a result, the profile of the programs has shifted much more toward analysis and dissemination, and away from data collection and processing.^{Footnote 29}

Expected outcomes

The MWTRSIP is part of the Economic and Environmental Statistics Program Activity (Sub-program 1.2, Industry Statistics) in Statistics Canada's Program Alignment Architecture. It contributes to the following Statistics Canada Strategic Outcomes (SOs):

• "Canadians have access to timely, relevant and quality statistical information on Canada's changing economy and society for informed debate, research and decision making on social and economic issues" (SO1).
• "Specific client needs for high-quality and timely statistical services are met" (SO2).

While Statistics Canada is part of the "Government Affairs" outcome area in the whole-of-government framework, the agency and its programs contribute to most federal government outcomes in the domains of economic, social, international and government affairs.

The following logic model describes MWTRSIP's key activities, outputs and outcomes. It is based on Statistics Canada's Quality Assurance Framework,^{Footnote 30} which describes the six dimensions for quality management (accessibility, timeliness, accuracy, interpretability, coherence and relevance^{Footnote 31}) and is applied to all Statistics Canada statistical data and products.

Figure 4 - MWTRSIP logic model