Consultation summary

This document contains excerpts from the 2009 Quadrennial Program Report for the Business Special Surveys and Technology Statistics Program covering the 2005/06 – 2008/09 reference period. Statistics Canada has established an integrated program evaluation and reporting system. In that system, statistical programs undergo a full program evaluation every four years. A component of the evaluation is the extent to which existing statistical products and services continue to meet the evolving needs of clients.

Results

Nature of client consultations

A client satisfaction survey was conducted by Client Services Division in late 2009 on behalf of Business Special Surveys and Technology Statistics Division (BSSTSD). A standard corporate electronic questionnaire was used to assess both the satisfaction and the importance of various issues. Statistics Canada confirmed that this questionnaire was received by 458 clients. Fifty-two clients replied, for a response rate of 11%. Given this low response rate, the composition of survey respondents was profiled to determine if it was representative. Survey results were supplemented by insight gained from formal and informal consultations.

Clients

Before results were analyzed, clients who responded to this survey were profiled by the organizations they represented, as well as by the roles they played within those organizations. Forty-six percent of responding clients were federal government users. Clients from provincial/territorial governments, the education sector, and the non-profit sector each made up an additional 10% of respondents, respectively. Two-thirds of responding clients (66%) were managers, senior managers or economists. Although the 11% response rate might not be strongly representative, this concentration of clients who are managers and economists in the federal sector is a significant community of BSSTSD clients.

Client uses of BSSTSD products and services

One-quarter to one-third of clients (depending on area of purchase: Science and Technology: Telecommunications and Broadcasting: Cost-Recovery Surveys and Data Integration) responded that they use data for research or analysis. An additional 20% to 25% of clients (again, depending on area of purchase) reported using our data for policy analysis or development. The remainder reported equally using our data to plan, develop, and evaluate programs and services and using our data to model, forecast, and plan business issues.

Satisfaction

For both the standard and custom outputs of the base and cost-recovery programs, timeliness is the key consideration for clients. Consistently timeliness was ranked at the top level of importance for Science and Technology (S&T) and Telecommunications and Broadcasting (T&B) base programs, as well as for cost-recovery surveys and data integration products. Satisfaction with timeliness was ranked lower than importance, suggesting work is needed to improve in this area. Since clients often view relevance as synonymous with timeliness, it was not surprising that relevance is the second-highest consideration of our clients. Ninety to one-hundred percent of them ranked it at the top levels of importance. Satisfaction with relevance is higher for the standard outputs of both base and cost-recovery programs than for their custom outputs, which is not surprising given the additional time required to specify, tabulate and undertake disclosure evaluations of these more complex outputs.

Action plan

Recognizing that widely-accessible content helps clients determine, refine and acquire data for themselves, BSSTSD increased its presence on CANSIM and on the Statistics Canada website area known during the review period as Canadian Statistics. The division also publishes metadata that addresses frequently-asked questions in our Working Papers series. Data are now available electronically on the federal R&D survey, on the annual and quarterly surveys of telecommunications, and on several occasional surveys. In response to user inquiries and client feedback, work is now underway to strengthen geography and industry Research and Development (R&D) detail in CANSIM with new matrices scheduled for release in the fall of 2010.

Statistics Canada thanks participants for their participation in this consultation. Their insights guide the Agency's web development and ensure that the final products meet users' expectations.

Consultation summary

This document contains excerpts from the 2009 Quadrennial Program Report for the Income Statistics Program, covering the 2005/06 – 2008/09 reference period. Statistics Canada has established an integrated program evaluation and reporting system. In that system, statistical programs undergo a full program evaluation every four years. A major component of the evaluation is the extent to which existing statistical products and services continue to meet the evolving needs of clients.

Results

The Income Statistics Division (ISD) conducted a Client Satisfaction Survey in September 2009. The main purpose of the survey was to seek client feedback on the quality of our products and services so that we could better understand our clients' needs and priorities going forward. The survey achieved a 33% response rate.

Overall, the results indicate that clients were from 'generally' to 'very satisfied' with the division's products and services. First concerning our products, on a scale of 1 to 5, our products typically scored 4 out of 5 or higher. Our Income Trends in Canada HTML product, our publications, our public use microdata files (PUMFs), our Standard Tables, our CANSIM data, and our other website products all received high scores from our clients in terms of relevance, accessibility, and interpretability. Those clients who bought custom data tables from us were also very satisfied with the outputs they received.

With respect to service (i.e. the promptness of initial response to client requests, accessibility, and overall quality of service delivery), clients were again generally satisfied to very satisfied. Ease of access, quality of information received, ability to receive service in their language of choice, the existence of knowledgeable and competent staff, and the perception that clients received fair treatment were all elements that received high satisfaction scores.

Three main issues were raised through our client satisfaction survey, including timeliness, level of estimates and the need for ongoing wealth data. First, concerning timeliness of our products and survey results, there is not a lot that can be done in some cases because of our dependence on tax files from Canada Revenue Agency that arrive about 10 months after the reference year. However, it is clear that some effort will have to be expended to determine what can be done to release in a more timely fashion, going forward. Second, a number of clients would like to have income and expenditure estimates at lower levels of geography, including all census metropolitan areas. In the case of wealth, it was mentioned that a minimum of provincial level estimates was required given that the 2005 Survey of Financial Security was on a reduced sample. Finally, it was obvious through the survey results that clients require regular information on the wealth holdings of Canadians — which is clearly a major gap in our program. Also, given the current fiscal climate and the question of Canadian's readiness for retirement, many comments were received regarding pensions data, emphasizing that an increasing number of users have a need for these data.

The survey also confirmed and supports the direction the Division has taken concerning increasing access to our products and data; the most recent initiative being free PUMFs, in addition to many already free products (e.g. Income Trends in Canada).

Relevance

In addition to our Client Satisfaction Survey, the relevance of our programs, products and services is evaluated regularly through consultation with both our internal and external clients and stakeholders. Key internal clients from both the social and economic areas depend on the Division's products in creation of their own products and services of both microdata and aggregate estimates. Specifically the System of National Accounts and Prices Division rely on data from our surveys for the accounts and the basket weights for the Consumer Price Index. Analytical divisions also make significant use of ISD data, including Labour and Household Survey Analysis Division, Business and Labour Market Analysis (BLMA) and Socio-economic Analysis and Modelling Division. Including our own 44 articles published in the Income Research Paper Series, the Expenditures Research Paper Series and the Pension and Wealth Research Paper Series, 9 papers from BLMA, 31 articles in Perspectives on Labour and Income and 2 in Canadian Social Trends also relied heavily on our data.

Our diverse external clients require information on what Canadians earn, spend and save so they can engage in political debate, undertake policy analysis, describe social or economic conditions of families and individuals, or conduct fundamental research to clarify and improve our collective understanding. External clients come from federal departments, provincial ministries, and crown corporations, as well as non-governmental organizations, lobby groups, the media and academics.

Timeliness

The Client Satisfaction Survey revealed that timeliness of ISD products was an important issue. The timeliness of the Survey of Labour and Income Dynamics (SLID) release is constrained by the fact that data from the T1 file for respondents who give tax permission is not available to Statistics Canada until ten months after the tax year. During the review period, the SLID release occurred between 15 to 17 months after the end of the reference period.

The Survey of Household Spending (SHS) release occurs approximately 11.5 months after the reference period. In 2007, the SHS was released 14 months after the end of the reference period due to a need for increased analysis related to the use of the computer-assisted personal interview application for the first time.

Information from the Quarterly Survey of Trusteed Pension Funds is regularly released 26 to 27 weeks after the quarter-end. While improvements were made to the follow-up procedures for the Census of Trusteed Pension funds, its 2006 release was delayed to allow transition and knowledge transfer to new team members following the retirement of three key members. For the Pension Plans in Canada program, required administrative data from federal and provincial pension authorities is sometimes not received until 15 months after the reference date of January 1st, each year. Although we have been working with the jurisdictions to improve timeliness, it is a challenge to release earlier than May of the following year.

Action plan

In the next four years, the Division will continue its activities with a view to fulfilling its mandate which is to provide information on the economic well-being of Canadians, including the production of annual estimates on income, spending, assets and debts. To do this the Division will focus on the strategic elements in order to ensure the ongoing relevance and quality of our income, expenditure and pension/wealth data.

Statistics Canada thanks participants for their participation in this consultation. Their insights guide the Agency's web development and ensure that the final products meet users' expectations.

Background

This document is a summary report on consultations held with visitors to the Statistics Canada website as part of the ongoing program of website improvement. These consultations focused specifically on a planned redesign of the website's Home Page with a view to enhancing its client-centricity and user-friendliness thereby enabling visitors to this particular page to complete their tasks as effectively as possible.

Summary

In March and April 2009, five focus groups were conducted with approximately 50 frequent and infrequent visitors to the Statistics Canada website. During the sessions, user needs, tasks and preferences were discussed. Based on the feedback received, three Home Page prototypes were developed.

In August 2009, as a second phase, seven one-on-one usability interviews were conducted, five with external frequent and infrequent visitors and two with new Statistics Canada employees.

During the usability tests, participants were asked to conduct a variety of tasks using prototype A as their starting point. For each task, participants' points of entry, paths taken, task completion and barriers were noted. Participants were also asked to review and discuss their likes and dislikes of the existing Home Page in comparison with the three prototypes.

Results

Prototype A was the preferred version (over the existing Home Page as well as prototypes B and C) by 6 of 7 participants. They found it less cluttered, there was a decrease in repeated features, information was above the fold and, generally, the page design was considered to be more user-friendly.

The task completion exercises using prototype A yielded positive results.

Action plan

Replace the existing Home Page with prototype A, modified with some of the features from prototypes B and C.

The newly redesigned Home Page was successfully launched on the Statistics Canada website on December 2, 2009.

Statistics Canada thanks participants for their participation in this consultation. Their insights guide the Agency's web development and ensure that the final products meet users' expectations.

Consultation summary

This document contains excerpts from the December 2009 Quadrennial Program Report for the Agriculture Statistics Program, covering the 2005/06 – 2008/09 reference period. Statistics Canada has established an integrated program evaluation and reporting system. In that system, statistical programs undergo a full program evaluation every four years. A major component of the evaluation is the extent to which existing statistical products and services continue to meet the evolving needs of clients.

Results

Using the services of Client Services Division, an electronic Client Satisfaction Measurement Survey (CSM) was sent out over May-June of 2009 to 621 users of the Agriculture Statistics Program. In four weeks, the survey generated a 25% response rate, which compares favourably to the normal 20% (or less) for this type of survey, and may indicate that clients feel some attachment and loyalty to the Division as a result of the good products and services it provides.

The main feedback was related to costs, timeliness, responsiveness and staff. Key recommendations concerning cost were that the program be expanded (at least not cut anymore), and that there be more detailed, lower cost and free data. In addition, clients said they wanted information that comes out faster and reacts to new topics and trends more quickly (increase timeliness and responsiveness). Finally, although clients said their experience in dealing with divisional staff had been good, they also said they would like to see more staff with agriculture background and training, as well as a more stable staff with less rotation and turnover.

Overall, this CSM survey has not only given the Division good feedback to work with on potential improvements to the outputs of the agricultural statistics program, but has provided validation of the great work being done throughout the Division every day, especially when it comes to serving clients.

Relevance

Stakeholders indicated that the Division's data products and services have been key inputs to research, but the level of detail was indicated as a weak point. In other words, clients preferred more detailed data, related mostly to geographical and commodity level breakdowns. Overall, clients rated the importance of the data as more than just "satisfactory", although there were strong suggestions for the Division to consider more custom, and smaller, levels of geography, and to perform more analysis.

Timeliness

Although timeliness was indicated as being a bit above "satisfactory", it came out as one of the weakest points. Clients felt that they needed data to come out faster, and thought the Division should react faster to emerging issues.

Accessibility

Clients rated accessibility as more than satisfactory and indicated that they valued the resource materials provided by the Division. They also rate their experience in dealing with staff as more than satisfactory. However, there were remarks that the United States Department of Agriculture website was a lot easier to access. It was also noted that payment for special, non-standard data requests was a sore point. Clients disliked the fact that they have to pay, especially for the Census of Agriculture special tabulations. Clients who were also media made the point about free data. Currently, media do get all standard outputs at no cost, but they would like to have any custom request, regardless of the complexity or cost involved to produce it, provided to them free of charge.

Action plan

Some efforts to increase access to data by users are already underway and will continue, along with new initiatives. The Division will continue to collaborate with researchers at Agriculture and Agri-Food Canada and in academia to promote the use of Canadian agriculture and rural data. The Division continues to add series to CANSIM so that more data can be downloaded quickly and easily and at relatively low cost, rather than forcing potential users to rely on special tabulations and extractions. The Census of Agriculture plans to continue to make a wide array of data available free one year after Census Day, with the first release of 2011 data in May 2012.

Statistics Canada thanks participants for their participation in this consultation. Their insights guide the Agency's web development and ensure that the final products meet users' expectations.

Consultation summary

This document contains excerpts from the 2009 Quadrennial Program Report for the Demography Statistics Program, covering the 2005/06 – 2008/09 reference period. Statistics Canada has established an integrated program evaluation and reporting system. In that system, statistical programs undergo a full program evaluation every four years. A major component of the evaluation is the extent to which existing statistical products and services continue to meet the evolving needs of clients.

Results

In the coming years, the Division will focus more on the following strategic elements:

• improving the coherence and quality of the Agency's demographic estimates;
• producing population projections on a more regular basis to be in a better position to respond to requests for custom projections.

External consultation – Client Satisfaction Survey

The standard Statistics Canada questionnaire for assessing client satisfaction was used to conduct the Demography Division survey. This questionnaire is designed to obtain importance and satisfaction ratings of products for four of the six dimensions of the quality assurance framework, namely relevance, timeliness, accessibility and interpretability. This standardized content is also used to obtain feedback regarding client services.

The survey was administered in May 2009 to 33 provincial and territorial focal points (that is, senior and assistant focal points) and 143 external clients. The response rates were respectively 46% and 28%. With these rates including partial responses, there were generally very low counts for many of the items measured in the survey. This complicates the task of evaluating the results. Subject to these qualifications, the survey results yield the following general findings:

• On the whole, the clients interviewed are fairly satisfied with Demography Division products with scores around 4 out of 5 for the overall assessment of products.
• External clients, like provincial and territorial focal points, showed themselves to be fairly satisfied with the relevance, interpretability and accessibility of products (see Charts 2 and 3). However, some reservations are expressed with respect to the timeliness of products. Average scores for this aspect of quality were 3.5 for focal points and 3.8 for external clients, the lowest scores for all quality indicators.
• The importance assigned to products and the various aspects of quality measured for each product was generally quite high. While the scores obtained for importance are almost always higher than those obtained for satisfaction, the satisfaction/importance ratio is high, often nearly 100% and never below 80%. A low ratio would indicate a problem requiring some improvements.
• A number of written comments were provided relating to the different questions. Comments from provincial and territorial focal points mainly refer to the good relations maintained with Demography Division. Comments from other clients express dissatisfaction with the timeliness of products and their accessibility, especially as regards the performance of the search engine on the StatCan website.
• External clients, like the focal points, said they were generally satisfied with client services (e.g., speed of initial follow-up, speed of service, accessibility and overall quality of service delivery). However, they expressed a desire to be informed on an ongoing basis about upcoming releases and any sporadic delays that might occur in releases of regular products.
• Focal points were asked about the process of validation of the results of census coverage studies. A high level of satisfaction was obtained for this process. In particular, the satisfaction/importance ratio was found to reach or even exceed 100% for the management of meetings, the quality of presentations and the choice of topics on the agenda. By contrast, the dates on which documentation became available obtained a ratio of 71%, the lowest for any component of the process.

Apart from the Client Satisfaction Survey, the relevance of the program, products and services of the Division is evaluated regularly through consultations with its partners and users. This periodic consultation is conducted by way of different committees and working groups. Also, the increase over time in the number of downloads and hits on divisional products gives some indication of their relevance. Requests for cost recovery projects, such as projections of minority ethnocultural groups and projections of Aboriginal populations, also show the relevance of products and services.

Statistics Canada thanks participants for their participation in this consultation. Their insights guide the Agency's web development and ensure that the final products meet users' expectations.

Background

This document is a summary report on consultations held with visitors to the Statistics Canada website (www.statcan.gc.ca) as part of the agency's ongoing commitment to continuous improvement of the site's Search tool.

Since 2006, many client consultation usability tests have been conducted focusing on different search features including: Search and navigation (2006), Faceted search (2007), card sorting to determine types of information (2008), Search by category (2008) and Search – Try also and Suggested links (2009, phase one).

Summary

In December 2009, the second research phase of the Search – Try also and Suggested links usability testing was conducted to verify the usability of the changes that were implemented as a result of the phase one consultation. The web pages that were tested were as follows:

• the Home page of the Statistics Canada website
• the Search results page
• the Advanced search page

Results

The majority of the changes that were implemented based on the results of phase one of this research were welcomed by the phase two participants.

Action plan

Participants' feedback from phase two was analysed and many of the recommendations are being implemented.