Audit of Pathfinder Projects

Audit Report

November 2018
Project Number: 80590-108

• Executive summary
• Conformance with professional standards
• Introduction
  • Background
  • Audit objectives
  • Scope
  • Approach and methodology
  • Authority
• Findings, recommendations and management response
  • Pathfinder project contributions to modernization
• Appendices
  • Appendix A: Audit criteria
  • Appendix B: Pathfinder project alignment with the pillars of modernization
  • Appendix C: Initialisms

Executive summary

In 2017, the Chief Statistician announced an Agency-wide modernization program recognizing that: "Statistics Canada has reached a point where not only do Canadians and businesses want to be informed, but they also want to get their hands on high-quality, real-time information to make informed decisions. If we can't provide the information that Canadians need when they need it, they will turn to other sources. And, unfortunately, not all produce the same high-quality statistics as Statistics Canada. So, to help Canadians and businesses make evidence-based decisions and stay relevant, we must modernize our programs, practices and methods."

The Statistics Canada modernization program is broad-based in its approach. It encompasses client relations, workplace culture, and the methods, policies and practices followed for collecting and producing statistics. The principles guiding the Agency's modernization are reflected in its five pillars of modernization: user-centric service delivery; leading-edge methods and data integration; statistical capacity building and leadership; sharing and collaboration; and modern workforce and flexible workplace.

As an early focus of the modernization program, the Agency identified four "pathfinder" projects with expectations that they contribute new and innovative thinking, as well as experimental approaches to program delivery:

1. Towards Measuring Cannabis – This project is focused on establishing data collection and measurement mechanisms where none previously existed to provide insight into cannabis usage, its social and economic impacts, and a variety of other factors.
2. Transition to a Low Carbon Economy – This project provides information on the role of clean technology as the country transitions to a carbon-free economy.
3. Measuring Growth in International Visitors to Canada – This project seeks to use administrative data, rather than survey data, to better track tourism in Canada.
4. Canadian Housing Statistics Program – This project is being developed in response to the Government of Canada's demand for a single, integrated, comprehensive source of information on housing.

The objectives of the audit was to provide the Chief Statistician and the Departmental Audit Committee with early insight and assurance that pathfinder projects are systematically contributing to the broader expectations for agency modernization and provide lessons learned to support modernization.

Why is this important?

The pathfinder projects are expected to make an important contribution in shaping the Agency's modernization agenda. In a July, 2017, communication, the Chief Statistician stated: "To lead the modernization initiative, we are moving forward with four pathfinder projects. The valuable lessons learned from these projects will help us further define and refine the modernization plan."

Key findings

The Modernization Management Committee (MMC) was established to oversee the Agency's modernization. The MMC has been active in directing many of the modernization activities throughout the Agency, but has not yet played an active role in overseeing the pathfinder projects.

Pathfinder project leads maintain their own individual approaches in managing project risks. However, a coordinated, horizontal process is not yet in place to support the identification, assessment and mitigation of risks to the achievement of pathfinder project expectations for modernization.

Many beneficial lessons in modernization have been learned as a result of the pathfinder projects. The Agency would benefit from a timely, coordinated, and horizontal process to identify and analyze the lessons learned from the pathfinder projects. Instituting such a process would help guide future pathfinder projects and would serve to support the Agency's modernization plan.

Pathfinder projects can demonstrate alignment to the Agency's pillars of modernization, but it is unclear whether they are meeting senior management's specific expectations in relation to modernization. Clear expectations would help project staff focus their work and would permit more effective monitoring of project performance to ensure that the pathfinder projects are contributing adequately to the Agency's modernization initiative.

The Agency's internal services are engaged with, and support, pathfinder project leaders in determining the needs of pathfinder projects. However, challenges remain for internal services to fully meet pathfinder project needs. Of note, balancing policy requirements and the rigour of internal service processes with the pathfinder projects' need for agility and speed presents difficulties. Further, as the volume of modernization activity increases, the type of service currently being provided to pathfinder projects will not be sustainable without realigning how internal services are delivered or finding ways to discontinue other work.

Overall conclusion

Pathfinder projects have demonstrated clear alignment with the Agency's pillars of modernization. However, the Agency's broader modernization initiative would benefit from improved horizontal oversight of the pathfinder projects. This includes the following: establishing clear expectations for modernization; challenging whether the projects are contributing adequately to the broader Agency modernization initiative; managing risks common across the projects; and identifying and assessing the lessons learned relevant to modernization.

Conformance with professional standards

The audit was conducted in accordance with the Mandatory Procedures for Internal Auditing in the Government of Canada, which includes the Institute of Internal Auditors' (IIA) International Standards for the Professional Practice of Internal Auditing.

Sufficient and appropriate audit procedures have been conducted and evidence has been gathered to support the accuracy of the findings and conclusions in this report, and to provide an audit level of assurance. The findings and conclusions are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria. The findings and conclusions are applicable to the entity examined, and for the scope and period covered by the audit.

Steven McRoberts
Chief Audit and Evaluation Executive

Introduction

Background

In 2017, the Chief Statistician announced a broad, Agency-wide modernization program, recognizing that: "Statistics Canada has reached a point where not only do Canadians and businesses want to be informed, but they also want to get their hands on high-quality, real-time information to make informed decisions. If we can't provide the information that Canadians need when they need it, they will turn to other sources. And, unfortunately, not all produce the same high-quality statistics as Statistics Canada. So, to help Canadians and businesses make evidence-based decisions and stay relevant, we must modernize our programs, practices and methods."

Modernizing Statistics Canada draws on the understanding that innovative solutions to economic, social, health and educational challenges necessitate an ability to absorb, integrate, leverage, and act upon the ever-increasing mounds of data that flow from the connectivity of the modern world. Clients want timely, useful, and insightful statistical products based on accurate and detailed analytical information. They want data that are clear, current, and relevant to their business needs. To this end, the modernization program is broad-based in its approach. It encompasses client relations, workplace culture, and the methods, policies and practices followed for collecting and producing statistics. Specifically, modernization is expected to

• provide more timely and responsive statistics, ensuring Canadians have the data they need when they need it
• provide leadership and stewardship of the Government of Canada’s data assets, improving and increasing alignment and collaboration with counterparts at all levels of government, as well as private-sector and regulatory bodies, to create a whole-of-government, integrated approach to collecting, sharing, analyzing and using data
• raise awareness of Statistics Canada’s data and provide seamless access to them
• develop and release more granular statistics to ensure Canadians have the detailed information they need to make the best possible decisions
• contribute to modern and flexible operations and cost-effective product and service delivery.

These expected results are reflected in the following five pillars of modernization, used to guide the Agency's modernization activities:

1. User-centric service delivery: Users have the information / data they need, when they need them, in the way they want to access them, with tools and knowledge to make full use of them. Partners and stakeholders use the information provided by Statistics Canada to make informed decisions. This mode of delivery results in a lower response burden for Canadians.
2. Leading-edge methods and data integration: Statistics Canada has access to new or untapped data; the role of surveys is modified; and greater reliance is placed on modelling and integration capacity through a research-and-development environment. Modern approaches are utilized to disseminate data and metadata by providing data through alternate mechanisms that are in line with today's technology.
3. Statistical capacity building and leadership: Whole-of-government, integrated approach to the collection, sharing, analysis and use of data is established; Statistics Canada is the leader in fostering savvy information and critical analysis skills; Statistics Canada increases statistical literacy among Canadians and facilitates the responsible use of data for decision making.
4. Sharing and collaboration: Programs and services are delivered taking a coordinated approach with partners and stakeholders. Partnerships allow for open sharing of data, expertise and best practices. Barriers to accessing data are removed. While privacy and confidentiality remain protected, the number of suppressions in data products are minimized.
5. Modern workforce and flexible workplace: The organization is agile, flexible and responsive to client needs. Statistics Canada has the talent and environment required to fulfill current business needs, and remains open and nimble in order to adapt to future demands.

In initiating its modernization agenda, the Agency selected the four pathfinder projects on the basis of their alignment with the expected outcomes of modernization. It is anticipated that each of these projects will contribute new and innovative thinking, as well as experimental approaches to program delivery. In announcing the pathfinder projects in July 2017, the Chief Statistician stated: "To lead the modernization initiative, we are moving forward with four pathfinder projects. The valuable lessons learned from these projects will help us further define and refine the modernization plan." In the future, additional pathfinder projects are expected to be chosen, and it is expected that pathfinder projects will eventually cease to exist and that all Agency business will be conducted in this new, modernized way.

Each of the pathfinder projects was already under way prior to the announcement of Agency modernization. Further, two of the projects were supported by previously established commitments and funding defined in approved Treasury Board submissions, with funding commencing in 2017/2018.

The four pathfinder projects are the following:

1. Towards Measuring Cannabis – Launched in anticipation of the legalization of cannabis in 2018, this project seeks to identify information needs and develop different ways of gathering and processing data. This project is focused on establishing data collection and measurement mechanisms where none previously existed with a view to providing insight into cannabis usage, its social and economic impacts, and a variety of other factors.
2. Transition to a Low Carbon Economy – Initiated through funding set out in Budget 2017, this project provides information on the role of clean technology as the country transitions to a carbon-free economy. This project is focused on generating insight into the clean-technology industry, including developing a survey of producers of clean-technology goods and services; integrating sector and economic statistics in a clean-technology satellite account; improving energy statistics on renewable electricity generation; and developing statistics on demand for clean-technology.
3. Measuring Growth in International Visitors to Canada – Supported by three years of funding from Budget 2017, this project seeks to use administrative data, rather than survey data, to better track tourism in Canada. The focus of the project includes: enhancing existing data collection tools and strategies; integrating alternate data to support the statistical needs of the program; partnering to provide tourism data for each of the three territories; producing Tourism Satellite Accounts for the provinces and territories every three years; and designing a user-centric dissemination strategy for analytical products on tourism data.
4. Canadian Housing Statistics Program – Supported by five years of funding from Budget 2017, this project is being developed in response to the Government of Canada's demand for a single, integrated, comprehensive source of information on housing. This program will leverage administrative data to expand the agency's data sets in order to generate better insight into national housing statistics, including geography, property characteristics, ownership characteristics, property use and property financing.

Audit objectives

The objective of the audit was to provide the Chief Statistician and the Departmental Audit Committee with early insight and assurance that pathfinder projects are systematically contributing to the broader expectations for agency modernization and provide lessons learned to support modernization.

The audit focused on the following lines of enquiry:

• Effective oversight;
• Modernization outcomes; and
• Internal services support.

The audit also examined the extent to which elements of the modernization pillars have been implemented within the various pathfinder projects, whether successes were replicated across the Agency, and the nature of lessons learned obtained from the projects.

Scope

The audit scope included an examination of the modernization oversight structures, implementation of modernization outcomes, and the extent of support from internal services, as they relate to the four pathfinder projects: Towards Measuring Cannabis; Transition to a Low Carbon Economy; Measuring Growth in International Visitors to Canada; and Canadian Housing Statistics Program. However, the audit did not assess the progress pathfinder projects have made against their operational objectives. The audit's focus was solely on the modernization-related aspects of the pathfinder projects.

Approach and methodology

The audit work consisted of an examination of documents, interviews with key senior management and Agency personnel, and a review of compliance with relevant Statistics Canada guidance (see Appendix A: Audit criteria for details).

Specifically, the examination phase included the following:

• Structured interviews with pathfinder project leadership and team members, key members of modernization and project governance / oversight bodies, and representatives from Agency internal service organizations;
• Review of selected Agency guidance on modernization, such as presentations, bulletins, and other key documents communicating the strategic direction of modernization activities;
• Pathfinder project documentation, including documents supporting project planning, project execution, and progress reporting; and
• Review of selected governance or oversight body meeting agendas, minutes, records of decision, and related documentation.

Authority

The audit was conducted under the authority of the approved Statistics Canada Integrated Risk-based Audit and Evaluation Plan 2018/2019 to 2022/2023.

Findings, recommendations and management response

Pathfinder project contributions to modernization

Oversight of pathfinder project modernization

As stated by the Chief Statistician, pathfinder projects are expected to help further define and refine the Agency's modernization efforts. To support this alignment, clear expectations with respect to modernization should be established for each of the pathfinder projects. Further, a structure and supporting process should be in place to provide oversight and guidance for the achievement of these expectations.

The Modernization Management Committee (MMC) was established in 2017 to oversee Agency modernization. This new committee has been active in obtaining funding for, and directing, many of the modernization activities throughout the Agency, but has not yet played an active role in overseeing the pathfinder projects.

Individual governance structures have been established to oversee each of the pathfinder projects. These structures, in the form of committees and working groups, typically include representation from external stakeholders, and are generally focused on supporting the project's operational objectives. Consequently, by design, they have minimal focus on assessing the project's impact on the Agency's modernization initiative, and do not take a horizontal, Agency-wide view of the potential issues, challenges, or lessons learned common across all four of the pathfinder projects.

The Agency established the MMC in 2017. This senior management committee was made responsible for the overall governance, direction setting, and planning of the modernization initiative and associated projects, including the pathfinder projects. The MMC is supported by a Modernization Secretariat and supports the Agency's Senior Management Committee (SMC). Among its many responsibilities, the MMC's Terms of Reference states that it is responsible for

• engaging in dialogue about the direction of the modernization strategy, vision and supporting projects to ensure that they are strategically aligned with modernization principles and pillars, and that duplication between projects is avoided; and
• where resources are constrained, developing the priorities for modernization projects.

MMC is also expected to provide strategic direction on modernization, including aspects of culture change, communications, results, human resources and people elements, and strategic enablers, such as information technology.

We found that MMC has been active in supporting the Agency's modernization efforts by making funding decisions, developing a Memorandum to Cabinet, and directing numerous modernization activities throughout the Agency. However, as a result of these competing priorities, the MMC has not been able to devote very much time to providing oversight and strategic direction for the modernization activities of the pathfinder projects. Notably, SMC discussed two of the pathfinder projects, but the discussions were focused on project status updates for information purposes, and did not include an examination of the projects' contributions to Agency modernization.

Although the five pillars of modernization and their related expected outcomes have been widely communicated across the organization, they provide only broad, strategic-level direction. Given that the pathfinder projects were under way when the modernization initiative was announced, more specific, project-level expectations for modernization were not defined. To illustrate, while Agency-wide communications made clear that modernization should result in more timely statistics for users, the MMC's specific expectations in this regard for the pathfinder projects were not defined. For example, what does the MMC consider to be "timely statistics" in relation to the Cannabis project? Is it monthly, weekly, daily, or some other frequency? Clear expectations such as these help project staff focus their work and permit more effective monitoring of project performance to ensure that pathfinder projects are contributing adequately to the Agency's modernization initiative.

As the pathfinder projects are expected to inform the Agency's modernization initiative, horizontal oversight processes and practices are required to establish clear expectations and systematically link project results to the broader Agency modernization initiative. Without it, the Agency risks not maximizing the contributions of pathfinder projects to the modernization initiative.

Managing risk

Pathfinder projects are meant to push the Agency's boundaries by attempting new ways of doing things. While these new ways can lead to better and more efficient services to Canadians, they may also introduce new risks for the Agency. Horizontal processes will be required to help identify, assess and mitigate the new risks introduced by modernization.

Pathfinder project leads maintain their own individual approaches in managing project risks. However, the Agency would benefit from a coordinated, horizontal process to collectively identify, assess and mitigate risks to the expected outcomes of modernization for the pathfinder projects.

Another of the MMC's responsibilities outlined in its Terms of Reference is to play a guiding role in risk management and to support the design of interventions when necessary in relation to modernization expectations. As the MMC has had limited oversight of pathfinder projects to date, there has been minimal focus on systematically identifying, assessing and mitigating the horizontal risks that arise from the modernization aspects of the pathfinder projects. As a result, opportunities may be missed in terms of coordinating risk mitigation efforts across pathfinder projects or the Agency as a whole.

To date, three of the four pathfinder projects have established regular risk reporting (of risks to operational objectives, rather than those related to modernization) through a monthly project dashboard supported by each project's assigned Departmental Project Management Office (DPMO) representative. The Transitioning to a Low Carbon Economy project has not yet initiated similar risk management reporting as DPMO, and project representatives indicated that this project is not sufficiently mature to warrant this level of reporting.

While no evidence of a horizontal risk management process was observed, one example of a pathfinder project undertaking a broader assessment of risk was noted. More specifically, a risk case study was developed in relation to the launch of a new Agency capability (i.e., measurement of cannabis use, pricing, etc.). While not specific to modernization, the case study included the identification of risks and mitigation approaches that touch upon the broad risk impacts of implementing new and innovative approaches, a challenge common to all pathfinder projects.

The following are some of the risks identified in the case study:

Risk example	Risk mitigation examples
Risk of poor data quality – Risk of applying new and untested methods, that is, data produced could not be analyzed using the same traditional methods and quality assurance standards as would normally be applied to Statistics Canada products.	Active engagement of internal stakeholders (including methodologists) through the initiative's formal governance mechanisms. Strong tone at the top (from senior management) to take calculated risks.
Risk of negative public reaction – Risk associated with the credibility of the project's use of innovative and "imperfect" data methods and how this might cause external audiences to call into question the credibility of other, more critical products produced by the Agency.	Proactive engagement of media to explain methods and the value to be realized while controlling the Agency's message.

While this example provides useful insight into some of the modernization risks and mitigation approaches associated with one pathfinder project (Cannabis), we also view these risks as relevant to all pathfinder projects. There is, however, no recognized process in place either to collectively assess the relevance of these approaches or to consider the application of these risk management approaches to other pathfinder projects or to broader Agency modernization efforts. As a result, each of the pathfinder projects is identifying and addressing risks in their own way. This has the potential to duplicate efforts, especially where there are common risks. Further, the Agency may be at risk of not effectively learning from the risk mitigation approaches taken by specific pathfinder projects to inform its broader modernization initiative.

Lessons learned

In launching the pathfinder projects, the Chief Statistician stated that "The valuable lessons learned from pathfinder projects will help us further define and refine the modernization plan." Systematic, coordinated and horizontal processes should be in place to identify, assess, and benefit from the lessons learned on modernization from the pathfinder projects.

Many beneficial lessons in modernization have been learned as a result of the pathfinder projects. The Agency would benefit from a timely, coordinated, horizontal process to identify and analyze the lessons learned from the pathfinder projects. Doing so would help guide future pathfinder projects and the Agency's modernization plan.

Interviews with pathfinder project leads identified several lessons learned, some of which touched on key aspects of the Agency, including organizational culture, employee health and well-being, project structure, business processes, and information technology infrastructure considerations. However, our interviews with MMC committee members and pathfinder project representatives confirmed that a coordinated and horizontal review of pathfinder project lessons learned has not yet been undertaken in relation to modernization. Those interviewed further confirmed that it would be valuable to the Agency to perform such a review, although the timing and responsibility for doing so has not been established.

As one example, pathfinder project leads collectively highlighted a range of lessons learned in relation to the Agency's approach to working with the private sector in accessing alternative sources of data:

• Establishing an understanding of "what's in it for them" before approaching an outside organization;
• Engaging the organization at a senior management level, while recognizing that multiple levels of engagement / contact may be required;
• Relationship building to foster trust. There needs to be a recognition that an organization may not trust the Agency with sensitive data in the early stages of a relationship, that is, "don't ask for too much, too soon";
• Respecting the organization's response timeframes; and
• Having Agency subject matter expertise involved in detailed data discussions with an organization.

While these examples do not provide an exhaustive summary of all the lessons learned, they do provide a cross section of the beneficial learnings that can accrue from a broader, systematic review of pathfinder projects. A timely, coordinated, horizontal process to identify and analyze the pathfinder project lessons learned would help guide future pathfinder projects and the Agency's broader modernization plan.

Alignment with the five pillars of modernization

Pathfinder projects are expected to lead the way in the Agency's modernization initiative by pushing the boundaries and testing new and innovative solutions that might benefit future modernization efforts. Consequently, management must ensure the projects align with the five pillars of modernization, and that they adequately contribute to the Agency's broader modernization efforts.

Pathfinder projects are aligned with the Agency's pillars of modernization, but it is unclear whether they are meeting management's expectations. The Agency would benefit from more clearly defining management's expectations for modernization within each pathfinder project and from monitoring performance in this regard to ensure the projects contribute sufficiently to defining and refining the modernization plan.

According to our review of project documentation and interviews with project representatives, it is evident that the pillars of modernization are being applied, to varying degrees, in each of the pathfinder projects. To illustrate, some examples of "leading-edge methods and data integration"^{Footnote 1} found within each of the pathfinder projects are as follows:

• The Towards Measuring Cannabis project included the launch of an online statistical application that generates crowdsourced cannabis price data from Canadians in every province and territory; leveraging social media data to measure prevalence of use; and piloting the measurement of cannabis concentrations in waste water as a means of establishing prevalence of use.
• The Transitioning to a Low Carbon Economy project included the establishment of a Clean-Technology Satellite Account, employing alternate administrative data sources, and greater use of geospatial analysis.
• The Measuring Growth in International Visitors to Canada project includes transition from a survey-based program to one where survey data is supplemented with alternative data sources, such as Canada Border Services Agency traveler data and traveler credit card data.
• The Canadian Housing Statistics Program includes leveraging alternative data sources, such as the municipal property database, to support implementation of the Housing Statistics Framework.

For further examples, Appendix B outlines how each pathfinder project aligns with the five modernization pillars.

While it was evident that pathfinder projects are engaged in activities aligned with the modernization pillars, as previously noted, clear expectations for modernization for each projects were not established. Further, the pathfinder projects have not been subject to a comprehensive analysis or challenge of their modernization activities to ensure that they are meeting management's expectations and adequately support the modernization effort.

One of the established roles of the MMC is to engage in dialogue about the strategic direction with respect to the modernization strategy, vision and supporting projects to ensure that they are strategically aligned with modernization principles and pillars, and that duplication between projects is avoided. As previously noted, the MMC has been focused on the Agency's many other modernization activities and has not yet performed this challenge function for pathfinder projects. Performing this challenge function would help ensure that the pathfinder projects have their intended effect of defining and refining the Agency's modernization plan.

Internal service support for pathfinder projects

The internal service needs of pathfinder projects vary by project, but typically include Information Technology, Project Management, Administrative Data, Methodology, Human Resources, and Communications services. In supporting pathfinder projects, internal services should be actively engaged with their pathfinder project clients in determining and responding to current and future project needs.

The Agency's internal services are engaged with and support pathfinder project leaders in determining the needs of pathfinder projects. However, challenges remain for internal services to fully meet these needs. Of note, internal services expressed difficulties in balancing policy requirements and the rigour of their processes with the pathfinder projects' need for agility and speed. Further, as the volume of modernization activity increases, the type of service currently being provided to pathfinder projects will not be sustainable without realigning how internal services are delivered or finding ways to discontinue other work.

Interviews with internal service representatives confirmed that the approach they are taking with respect to serving the needs of pathfinder projects is not significantly different from the approaches taken with respect to serving other Agency clients. However, some examples were noted of internal services introducing changes to their service approaches to be more responsive to the needs of clients. These include the following:

• Administrative Data Division (ADD) has a manager assigned to address the data acquisition needs of the pathfinder projects. ADD is also represented on specific pathfinder steering committees.
• Informatics Branch has assigned a Business Relationship Manager to each pathfinder project to provide a single point of contact to the IT organization and to better identify and coordinate the IT service response to project needs.
• For two of the four pathfinder projects (Cannabis and Tourism), DPMO has assigned and embedded a project manager on the project team to take a more hands-on project management role versus the typical DPMO approach of being a project management advisor to projects. In addition, DPMO has introduced a less onerous version of its project planning template to support experimental-type projects, versus those projects where business requirements are clearly defined.
• While not focused on the needs of pathfinder projects, Human Resources has assigned a business partner to each of the Agency's fields (led by an Assistant Chief Statistician) to better serve these internal clients.

These examples are illustrative of the efforts made by the Agency's internal services to be responsive to the needs of pathfinder projects. That being said, feedback from the pathfinder project staff was mixed as to whether these changes have been successful in improving the service provided by internal services.

While internal services are actively engaged in efforts to support the delivery of pathfinder projects, several significant challenges were identified that warrant additional consideration in the broader context of the Agency's modernization initiative. To illustrate, we noted several challenges associated with balancing the rigour and policy requirements of internal services with the pathfinder project need for agility and speed of service. In one instance, the introduction of new alternative data sources created challenges in balancing the rigorous data quality focus of internal services (such as ADD and Methodology) with the project's need to produce real-time information that, while possibly imperfect, would still have value to Agency clients.

Internal service leads also expressed concern regarding the sustainability of continuing to provide immediate, prioritized service to modernization projects as the volume of modernization activity increases. This has also led to a concern around the well-being of employees, as they are being asked to work longer hours and produce deliverables in short time frames. They further reported that these challenges were compounded by a lack of mechanisms in place to appropriately plan and fund the resource requirements necessary to service pathfinder project requests, while maintaining service for ongoing Agency business. These concerns take on additional importance with the next round of pathfinder projects being planned, and this new way of conducting business is envisioned as the future for the Agency.

While not a complete summary, other challenges and questions pertaining to internal service support that could have broader implications for Agency modernization include:

• How to satisfy Government of Canada accessibility standards in relation to pathfinder project implementation of new data visualization tools and information;
• How to address Agency Information Technology infrastructure limitations, including limitations in the Information Technology tools available to pathfinder projects to support the collection / transfer and storage of large data sets acquired to support project needs;
• How to address Agency Information Technology infrastructure limitations impacting user / client access to pathfinder project datasets;
• How to optimally support pathfinder project teams with internal service resources. There was recognized value by some pathfinder projects from embedding methodologists onto their project teams versus the typical centralized model of Methodology service delivery. This model was viewed as allowing the methodologists to gain greater insight into project needs and challenges to more effectively support the project; and
• How to establish the optimal approach, and corresponding Agency roles and responsibilities, for acquiring alternative sources of data to support project needs. ADD has been established as a centralized custodian of Agency administrative data and provides support in meeting the data acquisition needs of pathfinder projects. Questions were raised as to how best to leverage the roles of ADD and the subject matter expertise of project representatives to effectively and efficiently acquire the specialized and alternative data sources being generated for pathfinder projects.

These examples provide an indication of some of the internal service challenges arising from the pathfinder projects. Resolving these challenges will require a coordinated, horizontal approach with senior management support, as they will have broader effects across the organization. Areas requiring change may include the Agency's culture, the organizational structure, business processes, and the supporting infrastructure.

Recommendations

The Co-chairs of the Modernization Management Committee, supported by the DG of Modernization, should ensure that MMC supports pathfinder projects^{Footnote 2} by:

1. Providing appropriate guidance on the specific expectations for modernization as they pertain to individual pathfinder projects;
2. Providing an effective challenge function in relation to pathfinder project progress towards their expected modernization results;
3. Providing a forum for the review and consideration of modernization risks and opportunities stemming from pathfinder projects, and their impact on the Agency's broader modernization initiative; and
4. Undertaking a coordinated, horizontal pathfinder project lessons learned exercise, as it pertains to modernization.

Management response

Management agrees with the recommendations.

A common baseline approach will enable the MMC to clearly communicate management's expectations for modernization:

• The MMC will review, finalize and endorse, as needed, modernization end-state objectives, project assessment criteria and frameworks, and key performance indicators (KPIs) to guide decisions.
• Going forward, the MMC will require all project proposals to demonstrate alignment with priorities and pillars, culture change, and contribution to modernization objectives.
• The MMC will prioritize sharing these measures with all staff to facilitate wider understanding of expectations for modernization.
• In the coming year, the MMC will oversee the corporate modernization review of all active statistical programs.

The MMC will receive updates on a quarterly basis, starting immediately, on all four pathfinder projects. The MMC will finalize and endorse, as needed, modernization end-state objectives, project assessment criteria and frameworks, and KPIs to guide its decisions. These measures will contribute to the continuous refinement and evolution of Statistics Canada's modernization objectives, programs and pathfinders.

Pathfinder project health checks will be held regularly at the MMC and will include consideration of milestones achieved, risks and opportunities from modernization perspectives, and broader implications for the modernization agenda writ large.

Building on the director general engagement session of spring 2018, the Director General of Modernization will continue to leverage different forums to share progress and lessons learned. The Director General of Modernization will also develop a standard template to facilitate analysis, documentation and communication of lessons learned after implementation. These measures will contribute to the continuous refinement and evolution of Statistics Canada’s modernization objectives.

Appendices

Appendix A: Audit criteria

Appendix A: Audit criteria

Control objectives / Core controls / Criteria	Sub-criteria	Policy instruments / Sources
Objective: To provide the Chief Statistician and the Departmental Audit Committee with early insight and assurance that pathfinder projects are systematically contributing to the broader expectations for agency modernization and provide lessons learned to support modernization.
1.1 There is appropriate and effective oversight to support the achievement of pathfinder project modernization objectives.	1.1.1 A structure and supporting processes and practices exist to provide guidance in the achievement of pathfinder project modernization objectives. 1.1.2 A process exists to identify, assess and mitigate the risks to the achievement of pathfinder project modernization objectives. 1.1.3 A coordinating process exists to ensure the timely identification of lessons learned with respect to the pathfinder projects, in support of the modernization initiative.	Office of the Comptroller General (OCG) Audit Criteria related to Management Accountability Framework (MAF): A Tool for Internal Auditors—Accountability, AC-1, AC-4; Governance and Strategic Directions, G-1, G-2, G-3, G-6; Risk Management, RM-1, RM-2, RM-3, RM-4, RM-5, RM-6, RM-7; Learning, Innovation and Change Management, LICM-1
2.1 Pathfinder projects systematically contribute towards the Agency's modernization efforts.	2.1.1 Pathfinder projects apply, where applicable, the principles of the modernization pillar user-centric delivery service. 2.1.2 Pathfinder projects apply, where applicable, the principles of the modernization pillar leading-edge methods and data integration. 2.1.3 Pathfinder projects apply, where applicable, the principles of the modernization pillar statistical capacity building and leadership. 2.1.4 Pathfinder projects apply, where applicable, the principles of the modernization pillar sharing and collaboration. 2.1.5 Pathfinder projects apply, where applicable, the principles of the modernization pillar modern workforce and flexible workplace.	OCG Audit Criteria related to MAF: A Tool for Internal Auditors— Results and Performance, RP-1, RP-2, RP-3; Learning, Innovation and Change Management, LICM-2
3.1 Agency internal services effectively support the implementation of the pathfinder projects.	3.1.1 Agency internal services are engaging with clients in determining the current and future needs of pathfinder projects. 3.1.2 Agency internal services are responsive to, and are effectively supporting, the achievement of both the project and modernization objectives.	OCG Audit Criteria related to MAF: A Tool for Internal Auditors— Stewardship, ST-21

Appendix B: Pathfinder project alignment with the pillars of modernization

Presented in this appendix are examples of how each pathfinder project is aligned with the Agency's five pillars of modernization. These examples are not intended to represent an exhaustive list. Rather, they constitute a summary of the key modernization activities under way within each project.

Towards Measuring Cannabis

Modernization Pillars
User-centric Delivery Service	Leading-edge Methods and Data Integration	Statistical Capacity Building and Leadership	Sharing and Collaboration	Modern Workforce and Flexible Workplace
Development of a Cannabis Statistics Hub which packages all cannabis-related statistics in a single online resource, available via the Agency website. Working with Health Canada and provinces/territories to lower response burden by designing regulatory forms so information is only collected once.	Launch of an online statistical application which generates crowdsourced cannabis price data from Canadians in every province and territory. Launch of a quarterly Rapid Response Household Survey on cannabis use and expenditures. Leveraging social media data to measure prevalence of use. Piloting the measurement of cannabis concentrations in waste water as a means of establishing prevalence of use.	Working with Health Canada and the provinces and territories to: Ensure newly developed regulatory forms meet statistical needs and information is only collected once. Using Agency survey collection infrastructure to collect regulatory information.	Working with Health Canada and the provinces and territories to: Ensure newly developed regulatory forms meet statistical needs and information is only collected once. Using Agency survey collection infrastructure to collect regulatory information.	Established a project team to respond to high priority information need (impact of Cannabis) with no dedicated source of project funding

Transition to a Low Carbon Economy

Modernization Pillars
User-centric Delivery Service	Leading-edge Methods and Data Integration	Statistical Capacity Building and Leadership	Sharing and Collaboration	Modern Workforce and Flexible Workplace
Development of a measurement framework to articulate clean-tech statistical reporting outputs that support a broader view in support of the Clean Tech Data Strategy (GoC focus)	Development of a Clean-Tech Satellite Account	Whole-of-government approach in developing a measurement framework to articulate clean-tech statistical reporting outputs (We have not received a copy of the framework yet) that support a broader view in support of the Clean Tech Data Strategy (GoC focus). Working with partners (e.g. Canadian Space Agency) to improve Agency production and dissemination of geospatial information.	Working with NRCan and Innovation, Science and Economic Development (ISED) to better link the Clean Tech Satellite Account and the Natural Resources Satellite Account.

Canadian Housing Statistics Program

Modernization Pillars
User-centric Delivery Service	Leading-edge Methods and Data Integration	Statistical Capacity Building and Leadership	Sharing and Collaboration	Modern Workforce and Flexible Workplace
Implementing a new Housing Statistics Framework to address gaps in nationwide housing data and to address user information needs. Provision of access (e.g. to CMHC) to housing microdata Provision of enhanced housing data on Vancouver and Toronto markets (Release 1 – December, 2017) and British Columbia and Ontario markets (Release 2 – June, 2018) (ref: Housing releases).	Leveraging alternative data sources (e.g. municipal property information) to support implementation of the Housing Statistics Framework	Development of guidelines to facilitate data acquisition. Establishment of communication channels with provincial data providers to support implementation of the Housing Statistics Framework.	Partnering with CMHC in release 1 (December, 2017) of information under the Housing program. Establishment of relationships and arrangements with federal government entities (e.g. CMHC) provincial governments, municipal governments (e.g. Vancouver, Toronto), industry associations and private sector organizations (e.g. Teranet) in supporting the identification of alternative data sources to be leveraged.	Embedded Methodology resources within the project team to support more timely service delivery

Measuring Growth in International Visitors to Canada

Modernization Pillars
User-centric Delivery Service	Leading-edge Methods and Data Integration	Statistical Capacity Building and Leadership	Sharing and Collaboration	Modern Workforce and Flexible Workplace
Production of annual tourism data for the three territories More granular data for the provinces Developed a key indicator and database for daily cross border travel	Transition from a survey-based program to one where survey data is supplemented with new data sources (e.g. credit card data, payment processor data, CBSA traveller data) Exploring the use of machine learning and artificial intelligence as part of its data collection methods in Canada's north.	Piloting the 100 Points of Contact initiative (a partnership with Destination Canada) wherein international tourism data is collected at several tourism hot spots across the country. Provide conceptual and data training to a wide range of tourism data users via the Tourism Newsletter Work with Destination Canada to build statistical models	Working with the territories to exploit alternative data to produce data on tourism in the North Partnering with Destination Canada in the piloting of the 100 Points of Contact initiative wherein international tourism data is collected at several tourism hot spots across the country Secured Canadian passport data from CBSA to improve sampling for the National Travel Survey	Established a section within the Tourism division focused solely on data analysis. Embedded data scientists within the Tourism project team to support requirements related to working with large quantities of alternative data.

Appendix C: Initialisms

ADD

Administrative Data Division

DPMO

Departmental Project Management Office

IIA

Institute of Internal Auditors

KPI

Key performance indicator

MAF

Management Accountability Framework

MMC

Modernization Management Committee

OCG

Office of the Comptroller General

SMC

Senior Management Committee