May 26, 2015
Project Number: 80590-84
- Executive summary
- Introduction
- Background
- Audit objectives
- Scope
- Approach and methodology
- Authority
- Findings, recommendations and management response
- Governance and roles and responsibilities
- Risk and issue management and project interdependencies
- Project baselines including costs and efficiencies estimates
- Project monitoring and outcomes
- Appendices
- Appendix A: Audit criteria
- Appendix B: Acronyms and initialisms
Executive summary
Statistics Canada has a Departmental Project Management Framework (DPMF) in place that identifies the agency's standard project management processes, tools and templates. The DPMF was fully implemented in March 2012 and is mandatory for all projects valued at or above $150,000. It is also highly recommended for projects valued at less than $150,000. This is a key guidance tool for project management within the agency, as it provides context on the project life cycle and the project management deliverables required at these stages.
The Departmental Project Management Office (DPMO) is responsible for supporting the agency's project and portfolio management functions by serving as a resource for project managers and other stakeholders; offering standardized processes, practices and tools across the agency; and supporting professional development for project managers.
The objectives of the audit were to provide the Chief Statistician and the Departmental Audit Committee with assurance that
- the agency has an effective and adequate project management framework to ensure a systematic approach to managing its projects and associated interdependencies in compliance with the Treasury Board Policy on the Management of Projects
- projects are managed in compliance with the DPMF and its related tools, and are monitored against established milestones and expected results.
Key findings
Statistics Canada has a history of successfully delivering projects, and it is assessing ways of improving its business practices. The DPMF, developed by the DPMO, communicates expectations as well as the agency's strategic direction for project management. It outlines the authorities, roles and responsibilities of the DPMO, of key roles within project management teams, of oversight committees and of other key stakeholders.
The DPMO supports the agency's project and portfolio management functions, as well as its governance committees, by serving as a resource; offering standardized processes, practices and tools; and conducting project portfolio management. The audit revealed inconsistencies in governance practices. Combined with project managers who have limited experience or expertise in project management, this results in the DPMO spending a significant amount of time following up with project managers on missing deliverables rather than providing strategic project management support to the agency.
The DPMF recognizes the importance of risk and issue management as part of strong project management, but it currently provides limited guidance regarding the processes for identifying, assessing, communicating and monitoring project risks and interdependencies within the project life cycle. Additionally, the roles and responsibilities of key project team members and the oversight committees with respect to risk management and monitoring project interdependencies have not been clearly defined within the DPMF.
The DPMO has developed and introduced the Change, Issue, Risk Management Tool (CIRMT) to help document and monitor project risks and issues, but its use has not been made mandatory for all projects. Further, at the time of the audit, a project interdependency tool was being piloted only for Corporate Business Architecture projects.
The DPMF currently outlines the requirement for documenting project baselines, including estimated costs and planned efficiencies, at various points in the project life cycle. It also provides templates to be completed by the project team. The assumptions made in determining the cost and efficiency baselines are not consistently being formally documented.
The audit team was unable to assess the reliability and completeness of these baselines or to determine whether all project stakeholders were consulted in determining or reviewing the cost and efficiency assumptions. Amendments made to a project's scope, schedule or cost and efficiency baselines are not consistently being formally documented as expected, which affects the transparency of project amendments—i.e., the number of amendments made to a project and its overall final cost or expected efficiencies.
As outlined in the DPMF, the monthly executive project dashboards are to be prepared and reviewed according to the established governance structure for each project. As the audit team identified weaknesses in the reliability and completeness of the reporting being made through a sample of dashboards, the DPMO is revising its Executive Project Dashboard template to make a project's status more transparent.
Limited evidence is maintained to demonstrate that project oversight committees are consistently reviewing and approving the monthly executive project dashboards in a timely manner.
Expected business outcomes are being documented in the business case; however, in some instances, these outcomes are not sufficiently developed to provide an understanding of the benefits of the project to the agency, nor are they measurable. Without clearly defined measurable outcomes, it is difficult to assess whether a project has achieved what it was expected to.
From the sample of projects tested that had reached completion/close-out, the audit team noted that project managers are not consistently completing a full assessment of the outcomes against the original or revised targets.
Conclusion
The agency, through the DPMO, has developed and implemented a project management framework aligned with the requirements outlined in the Treasury Board Policy on the Management of Projects and encourages a systematic approach to the management of its projects. The establishment of this framework was essential given the significant and complex transformational projects being carried out within the agency.
Greater clarity of roles and responsibilities, guidance for all stakeholders, and mandatory use of CIRMT in the areas of risk management are required to strengthen the ability of governance bodies and key participants to effectively identify, assess, communicate and monitor project risks and issues and interdependencies. These actions will ensure that projects are managed efficiently and effectively and will lead to greater compliance with the DPMF. As well, they will help ensure that project baselines, including cost and efficiency estimates, are established with sound stewardship; that the status of projects is more effectively monitored; and that reporting against expected results is improved.
Conformance with professional standards
The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which include the Institute of Internal Auditors International Standards for the Professional Practice of Internal Auditing.
Adequate and appropriate audit procedures were followed and evidence gathered to support the accuracy of the findings and conclusions in this report and to provide an audit level of assurance. The findings and conclusions are based on a comparison of conditions, as they existed at the time, against pre-established audit criteria. The findings and conclusions are applicable to the entity examined for the scope and time period covered by the audit.
Patrice Prud'homme
Chief Audit Executive
Introduction
Background
As Canada's national statistical office, Statistics Canada has a mandate to serve the statistical needs of all levels of government, Canadian institutions and Canadians. Providing high-quality data that support core economic, environmental and social statistics programs is key to determining how Statistics Canada allocates its funding. In its Departmental Investment Plan, Statistics Canada presented an investment of $512 million on projects (all from within existing reference levels) over the years 2011/2012 to 2015/2016 to meet its operational requirements. The largest project was the 2011 Census of Population and the 2011 National Household Survey, estimated at $360 million.
In April 2010, Statistics Canada created the Departmental Project Management Office (DPMO) to provide leadership, training and support in developing common project management processes and tools to be used by all projects to improve the timely delivery of projects within cost, within scope and to required quality standards.
In April 2011, the DPMO launched the Departmental Project Management Framework (DPMF) and implemented the Treasury Board Policy on the Management of Projects. The framework is a set of standard project management processes, templates and tools used throughout a project's life cycle to initiate, plan, execute, control and close a project.
All of Statistics Canada's projects with total costs of $150,000 and above are required to comply with the DPMF, although it is also recommended for projects valued at less than $150,000. Statistics Canada has based its DPMF on international industry standards, including the Project Management Body of Knowledge Guide, published by the Project Management Institute.
Over the last few years, over 1,000 participants have attended training and information sessions to gradually raise awareness of the DPMF's use and importance.
Statistics Canada's project approval processes are integrated through the DPMF and are aligned with its Integrated Strategic Planning Process (ISPP). The ISPP includes project management governance, performance monitoring and risk management. The ISPP follows an annual calendar of events with clearly identified governance, elements, activities and expected outputs throughout each of the planning phases.
There are six stages within the DPMF: (1) idea generation; (2) project assessment; (3) project initiation; (4) project planning; (5) project execution; and (6) project close-out. When the project is completed, a post-launch review is conducted. The project management deliverables and requirements are defined in the DPMF depending on the Project Complexity and Risk Assessment level of the specific project and its current stage. At each of these stages, key documents are expected to be developed and submitted to the relevant governance bodies for review and gating approval.
Managing projects effectively is key to ensuring that Statistics Canada is providing value-for-money and demonstrating sound stewardship in program delivery. Specifically, a comprehensive approach to managing projects that is integrated across the department and is appropriate to the level of project risk and complexity will enhance the likelihood of realizing project outcomes.
Audit objectives
The objectives of the audit were to assure the Chief Statistician and the Departmental Audit Committee that
- the agency has an effective and adequate project management framework to ensure a systematic approach to managing its projects and associated interdependencies in compliance with the Treasury Board Policy on the Management of Projects
- projects are managed in compliance with the DPMF and its related tools, and are monitored against established milestones and expected results.
Scope
The scope of this audit included examining the adequacy and effectiveness of the governance and monitoring practices that support clear accountability, oversight, scrutiny and challenge functions. It also included examining the effectiveness of risk management processes for identifying the key risks facing projects, including emerging risks, as well as for developing and monitoring risk management strategies.
Lastly, the scope included examining the operational processes and controls that enable the consistent application of a common project management framework as well as the tools, training and information management practices that support informed project management decisions.
The scope of the audit included assessing project management activities from April 1, 2011, to March 31, 2014.
Approach and methodology
The audit work consisted of examining documents; reviewing relevant procedures, guidelines and frameworks related to project management processes; and interviewing key senior management representatives, a sample of project managers, representatives of the DPMO and a sample of oversight committee members. Additionally, the field work consisted of reviewing a sample of project files to assess the consistency of project management activities and compliance with the DPMF.
The selection of a sample of 12 projects was based on a judgmental sampling approach that considered the size, number of adjustments and complexity of each project. The following table presents the breakdown of the sample:
Population of projects Table 1 - Footnote 1 | Sample size | |
---|---|---|
Footnotes
|
||
Project category | 94 | 12 |
Corporate Business Architecture (CBA) | 30 | 5 |
Significant non-CBA Table 1 - Footnote 2 | 17 | 3 |
Non-significant non-CBA Table 1 - Footnote 3 | 47 | 4 |
This audit was conducted in accordance with the Standards for the Professional Practice of Internal Auditing of the Institute of Internal Auditors and the Treasury Board Policy on Internal Audit.
Authority
The audit was conducted under the authority of the approved Statistics Canada integrated Risk-based Audit and Evaluation Plan 2014–2019.
Findings, recommendations and management response
Governance and roles and responsibilities
As Statistics Canada continues to carry out a significant number of large-scale projects as part of its business architecture transformation, it is critical that project management expectations throughout the agency be communicated effectively and consistently. The Departmental Project Management Framework (DPMF), developed by the Departmental Project Management Office (DPMO), is the mechanism used to communicate these expectations as well as the agency's strategic direction for project management. The DPMF further outlines the authorities, roles and responsibilities of the DPMO, of key roles within project management teams, of oversight committees and of other key stakeholders.
The main responsibilities of the DPMO, as defined in the DPMF, are to support the agency's project and portfolio management functions, as well as governance committees, by serving as a resource; offering standardized processes, practices and tools; and conducting project portfolio management. As a result of inconsistencies in the governance practices for past and ongoing projects and given that many project managers have limited experience or expertise in project management, the DPMO continues to spend a significant amount of its time following up with project managers on missing project management deliverables and saving them in the DPMO's repository rather than providing strategic project management support to the agency.
Statistics Canada's strategic direction for project management should be documented and adequately communicated by establishing a framework that clearly defines authorities, responsibilities and accountabilities with respect to initiating, planning, executing and closing-out projects. Additionally, the DPMO should provide employees with the necessary tools, resources, information and training to discharge their responsibilities and achieve expected results.
Strategic direction for project management as well as key roles and responsibilities have been clearly defined and documented in the DPMF.
Strategic direction for project management within the agency has been formally documented and communicated through the DPMF guidelines. Not only does the DPMF define project management for the agency, it has also established a set of standardized project management processes, templates and tools to guide project management activities across the agency. Further, the DPMF has defined the project management deliverables required for each stage of a project: (1) idea generation; (2) project assessment; (3) project initiation; (4) project planning; (5) project execution; and (6) project close-out.
The audit noted that the DPMF has clearly defined the roles, responsibilities and authorities of the project team, including those of the project manager, executives and business sponsors. It has also defined the responsibilities of key oversight committees, including the project steering committees, the Corporate Business Architecture (CBA) Management Committee (CBAMC), the Systems Architecture Review Board and the field planning boards (FPBs).
The audit team did, however, identify opportunities to clarify the roles, responsibilities and authorities of oversight committees, project managers and other stakeholders in the areas of risk management; identifying, assessing and communicating project interdependencies; and project costing and forecasting. These opportunities are presented in more detail in the sections "Risk and issue management and project interdependencies" and "Project baselines including costs and efficiencies estimates" in this report.
A complete list of ongoing projects, including their total approved budget and expenditures, was not maintained by the DPMO.
The audit noted that the DPMO recently created a database to track all ongoing initiatives and projects across Statistics Canada. Given the limitations of the Budget and Revenue Management System, the organization's budget system, and the Financial Reporting System, financial information can be generated for no more than a single fiscal year at a time.
Therefore, to obtain the total approved budget and incurred expenditures for a project, multiple reports need to be generated for the analysis. At the time of the audit, the total approved budget and expenditures for each project were not being maintained by the DPMO.
Project governance and oversight is consistent for CBA projects; however, practices varied for non-CBA projects.
Although the DPMF has outlined the agency's expectations regarding the level of governance and oversight to be provided for projects according to their level of complexity, the audit team noted inconsistencies in the governance and oversight practices established for non-CBA projects.
Specifically, while the audit team noted that steering committees were created for CBA projects and that FPBs consistently oversaw CBA projects, the level of oversight provided by project steering committees and FPBs varied for non-CBA projects. For example, although the DPMF requires that all projects establish a steering committee, the audit team identified three projects (out of seven projects tested) that were not overseen by a steering committee but rather were overseen solely by the project sponsor.
Additionally, for stages 4 to 6 of the project management life cycle, the audit noted that the roles and responsibilities of the FPBs varied according to the type of project and its field, including inconsistencies in the level of documentation maintained as evidence of decision making and gating for these stages, specifically for non-CBA projects.
The audit noted that the senior official accountable for the DPMF is not part of key management governance committees. Including the assistant chief statistician responsible for this function in those committees would ensure that considerations regarding the DPMF are regularly included in deliberations, provide for a more comprehensive and robust review function and increase the rigour of the project management challenge function.
Without the establishment of appropriate governance and oversight for all projects, including non-CBA projects, there is an increased risk that project management deliverables will not be challenged and approved at the appropriate level, resulting in agency resources being used inefficiently.
Expectations for project managers' experience and certification have not been established.
The DPMF does not currently establish any minimum requirements regarding the level of experience or certification for project managers. Project managers are generally assigned on the basis of availability and their specialized expertise in the project topic rather than their experience or expertise in project management. As such, most project managers interviewed indicated having little to no previous project management experience, and only 1 project manager (out of 12 interviewed) held any related certification. In some cases, these individuals were managing large, complex projects.
Without a minimum level of project management experience (or certification) held by those managing complex projects on behalf of the agency, there is an increased risk that project managers may not efficiently and effectively manage their projects in accordance with the expectations outlined in the DPMF.
Project management training has been developed for key roles, but it has not been made mandatory.
A series of training programs have been developed and made available to project managers and executives. Specifically for project managers is a three-day training session on basic project management skills and a one-day in-house training session on the requirements of the DPMF, the gating process and how to develop the project management deliverables. The DPMO has further developed individual training sessions for key project management tools, such as the Change, Issue, Risk Management Tool (CIRMT) and the Executive Project Dashboard.
The audit noted that the project managers for the projects tested generally had not completed the project management training sessions offered to them, as these are not currently mandatory. Project managers' lack of training, as well as the fact that there is no minimum project management experience or certification requirement for project managers based on the complexity of projects, has resulted in inconsistencies in the level of understanding and use of the DPMF and its related tools. When the sampled projects were tested, this was demonstrated by the inconsistent use of the project management tools by project managers and the level of sophistication of certain project management deliverables.
Additionally, a training course designed to introduce executives to the DPMF and their roles and responsibilities, entitled Departmental Project Management Framework for Executives, was developed in 2012. In addition to other topics, this course covers the following areas: interpreting and approving project dashboards, issues and risk management, why governance is key in project management, the DPMF stages, and the gating process. Because this course was introduced as a pilot, it is not currently being offered to executives and, similar to the training offered to project managers, has not been made mandatory for executives who typically have oversight responsibilities relative to projects.
Recommendation
The assistant chief statistician of Corporate Services (and chief financial officer) should ensure the following:
- Governance expectations outlined in the DPMF—including the composition of committees, the role and responsibilities of the DPMO in projects, and the requirement to set up a project steering committee on all projects—are reiterated to senior management and project sponsors to ensure consistency across the agency.
- Criteria are established regarding minimum experience or certification in project management according to the size, risk, complexity and type of project established and documented in the DPMF.
- The basic project management skills and DPMF training courses are made mandatory for all project managers.
- The training course developed for executives on the DPMF is made a permanent course and, as possible, is embedded within the training offered to new executives.
Management response
Management agrees with the recommendation.
Following the time period for this March 2014 audit, additional activities occurred to clarify roles and responsibilities related to project management. In 2014/2015, the roles of the FPBs, the Senior Management Review Board and CBAMC were revised and communicated to the governance committees and the project managers. Further actions will be considered to address the recommendations:
- The director of Corporate and Financial Planning Division (CFPD) will update the governance expectations to clarify the composition of governance committees, the role and responsibilities of the DPMO in projects, and the requirement to set up a project steering committee based on size and scope, during the review of the DPMF by March 2016. Following the review, this clarification will be communicated to project managers and executives.
- The director of CFPD will ensure that criteria regarding recommended minimums for project management experience or certification are established and documented during the review of the DPMF in 2015/2016.
Deliverables and timeline: Update guidance and communication of governance expectations in the DPMF by March 2016.
Since 2012/2013, over 1,000 participants have attended project management training courses, such as those on the Executive Project Dashboard and CIRMT. Large and complex projects were initially focused on to ensure that the project management tools were adopted consistently. Attention will now move to the other projects. These training sessions were essential in providing guidance to project managers. The following action will be addressed shortly:
- The director of CFPD will recommend to the Learning and Development Committee that mandatory training be offered to all project managers and new executives. Pending discussions surrounding the transfer of training to the Canada School of Public Service may affect this action.
Deliverables and timeline: Obtain approval from the Learning and Development Committee to provide mandatory project management training by September 2015.
Risk and issue management and project interdependencies
The Departmental Project Management Framework (DPMF) recognizes the importance of risk and issue management as part of strong project management, but it currently provides limited guidance on the processes for identifying, assessing, communicating and monitoring project risks and interdependencies within the project life cycle. Additionally, the roles and responsibilities of key project team members and oversight committees with respect to managing risks and issues and monitoring project interdependencies have not been clearly defined within the DPMF.
The Departmental Project Management Office (DPMO) has developed and introduced the Change, Issue, Risk Management Tool (CIRMT) to help document and monitor project risks and issues, but its use has not been made mandatory for all projects. Further, at the time of the audit, a project interdependency tool was being piloted only for Corporate Business Architecture (CBA) projects.
The audit team believes that the lack of guidance and standardization of tools has led to the inconsistent management and documentation of project risks and interdependencies, as well as little evidence that project risks and interdependencies are being actively monitored.
The DPMF should require and provide the tools so that project risks and interdependencies, both internal and external to Statistics Canada, can be continually identified, assessed, monitored and reported.
The DPMF provides limited guidance regarding managing risks and monitoring project interdependency within the project life cycle.
The DPMF currently provides high-level guidance on the risk and issue management process; however, the audit noted that it does not detail how to embed the expected practices for risk management throughout the stages of the project's life cycle. Additionally, it was noted that there is no guidance in the DPMF regarding how to identify, assess, communicate and monitor project interdependencies.
Although the DPMF identifies the roles and responsibilities of project team members in other project management activities, it outlines the roles and responsibilities of the project manager only in key risk-management activities. The roles of other key project team members and the role of oversight committees regarding risk management within the context of a project have not been formally documented.
Similarly, the DPMF identifies the role of the CBA Secretariat only in relation to monitoring project interdependencies for CBA projects, and does not define the roles, responsibilities and accountabilities of other key project stakeholders, including those of the project manager, in identifying, communicating and monitoring project interdependencies.
As a result of this lack of guidance, key project team members and oversight committees may not have a consistent understanding of their roles in effectively and quickly identifying, assessing, communicating and monitoring project risks, issues and interdependencies. Consequently, the audit team noted inconsistencies in the level of documentation maintained to support identifying and monitoring project issues, risks and interdependencies.
Specifically, the audit identified 6 projects (out of 12) with limited evidence of risk management activities, including identifying relevant project-specific risks. Further, the audit identified four projects for which there was only limited or no evidence that project interdependencies were considered in the project planning and in dealing with project issues and risks.
CIRMT has been created; however, its use has not been made mandatory.
In 2013, the DPMO introduced CIRMT as the corporate risk and issue management tracking tool for documenting issues, risks and changes to a project's scope, schedule, costs and efficiency estimates. The use of a corporate risk and issue management tracking tool forces a level of discipline in identifying and assessing a project's issues and risks. As per the DPMO, all CBA and key non-CBA projects were to be migrated to the new tool by March 31, 2013; however, the use of the tool by all projects has not been made mandatory.
Through a review of CIRMT for the sample of 12 projects selected, it was determined that only 7 projects were using the tool and that only 4 of these 7 projects were using it consistently. The other five projects noted using other tools, such as Excel spreadsheets or Access databases, to log and monitor issues, risks and changes. However, the level of documentation maintained to support project changes as well as issue and risk management activities varied for the five projects.
Although these projects maintained an alternate tool, little or no documentation supporting project amendments and little or no evidence that risk and issue management activities were conducted was available. Without a consistently applied tool that forces a level of discipline when identifying and assessing risks and issues, it is possible the analysis and supporting documentation relative to risk and issue management will be insufficient.
An interdependency tool for tracking and monitoring project interdependencies is currently being piloted for CBA projects.
An interdependency tool has been developed by the DPMO to document interdependencies that exist between projects. This tool was in its pilot stage at the time of the audit and is currently limited to CBA projects.
The tool is maintained by the DPMO, but CBA project managers are asked to review their interdependencies monthly and communicate any changes to the DPMO in a timely manner. Although the DPMO continues to assess the tool as a result of the pilot, project management best practices indicate that a tool for managing project interdependencies should be maintained for all projects.
Recommendations
The assistant chief statistician of Corporate Services (and chief financial officer) should ensure that the DPMF is reviewed and updated to include
- the roles of key project team members and oversight committees with respect to risk and issue and project interdependency management
- the mandatory use of CIRMT and a corporate-level interdependency tool to track details of projects' risks, issues and interdependencies.
Management response
Management agrees with the recommendation.
In 2014/2015, the DPMO completed a pilot to monitor and report on project interdependencies. The result of this pilot was shared with various governance committees. Its implementation in 2015/2016 will address this recommendation. In addition, the following actions will be addressed:
- The director of Corporate and Financial Planning Division (CFPD) will ensure that roles and responsibilities for project managers and oversight committees with respect to project interdependency management will be added during the review of the DPMF by March 2016. In addition, an interdependency tool will be launched to track interdependencies between projects.
Deliverables and timeline: Update guidance in the DPMF by March 2016 and implement a corporate interdependency tool by October 2015. - The director of CFPD will ensure that all projects use CIRMT to track and manage changes, issues and risks once the new Executive Project Dashboard is launched in 2015/2016.
Deliverables and timeline: Make use of CIRMT mandatory by June 2015.
Project baselines including costs and efficiencies estimates
The Departmental Project Management Framework (DPMF) currently outlines the requirement for documenting project baselines, including estimated costs and planned efficiencies, at various points in the project life cycle, and provides templates to be completed by the project team. The audit team identified opportunities to strengthen current cost-management practices.
The assumptions made in determining the costs and efficiency baselines are not consistently being formally documented. The audit team was unable to assess the reliability and completeness of these baselines or to verify that all of the projects' stakeholders were consulted in determining or reviewing the cost and efficiency assumptions.
The audit further revealed that amendments made to a project's scope, schedule or cost and efficiency baselines are not consistently being formally documented as expected. This affects the transparency of project amendments, i.e., the number of amendments made to a project and its overall final cost or expected efficiencies.
Reliable project baselines should be clearly defined through consultation with all relevant stakeholders, and any amendments should be documented and approved as per the established governance structure.
Improvements to the documentation of assumptions underpinning project financial cost and efficiency estimates are necessary.
The audit noted that project managers are generally documenting estimated project costs, as well as any planned efficiencies for the project, in both the Business Case Costing template and the Project Plan, both of which are subject to the project management gating process. In most cases, however, the audit team was unable to obtain documentation outlining the assumptions made in determining the cost and efficiency baselines, as well as any ongoing maintenance costs associated with the initiative. Without these assumptions, the audit team was unable to assess the relevancy and completeness of the assumptions and, as a result, the reliability and completeness of the project costs and efficiencies.
Additionally, since project managers indicated that the assumptions and risks underpinning the cost and efficiency baselines were not discussed formally with relevant stakeholders, the audit team could not confirm that all relevant project stakeholders were consulted in determining or reviewing the cost and efficiency assumptions, such as any system architecture needs.
Project management best practices dictate that not only should project assumptions be formally documented, reviewed and approved to show sound stewardship, but also that project managers should be required to conduct a sensitivity analysis on the costing assumptions (i.e., provide scenarios on costing results should specific assumptions change). This analysis would help project managers, project sponsors and the oversight committees understand what would happen to the project estimates if the assumptions became unreliable or changed over the course of the project.
Sensitivity analysis involves changing the assumptions in a cost-efficiency calculation to see the impact on the project's finances. This analysis helps stakeholders make better-informed investment decisions.
Inconsistencies were noted in the documentation maintained to support scope, schedule and cost and efficiency amendments.
Although there is an expectation that all amendments to the scope, schedule and cost and efficiency baselines will be documented in both the Change, Issue, Risk Management Tool (CIRMT) and Appendix A of the Project Plan, the audit team noted difficulties reconciling the adjustments being made with the projects' baselines. In the absence of a formal audit trail supporting updates made to the scope, schedule or cost and efficiency baselines, the audit team was forced to review email correspondence, review other project management deliverables, and hold discussions with project managers to identify project amendments.
Even through these additional procedures, the audit team was unable to reconcile the cost and efficiency baselines reported in the executive project dashboards and the Project Outcome Report with approved adjustments for 4 of the 12 projects selected for review.
With the Departmental Project Management Office's (DPMO) proposed revisions to the Executive Project Dashboard (discussed in more detail below), project managers will be required to document their proposed amendments to the scope, schedule and cost and efficiency baselines in CIRMT, which will then be populated directly in the Executive Project Dashboard. The Executive Project Dashboard will also track and report on the number of amendments made to the project and will require the project manager to attach evidence of the approval for all amendments directly to the dashboard.
The DPMF provides limited guidance on the roles, responsibilities and accountabilities of key project team members and oversight committees in cost-management activities.
The DPMF outlines the gating process for the Business Case Costing template and the Project Plan, both of which present cost and efficiency estimates. The DPMF does not currently outline expectations regarding the roles, responsibilities and accountabilities of each project stakeholder—including the project manager, the financial management analyst, the project sponsors and the oversight committees—in contributing to or reviewing and approving the cost and efficiency estimates and their underlying assumptions. As such, the audit revealed inconsistencies in the level of engagement of these key stakeholders in developing and reviewing the cost and efficiency estimates, including the financial management analyst's level of engagement.
Recommendations
The assistant chief statistician of Corporate Services (and chief financial officer) should ensure that
- the DPMF and its related templates and tools are reviewed and updated to reflect cost-management expectations, including
- the documentation of all assumptions considered in developing the cost baselines, including any potential ongoing maintenance costs associated with the project
- formally documenting the consultations carried out with relevant project stakeholders in developing the cost and efficiency estimates and associated assumptions
- the roles and responsibilities of each project team member and oversight committee in contributing to and reviewing and approving cost and efficiency estimates and assumptions.
Management response
Management agrees with the recommendation.
- The director of Corporate and Financial Planning Division (CFPD) will include a new checklist to be used by all projects to ensure that key assumptions are sufficiently documented at the beginning of a project and updated during the roll-out of the project. The checklist will require the project manager to obtain approval from key stakeholders on assumptions. The checklist will include detailed planning, costing and information-technology related questions. This checklist will amalgamate assumptions from various documents and replace existing documents.
Deliverables and timeline: Implement new checklist that documents detailed assumptions for all projects by March 2016. - The director of CFPD will ensure that the roles and responsibilities of each project team member and oversight committee in contributing to and reviewing and approving cost and efficiency estimates and assumptions will be added during the review of the DPMF by March 2016.
Deliverables and timeline: Update guidance in the DPMF by March 2016.
Project monitoring and outcomes
As outlined in the Departmental Project Management Framework (DPMF), the monthly executive project dashboards are to be prepared and reviewed as per the established governance structure for each project. As the audit team identified weaknesses in the reliability and completeness of the reporting being done through a sample of dashboards, the Departmental Project Management Office (DPMO) is revising its Executive Project Dashboard template to make a project's status more transparent.
The audit also noted that limited evidence is maintained to demonstrate that project oversight committees are consistently reviewing and approving the monthly executive project dashboards in a timely manner.
Expected business outcomes are being documented in the business case; however, in some instances, these outcomes are not sufficiently developed to provide an understanding of the benefits of the project to the agency, nor are they measurable. Without clearly defined measurable outcomes, as well as an evaluation approach to their measurement and expected targets, it is difficult to assess whether a project has achieved what it was expected to.
From the sample of projects tested that had reached completion/close-out, the audit team noted that project managers are not consistently completing a full assessment of the outcomes against the original or revised targets.
The established oversight bodies should adequately monitor projects, and evidence that they reviewed and approved the monthly executive project dashboards should be maintained. Additionally, relevant project outcomes should be formally established and projects should be assessed against expected results.
The Executive Project Dashboard template is currently being updated to improve its reliability and completeness to ensure effective project oversight.
The monthly Executive Project Dashboard is the main tool used for reporting on a project's status through its established governance structure, including disclosing any emerging risks and issues. The dashboard contains five indicators for monitoring purposes: scope, schedule, cost, risk and issues. At the time of the audit, the Executive Project Dashboard provided solely in-year reporting, fed by both the Financial Reporting System and the project manager. An assessment against each of the five indicators is required (red, yellow and green) and is rolled-up to derive the health of the project's overall status.
The DPMO has revised the dashboard template to make the information being presented in the dashboards more reliable and complete. The revised template—extracted directly from the Financial Reporting System—will present the variance of the project's in-year actual costs vs. budget per quarter and will include a graph demonstrating the same variance for all fiscal years.
This improvement will allow governance committees to review the total costs incurred on the project to date vs. the approved budget, instead of only in-year costs. Additionally, the risks, issues and project amendments presented in the dashboard will be sourced directly from the Change, Issue, Risk Management Tool (CIRMT). This will help ensure the reporting of risks, issues and project amendments are complete. The revised Executive Project Dashboard template is expected to be available for use by all project managers starting April 2015.
The audit team believes that the proposed amendments to the Executive Project Dashboard—specifically that the risks, issues and project amendments will be linked to CIRMT and the financial information will be presented for all fiscal years—may address some of the inconsistencies noted through a review of the dashboards for the 12 projects selected for testing.
Specifically, for a number of projects, the audit team noted difficulties in reconciling the financial information presented in the dashboards with the approved budgets and the risks and issues within CIRMT. However, with the revised template, since the data reported will be sourced directly from the corporate project management tools, the content will be more reliable, ensuring that the project's status is transparent.
Further, the new template may also help ensure that project managers are keeping the content in the project management tools up-to-date, as the governance committee will review them through the Executive Project Dashboard.
Governance committees noted exercising their project oversight responsibilities; however, limited documentation exists to confirm these critical activities.
There are a number of committees tasked with overseeing projects, including the individual project steering committee and the field planning boards (FPBs), as well as the Corporate Business Architecture (CBA) Management Committee (CBAMC) for CBA projects and other significant or complex projects as requested by senior management. The audit noted that the committees are generally exercising their monitoring responsibilities; however, this is being done informally.
Specifically, interviews revealed that the monthly executive project dashboards are generally presented to and discussed by the project steering committees and FPBs. For seven projects out of nine (three projects did not prepare any executive project dashboards), the audit team was unable to obtain evidence that these discussions took place because meeting minutes, action items or resulting decisions were not being consistently documented.
Similarly, limited evidence is maintained to demonstrate that CBAMC reviewed and approved the executive portfolio summary reports and periodic risk reports being prepared by the DPMO for CBA projects and other significant projects.
Expected business outcomes are difficult to assess and measure.
The audit team noted that project managers are documenting expected business outcomes in the project's business case. For 3 out of the 12 projects, the business outcomes were not sufficiently developed to provide an understanding of the benefits of the project to Statistics Canada, nor were they measurable. Without clearly defined and measurable outcomes, as well as an evaluation approach to their measurement and expected targets, it is difficult to assess whether a project has achieved what it was expected to.
The Project Close-Out Report is the mechanism used to communicate the results of the project against its intended benefits. If the expected business outcomes are not achieved, the project manager is provided with an opportunity to explain any deviations within this report. Of the 12 projects selected for review, 4 projects were completed. Of these completed projects, the audit team noted that a full assessment of results against all expected business outcomes (as originally documented within the business case or revised in other project documentation) was not available for two projects.
The Outcomes Realization Report is part of the post-launch review within the project life cycle and complements the Project Close-Out Report, as it allows business outcomes to continue to be measured and reported on beyond the life of the project. This report is especially important because it allows the measurement of outcomes, which it may not be feasible to measure in the short-term.
Even though, as noted above, an evaluation against all expected business outcomes as part of the project close-out was not available for two projects, an Outcome Realization Report was not completed. The audit further noted that project managers generally were not aware of the requirement to complete this report, nor did they understand its purpose.
Recommendations
The assistant chief statistician of Corporate Services (and chief financial officer) should ensure that the DPMF is reviewed and updated to include the following requirements:
- The monthly executive project dashboards be formally reviewed and approved as per the established project governance structure, and evidence of their review and approval be maintained.
- Project managers, project sponsors and oversight committees review the expected business outcomes to ensure that they provide a comprehensive explanation of the benefits of the project and that they are measurable before the business case is approved. Additionally, during the project planning stage, a timeline for measuring the business outcomes as well as what outcomes will be measured as part of the Project Close-Out Report and which will be reported using the Outcomes Realization Report (medium- and long-term impacts) should be established.
Management response
Management agrees with the recommendation.
- The director of Corporate and Financial Planning Division (CFPD) will ensure that evidence of approval for all monthly executive project dashboards will be maintained by the DPMO and that monthly compliance reports will be submitted to the Corporate Planning Committee.
Deliverables and timeline: Implement monthly compliance reports for the Corporate Planning Committee by September 2015. - The director of CFPD will ensure that roles and responsibilities for project managers and oversight committees for reviewing expected business outcomes will be further documented during the review of the DPMF by March 2016.
Deliverables and timeline: Update guidance in the DPMF by March 2016.
Appendices
Appendix A: Audit criteria
Control objective, core controls and criteria | Sub-criteria | Policy instrument |
---|---|---|
1) The agency has an effective and adequate project management framework to ensure a systematic approach to managing its projects and associated interdependencies in compliance with the Treasury Board (TB) Policy on the Management of Projects. | ||
Governance and strategic directions 1.1 Statistics Canada has strategic direction in place for project management, including the establishment of a framework that is documented and adequately communicated. (G3 & G4). |
1.1.1 Strategic direction by the Departmental Project Management Office (DPMO) is documented and communicated to all relevant stakeholders. 1.1.2 A departmental management framework exists, along with associated guidance and processes, which complies with the TB Policy on the Management of Projects and reflects best practices. |
Statistics Act Policy on the Management of Projects (TB) Integrated Strategic Planning Process Departmental Project Management Framework (DPMF) Guidelines (Statistics Canada) |
Accountability and stewardship 1.2 Authorities, responsibilities and accountabilities with respect to initiating, planning, executing and closing-out projects have been established and have been formalized and communicated. (AC-1; AC-2; AC-3; AC-4 & ST-13) |
1.2.1 Employees' duties and control responsibilities as they relate to project management are clearly defined, documented and communicated to the relevant stakeholders. 1.2.2 Authority, responsibility and accountability are exercised as intended. 1.2.3 Authority is formally delegated, and delegated authority is aligned with individuals' responsibilities. Where applicable, incompatible functions are not combined. |
Statistics Act Financial Administration Act Policy on the Management of Projects(TB) Integrated Strategic Planning Process DPMF Guidelines (Statistics Canada) |
Risk management 1.3 The DPMF requires and provides the tools for identifying, assessing, monitoring and reporting project risks and interdependencies, both internal and external to Statistics Canada. (RM-3 & RM-6) |
1.3.1 Formal processes and guidelines exist and are applied to identify project risks more easily. 1.3.2 Risk information is regularly presented to and discussed at established management and oversight committees and embedded in key performance reports. 1.3.3 Processes have been established to identify and communicate internal and external project interdependencies, such as deliverables, milestones and resource requirements. |
Statistics Act Policy on the Management of Projects(TB) Integrated Strategic Planning Process DPMF Guidelines (Statistics Canada) Risk Management Policy (TB) |
People 1.4 The DPMO provides employees with the necessary tools, resources, information and training to discharge their responsibilities and achieve expected results. (PPL-4) |
1.4.1 A suitable training and development plan exists for stakeholders involved in the project management process. 1.4.2 Employees have access to sufficient tools, such as software, equipment, work methodologies, and standard operating procedures. 1.4.3 An information-sharing process exists to support the efficient and targeted dissemination of relevant and reliable information to the appropriate stakeholders. |
Statistics Act Policy on the Management of Projects(TB) Integrated Strategic Planning Process DPMF Guidelines (Statistics Canada) |
2) Projects are managed in compliance with the DPMF and its related tools, and are monitored against established milestones and expected results. | ||
Policy and programs, and stewardship 2.1 Applicable DPMF project management deliverables are adequately completed as per established timelines, and approved by individuals with the appropriate delegated authority. (PP-4, ST-15 & ST-18) |
2.1.1 The project management deliverables for initiating, planning, executing and closing-out projects are comprehensive, clear and well documented, and are supported by substantiated information, as per established timelines. 2.1.2 All project management deliverables are formally reviewed and approved in a timely manner, as per the established governance structure. 2.1.3 As required, project management deliverables are submitted to TB for review and approval. |
DPMF Guidelines (Statistics Canada) |
Results and performance, stewardship, and people and risk management 2.2 The project baselines have been clearly defined through consultation with all relevant stakeholders. (RP-2, ST-1, ST-2, ST-3, ST-15, ST-18, PP-4 and RM-2) |
2.2.1 Project baselines are developed before projects are implemented. 2.2.2 Evidence exists demonstrating that interdependencies and risks were considered when developing project assumptions. 2.2.3 Relevant stakeholders are involved in developing the project baseline, including Finance and Information Technology (as appropriate). |
DPMF Guidelines (Statistics Canada) |
Stewardship, risk management, and policy and programs 2.3 Project risks, issues and changes are being escalated as per the established governance structure, and the impacts of project interdependencies are considered. (ST-15, ST-18, RM-1, RM-2, RM-5, RM-6 & PP-4) |
2.3.1 An established process exists for escalating risks, issues and changes for review and approval as per the approved governance structure. 2.3.2 Evidence of decision making by the appropriate authority level is maintained. 2.3.3 An established process exists for identifying, assessing, communicating and monitoring project interdependencies, including interdependencies involving external parties. |
DPMF Guidelines (Statistics Canada) |
Results and performance, stewardship, policy and programs, governance and strategic direction, and risk management 2.4 Effective and timely project monitoring against the established baseline is conducted, and evidence is maintained to support decision making. (RP-2, ST-1, ST-2, ST-3, PP-3, ST-15, ST-18, PP-4, G-6, RM-2 & RP-3) |
2.4.1 Monthly status reports and dashboards are being completed by project managers and reviewed as per the established governance structure in a timely manner. 2.4.2 The status reports and dashboards provide sufficient information to monitor progress as per the approved project baseline (e.g., milestones and costs). |
DPMF Guidelines (Statistics Canada) |
Results and performance, stewardship, and policy and programs 2.5 Relevant project outcomes have been formally established, and projects are assessed against expected results, including their completion on time and on budget. (RP-2, ST-1, ST-2, ST-3, ST-15, ST-18 & PP-4) |
2.5.1 Relevant project outcomes (i.e., expected results) are established and approved in the planning stages of the project. 2.5.2 Evidence exists that project outcomes have been assessed in a timely manner against the approved expected results, including expected efficiencies (either by completing the Project Close-Out Report or, as required, through additional monitoring). 2.5.3 The Project Close-Out Report includes an assessment of the project's completion on time and on budget as per the approved original baselines. |
DPMF Guidelines (Statistics Canada) |
Appendix B: Acronyms and initialisms
Acronym | Description |
---|---|
CBA | Corporate Business Architecture |
CBAMC | Corporate Business Architecture Management Committee |
CFPD | Corporate and Financial Planning Division |
CIRMT | Change, Issue, Risk Management Tool |
DPMF | Departmental Project Management Framework |
DPMO | Departmental Project Management Office |
FPB | Field Planning Board |
ISPP | Integrated Strategic Planning Process |
TB | Treasury Board |