Audit of Agency's Long Term Planning Process

Audit Report

April 22, 2013
Project Number: 80590-76

Executive summary
Introduction
- Background
- Audit objectives
- Scope
- Approach and methodology
- Authority
Findings, recommendations and management responses
- Long Term Planning process control environment
- Oversight and scrutiny for decision-making
Appendices
- Appendix A: Audit criteria
- Appendix B: Acronyms
Figures
- Figure 1 - Integrated Strategic Planning Process (ISPP)
- Figure 2 - Stages of the Long-Term Planning Process

Executive summary

With the mandate to serve the statistical needs of all levels of government, Canadian institutions and all Canadians, Statistics Canada seeks to ensure that Canadians have access to a trusted source of statistics on Canada that meet their highest priority information needs. The provision of high quality data supporting core economic, environmental, and social statistics programs is a key driver in determining how Statistics Canada allocates its funding. As such, strategic priority setting and funding of investments is achieved through the Integrated Strategic Planning Process (ISPP) established in 2010-11. The ISPP is an annual six step cycle, the first four steps of which are known as the Long Term Planning (LTP) process established in 1984.

Statistics Canada plans to invest $134 million in assets, $171 million in acquired services and $512 million in projects (largest project is the 2011 Census of Population and National Household Survey) over the 2011-12 to 2015-16 planning horizon. The planned investments meet the Agency's operational requirements and are within the Agency's existing reference levels.^{Footnote 1}

The objectives of this audit were to provide assurance to the Chief Statistician (CS) and Statistics Canada's Departmental Audit Committee (DAC) that governance mechanisms are in place to support the planning, resource allocation and monitoring of LTP projects, and the identification and approval of LTP projects is enabled by a structured and systematic process.

The audit was conducted by Internal Audit Services in accordance with the Government of Canada's Policy on Internal Audit.

Key findings

The Agency has a formal LTP process that is documented and generally understood by participants. However, some important roles and responsibilities are not well defined, leading to a lack of clarity in relation to how key stakeholders are involved in the process. In other instances, roles and responsibilities of key governance bodies are not well defined, or defined with a high degree of flexibility, creating opportunities for inconsistency in governance over the planning process.

There is currently no requirement to formally document planning assumptions in support of costing and efficiencies estimates; and formal guidance for the preparation of the estimation of costs, planned and actual efficiencies is not provided. As a result, the approaches used to generate and track these estimates are inconsistent.

Currently, Corporate Business Architecture (CBA) projects do not always take into consideration the interdependencies that exist within Statistics Canada's matrix environment. This may result in the loss of opportunities for greater efficiencies. An 'Interdependency Model' has been developed by Corporate Planning Division (CPLD) to better identify and manage interdependencies. Approval of the model will make it mandatory for CBA projects to formally track, identify, manage and monitor interdependencies amongst CBA projects.

Governance and operational practices for scrutiny and challenge vary across Fields, with limited substantive review and challenge of the financial and non-financial information in the LTP forms by PAs and FMAs. Challenges by key participants and within governance bodies are not documented. This may result in inconsistent assessment of proposals across Fields and incomplete information to support defensible planning and approval decisions.

There is general compliance with the LTP process as mapped and value of the LTP process is increasingly being realized with the considerable evolution in the use of the DPMF and related tools. However, original baseline costs and efficiencies estimates of projects should be included in the Executive dashboard to systematically track, monitor, understand and challenge the impact of project adjustments from original cost and efficiency estimates.

Record-keeping and information management practices related to ownership, version control and transmission should be strengthened to ensure sound stewardship and data integrity.

Overall conclusion

The LTP process has been established for some time, but has recently seen a major evolution with the advent of the DPMF and related requirements. The new structures and tools for the LTP process provide a good foundation for discipline and defensible decision-making, but more progress is needed – both process wise and culturally.

Greater clarity on roles and responsibilities; guidance on cost and efficiency estimation; and formal documentation of planning assumptions for costing and efficiency estimates will strengthen the ability of governance bodies and key participants to effectively discharge their scrutiny and challenge responsibilities. Inclusion of the original baseline costs in the project dashboards will strengthen the ability of project sponsors and governance bodies to effectively track and monitor the cumulative impacts of project adjustments to ensure sound stewardship of resources.

Opportunities exist to strengthen record-keeping and information management practices related to ownership, version control and transmission to ensure sound data stewardship and data integrity.

Conformance with professional standards

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors (IIA) Internal Standards for the Professional Practice of Internal Auditing.

Patrice Prud'homme
Chief Audit Executive

Introduction

Background

As Canada's national statistical office, Statistics Canada has a mandate to serve the statistical needs of all levels of government, Canadian institutions and all Canadians. Statistics Canada contributes to most federal government outcomes in the domains of economic, social, international and government affairs. In so doing, it seeks to ensure that Canadians have access to a trusted source of statistics on Canada that meet their highest priority information needs.

The allocation of funding in support of the Agency's mandate and priorities is done through the Integrated Strategic Planning Process (ISPP) established in 2010-11.The ISPP describes how decisions are made through the long term planning (LTP) process. The ISSP is an annual process made up of six steps. The first four steps are the LTP process and the fifth and sixth steps relate to the communication of plans and the monitoring of LTP projects. Established formally in 1984, the LTP process has evolved over the years, most recently adapting to the application of the tools implemented as part of the Departmental Project Management Framework (DPMF) which itself was launched as a pilot in 2011-12. Project management functions are supported by the Departmental Project Management Office (DPMO).

Through the LTP process, the Agency annually assesses its programs and resources to adjust to changing client needs and priorities. It also provides a forum for the identification and discussion of cross-cutting issues, fostering efficiency improvements and encouraging innovation. To ensure ongoing program continuity and quality, the ISPP exercise features a ten-year planning horizon called "Funding the Future" to help the Agency plan significant expenditures well in advance, manage operational risks and align its programs to the needs of Canadians. Through the ISPP, the Agency integrates sound management practices such as risk management, investment planning, project management and evaluation into the planning process.

Figure 1 - Integrated Strategic Planning Process (ISPP)

Description for Figure 1 - Integrated Strategic Planning Process (ISPP)

The first step of the LTP process starts in April in each year with a "strategic direction" session to align strategic direction with priorities and emerging issues, update the Corporate Risk Profile and focus the discussions around mitigating important corporate risks. At the second step from May to June, managers update their 10 Year Investment Outlook to reflect the most current information available, and develop a high level business proposal to request funding through the LTP process. Proposals are grouped into three main categories for decision-making:

Corporate Business Architecture (CBA) improvements and initiatives;
Continuity and Quality maintenance (CQM) of existing programs; and,
New Initiatives / Enhancements.

Each category reflects the different aspects of strategic planning. CBA initiatives find new and better ways to do business and either generate efficiencies in corporate business processes or improve their robustness and responsiveness. Net efficiencies generated by CBA projects are used to maintain the continuity and quality of existing programs through the CQM projects, such as, implementation of new international standards and classifications, system redesigns, sample redesign, etc. New initiatives or program enhancements are funded by re-allocating existing budget or new monies (Treasury Board allocations or cost-recovery funds).

During the third step, from July to October, a detailed business case for each approved proposal is developed by the managers. Full costs, options for investments and anticipated savings are identified for review and approval by the Field Planning Boards (FPBs).

At the fourth step in November, business cases are presented to the Senior Management Review Committee (SMRC). SMRC ensures that all information and implications have been duly considered, and determine which investments will be funded. The final decisions are recorded in the Decision Record (Blue book) which is distributed to the Agency's senior management team at the conclusion of the process.

Audit objectives

The objectives of this audit were to provide assurance to the Chief Statistician (CS) and Statistics Canada's Departmental Audit Committee (DAC) that:

The Agency has appropriate governance mechanisms in support of the planning, resource allocation and monitoring of LTP projects; and
The identification and approval of LTP projects is enabled by a structured and systematic process, supported by appropriate knowledge and defensible information inputs.

Scope

The scope of this audit included an examination of the adequacy and effectiveness of the controls in place over the Long-Term Planning process. Specific areas that were examined included:

Governance and monitoring practices that support clear accountability, oversight, scrutiny and challenge functions; and
Operational processes and controls that enable the consistent application of a common LTP methodology as well as the tools, training and information management practices that support informed LTP decisions.

The scope of the audit included all Fields of the Agency and examined a sample of 20 project files approved in 2010-11, for 2011-12 LTP funding. The audit acknowledges that the fiscal period selected for testing was the first year of launching the DPMF as a pilot and the programs and the DPMO were adapting to the new process.

Approach and methodology

The audit work consisted of examination of documents, interviews with key Senior Management and personnel, and a review of relevant procedures, guidelines and frameworks related to the Long Term Planning process.

The field work consisted of a review of applicable procedures and documentation to support the governance and monitoring practices; and testing of operational processes and procedures to verify if appropriate and robust information management practices are in place; if the LTP process is adhered to by all key stakeholders; and if robust scrutiny and challenge mechanisms exist and are applied in support of defensible planning and approval decisions.

This audit was conducted following the Standards for the Professional Practice of Internal Auditing of the Institute of Internal Auditors (IIA) and in accordance with the TBS Policy on Internal Audit.

Authority

The audit was conducted under the authority of Statistics Canada Integrated Risk-Based Audit and Evaluation Plan 2012/13-2014/15 recommended by the Departmental Audit Committee, April 2012 and subsequently approved by the Chief Statistician.

Findings, recommendations and management responses

Objective # 1: The Agency has appropriate governance mechanisms in support of the planning, resource allocation and monitoring of LTP projects.

Long Term Planning process control environment

The Agency has a formal LTP process that is documented and generally understood by participants. However, roles and responsibilities of the PA, FMA, and FPM, are not defined, leading to a lack of clarity in relation to how key stakeholders are involved in the process. In other instances, roles and responsibilities of the FPB are defined with a high degree of flexibility, creating opportunities for inconsistency in governance over the planning process.

Currently, CBA projects do not always take into consideration the interdependencies that exist within Statistics Canada's matrix environment. This may result in the loss of opportunities for greater efficiencies. An 'Interdependency Model' has been developed by CPLD to better identify and manage interdependencies. Approval of the model will make it mandatory for CBA projects to formally track, identify, manage and monitor interdependencies amongst CBA projects.

Treasury Board Policy on Investment Planning – Assets and Acquired Services requires Deputy heads to ensure that appropriate governance, systems and resources for investment planning are in place, and maintained based on principles of sound stewardship and value for money. As well, an effective investment management framework, including departmental procedures, processes and systems should be implemented.

Long Term Planning process

A description of the Long Term Planning process is embedded in two key documents, "Integrated Strategic Planning Process at Statistics Canada: An Overview" and the "Departmental Project Management Framework (DPMF) Guidelines" (the Guidelines) developed by the Corporate Planning Division (CPLD).

The Integrated Strategic Planning Process at Statistics Canada: An Overview provides an explanation of the ISPP; a description of the governance structure specific to the ISPP with a description of the key responsibilities of its governance committees. It includes a separate section for each of the six steps, with a description of the specific activities, key stakeholders, responsibilities and the documents, deliverables and outputs related to each activity.

The LTP process consists of two stages within the project management framework, and the Guidelines provide a description of project management concepts; project risk management; project management stages, deliverables and documentation; the governance process, governance committees and description of their roles and responsibilities; and a description of the roles and responsibilities of the project manager and project teams.

Both documents are easily accessible to all key process participants on Statistics Canada's intranet site. Training on the ISPP is in place and has been provided on a pilot basis. The training material provides a good overview and description of the LTP process and is consistent with the Guidelines.

Figure 2 - Stages of the LTP Process

The two stages of the LTP process are illustrated below.

In Stage 1 programs develop a Business Proposal (BP) for approval. Corporate Business Architecture (CBA) proposals go through an approval process by both the CBA management committee and FPBs and non-CBA projects go through an approval process by the FPBs. Proposals approved by the FPBs are then presented to the SMRB in June, for gate 1 approval and consideration for LTP funding. Proposals approved for consideration in June, move to Stage 2. In Stage 2 programs prepare a Project Complexity and Risk Assessment (PCRA), Business Case and Business Case Costing (BCC) template. These forms then go through the same approval process as in Stage 1 and are presented to the SMRB in November for gate 2 approval.

Roles and responsibilities

Roles, responsibilities and accountabilities for the following governance committees involved in the LTP process, are formally defined and documented in the ISPP document and the Guidelines:

Field Planning Boards (FPBs)
Corporate Business Architecture (CBA) management committee
Corporate Business Architecture Secretariat
Senior Management Review Board (SMRB)
Branch Steering Committee and
Project Steering Committee

The role of the FPB is defined in general terms in the Guidelines. The Guidelines stipulate that

"each FPB has the flexibility on how they conduct their gate reviews".

This introduces a measure of inconsistency into the planning process, including the timeliness of DPMF document submissions for the LTP process. This could negatively affect the consistency of governance practices across Fields.

The audit noted that formal Terms of Reference do not exist for any of the oversight bodies, although, one of the Fields has developed a 'Governance Model' document which describes the purpose of the FPB and other branch level oversight bodies and their roles and responsibilities with respect to the LTP process. Terms of Reference are written guidelines that clarify the role, purpose and responsibilities given to a committee. SMRB is ultimately responsible for the decisions taken by the governance bodies; therefore it is important to establish a clear reporting mechanism and to set the perimeters within which authority is delegated to the FPBs and the CBA management committee.

Interviews with the FPMs for the other fields revealed their intention to replicate the 'Governance Model' for their own Fields. The CBA secretariat also confirmed its intention to develop a formal Terms of Reference for the CBA management committee.

Roles, responsibilities and accountabilities for the Project Manager (PM) and the Project Team are formally defined and documented in the Guidelines. This is not the case for the Planning Analyst (PA), Field Portfolio Manager (FPM) and the Financial Management Advisor (FMA) which has resulted in some confusion and overlap of activities.

The role and responsibilities of the PM varies across Fields and programs with respect to the LTP process. PMs at assistant director level and above were more engaged in the process from the idea identification and generation stage and participated at branch and divisional management committees and at the FPBs. The role of the PMs at the Chief level generally started with the preparation of the DPMF documents for stage 2 of the LTP process and they do not benefit from the knowledge of the complete life-cycle of the proposal.

The Planning Analyst is understood to be the owner of the LTP process but a common understanding of the nature of the scrutiny they should be exercising is not defined. As owners of the LTP process, they exercise their process knowledge, by ensuring that documents presented before the FPBs align with corporate objectives and the DPMF requirements, with a focus on their completeness and reasonableness, but not on their substance or content of the information submitted. Interviews with Project Sponsors (DG level) confirmed that PAs assist in data entry and ensure comparability of proposal submissions, and that a common approach is followed for all projects, but do not perform a challenge function.

The position of the FPMs who report to Assistant Chief Statisticians (ACS) is new and has emerged due to the increase in oversight and documentation requirements as part of the uptake of the DPMF. The FPMs assist with the management and monitoring of project activities within their Field. As such, they review the DPMF documentation for completeness before submitting to the FPB and submit the final version to the DPMO. There is overlap and duplication in the roles and responsibilities of the PA and FPM.

The role of the FMA, like the PA, varies across Fields. For instance, in some Fields, they are present at the Field Planning Board on a mandatory basis. In others, they are invited on an as-and-when-required basis. Their roles and responsibilities with regards to scrutinizing and challenging financial estimates during the LTP process are not clear.

A formal LTP process is documented and generally understood by participants, but some key roles and responsibilities are not well defined, leading to a lack of clarity in relation to how key parties are involved and contribute to the process.

Guidance

The LTP process is embedded in the broader guidance around the ISPP and the DPMF, and is described at a high level, stage-by-stage in the portion of the DPMF mechanisms in the Guidelines related to the LTP process. Direction for each stage focuses on the forms to be prepared and describes who is responsible, accountable, consulted and informed for each deliverable.

Interviews identified that there was insufficient guidance on the preparation of cost estimates and the estimation of planned efficiencies and actual efficiencies realized (for CBA projects). Review of the Guidelines confirmed this to be the case and revealed that assumptions are not required to be documented as supporting explanation for how the costs or efficiencies estimates presented were arrived at. Interview evidence suggests that discussions on planning assumptions take place informally and bilaterally at various stages of the LTP process. The discussions are based on the accumulated and combined experience and knowledge of the key parties engaged in the process, however, little to no documented evidence exists of these discussions.

Review of the DPMF documents for the LTP process (the Business Proposal, the Business Case and the Business Case Costing templates), revealed that each form template is supported by a guide, which provide detailed instructions on how to prepare each form. In the guide for the Business Proposal template, the definition that is included for an assumption is

"that which would make the proposal successful or unsuccessful".

It does not, however, specify that the planning assumptions for costs or efficiencies estimates should be included.

At the time of the audit, CPLD indicated that they will be implementing a 'Project Costing' template, which would require the documentation of financial and non-financial inputs, and underlying assumptions. This will reduce the risk of an inconsistent and incomparable approach to costing and efficiencies estimates and increase the ability to track the achievement of efficiencies.

As a matrix organization, functional responsibilities at Statistics Canada are organized into six Fields, with the management of product development processes for large projects taking place across the Fields. As a result, most projects are inter-dependent in nature, and require consultation across the organization to identify potential synergies, allow for appropriate linkages, and manage changes across Fields. Projects not managed in an inter-dependent fashion could result in overlap, duplication and unnecessary costs.

The audit found that when resource estimates are being prepared, the Guidelines do not outline any specific requirements for information gathering or consultation across Fields or with internal service providers. In practice, some cross-Field consultation takes place informally or bilaterally at the project level, but there is no documented evidence to suggest that this is done. As a result, limited audit trail is in place regarding the nature and extent of deliberations and discussions held, and of any assumptions made. Sign-off from inter-dependent Fields was a requirement as part of the oversight processes in the past, but with the DPMF, forms are now submitted electronically and physical sign-offs by partners are not required.

An 'Interdependency Model' has been developed by CPLD to better identify and manage interdependencies. Once the model is approved, it will be mandatory for CBA projects to formally track, identify, manage and monitor interdependencies amongst CBA projects.

Recommendations:

The Assistant Chief Statistician (ACS) Corporate Services should ensure that:

The roles and responsibilities of the Planning Analyst, the Field Portfolio Manager, the Financial Management Advisor, the Program Manager and the Field Planning Boards are clearly defined, documented and communicated in the Guidelines; and formal Terms of Reference for the governance bodies (Senior Management Review Board, CBA management committee ands the FPBs) are developed, approved and reviewed periodically to establish role, purpose and responsibilities and include clearly identified timelines between stakeholders for delivery of key documents for the LTP process.
The DPMF LTP process templates (Business Proposal, Business Case and Business Case Costing) are updated to require that planning assumptions and rationales for the cost and efficiency estimates (where applicable) be documented; and a Project Costing template is developed and adopted for a consistent and comparable approach to cost and efficiency estimation.
Guidance is developed on the conduct and documentation of interdependency consultations between Fields, including internal service providers, to formally track, identify, manage and monitor interdependencies.

Management response:

Management agrees with the recommendations.

The Director CPLD will ensure the documentation of the roles and responsibilities of the Planning Analyst, the Field Portfolio Manager, the Financial Management Advisor, the Program Manager and the Field Planning Boards and this is in progress.

Deliverable and Timeline: The "Field Planning Board Roles and Responsibilities: Terms of Reference" document will be made available to all managers on CPLD ICN site, by June 30, 2013.
The Director CPLD will ensure the documentation of Formal Terms of Reference for the governance bodies (Senior Management Review Board, CBA management committee ands the FPBs) with clearly identified timelines for delivery of key documents for the LTP process will be documented.

Deliverable and Timeline: Formal Terms of Reference will be made available to all managers on CPLD ICN site, by October 31, 2013.
The Director CPLD launched a process and tool to monitor and control CBA Portfolio risks, issues and interdependencies for CBA projects and 16 non-CBA projects as of Feb 28, 2013.

Deliverable and Timeline: Completed interdependencies spreadsheet and analysis of project key deliverables and timelines, by June 30, 2013. Training and support to project managers, by August 31, 2013 and Quarterly updates to CBA management committee by October 31, 2013.

Objective #2: The identification and approval of LTP projects is enabled by a structured and systematic process, supported by appropriate knowledge and defensible informational inputs.

Oversight and scrutiny for decision-making

Record-keeping and information management practices related to ownership, version control and transmission should be strengthened to ensure sound stewardship and data integrity.

Appropriate and robust stewardship practices, scrutiny and challenge mechanisms should exist to support defensible LTP decisions; provide stewardship of project resources; create accountability for projects and align projects with departmental strategic outcomes.

The audit tested a sample of 20 LTP project files approved in 2010-11, for 2011-12 LTP funding to verify if appropriate and robust information management practices are in place; if the LTP process is adhered to by all key stakeholders; and if robust scrutiny and challenge mechanisms exist and are applied in support of defensible planning and approval decisions. The audit acknowledges that the fiscal period selected for testing was the first year of launching the DPMF as a pilot, and notes that both the programs and the DPMO were adapting to the new process.

Seven CBA and thirteen non-CBA projects were judgementally selected for testing to ensure coverage of the three categories of projects from all the Fields. Completed Business proposal (BP), Business Case and Business Case Costing (BCC) forms for each project were requested from the Corporate Planning Division (CPLD) which is

"the central repository of all electronic copies of final documents"

per the Guidelines.

Information management

Review of the Guidelines revealed that although document management is discussed, the focus is largely on the templates to be used, not the information management practices related to ownership, version control, controlled transmission and data integrity. The audit sought to trace the sampled BP, Business Case and BCC forms held in the central repository and confirm that they were in agreement with those records held by the project manager. This test was not completed because the audit team encountered difficulty in obtaining the final documents. For the CBA projects, CPLD was only able to provide two thirds of the documents requested and for non-CBA projects they were only able to provide half of the documents requested. A common repository for non-CBA projects was not required for the year selected for our audit test, but has since been established in the subsequent year.

A review of the structure of the DPMO repository revealed that it is configured to follow the stages of the ISPP and all the LTP project folders were consistently mapped to this structure. As noted above the project folders for CBA projects did not contain the complete project documentation. Further enquiry revealed that record keeping challenges were attributed to PA turnover and the volume of the projects to manage. The audit noted that with the advent of the FPM, the role of the PA as coordinator and manager of information is not clear and may result in confusion over version control and ownership of the LTP documents.

Adherence to the LTP process

The BP forms for 6 CBA projects and 9 non-CBA projects received were completed in compliance with the instructions in the guidance included in the form.
Only 3 CBA projects and 6 non-CBA projects had developed a Business Case. For the CBA projects, 1 of the forms had been well completed with assumptions provided, while those for the other 2 appeared to be under development, with many sections uncompleted.
Of the projects sampled, 2 did not have Business Case Costing forms. The remainder contained the estimated costs and planned efficiencies.

For projects that had completed forms on file, the audit noted that little to no supporting documentation of assumptions or supporting analysis was available for the cost and efficiency estimates.

The audit noted that there has been a considerable evolution in the use of the DPMF and related tools, particularly with respect to CBA projects and the value of the process are increasingly being realized. For non-CBA projects, the LTP process is still being viewed as a 'compliance' requirement rather than a valuable management practice. However, record-keeping and information management practices related to ownership, version control and transmission should be strengthened to ensure sound stewardship and data integrity.

Scrutiny and challenge mechanisms

Scrutiny and challenge of an LTP project proposal from idea generation to gate 1 and gate 2 approvals takes place at the Field level and at the departmental level. Given the volume of project proposals that are identified for LTP funding, much of the scrutiny and challenge function takes place in advance of the departmental level governance meetings. This is appropriate both from a process flow and from an efficiency perspective.

At the Field level, interview evidence revealed that a considerable amount of bilateral verbal discussions occur in varying ways by key participants in the scrutiny and challenge of project proposals and the LTP documents. Rigour and due diligence is performed by senior management (Assistant Directors, Directors, DGs, ACSs), at various branch and divisional meetings. Project sponsors, i.e. DGs are involved from the idea generation stage and see all the projects as part of the early decision making within their areas. According to them, assumptions are not documented in the DPMF forms, but are discussed verbally and challenged at the various Field level meetings; Project Steering Committee meetings; and finally the FPB. The nature of the scrutiny exercised by the key participants could not be independently examined due to lack of documentary evidence.

The manner in which PAs, FPMs and FMAs within a Field interact with each other and management in the substantive scrutiny and challenge of project proposals and LTP documents varies, and is based on the individual Field and personality of players.

Interviews revealed that PAs do not always feel they have the authority to exercise due diligence or the time for a substantive review and challenge of the financial and non-financial information in the LTP forms. FMAs involvement in the LTP process starts mainly at project implementation. Like the PAs, they also feel they do not always have the authority to exercise due diligence or the time for a substantive review and challenge of the financial information in the LTP forms. FPMs are increasingly playing a coordination and scrutiny role similar to that performed by the PAs.

At the Field level, the FPBs are in place and play an active role in the scrutiny and challenge function of the LTP project proposals. However, the Guidelines stipulate that

"each FPB has the flexibility on how they conduct their gate reviews".

This introduces a measure of inconsistency into the planning process which could result in variable governance practices across Fields. Review of FPB meeting presentations and minutes revealed that their practices do vary across Fields.

At the departmental level, CBA management committee and the SMRB are in place. Interview evidence, review of CBA management meeting presentations, and attendance as observers by the audit team at a CBA management committee and the SMRB conference in November 2012, revealed that bilateral verbal discussions and deliberations, where feedback is sought and provided, take place. Information, knowledge, and expertise are exchanged and assumptions on cost and efficiency estimates are covered, challenged and discussed and approval is sought, given, and the decisions are documented in the records of decisions or minutes of meetings. The audit concluded that the CBA management committee plays an active and robust role in scrutinizing and challenging CBA LTP projects.

The audit reviewed the minutes of the FPBs and the CBA management committees charged with approval, for the projects in our sample to verify if gating and key decisions were formally approved for LTP funding. The audit noted that the CBA management committee reviewed and formally approved all of the CBA projects at stage 1 and stage 2. The FPBs reviewed and discussed the projects over several sessions before formally approving them and submitting them to the SMRB for gate 1 and 2 approval.

Monitoring: Post-approval feedback loop for variances and adjustments

The audit examined, in Stage 4 of Project Implementation, the nature of the monitoring controls that allowed the governance bodies, charged with approval, to understand and oversee subsequent adjustments to key project variables. This did not constitute a full examination of project monitoring controls, but was based on targeted testing of the feedback loop that informs governance bodies of major variances from the original commitments. This was done because there is inherent risk that project proposals are optimistically presented for the purposes of project approval, only to have the cost estimates adjusted upwards, or the efficiency adjusted downwards, post-approval.

The audit found that once the project is in production, monitoring of project variables is undertaken through a variety of mechanisms.

For non-CBA projects, responsibility for project monitoring rests with the project steering committee and other governance bodies established for a particular project. Adjustments to the original baselines (cost, timelines, etc), are required to come before the FPB for review and approval. This is done through a formal adjustment process, whereby the adjustment is put forward through the LTP process as a new proposal and follows the overall LTP process (i.e., uses the same templates and governance structures, etc.). Since the assumptions underpinning the original cost estimates are not formally documented, the FPB relies on the corporate memory of those involved. This is a compensating control, but it is a weak control as it relies on the permanency and availability of the key participants and could therefore limit the degree and ability of the FPBs to effectively challenge and scrutinize the changes. In April 2012, ten top non-CBA projects were transitioned by the DPM office to use the Executive Dashboard for review and monitoring by the CBA management committee.

For CBA projects, steering committees and FPBs also exercise monitoring responsibilities. The CBA management committee also exercises review responsibilities, using the Executive Dashboards, Executive Portfolio Summary and periodic risk reporting as their main tools. The Dashboards provide in-year reporting, fed both by FRS and by the project managers themselves. In-year variances in the cost or projected efficiencies are tracked, but no link to the original baseline data is presented or tracked. Therefore, the health of the project is compared to the adjusted estimates of cost and efficiencies and not to the original baseline costs. In examining the degree of scrutiny exercised by the CBA management committee over adjustments, the audit reviewed minutes and found that there was thorough scrutiny of both non-financial and financial factors. Nonetheless, because the original baseline costs and efficiencies estimates are not noted in the project dashboard, the ability of governance bodies to systematically track, monitor, understand and challenge the changes is limited and weakens the quality of information required for defensible sound decision-making.

Corrective action and feedback loop appears to be for tracking milestones and not for measuring if funding requested was sufficient and if efficiency estimates are realistic. There is also a risk that efficiencies may be "harvested" from a program area on the assumption that efficiencies have been realized, but due to the inability to effectively measure them, they have not. This could result in an unfair "taxing" of the organization which could have negative consequences in terms of program integrity.