Evaluation Report

June 2022

How the report is structured

• The report in short
• Acronyms and abbreviations
• What is covered
• What we learned
  1. Relevance
  2. Lessons learned and impact
• How to improve the program
• Management response and action plan
• Appendix A – Variables and data products released by the CHSP included in evaluation scope

The report in short

The Canadian Housing Statistics Program (CHSP) was launched as a modernization pathfinder project^{Footnote 1} in 2017 to provide comprehensive information on non-resident ownership and the financing of residential properties. Its first release focused on the cities of Toronto and Vancouver and has since expanded to cover other geographies and topical issues through the creation of a micro-level database. This standalone database was one of the CHSP's main deliverables and involved standardizing, cleaning, and integrating various databases (e.g., property assessment rolls, land titles, Census of Population, tax data, the Business Register, and the Longitudinal Immigration Database) from internal and external data providers.

This evaluation was conducted by Statistics Canada in accordance with the Treasury Board's Policy on Results and Statistics Canada's Risk-based Audit and Evaluation Plan (2021/2022 to 2025/2026). The objective of the evaluation was to provide a neutral, evidence-based assessment of the CHSP. The evaluation aimed at providing valuable information about the relevance and usefulness of data produced by the CHSP. It also looked at some of the lessons learned from the CHSP so far to inform future direction.

The evaluation methodology consisted of a document review and interviews. Interviews were done with Statistics Canada staff (i.e., CHSP staff, staff from divisions that partnered with the CHSP, and staff from divisions that used CHSP data) as well as with users and data providers external to Statistics Canada. The findings outlined in this report are based on the triangulation of these data collection methods.

Key findings and recommendations

Relevance

Overall, users reported that the CHSP database and data products were relevant and useful and filled important existing data gaps. CHSP data were used to generally understand the housing market, prepare reports, inform policy, conduct research, and communicate with the public.

Possible improvements in areas such as timeliness, accessibility, and available data were noted. Users also identified several needs including information about the rental market, increased granularity and increased geographic coverage. The CHSP is aware of these needs and is currently exploring options to meet them moving forward.

Lessons learned and impact

Several key lessons can be learned from the CHSP as a modernization pathfinder project. These lessons include the importance of supporting innovation, recruiting and retaining skilled staff, and developing relationships with stakeholders. The CHSP also highlighted the complexity involved and the resources required to work with administrative data.

At the broader agency level, the CHSP has highlighted the need for Statistics Canada to be clear about the complexities of working with administrative data, the opportunity to continue to support partnerships and coordination across housing divisions, and the importance of supporting innovation and expediency while managing risk.

Recommendation 1:

The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure a comprehensive strategic plan is developed that defines the CHSP's core priorities:

• The strategic plan should consider the development of new products that meet users' needs and existing gaps, the CHSP's communications goals, and provide a roadmap on how to efficiently achieve these in a standardized and sustainable way.
  • The plan should be based on a risk analysis that accounts for the CHSP's evolution from a developing program to a more established one — thus impacting the balance between innovation, expediency, and risk appetite.
• The strategic plan, either annual or multi-year, should be reviewed periodically by the ACS or appropriate oversight group.

Recommendation 2:

The ACS, Economic Statistics (Field 5), in consultation with relevant partner ACSs, should ensure that there are processes in place, informed by CHSP's lessons learned, to support the CHSP's continued collaboration with other partners across the agency. This includes:

• Developing mechanisms and/or governance structures that support coordination and collaboration across divisions that work on housing as well as clearly defining the housing divisions' roles and responsibilities.
• Assessing the CHSP's relationships with internal corporate partners (e.g., Stakeholder Relations and Engagement, the Data Integration Division, and the International Cooperation and Methodology Innovation Centre) given it is nearing the end of the first developmental phase. This assessment should identify opportunities for further collaboration, including sharing innovations the CHSP has developed, identifying opportunities to leverage internal partners' expertise, and defining their roles moving forward.
• Reviewing and documenting lessons learned from the CHSP, and sharing these lessons, including innovative in-house solutions, with key partners to promote innovation and expediency.

Acronyms and abbreviations

ACS

Assistant Chief Statistician

CHSP

Canadian Housing Statistics Program

CMHC

Canadian Mortgage Housing Corporation

CODR

Common Output Data Repository

CREA

Canadian Real Estate Association

PUMF

Public Use Microdata File

What is covered

The evaluation was conducted in accordance with the Treasury Board Policy on Results and Statistics Canada's Integrated Risk-based Audit and Evaluation Plan (2021/2022 to 2025/2026). In support of decision making, accountability and improvement, the objective of the evaluation was to provide a neutral, evidence-based assessment of Statistics Canada's Canadian Housing Statistics Program (CHSP). As a modernization pathfinder project, the CHSP pursued new and innovative approaches, which presents the opportunity to gain useful lessons and insights.

The evaluation aimed at providing valuable information about the relevance and usefulness of data produced by the CHSP. It also looked at some of the lessons learned from the CHSP so far to inform the future direction of the program as well as considerations for the broader agency.

The CHSP database and data products

The CHSP was launched in 2017 to provide comprehensive information on non-resident ownership and the financing of residential properties. As part of its work, the CHSP developed a database by standardizing, cleaning, and integrating data from multiple internal and external sources (e.g., property assessment rolls, land titles, Census of the Population, tax data, the Business Register, and the Longitudinal Immigration Database). The database contains information about residential properties and residential property owners (excluding Indian reserves and collective dwellings).

The initial rationale for the database was to provide data about non-resident property owners in Vancouver and Toronto. The scope has since been expanded to continue to address users' needs by releasing new indicators and more jurisdictions. Some examples of variables within the database include the assessed value of the property, property type, square feet of living area, age of the owner, first-time home buyer status, and residency status of the owner^{Footnote 2}. For a complete list of variables released by the CHSP at the time of the evaluation, refer to Appendix A. The CHSP intends to continue to evolve to meet user needs as it completes its initial development stage.

Using an asymmetric approach, data and data products were released as soon as development and analysis were completed. As of January 2022, the released information covers residential properties and property owners for British Columbia, Ontario, New Brunswick, Nova Scotia, Newfoundland and Labrador, Yukon, Northwest Territories and Nunavut. Refer to Figure 1 for a timeline of the CHSP's data releases. Work is underway to add the remaining jurisdictions.

The evaluation

The scope of the evaluation encompassed data products and the underlying database that were produced between the program launch to January 2022. The scope was established in consultation with CHSP leadership.

The evaluation was conducted from January to May 2022 and covered products listed in Appendix A.

Two evaluation issues and four evaluation questions were identified for review (Table 1).

Guided by a utilization-focused evaluation approach, the data collection methods outlined in Figure 2 were used.

Three main limitations were identified, and mitigation strategies were employed (Table 2).

What we learned

1. Relevance

Summary

Overall, external and internal users reported that the CHSP database and data products were relevant and useful and filled important existing data gaps. Data products, especially data tables, were used for several purposes including preparing reports, informing policy, conducting research, and communicating with the public. Possible improvements in areas such as timeliness, accessibility, and available data were noted.

The CHSP was viewed as useful and relevant and was expected to continue to be so. Data were used to generally understand the housing market, prepare reports, inform policy, conduct research, and communicate with the public.

Overall, almost all external and internal users reported that the CHSP was useful and relevant and indicated they would be interested in using the data again. Users reported several benefits of the CHSP including filling long-standing data gaps, informing conversations on hot topics, and providing evidence to help inform policy. For example, users reported that without the CHSP data, there would be a lot more guessing and estimating, particularly with topics like non-resident ownership, housing stock, and owner occupancy. As shown in Table 3, CHSP data were used for multiple purposes depending on the type of user. A few users indicated that the analytical products were useful to inform the methodology of their work.

Table 3. Common uses of CHSP data by user group

Many of the variables in the CHSP were relevant to users. Topics that were of particular interest included understanding participants in the housing market (including non-resident and immigrant owners, investor ownership, and owners' incomes), owner occupancy, and the housing stock (including property characteristics).

Users also appreciated that the CHSP data supported comparisons across jurisdictions. Even if users primarily focused on data from certain jurisdictions, such as Ontario or British Columbia, it was perceived as valuable to have multiple jurisdictions to be able to compare with and to provide additional context. Another benefit was that the CHSP published data products that were publicly available.

The relevance and usefulness were expected to increase for both external and internal users as the program expands its coverage and develops a time series, as housing was predicted to continue to be a topic of importance moving forward.

Some data gaps, including the granularity of available data, geographic coverage, and not including the rental market, influenced its relevance and usefulness.

While users were appreciative of the information available through the CHSP, they identified several gaps that affected its relevance. The most common need for external users was for information about the rental market. Currently, the CHSP covers owners and properties. It does not cover the number of dwellings (i.e., units) in a property, which impacts understanding the rental stock and the housing stock more generally. Many users wanted information about the profile of renters and rental housing (e.g., condition of buildings, vacancy rates, evictions). This was perceived as important to answer key questions about affordability especially given the proportion of the population that rents. Within the rental market, some users also wanted more information about subsidized or affordable housing, including demographics and whether residents are receiving rent supplements versus living in public or non-profit housing. There was also interest in gaining a better understanding of who owns rental structures and rental housing developments. This included specifying whether owners are institutional investors, corporations, pension funds, or individuals. Users recognized the difficulties in acquiring and integrating rental data.

Many external users also wanted more granularity in the data. While some users had a higher-level or more regional lens to their work, others, such as municipalities, needed more detailed data to make the most use of it. They reported having information at the census tract as well as more easily accessible microdata, where possible given privacy considerations, would be helpful.

Another key gap that impacted many external and internal users was that the CHSP does not yet cover all provinces and territories in Canada. For some users, this meant that the CHSP could not be their primary or only source of data or could not be used in reports given they needed a nationwide perspective.

In addition to these three gaps, external and internal users also identified the following areas as needs:

• Information about financing (e.g., how are people getting in the market and staying in the market, including topics like mortgages, gifts, the size of down payments made, levels of indebtedness)
• Building a time series, both moving forward as well as a historical time series to inform unresolved questions from 2016 and earlier
• Additional information about the properties (e.g., number of dwellings, information about additional types or uses of properties like laneway houses or vacation properties, what the property is being used for if it is not owner-occupied)
• Information about market price instead of only assessed value
• Additional information about owners and residents, including equity deserving groups, non-individual owners, inter-jurisdictional or intra-jurisdictional property owners, and who else is living in the household
• Information about mobility and migration (e.g., how often and where are people moving from/to)
• Bridging information about housing stock with information about the flow

While users accessed other housing data sources, they were viewed as complementary to the CHSP.

External and internal users often used other housing data sources, including:

• Other Statistics Canada products (e.g., the Census, Canadian Housing Survey)
• Canada Mortgage and Housing Corporation (CMHC) products (e.g., housing market reports, rental market reports, housing starts and completions data tables)
• Municipal, provincial, or territorial databases (e.g., BC Assessment, BC Housing, Yukon Housing Corporation)
• Teranet-National Bank House Price Index
• Canadian Real Estate Association (CREA), MLS, and local real estate boards
• Rental websites that they web-scraped

Users perceived minimal duplication between these other data sources and the CHSP (excluding cases where the source was an input into the CHSP like BC Assessment data). They were generally viewed as complementary sources.

Other sources were used because they contained additional variables (e.g., flow information, rental information), were timelier (e.g., sales data), had more granularity, had better geographic coverage, or the user had direct access to them (e.g., a jurisdictional data provider). Because of these reasons, a few users noted that they used the CHSP as a secondary or supplemental source of data. However, not all other data sources were perceived to be of high quality. Users also reported gaining access to other data sources could be costly or subject to non-disclosure agreements, which limited how their analyses could be used.

External users primarily used the data tables on the website to support their needs.

The CHSP developed, contributed to, and produced several data, methodological and communication products based on the underlying database, including:

• Common Output Data Repository (CODR) tables available through the website
• The Daily releases
• Analytical products that are published in the Economics Insights publication or the Housing Statistics in Canada publication (which was developed by CHSP)
• Standalone or embedded infographics
• Thematic maps
• The Canadian Statistical Geospatial Explorer Hub (formerly the Housing Data Viewer)
• Customized data products
• Microdata files available through Research Data Centres
• Metadata
• A Quality Assurance Framework which provides quality indicators for administrative data

More information about these products can be found in Appendix A.

According to external users interviewed, the most common method of accessing the CHSP data was through the CODR tables. They reported they liked to access data directly from the tables and then use it for their own purposes. The second most common method was through the analytical products such as publications in Economic Insights or Housing Statistics. According to web metrics acquired by the program, the tables and The Daily articles have the most views per month—comprising 73% of the CHSP's website views.

While experienced external users found that data products were accessible, less experienced users reported some challenges navigating the website or finding relevant tables. Potential opportunities for improvement were noted.

External users who regularly used Statistics Canada's website generally found the data products, especially the tables, easy to access. A few used an R statistical package to pull the data, which facilitated their access.

Some external users though, especially those who did not regularly use Statistics Canada's products, experienced challenges navigating the website or accessing the data. Challenges included using the search engine on the website and understanding which table was relevant for them or the differences between the tables, especially given the diversity of data the CHSP covers. A few users noted it was easier to go through an external search engine like Google instead of searching on Statistic Canada's website. Although they were not asked specifically about the Housing Portal, only one user mentioned the Housing Portal by name as a way of finding data. No other users commented on the Housing Portal, either negatively or positively.

CHSP staff helped to facilitate external users' access. Some users reported they engaged with CHSP staff who directed them to the appropriate table because it was easier than trying to find it themselves. CHSP staff also reached out to several external users to inform them of an upcoming release that may be of interest, which supported their awareness of data products.

Overall, users found the methodology clearly communicated and perceived the CHSP data to be of high quality.

Most external and internal users indicated the methodology was sufficiently clear to them for their purposes and perceived the data to be of high quality. In some cases, users shared that they had not looked for detailed information about the methodology and trusted Statistics Canada to do appropriate checks on the quality. A few external users indicated that the data quality indicators were helpful and informed their use of the data.

When external users reached out to clarify methodology with program staff, they were perceived as very helpful and contributed to reducing misinterpretations. Areas where external users needed more information or had observed others misinterpreting the data included:

• Interpreting specific definitions (e.g., non-resident vs. foreign, property vs. dwelling, assessed versus market value, the different categories of corporate investors, affordability)
• Understanding transformations performed and/or the linkage process (e.g., which years of data had been linked for tax and assessment value data, what uncertainties there were with the linkage process)

One external user also noted they could not identify which data were used in analytical products when comparing with published data tables.

While the CHSP data were timely for some users, others noted the lag time impacted their ability to use the CHSP data for up-to-date monitoring of the housing market.

While some users indicated the data were timely, especially as housing is a hot topic, others noted that the lag time in data releases impacted how they could use the CHSP data. In particular, not having up-to-date data during the COVID-19 pandemic affected its usefulness. They felt the CHSP was better suited to inform longer-term understanding instead of current or up-to-date monitoring. Users acknowledged the difficulties in having timely data due to the delays that are outside the CHSP's control, such as in processing assessment and tax data.

Some external users expressed that having more frequent releases (e.g., quarterly for some indicators) would be beneficial. However, annual releases, especially once a time series was built up, were still appreciated and were viewed as filling a gap between census years.

Summary

Users identified several areas of improvement for the CHSP that would support data products to be more relevant and useful for their needs. These needs include filling data gaps, increasing timeliness, improving accessibility, and improving clarity about the methodology. Due to ongoing engagement with users, the CHSP is aware of these key data gaps and needs and is currently exploring options to meet them.

Engagement with users has supported the CHSP to be aware of key gaps that affect the relevance of its data. The CHSP plans to address these gaps when possible given limitations in data availability.

The CHSP regularly engages with its external users to inform the program's direction. They have a "parking lot" of ideas where they store and prioritize users' requests. Through their user engagement, the CHSP is aware of the gaps identified in this evaluation, such as the granularity of data, geographic coverage, and the rental market.

They have plans to explore opportunities to address users' needs where possible recognizing that there are several challenges including data availability, data quality, and cost as well as privacy considerations. In addition to filling gaps in available data, they also are working on streamlining and standardizing their processes to increase the timeliness of data releases. Given the number of users' needs, staff identified that it will be important for the CHSP to prioritize needs (considering importance as well as feasibility) to allow staff to focus on completing key tasks.

Some ways the program is currently aiming to meet users' needs include expanding geographic coverage. This is a high priority, and they are currently working on adding the remaining jurisdictions. They also are looking into acquiring additional data sources and partnering with new data providers to help address some of the identified gaps. Another way they are trying to fill data gaps is by developing innovative methods, such as using artificial intelligence and Google Maps.

Additionally, the CHSP wants to increase the granularity of data released by developing a synthetic micro-level data set for the general public. This is a similar concept to Public Use Microdata Files (PUMFs) and addresses key privacy and confidentiality concerns because all the data are synthesized.

While the CHSP is aware of data gaps, a few external users identified that it would be beneficial to conduct further engagement with users and partners about which analytical products would be helpful. This is particularly the case if an analytical product focuses on one specific jurisdiction instead of looking across all available jurisdictions.

Users suggested several ways to improve the accessibility of the data and the clarity of the methodology.

In addition to filling data gaps, external users identified a need to improve the accessibility of CHSP data. Suggestions included:

• Having videos or webinars to orient users to Statistic Canada's website/tables
• Making a specific page for the CHSP that shows the latest releases or data
• Clarifying what information is included in each data table
• Categorizing tables thematically
• Having non-static tables that allowed users to select the data for the subset they were interested in (versus the current tables that are fairly fixed in terms of what can be included/excluded)
• Allowing users to calculate percentages within the CODR tables
• Making the microdata more accessible (e.g., through a PUMF or improving remote access options)
• Providing examples for different audiences on the website (e.g., "if you are municipal staff, you may want to look here")
• Providing a publishing schedule so users can prepare for the data

While generally the methodology used was clear, external users identified opportunities to further improve the clarity and reduce misinterpretations. Suggestions included:

• Highlighting when terminologies may be different than other Statistics Canada products (e.g., non-resident versus foreign)
• Providing definitions in both technical and plain language and ensuring all definitions are included in the glossary
• Highlighting any key differences in data that would affect comparability between provinces or overtime
• Increasing accessibility of metadata and methodology (e.g., a directory of methodological papers or tab showcasing the structure to different methodology-related pages)
• Keeping historical records of data tables that get updated to support reproducibility of previous work
• Explaining why some variables are only available at certain levels (e.g., provincial, census metropolitan area, census subdivision)
• Providing more support to help users understand what can be done with the data or what can be asked to support use and reduce misinterpretation (e.g., the current data are about stock, not about flow)

2. Lessons learned and impact

Summary

Several key lessons can be learned from the CHSP as a modernization pathfinder project. These include the importance of supporting innovation, recruiting and retaining skilled staff, and developing relationships with stakeholders. The CHSP also highlighted the complexity involved and resources required to work with administrative data and the need to balance innovation and expediency with risk management.

The CHSP has had many successes so far, including creating a comprehensive database of residential properties and owners despite the complexity of this task. Several factors contributed to their successes, including encouraging and implementing processes to increase efficiency as well as having a strong team, intentional stakeholder engagement, and the ability to be flexible, dynamic, and innovative.

The CHSP has had many successes over the last five years, including those that move forward Statistics Canada's modernization program and each of the modernization pillars^{Footnote 3}. One of the CHSP's most notable successes, creating a comprehensive database of residential properties and owners by linking over 20 different data sources (including unstructured and unformatted data), advanced the Leading-edge Methods and Data Integration pillar. This task was very complex and involved many internal and external partners.

Through their work creating the database, the CHSP also contributed to leading-edge statistical advancements, frameworks, and processes. These advancements include developing data linkage processes, partnering with Statistics Canada's International Cooperation and Methodology Innovation Centre to create quality indicators for administrative data (a first internationally), and exploring the use of tools like artificial intelligence to fill data gaps. This is another example of how the CHSP contributed to the Leading-edge Methods and Data Integration pillar as well as the Sharing and Collaboration Modernization pillar due to its work with internal partners. The CHSP's work also led them to contribute more broadly to Statistics Canada's frameworks and processes. For example, CHSP staff supported the development of Statistic Canada's Necessity and Proportionality Framework and have helped to refine data acquisition processes for the agency.

One factor that supported the CHSP's achievements was staff and management's desire to develop and implement processes to innovate and increase efficiency to meet their project timelines and deliverables. Some examples of processes they developed, in addition to their work standardizing and streaming processes across multiple phases (e.g., processing, production), include:

• A web metrics dashboard from web scraping the Statistics Canada's website
• A machine learning Python algorithm to review resumes to identify skilled candidates
• An online platform to record questions received from users and related documentation
• An integrated release workflow to reduce the amount of manual work required for The Daily releases by using LaTeX

Another enabler was their ability in assembling a team of passionate individuals with diverse skills. The CHSP was able to do this in part because candidates were interested in contributing to highly innovative, leading-edge work. They also used various recruitment mechanisms, such as requesting Human Resources provide the entire list of candidates so they could use algorithms to review resumes, recruiting from across the country, and promoting that they were hiring at public events they were at. This supported them to hire many skilled and knowledgeable candidates. Once candidates were hired, the CHSP provided support such as assigning mentors and promoting professional development. They also started having team members work on multiple stages of a process, which has helped staff to stay engaged and develop more knowledge about the program. The work done to hire talented staff and develop a supportive environment supports the Modern Workforce and Flexible Workplace pillar.

The CHSP also actively engaged with and developed relationships with users and data providers to help better understand their needs. This allowed them to be up to date about user needs, support users to access and use the data (including customized requests), and change directions or priorities according to users' needs. They have also reached out to international partners, including the Organisation for Economic Co-operation and Development, and built relationships with academic institutions, such as Ryerson University. This work has helped the CHSP advance the User-centric Delivery Service pillar as well as the Statistical Capacity Building and Leadership pillar as staff supported users to understand how to use the data correctly. CHSP staff reported that they benefited from being a new program as they had the flexibility and opportunity to respond to users' needs in ways that were not possible for more established Statistic Canada programs. Developing relationships with data providers also led to stronger partnerships and a smoother data acquisition process. Staff reported that it was beneficial having a direct link with data providers so that their team could communicate efficiently to troubleshoot technical problems.

The CHSP's experience highlighted the challenges of working with administrative data, developing an adaptable framework that can accommodate future changes, and coordinating housing work done across Statistics Canada.

The CHSP's work highlighted the challenge and complexity involved in working with administrative data. Challenges were encountered across all stages of working with the data, including addressing privacy concerns to acquiring, cleaning, validating, and processing data. In particular, the CHSP highlighted the general challenges associated with the privatization of data as well as the associated costs of acquiring data, which in some cases can be prohibitive. The CHSP also experienced challenges in making sure that the data ultimately are useful and comparable, especially because data providers across jurisdictions collect and store data differently. Addressing these challenges related to working with administrative data required sufficient resources, having time to make sure things were done properly, and working on standardization processes to increase timeliness. It was also beneficial for the CHSP to partner with two internal divisions, Strategic Analysis, Publications and Training and the Social Analysis and Modelling Division, to prepare analytical products early on as this helped to test the data.

Given the dynamic nature of the program and the asymmetrical approach it took to releasing data, the CHSP also demonstrated the need to have an adaptive framework and strategic plan that can accommodate future changes. The CHSP was under pressure to deliver results quickly, which required working with data as it was acquired instead of waiting to acquire and process all data simultaneously. Given challenges in acquiring data from all jurisdictions in a timely manner, this approach meant the CHSP was able to publish data for at least some jurisdictions. However, this approach also meant that the CHSP had to consider how easy it would be to revise previous data. The pressure to deliver also meant that there was a lot of focus on getting things done and evolving to meet users' many and immediate needs. Staff noted though that it could be beneficial to take the time to plan a roadmap and ensure priorities and processes are efficient, standardized, and focused.

In Statistics Canada, several divisions work on or with housing data. The CHSP's experiences highlighted that there are opportunities to continue to promote coordination amongst these divisions. While the CHSP convened a working group for these divisions and participates in joint committees with external partners, it was noted there were varying levels of participation and that more work could be done to have improved coordination and one voice for housing. Collaboration was viewed as more dependent on an individual's actions instead of structures or mechanisms, which could result in duplication or gaps. Additionally, it was reported that the dispersed nature of housing meant external partners did not always know which division to approach with housing-related requests. This meant there were occasions where divisions responded to external requests when another division could have also contributed to the work.

The CHSP has shown potential risks that need to be considered, including the need to support staff retention, the reliance on being able to continue to obtain external data sources, and the balance required to manage innovation and expediency with risk.

There are also lessons to be learned from the potential risks the CHSP faces. One risk is staff retention and the associated risk of loss of knowledge. The CHSP has experienced staff turnover due to a variety of factors, such as moving due to promotions, being on rotations, or language requirements. Some staff attributed turnover in part due to their success in attracting strong candidates who end up advancing quickly in their careers. They also noted that while remote work was beneficial because it could allow for recruitment from across Canada, it also made it more difficult to develop a team dynamic and onboard people in a large agency like Statistics Canada. Suggestions to reduce turnover included continuing to recognize staff achievements, assign varied and interesting work, continue to innovate, support staff development, have a webinar that walks through Statistics Canada's organizational structure, allow staff to focus on a few priorities, and communicate a vision.

Another risk is the reliance on external data providers for data. Because the CHSP does not collect its own data, it must be able to continue to acquire data from its external partners. This makes the CHSP vulnerable to risks such as increasing data costs or data providers no longer being willing to provide the data. Given the importance of continuing to acquire the data, it will be important to maintain relationships with data providers. Some data providers suggested that it would be beneficial to close the loop about what has occurred with the data and share opportunities on how to use the data with them. The CHSP is also reliant on being able to obtain additional data that are important for its continued relevance, such as financial data. This requires overcoming challenges such as privacy concerns, costs, and availability.

As a pathfinder project, the CHSP was expected to contribute to new and innovative thinking as well as use an experimental approach to program delivery. They had pressure to deliver a program quickly, and the work was very complex due to the amount and type of administrative data they acquired. To accomplish its deliverables, the program pushed for innovation and efficient solutions. As a result of this, and because of the skills their staff had, they developed many in-house processes. While this supported their goals of delivering the database under a time pressure, their experience also highlights the balance between promoting innovation with managing risk. This can be seen across several of their processes, including data linkage, methodology, and communications. For example, while their in-house processes have supported them to deliver results, the CHSP is more reliant on individual staff's knowledge than it would be if it partnered more extensively with Statistics Canada's service providers, such as the International Cooperation and Methodology Innovation Centre, the Data Integration Division, and Stakeholder Relations and Engagement. Another example is the CHSP's relationships with media stakeholders. They have developed their own relationships which are beneficial and allow for efficiency. However, Statistics Canada's Communication Branch has processes to manage risk, so if they are not engaged early in the process there is a trade-off between expediency and risk. This example highlights the inherent trade-off tension between delivering results quickly while still needing to go through all the agency's existing approval processes.

Summary

Learnings from the CHSP can also be applied to the broader context and practices of Statistics Canada. Key lessons include the importance of being clear about the complexities of working with administrative data, supporting partnerships and coordination across housing divisions, and supporting innovation and expediency while managing risk.

At the broader agency level, the CHSP highlighted a key lesson of having clear expectations about the role of administrative data as well as the importance of having sufficient resources, relationships with data providers, and social consent.

As a pathfinder project, lessons from the CHSP can be considered more broadly in the context of Statistics Canada's practices. One important lesson from the CHSP is the importance of clear expectations, internally as well as externally, about the role of administrative data. This includes acknowledging what it can and cannot do. While there was perceived value in acquiring administrative data to answer key questions, staff identified the importance of understanding the resources (i.e., expertise, time, and funding) required to acquire and work with administrative data. Additionally, staff suggested that future projects with administrative data should not be rushed and ensure there is time to understand the data.

Working with administrative data at this scale also highlighted that Statistics Canada will have to consider what approaches it is willing to take to acquire data. While Statistics Canada's legislative authority was a factor in why data providers were able to share data with the CHSP, it was still important to build relationships with them. Data providers indicated that being open, responsive, and respectful as well as taking time to build trust and a partnership facilitated their relationship with Statistics Canada. To support acquiring data for programs such as the CHSP, internal staff also felt it was important for Statistics Canada to continue to demonstrate the value of this work to Canadians and to get social consent.

Supporting internal partnerships was a key lesson in strengthening work and reducing duplication.

The CHSP showed the importance of supporting internal collaboration within Statistics Canada. Collaboration helped to strengthen divisions' work and was perceived as a way to develop sustainable, standardized solutions. For example, the International Cooperation and Methodology Innovation Centre helped develop quality indicators for the CHSP and the CHSP helped improve corporate tools developed by the Data Integration Division with their expertise in data matching and property assessments. The internal partners shared that it was beneficial to develop partnerships instead of only being seen as service providers. They also highlighted that being engaged early in the project would increase the value of their contributions, reduce duplication, and would support proactive rather than reactive engagement. Sharing goals, deliverables, and long-term projects would also help with alignment between divisions. One internal partner suggested that formalizing collaborations on projects at the working level, in addition to the senior executive level, would be valuable.

There are opportunities to continue to support coordination between divisions that work on housing.

The CHSP also highlighted the importance of continuing to support coordination and alignment between divisions that work on housing across Statistics Canada. This includes establishing clear roles and responsibilities for these divisions. Time restraints and the need to focus on their own deliverables were perceived as barriers to collaboration. Internal staff reported that current systems were not robust enough to avoid duplication or gaps amongst housing divisions. Suggestions to support coordination included having one housing division, more joint collaborative meetings, program leadership on other programs' steering committees, or a focal point for housing. Past presentations about the CHSP and its database were also seen as helpful. Additionally, supporting increased internal collaboration was perceived as not only beneficial for internal processes but would also support engagement with key external partners.

The CHSP used innovative approaches, but their experience highlights the inherent trade-off between risk management and expediency.

The approaches the CHSP used to support innovation provide an important lesson for the broader agency. Having a real project with objectives, a need to produce, and constraints gave focus to the work done by the CHSP. Staff reported that having an entrepreneurial spirit and an openness to being innovative were beneficial. One suggestion to support innovation moving forward was to root development initiatives within a program. Co-locating teams could help develop integrated, sustainable solutions. While there were trade-offs, using an agile continuous development approach that pushed things out before acquiring data for all jurisdictions was generally viewed as a novel approach and a strength. Being able to develop in-house solutions also supported the CHSP try new things without being tied to existing processes. The CHSP's innovative work though also highlights the inherent trade-off between expediency and risk and the need to balance these. Some staff reported that a higher risk tolerance, greater staff empowerment, and streamlining approvals would support them to do their work. There is a future opportunity to determine the desired risk tolerance for pathfinder projects, such as the CHSP, while acknowledging that individual programs do not stand alone and have an impact on the agency's processes and reputation.

How to improve the program

The CHSP has had a strong start in developing relevant and useful data for users, especially accounting for the complexity of the data and length of time it has been operating. The CHSP's users have many and evolving housing data needs, many of which are difficult to meet with available data sources. Given that the CHSP is now transitioning out of its initial development phase, there is the opportunity for the CHSP to work on its strategic plan to define its priorities and provide a roadmap of how to achieve their goals. A strategic plan would also help to prioritize the development of new data products that fill existing gaps and meet users' needs.

Recommendation 1:

The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure a comprehensive strategic plan is developed that defines the CHSP's core priorities:

• The strategic plan should consider the development of new products that meet users' needs and existing gaps, the CHSP's communications goals, and provide a roadmap on how to efficiently achieve these in a standardized and sustainable way.
  • The plan should be based on a risk analysis that accounts for the CHSP's evolution from a developing program to a more established one — thus impacting the balance between innovation, expediency, and risk appetite.
• The strategic plan, either annual or multi-year, should be reviewed periodically by the ACS or appropriate oversight group.

As a pathfinder project, the CHSP has several lessons learned that can be considered more broadly at the agency level such as the importance of collaboration within Statistics Canada. Moving forward, continuing to support collaboration with key partners could help to increase coordination amongst divisions that work on housing, identify how to best leverage partners' expertise to support the CHSP as it moves forward into its next phase, and share learnings and innovative processes from the CHSP.

Recommendation 2:

The ACS, Economic Statistics (Field 5), in consultation with relevant partner ACSs, should ensure that there are processes in place, informed by CHSP's lessons learned, to support the CHSP's continued collaboration with other partners across the agency. This includes:

• Developing mechanisms and/or governance structures that support coordination and collaboration across divisions that work on housing as well as clearly defining the housing divisions' roles and responsibilities.
• Assessing the CHSP's relationships with internal corporate partners (e.g., Stakeholder Relations and Engagement, the Data Integration Division, and the International Cooperation and Methodology Innovation Centre) given it is moving out of its development phase. This assessment should identify opportunities for further collaboration, including sharing innovations the CHSP has developed, identifying opportunities to leverage internal partners' expertise, and defining their roles moving forward.
• Reviewing and documenting lessons learned from the CHSP, and sharing these lessons, including innovative in-house solutions, with key partners to promote innovation and expediency.

Management response and action plan

Recommendation 1:

The ACS, Economic Statistics (Field 5), should ensure a comprehensive strategic plan is developed that defines the CHSP's core priorities:

• The strategic plan should consider the development of new products that meet users' needs and existing gaps, the CHSP's communications goals, and provide a roadmap on how to efficiently achieve these in a standardized and sustainable way.
  • The plan should be based on a risk analysis that accounts for the CHSP's evolution from a developing program to a more established one — thus impacting the balance between innovation, expediency, and risk appetite.
• The strategic plan, either annual or multi-year, should be reviewed periodically by the ACS or appropriate oversight group.

Management response

Management agrees with the recommendation.

The CHSP will develop a comprehensive strategic plan outlining CHSP's short (1 year), medium (3 years) and long term (5+ years) priorities based on a risk analysis. The plan will include:

• An elaboration of new data and data products based on the needs of its users founded on extensive and ongoing stakeholder engagement. As part of the risk analysis, the development of new data and data products will consider quality, relevance, timeliness, data availability, processing complexity and the costs involved in creating new data products.
• A road map, which includes optimization and standardization strategies to produce efficiencies. Consultations with internal stakeholders will be conducted as part of the risk analysis to ensure the solutions being proposed lead to a balanced risk profile for the stability and agility of the program.
• A proactive CHSP communications plan developed in collaboration with Field 4 that seeks the right balance between innovation, expedience and risk appetite to enhance Statistics Canada's role as a leading centre for Canadian housing data and expertise.

Deliverables and timelines

A comprehensive 5-year strategic plan approved by ACS. The strategic plan will be reflective of a risk analysis, address new products that meet users' needs and existing gaps, and include a communication plan and a roadmap. (February 2023 and annually refreshed afterwards)

Recommendation 2:

The ACS, Economic Statistics (Field 5), in consultation with relevant partner ACSs, should ensure that there are processes in place, informed by CHSP's lessons learned, to support the CHSP's continued collaboration with other partners across the agency. This includes:

• Developing mechanisms and/or governance structures that support coordination and collaboration across divisions that work on housing as well as clearly defining the housing divisions' roles and responsibilities.
• Assessing the CHSP's relationships with internal corporate partners (e.g., Stakeholder Relations and Engagement, the Data Integration Division, and the International Cooperation and Methodology Innovation Centre) given it is nearing the end of the first developmental phase. This assessment should identify opportunities for further collaboration, including sharing innovations the CHSP has developed, identifying opportunities to leverage internal partners' expertise, and defining their roles moving forward.
• Reviewing and documenting lessons learned from the CHSP, and sharing these lessons, including innovative in-house solutions, with key partners to promote innovation and expediency.

Management response

Management agrees with the recommendation.

The CHSP will engage in a consultation/review exercise:

• With internal housing stakeholders to determine appropriate mechanisms or governance structures to better strategically align the work of the CHSP with other housing program areas, enable more collaboration, and avoid duplication, to better serve Canadians and meet the evolving housing data needs of its users.
• With internal corporate partners to assess current collaborations, identify opportunities for further collaboration, and share innovations with a goal of defining roles moving forward with the aim of reducing costs, improving timeliness and data quality. As part of this assessment, readiness and capacity of internal partners would be considered.
• Internally, to identify key lessons learned that are deemed relevant and useful for the agency with its key partners. Lessons will be shared with divisions that are implicated in the lessons learned.

Deliverables and timelines

• Documented governance structures, processes and/or roles and responsibilities developed in collaboration with internal partners. (April 2023 and periodically refreshed)
• Documented assessment of internal partnerships and identification of opportunities for further collaboration. (January 2023 and periodically refreshed)
• Lessons learned from the CSHP documented and shared with key partners. (November 2022)

Appendix A – Variables and data products released by the CHSP included in evaluation scope

List of variables released by the CHSP included in the evaluation scope

• Admission category of immigrant, category
• Age of property owner, category
• Age of property owner, number
• Assessment value of residential property, category
• Assessment value of residential property, value
• Birth year of property owner, category
• Condominium status of residential property, category
• Employment income of person, status
• Family income of person, value
• Family size of census family, number
• First-time home buyer status of person, status
• Geographic location of residential property, name
• Home buyer's amount claimant status of property owner, status
• Immigrant status of person, category
• Industry of property owner, type
• Legal type of property owner, category
• Living area of residential property, area
• Living area of residential property, category
• Marital status of person, category
• Number of buyers as part of a property sale of property buyer, category
• Number of owners of residential properties, category
• Number of residential properties, count
• Number of residential properties owned of property owner, category
• Ownership type of property owner, category
• Ownership type of residential property, category
• Owner status of person, category
• Period of construction of residential property, category
• Place of birth of person, name
• Price-to-income ratio of property buyer, number
• Property type of residential property, category
• Property use of residential property, category
• Residency status of property owner, category
• Residency status of residential property, category
• Residential properties assessment value of property owner, category
• Sale price of property, value
• Sales of property, status
• Sex of person, category
• Tax filer status of person, status
• Total income of person, value
• Type of census family, category