PDF Version (PDF, 889.67 KB)
Description - Formative Evaluation of Microdata Access: Virtual Data Lab Project - Page 1
Formative Evaluation of Microdata Access: Virtual Data Lab Project - Page 1
About the evaluation
Statistics Canada evaluates programs like the Virtual Data Lab (VDL) project to ensure that they align with user needs and related internal processes are effective and efficient. The evaluation was conducted in accordance with the Treasury Board's Policy on Results (2016) and Statistics Canada's Risk-Based Audit and Evaluation Plan (2019–20 to 2023–24).
The formative evaluation examined whether existing VDL assessments were adequately comprehensive in their coverage, and whether findings and recommendations from these assessments were integrated into the design, delivery, strategic planning and implementation of the VDL project. In addition, the evaluation assessed the extent to which performance measurement and risk assessment frameworks have been implemented. Formative evaluations, the approach taken here, are conducted during program implementation in order to gather insights on how to improve or strengthen program implementation.
The conducting phase of the evaluation was completed before the COVID-19 pandemic. As a result, the findings and recommendations do not consider the activities or decisions that took place after March 2020.
About the VDL pilot
Building on the foundation of the research data centres (RDC) and the Canadian Centre for Data Development and Economic Research (CDER), Statistics Canada created the VDL pilot that—in its end state—will provide remote access to detailed, anonymized social and business microdata for research and analysis through a secure cloud-based interface.
Where is the VDL on the access continuum?
The continuum of access begins with less sensitive, self-service access and ends with most sensitive, restricted access:
- Data Liberation Initiative - This is a subscription-based service that provides unlimited access to available, anonymized and non-aggregated data in the collection.
- Access to public use microdata files - Postsecondary institutions and Statistics Canada partnerships provide faculty members and students with unlimited access to a variety of public use data and geographic files.
- Real Time Remote Access - This is an online service that allows users to run SAS programs in real time using data located in a secured location.
Research data centres, the Canadian Centre for Data Development and Economic Research, Virtual Data Labs - Secure Statistics Canada physical environments are made available to accredited researchers and federal government employees to access anonymized microdata for research purposes, ensuring that all personal information is removed from outputs.
Five Safes framework
The Five Safes framework is a holistic approach toward data and risk management. It is used to mitigate security risks, protect respondents and maintain data confidentiality. Used internationally, it originates from the United Kingdom's Office for National Statistics. The framework ensures security measures are in place and followed.
Description - Evaluation of the Consumer Price Index Program - Page 2
Evaluation of the Consumer Price Index Program - Page 2
The figure depicts the five safes along with 4 elements that tie the safes together. Overarching the safes is the governance. The five safes are considered in combination and on a sliding scale, depending on the type of researcher, access and data.
The five safes are:
- Safe people
- Safe projects
- Safe data
- Safe setting
- Safe outputs
The 4 elements are:
- Security and monitoring requirements
- Accreditation of data users and sharing accountability with the host organization
- Treatment of datasets – vetting of outputs
- Review and approval process
Microdata access user process
The user process from start to finish:
- proposal
- application
- security clearance, MRC (microdata research agreement), and oath
- deemed employee
- training and orientation
- controlled access
- vetting
What we learned
- The assessments are comprehensive in their coverage of existing users and in identifying user needs and feedback trends.
- Collectively, existing assessments are comprehensive in their coverage of all project dimensions and in alignment with desired outcomes.
- Assessments of user experience (UX), privacy and security have been leveraged to inform strategies and planning.
- An international scan of national statistical offices (NSOs) with similar capabilities and programs took place to determine best practices and assess similar program offerings around the world.
- Performance indicators have been articulated and a risk management framework exists and active risk management is ongoing.
What can be improved
- Given the level of complexity and dependence on other parts of Statistics Canada, contingencies and clearer timeframes should be articulated for activities.
- Ensure that governance mechanisms are in place that effectively manage horizontal activities in a holistic manner including the clear establishment and understanding of roles and responsibilities.
Update
Since the end of the reference period for the evaluation (March 2020), the VDL Project team has launched and completed a variety of initiatives to meet urgent emerging needs and at the same time accelerate project development. These initiatives included: launching an interim access solution to facilitate COVID-19 research; piloting a cloud environment for a project on opioids with federal partners; working closely with governing bodies to approve frameworks and approaches; and creating and revising key documents such as contracts and agreements to ensure they reflected the VDL framework and governance. The team also updated the project plan to account for progress in several areas.