(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Geoprocessing Tools (Part 1)")
So today we'll introduce geoprocessing tools, which enable layers to be spatially overlaid and integrated in a variety of ways. These tools epitomize the power of GIS and geospatial analysis, facilitating combining feature geometries and attributes, whether it be assessing spatial relations, distributions or proximities between layers and associated variables of interest.We'll demonstrate these tools with a simple case-study, examining land-cover conditions near water features, also known as riparian areas, in southern Manitoba.These tools can be reapplied and iterated with multiple layers, enabling you to combine, analyse and visualize spatial relations between any variables, geometries and layers of thematic relevance to your area of expertise.
So first, the Merged Census Division feature from the AOI layer was selected and subset to a new layer – CAOI – since Selected Features is not available when running tools as a batch process.
In addition to the interactive and attribute selection tools covered previously, there is one final type – Select by Location. This selects features from the input layer according to its spatial distribution relative to a second layer and the selected geometric predicates. The predicates define the particular spatial relations used when selecting features. We'll use Intersects, Overlaps and Are Within. Multiple predicates can be used, provided they do not conflict. And processing times increase with the number of selected predicates. At the bottom, the alternative selection options are available in the drop-down, but we'll run with the default.
So most selected features match the predicates but two spatially disconnected features were also returned due to a common attribute. So now we'll use the Multipart to Singlepart tool to break the multi-polygons into separate features, running with Selected Features Only.
Now we'll use a slight variation of Select by Location - Extract by Location. Instead of creating feature selections in our input layer, this will generate a new layer. So matching the predicates and comparison layer to those used in Select by Location, we'll click Run. In addition there is also Join by Location, which enables fields from the second layer to be joined to the first according to the predicates and the specified join type – as one-to-one or one-to-many. So these by Location tools enable features to be selected or extracted and field information joined between layers according to their relative spatial distributions.
So now we'll merge the land-cover 2000 layers into one file with the Merge Vector Layers tool. Open the Multiple Selection box and select the four land-cover files. We'll also switch the Destination Coordinate Reference System to WGS84 UTM Zone 14 for spatial analysis. Click run with a temporary file. So merge can be applied to vectors of the same geometry type. It works best when layers contain the same fields and cover distinct yet adjacent areas – making the land-cover layers highly suitable. Two additional fields specifying the originating layer and file path for each of the features is included in the output.
While Merge is running, we'll reproject the watershed layer to the same Coordinate Reference System for consistency in our spatial analysis.
Now we'll join the provided classification guide with the class names to the merged output, using the Joins tab. So code is the Join Field and COVTYPE - the Target Fields. We'll join the Class field and remove the prefix. Now we can run the merged layer through the Fix Geometries tool to accomplish two tasks simultaneously. First it will fix invalid geometries – critical for adding spatial measures and applying geoprocessing tools, while also permanently joining the Class fields. The process may take a few minutes to complete.
So now we'll rename the Reprojected and Fixed layers to PTWShed for projected tertiary watershed and FMLC2000 for fixed merged land-cover 2000. This will enable us to use the autofill settings to populate the file paths and names when running Clip as a Batch Process. So open Clip from the Toolbox and click Batch Process.
As we've covered, the Clip tool helps standardize the extent of analysis for multiple layers to an area of interest, or reduce processing times and file sizes in a workflow. The inputs can be of any geometry type while the Overlay Layer is always a polygon. Features and attributes that overlap with the Overlay Layer are retained, with the Overlay Layer acting like a cookie cutter on the input.
So select FMLC2000and PTWShedas the input and select CAOIas the Overlay Layer. We can then Copy and paste it into the next row – which we could repeat for as many entries as required. We'll click the plus icon and copy PTWShed for the Input to prepare this layer for an upcoming demo. Here we'll use Manitoba Outline as the Overlay layer. For the output files we'll store them in a Scratch folder, for intermediary outputs in our workflow which can then be deleted at the end of part 2 of the demo. Enter C for the filename, and click Save and then use Fill with Parameter Values in the Autofill settings drop-down. This adds a C prefix to our existing layer names. We'll store the last file in the Geoprocessing folder so that it is retained. Click Run and we'll pick back up once completed. The process takes around five minutes to complete.
So with the clipped layers complete, load them into the Layers Panel. I'll move them back into the Processing Group for organization purposes and then zoom in on the layers.
We can load the provided symbology file to visualize the different land-cover classes.
Then we'll add an area field to the clipped land-cover file. Call it FAreaHA for field area, using a decimal field type with a length of 12 and a precision of 2. We'll reuse these parameters for adding subsequent numeric fields. Enter the appropriate expression - $area divided by 10000.
Now we'll use Select by Expression to isolate 'Water' features using "COVTYPE" = 20 or "Class" LIKE 'Water' – and then click Select Features.
Now we'll generate a Buffer around the selected features to begin creating the Riparian area layer. There are many Buffer tools available in the Processing Toolbox – which we'll demonstrate in Part II – here using the default tool.
We'll check 'Selected features only' box and enter 30 for the distance – a common riparian setback in land-use planning and policies. Change the End Cap Style to Flat and check Dissolve Results, so that any overlapping buffers are merged to avoid conflating total area estimates. Run with a temporary output file. We'll rerun the tool toggling back to the Parameters and changing the distance to 0, to output Water features as their own temporary layer – reducing processing times for the next tool.
Buffer tools can be applied to any vector geometry type. And they are used to assess the proximity of features to those in other layers. We can also use buffers to facilitate combining our geometries and attributes with other layers – like buffering lines or points to use them as a difference layer. The buffer contains the input layer's attributes, which can be used for further analysis. The outputs are often applied with other geoprocessing tools for further examination.
So we'll rename the outputs, naming the first B30W and the second LC2000Water, to facilitate their distinction.
Zooming in on the buffer, the input water features were also included in the output geometry.Since we are not interested in water features but the land-cover conditions around them we'll run the water buffer through the Difference tool using LCWater2000 as the Overlay Layer to retain only the buffered area. So difference is the opposite of Clip – retaining only input features that do not overlap with the Overlay layer. Like Clip – the input can be any geometry type, while the overlay layer is always a polygon. Difference can be used whenever we are interested in features that do not overlap with a specific polygon, such as areas external to a certain drive or distance from hospitals or farm fields, roads or grain elevators not impacted by historical flooding.So click Run and we'll continue once the output is complete.
Toggling the water layer off, we can see that the Difference has retained only our 30 metre buffer. So now we've successfully generated our riparian area layer but need to follow up with the Intersection tool – running it twice to extract watershed codes and land-cover classes to our layer. Intersection retains the overlapping feature geometries of the input layers and any selected attributes of interest in the Fields to Keep parameter. If geometry types differ between layers, the first layer's geometry is used in the output. Thus, Intersection can help combine variables of interest from multiple layers.
For the first run we'll use the Difference and clipped watershed layers as the inputs to assign watershed codes to the riparian buffer. This will enable us to examine land-cover conditions by watershed in Part II of the demo. And for PTWShed check the sub-basin code field in the Multiple Selection box. For the Difference layer, we'll select an arbitrary field for the Fields to Keep parameter – here selecting the "layer" field, clicking OK and then clicking Run. This process takes around 5 minutes and we'll continue when complete.
Within the Attribute Table we can see watershed codes have been successfully assigned to the riparian layer. Now we'll run the tool again, using the intersect as the Input and the clipped land-cover file as the Overlay layer to integrate the land-cover features in the riparian areas. We'll retain the watershed code field from the first layer and the "Class" and "FAreaHA" fields from the land-cover. We'll save it to file, storing it in the main geoprocessing folder and calling it RipLC2000 for riparian land-cover 2000. If the tool fails, use Fix Geometries tool and rerun the Intersection with the fixed outputs. We'll pick back up after the layer is created, which may take up to 20 minutes.
With the riparian land-cover layer loaded copy and paste the style from the clipped land-cover to visualize the different feature classes occupying these areas. Now we've successfully combined the riparian buffer by watershed with the land-cover layer. And for the final component of Part I we'll add four new fields with the Field Calculator, specifically the intersected area in hectares, to determine the area of each land-cover feature within the buffered riparian area. Use the same parameters and expression as applied for creating the FAreaHA field.
So next we'll calculate the percentage of each feature within the 30 metre buffer, to assess the relative distribution of the original features within the riparian setback and isolate any potential violating land-uses.We'll call the field PrcLCinRip, for percent land-cover in riparian area, with the same parameters as the previous fields. Expanding the fields drop-down, we'll divide IAreaHA by FAreaHA and multiply by 100.
The next two fields are to create an identifier which combines the subwatershed codes and land-cover class fields which we'll use to aggregate and assess riparian land-cover by watershed. First is an FID field or FeatureID, which we'll use for the Group_By parameter when using the concatenate function. Leave the parameters in their defaults and double-click the @row_number expression.
Now we can use Concatenate to combine our fields in creating the ID. This is extremely helpful for further processing and analysis, such as distinguishing and rejoining different processed layers to original features or aggregating datasets by different criteria. So we'll change to a text field type with a length of 100 and call it "UBasinLCID".
So type concatenate in the expression box – specifying the function to apply, and then open bracket and double-click SUBBASIN in the fields and values drop-down. Using the separators and adding a dash in single quotes will help separate the codes and class fields for interpretability. As noted, the FID field is used for the Group_By parameter, writing group underscore by, colon, equal sign and then double-clicking the FID field.
We can see the combined fields in the output preview. Given the number of features, the concatenated function can take up to 30 minutes to create. After it's complete, ensure to save the edits to the layer and the project file with a distinctive name for use in Part II of the demo.
By the end of this video, you will learn about the basic concepts of the analytical process:
the guiding principles of analysis,
the steps of the analytical process and
planning your analysis.
Data journey step
Analyze, model
Data competency
Data analysis
Audience
Basic
Suggested prerequisites
N/A
Length
8:13
Cost
Free
Watch the video
Analysis 101, part 1: Making an analytical plan - Transcript
(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Analysis 101, part 1: Making an analytical plan")
Analysis 101: Part 1 - Making an analytical plan
Hi, welcome to analysis 101 video one making ananalytical plan.
Learning goals
By the end of this video you will learn about the basicconcepts of the analytical process: the guiding principlesfor analysis, the steps in theanalytical process and planningyour analysis. This video is intended for learners who wantto acquire a basic understanding of analysis. No previousknowledge is required.
Analysis at your organization
Take a second to think about analysis at your organization.What role does analysis play?Are you and your colleagues producing briefing notes forsenior leadership? Are you writing reports for clients orfor your website? Are you doing more technical ordescriptive work?Does your organization have guiding principles that youshould be aware of?You'll be taking these into consideration whenyou plan your analysis.
Steps in the analytical process
(Diagram of 6 images representing the steps involved in the analyze phase of the data journey where the first steps represent the making of an analytical plan, the middle steps represent the implementation of said plan and the final steps are the sharing of your findings.)
On this slide you can see that there are six main steps in theanalytical process and each is related to making a plan,implementing that planor sharing your findings.We will explain the main activities that you willneed to undertake within each step. If you've beenwatching statistics candidates, data literacyvideos, you'll recognize that this work is part ofthe third step: the analyze phase of the data journey.This diagram is the backbone of our analytical process. We willcome back to it in each of the videos in this series.
Step 1: What do we already know?
For this video and planning, your analysis will start byunderstanding the context and investigating what we alreadyknow about a topic.Start by ensuring you fully understand the broader topic andthe context surrounding it, and think through the following questions.What do we already know about the topic? Has one ofyour colleagues already done asimilar exercise? Start by reviewing any previous work doneon the topic.Once you've read up on the topic, you can identify theknowledge gaps. What is missing in the previous work?This will help you realize how your projects add value.
Example
To make sure you understand these steps, let's go through anexample together. This is from a study on the economic well beingof millennials. This study was motivated by a lack ofinformation on financial outcomes for Canadianmillennials.
Millennials-Context
When we began work for this study, we knew that Millennialswere often stereotyped by the media's still living in theirparents basement. Spending too much on takeout food, and so on.We also knew that a study by the United States Federal ReserveBoard had shown that American Millennials had lower incomesand fewer assets than previous generations had at the same age.What were the knowledge gaps? Despite anecdotal media reportson millennials, we knew that there wasn't a detailed studyassessing the economic well being of Canadian millennials.
Millennials-Relevance
Why is our analysis relevant? Well, housing affordability andhigh debt levels have been identified as concerns foryounger generations earlier intheir life. From this we knew the topic was relevant forpolicy makers. Journalists and Canadians will return tothis example later.
Step 2: What do we already know?
Back to our analytical process.The next step is to define your analytical question.
What is your analytical question?
How do you define youranalytical question? Very clearly state the question youare trying to answer and use plain language simply.This means using vocabulary that an eighth grader could understand.You might have one main analytical question followed bysome supporting questions.Why is your question relevant? Why should we care about your work?Define the value that your analysis adds either to yourorganization, your client, or to our understanding of the topic.
Plan your analysis
Now that you have a relevant question, how will you answer it?This is the perfect time for you to put together ananalytical plan which provides a road map for answering youranalytical question. You will need to think about the contextof your topic and how you will answer your question.What data and methodology are needed to answer your question?You will also need to think about how you will communicateyour results, whether through a briefing note, analytical paper,infographic or presentation.
Identify your resources
Now that you have your analytical plan, think aboutyour resources. Feedback is an essential element of youranalytical journey and you should leverage input fromcolleagues at every step. Typically we will put together ashort plan for colleagues and management to review.Maybe some of your colleagues have expertise on the topic youare working on. Colleagues might also have expertise in the datayou are using or yourmethodology. Our colleagues are often in the best position toprovide tips and feedback and to help us work through problems.
Millennials-Analytical question
For our example about the economic well being of CanadianMillennials. Our main analyticalquestion was: Are Millennials better or worse off thanprevious generations of the same age in terms of income levels,debts, assets, and net worth?Given the level of interest on Millennials and debt levels, wewrote a short, analytical paper that answered this question.
Your analytical journey
Remember it this way analysis is like taking a canoe trip.You need a good plan.You should map out where you are going and how you will get there.That's your analytical plan.You will also need a strong analytical question,solid data, and good methodology. That's your canoe.
Remember: Define your analytical question
The key takeaway from this video is to remember to develop aclearly defined analytical question even with a great topicand high quality data you cannot produce good results without awell defined question.
Summary of key points
To summarize, the analytical process can be viewed as aseries of steps designed to answer a well defined question.Once the topic has been defined, the next step is to create ananalytical plan. And always incorporate the feedback youreceive during the planning stage of your analyticalproject.Before the next video, take a few minutes to identify twoanalytical questions and think through why these questions arerelevant for your organization.Stay tuned up next. We'll share tips on how toimplement your analytical plan.
(The Canada Wordmark appears.)
What did you think?
Please give us feedback so we can better provide content that suits our users' needs.
Semi-Automated Mapping in QGIS with the Atlas Panel - Video transcript
(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Semi-Automated Mapping in QGIS with the Atlas Panel")
So following up from Creating Maps in QGIS, today we'll discuss using the Atlas Panel in the Print Layout to rapidly generate multiple maps. The Atlas panel uses a specified 'Coverage Layer' to define the geographies of the outputs. Today we'll use it to map population dynamics in Census Metropolitan Areas - or CMAs for short - across Canada - effectively semi-automating the map production process.
So once again the preparation steps are provided in the video description – but to summarize quickly, we used the one-to-one join procedures to link the population table to the Census Tract layer. The Refactor Fields tool was then applied to save to a permanent file with correctly attributed field types. We also dissolved the Census Tract layer using the Census Metropolitan Area Name (CMANAME) field to create the Coverage Layer for our Atlas, and with the fill colour set to fully transparent, outline the Census Tracts within our main map group.
The cartographic Census Subdivision layer was then run through the Fix Geometries tool, dissolved by the Provincial Unique Identifier and run through the Multipart to Single Part tool, ensuring that all features were separate entries within the attribute table. Area fields were added to it, and the Lakes and Rivers polygon, using the Field Calculator, and the Select by Expression tool used to subset features with areas greater than 2500 and 500 square kilometres respectively and grouped together to comprise our Inset Map.
The Labels applied to our coverage and province layers had similar settings. So to quickly summarize - in the Formatting subtab for the coverage Layer we specified to wrap text on the dash character. A text buffer was also applied – with the coverage layer set to 75% Opacity. The Placement was set to Horizontal (slow) ensuring the legibility of the text-based labels. And in the Rendering tab only draw labels which fit completely within the feature was checked and discourage labels from covering the feature's boundaries was selected with the highest weight applied. So now we can toggle off the Prep Layers group.
Now in the Print Layout I used the add shapes tool, specifically Add Rectangles, to divide the layout for the map items, which were then locked within the Items panel. The alignment tool on the Actions toolbar was then used to ensure that added items were placed above the rectangles. I've also already added many of the mandatory map items – including the additional information, scale bar, legend and title. The title uses an expression, including a generic text prefix within single quotes followed by the vertical separators and then Census Metropolitan Area name field to label by metropolitan area, which will update automatically once our Atlas is generated.
Just a quick tip for map item formatting - if greater control was needed – the Print Screen or Snipping tools could be used to export, externally format and re-add items, such as the legend, diagrams or table, as a picture.
On that note our North Arrow is still missing. So rather than using the add Arrow and Text label function, let's add it from Picture this time. Clicking and dragging across the desired location in the Print Layout, we can then go to the Item Properties Panel and expand the Search Directories. And once loaded select the desired icon. I'll alter the Fill Colour to be darker, to ensure its visibility against the main map, and then clicking Back, we'll also switch the placement to Middle.
So now we can add both maps simultaneously – placing the main map in the larger box of the layout and the inset map in the smaller box on the right. The main map is currently being rendered over our north arrow, so once again we'll select it and use the Lower in the Alignment tools to ensure the north arrow is visible.
While they're rendering I'd also like to highlight that the scale-bar for the main map is set to Fit Segment Width as opposed to the Fixed Width used in the previous demo, which will update the scale-bar according to the size of the census metropolitan area being mapped.
Now for our second map, let's add a Grid, expanding the drop-down and clicking the Plus icon – and then select Modify Grid Properties. We'll also change the CRS to WGS 84, entering 4326 in the system selector, so that we can show coordinates in decimal degrees. We'll specify an interval of 2 degrees, which at the moment adds many lines, but will be more appropriate once we generate the Atlas. Check the Draw Coordinates button. The format of the coordinates can then be selected in the drop-down, here we'll leave it with the default. We'll specify to show latitude only for the Right and Left parameters and longitude only for the Bottom and Top. At the bottom of the panel we'll change the precision for the Grid units to 1.
Back in the Item Properties Panel for the Inset Map, we'll also add an Overview – clicking the plus icon and specifying the map being overviewed in the drop-down, which is Map 1.
Now we can generate the Atlas. So in the Atlas drop-down on the Menu-bar select Atlas Settings. And within the Atlas panel check Generate Atlas and specify the Coverage Layer – in this case DCMAAtlas or the dissolved Census Metropolitan Areas. We'll specify the field used for the Page Name, in this case the CMANAME field and use the same field for Sort by. This will sort them alphabetically when we preview our Atlas. Uncheck the Single Export option, as we want each metropolitan area to be a separate map and select the desired output file format. Additionally we'll change the output filename to something more intuitive than just the feature IDs. So we'll switch 'Output' to JCTPop and then click on the Expression box. In the Variables drop-down, we will replace featureID with @atlas_pagename, which we set to CMANAME field, so the outputs will be named according to the census metropolitan area. This did involve a trade-off– requiring the entries within the CMANAME field to be reformatted – removing periods, slashes, question marks and other symbols that caused erroneous filenames, which would lead to the atlas output failing. So having replaced these characters I can just click OK.
Now we can Select Map 1 and within the Item properties Panel check the Controlled by Atlas box. For the main map we'll specify 5% for the Margins Around the Feature. In the main interface we can now toggle off the inset map group. And back in the Print Layout select the main map and in the Item Properties Panel check Lock Layers and Lock Styles.
Repeating with Map 2, we'll check controlled by Atlas once again and enter a larger margin of 750% to ensure the broader geographic location is shown. Then in the main interface toggle off the main map group and back in the layout select Lock Layers. And this is so that the inset map does not show the layers of the main map and vice versa.
Now on the Atlas toolbar we can select the Preview Atlas icon. So we can toggle through the different CMAs alphabetically, or select ones of interest from the drop-down. The next metropolitan area is Barrie. So as you can see, the title, grid and scale-bar update rapidly, while the maps, particularly the Inset Map takes longer. This is likely due to the detail of the cartographic boundary file, combined with the broader extent being mapped within the inset.
I ran the Atlas output earlier – as we can see scrolling through the maps, most are appropriately formatted and ready for use as supporting figures or stand-alone documents. Relatively few maps require manual editing and individual export to maintain intuitive values for map items such as the scale or grid intervals. For example, for Edmonton we would want to use a larger interval for the Grid coordinates, such as 5°. And similarly, for Granby, we would want to alter the scale to Fixed Width and enter 10 for a more intuitive break value. Then we could use the export procedures from the making maps demo to individually export these specific maps. On the whole, the Atlas Panel has facilitated rapidly mapping multiple locations and variations in attributes of interest with relatively little input or effort.
Toggling back we could now select another CMA of interest, such as Drummondville – or one of the outputs from the Atlas that needed edits such as Edmonton. Then we could select the Inset map, re-expand the Grids drop-down and click the Modify button – updating the Grid Interval for X to 5 degrees. We could then resize the North Arrow to ensure it's not obscuring the main map features, and then individually export this map using the procedures covered in the previous demo.
So with the Export Atlas tool we can specify the for..file format to use. The same formats from the single export options are available. It is a good idea to create a separate directory for the output maps. Then specify the output resolution and click Save to run. We won't actually run the output as it's a time-intensive process, taking around 35 minutes.
The last thing that we would want to do is save the print layout as a template for further use, such as reapplying for map production in the next census collection period or to generate maps with a different variable of interest at the Census Tract level.
So use the Atlas panel with a coverage layer to rapidly and easily generate multiple maps for particular areas of interest. Save the template for re-use in examining other variables of interest or applying in another time-period. Apply these skills to your own areas of expertise and datasets of interest for semi-automated map production.
Making Maps in QGIS with the Print Layout (Part 2) - Video transcript
(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Making Maps in QGIS with the Print Layout (Part 2)")
So using the Layout Manager, we can reopen our Layout from part one. And now we'll cover adding some additional optional map items. When used judiciously, these items can help enhance the interpretability and aesthetic of a map. One of the final procedures we covered in Part I was locking the Layer and Style of Layers for our Main Map, meaning that changes to the main interface will not impact its appearance in the Print Layout.
So the first item that we'll add is the Inset Map. Back in the main interface we'll toggle the main map group off, toggle the inset on and zoom to the provincially aggregated layer. Now back within the Layout we can add the Inset map, with the Add Map to Layout tool – and left-clicking and dragging across for its placement within the Layout.
We'll then add another scale-bar item - for Map 2 this time. Select Numeric from the Format drop-down. And we'll place it below the Inset map, altering the placement parameter in the display drop-down to Center and adjusting the placement within the Layout. To ensure an intuitive and interpretable scale ratio once again we'll enter a fixed scale value for the Inset Map using the Data Defined Override drop-down, in this case entering 55 million in the Expression window.
Now let's add a picture. So with the tool engaged click and drag where it should be placed within the Layout. Now we can load the image from our Directory by clicking the triple dot icon. It can then be resized and placed within the Layout as needed.
For the final optional item, let's add part of an attribute table to the layout. Click the Attribute Table icon and drag in the layout for its positioning. We can then specify which layer to use, selecting our subset layer – JMBCDPop - in the Layers drop-down. Clicking the Attributes box, we can then specify which fields should be included or removed from the table. So we'll remove the Census Division Unique Identifier, Census Division Type, the provincial fields, as well the Total Private Dwelling counts, percent change and area fields. So we'll rename the remaining fields in the Heading column, which can be of any length. So we'll rename them to Name, Population 2011 and 2016 written in full, Percent Change, Density, Rank (CA) for the national level and Rank (MB) for the provincial level. We can then specify the fields to sort the table by. Here we'll use the CDName field. We could also add additional sorting rules, much like in Excel, here using the provincial population rank.
So now that the table is added we can nudge it down within the layout, and as we resize it within the layout the number of features in the table changes. We could also control this using the parameters in the Feature Filtering drop-down.
In the Appearance section, select Advanced Customization. Check even rows and we'll alter the colour formatting to be a light grey to distinguish the individual rows in the table. In the Show Grid drop-down uncheck the Draw Horizontal Lines box.
Now we'll add a Node Item, using the Add Polyline function, to the Print Layout. We'll use the lines to form the horizontal border lines for the attribute table and header row. So left click twice for the beginning and end of the line, and right-click to complete. We'll then edit the length of the line to ensure that it perfectly matches the width of the attribute table. Then we can sse.. copy and paste the first line and place it in the other two locations. Once this is done we can select the items by clicking and dragging over the Layout, and once again using the Group tool on the Actions toolbar and lock their position in the Items Panel.
With all map items formatted, we can now export the map. So the Map can be exported as an image or as a .pdf. The image file format enables it to be rapidly added within a document as a figure or supporting information, while the .pdf can be used to share the map with others in a widely accessible but protected file format. Here we'll export the map as an image. Navigate to the desired directory and provide an output filename. Then we can enter the desired resolution. In general, 300 dots per inch will suffice for most applications. But say we want to include the map on a poster, then we could use a finer resolution of 600 or even 1200 dots per inch as required. Then click Save, and the export procedure takes about a minute to complete.
When it is completed, click the Hyperlinked Filename at the top of the Layout. And then we can open and examine the output map. If we need to make any adjustments we could easily return to the layout, incorporate them and repeat the export procedure.
So in this demo we explored the principles, procedures and tools for creating a map in the Print Layout. Specifically, users should now have the knowledge and skills to: Distinguish mandatory and optional map items; Use available tools in QGIS's main interface and Print Layout to prepare map data Add map items to the Print Layout and alter their properties using available panels, such as using the lock layers and Style functions in the Item Properties panel to add inset maps, and using the group and lock functions in the Items panel to fix item positions in the Layout.
Finally you should also feel comfortable saving and exporting finalized maps. So apply these skills to your own areas of expertise to create well-balanced, easy-to-interpret maps.
Making Maps in QGIS with the Print Layout (Part 1) - Video transcript
(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Making Maps in QGIS with the Print Layout (Part 1)")
Hello everyone. Today we'll learn how to create maps in QGIS using the Print Layout– which is a separate window from the main interface used for mapping. Specifically we'll cover navigating the window, and using its tools and panels; adding map items, and distinguishing those that are mandatory versus optional; and saving and exporting a map.
Cartography, or map-making, blends the science and art of GIS. Maps are powerful tools for conveying information to a wide audience. The creator chooses which features are included, how they are visualized and how to best convey the information. Maps should be intuitive and readily interpretable. Important factors to consider are similar to those introduced in the vector visualization tutorials, including:
What is the main theme or message of the map and who is the target audience?
This helps define essential layers for the map. And generally you should exclude peripheral layers that may overcrowd your map or message.
Second, is the visualization logical and facilitate the distinction of layers or features?
And similarly, is the level of detail or generalization within the layers suited to the map scale?
Finally, have you selected an appropriate projection for the location of your map?
With the joined division and aggregated provincial layers from the one-to-one join by attributes demo loaded in the Layers Panel, division features in Manitoba were selected and subset to a new layer – JMBCDPop - which will be our main map. Selected features were also run through the Dissolve tool using the province name field to create the MB outline layer, which was grouped with the province layer to create the Inset Map. So inset maps just show the broader geographic location and context of a main map. The groups in the Layers Panel will help us add our two maps separately within the Print Layout, important since the Layout is actively tied to the Canvas in the main interface.
So with the map layers created and grouped, we now need to establish our visualizations. Instead of using the Layer Properties Box, today we'll use the Layer Styling panel, right-clicking on an empty toolbar area and selecting it from the drop-down. The panel contains the main visualization tabs from the Layer Property Box, and layers can also be selected from the drop-down at the top – enabling the rapid visualization of multiple layers.
So we'll apply a graduated symbology to the Pop Percent Change field, using the Spectral Colour ramp, Pretty Breaks as the Mode and 8 classes. We'll also change the precision to 1. Then in the Labels tab, we'll once again specify the Percent Population Change field to use for labelling. And in the Formatting subtab check Formatted Numbers and change the decimal places to 1. Since the visualized field includes negative values we could also check 'Show plus sign' if desired, but here we'll leave it unchecked.
We'll add a small text buffer around the labels using the default values. And for the Placement, we'll select Free (Slow).This will rotate the labels to fit them within the feature – but will still be interpretable since we have NEVER selected for show upside down labels in the Rendering tab.We'll also check only draw labels for fit completely within the feature and discourage labels from covering the feature's boundary. If you've noticed I haven't clicked Apply yet, because the Live Update box is checked, meaning the edits are being applied on-the-fly as they're being entered.
Now we'll select the Manitoba outline layer from the drop-down. Switch back to the Symbology tab. Click on Simple Fill and Fill Colour, which will alter the Opacity to 0% or fully transparent. Then we can change the stroke colour to a dark red and enter a width of 0.75 – creating the outline of the main map.
Closing the Styling Panel we'll copy and paste the style from our subset layer – selecting all categories – - to the aggregated province layer to ensure consistent visualization of population changes across the two layers and levels. Now we can toggle off other layers leaving only the main map group. If a bookmark was created it can now be used, or in this case since our main map is one layer, the Zoom to Layer tool. And we can use other zoom tools to refine the scale of the Canvas as needed. The scale value at the bottom of the interface is approximately 1 in 7 million.
So to access the Print Layout window, click the New Print Layout icon on the Project Toolbar. The layout manager icon to the right can be used when there are existing layouts that you want to access for further use. Clicking on the New Layout icon we need to provide a name – which we'll call Making Maps in QGIS. Obviously a more specific title is helpful to distinguish different layouts once multiple maps have been created.
So the Print Layout appears as such. There are a variety of panels on the right-side of the window, the most important being the Item Properties Panel, where all items in the Layout are formatted –defaulting for the selected item. On the left-hand side are a variety of tools to add different map items to the Layout.
So the first mandatory component is the main map – so we'll click the Add New Map icon and then click and drag to place it within the Layout. There are two interaction icons. The Select/Move Item, enabled by default is used to move, place and resize items within the Layout, while the one below it, Move Item Content, applies to Map Items only and can be used to alter the Canvas location and scale from within the Print Layout. Map items may take a moment to update when the formatting parameters are changed.
So here we can specify the properties for Map 1, such as setting the scale to that of the Canvas, or entering a specific value for the scale in the Main Properties drop-down. However, the scale will adjust automatically if the map item is resized, which is not ideal. So to prevent this we can click on the data defined override box, select edit, and enter the desired scale in the expression box, in this case seven and a half million. Now we can adjust the size without it impacting the scale.Engaging the Move Item Content tool we'll move the canvas for our main map feature so that it is fully visible within the Layout. And we'll also enter -5.0 for the Map Rotation to remove the tilted appearance of Manitoba, which is tied to the applied map projection.
The Guides panel can help us place items within the Layout. So we can click the Plus Icon and specify a distance for indentation. The guides then appear as dotted red lines. In addition there are a variety of alignment and distribution tools on the Actions toolbar to facilitate laying out and distributing items to create an aesthetically pleasing, well balanced map.
The second mandatory item is the North Arrow, which can point towards true, magnetic or grid north – particularly relevant for mapping at higher latitudes. So we'll use the Add Arrow function. Left-clicking twice, to define the start and end of the arrow – drawing a vertical line – and then right-clicking to finish. The North Arrow does not automatically point towards North, so expand the Rotation drop-down and enter -5 to match the applied rotation to our Main Map. We'll add a label above it, replacing the default text with capital N. Clicking on the Font box we can alter the size to a more appropriate value, 20 in this case should suffice and the alignment to Center and Middle. Once again we'll rotate the label.
Dragging across the Layout we can select both items and group them using the Group tool on the Actions toolbar. Now we can resize and reposition them within the Layout. In the Items panel we can then toggle Items on and off, as well as lock their position within the Layout. So now clicking and dragging across the Layout only the main map is selected.
The third mandatory item is a scale-bar - enabling real-world distances between features to be approximated from the map. So click the scale-bar icon and click within the Print Layout. We can select the map to which it applies, and the format. Here we'll stick with the default style - single box. Depending upon the map scale we could change the desired units to use in the drop-down. However, for our map kilometers is most appropriate. We'll include 4 segments to the Right of 0, and replace 75 with more interpretable break values, entering 200 in this case. We can use the arrows on the keyboard to nudge items in a direction of interest within the layout to facilitate their positioning.
The fourth mandatory item is a legend to interpret and distinguish the mapped features. By default the legend includes all layers in the Layers Panel. We can include a generic legend title at the top if needed, but here we'll leave it blank. Then we'll uncheck the Auto-Update box to enable the editing functions and ensure formatting changes are retained. We can reorder legend entries with the arrows and use the minus icon to remove them. So here we'll remove the Manitoba outline and aggregated provincial layers. And right-click on our main map group title and select hidden to remove it from the Legend. We can also rename the layers in the Legend Entries drop-down by double left-clicking. So we'll rename JMBCDPop to Percent Changes. We can also edit value ranges from within the Layout by expanding the Layers drop-down and double left-clicking. Here we'll change the upper and lower break values for the legend to less than -4.0 and greater than 14.0. We'll also remove its background and alter its placement to align with the scale-bar and the main map.
The fifth essential component is a title. It should be simple and quickly convey the map content, including the theme, location and level. So here we'll call it Percent Population Changes in Manitoba (2011-2016): Census Divisions. We'll change the font size to 34, specify the alignment and resize the text box accordingly.
So the final mandatory component are some additional text items that specify the map projection, creator and source references - particularly important when the map will be released as a stand-alone document. We can enter the information manually or use expressions to semi-automate the entry of this information. So we'll enter Prepared by: Insert Name or click the Insert Expression button and in the variables drop-down double-click user full name. Then type Projection colon, NAD83 Statistics Canada Lambert open-bracket. Reopening the expression box once again we can double-click project_CRS in the variables drop-down. So as shown, the manually entered information and expressions are being automatically formatted in the Item Properties panel. Then we need to specify the source references, typing datasets accessed from Statistics Canada. Finally, include an additional expression to include the date created and the program used. So we'll use the concat function and commas to separate the different components. So first - open single-quote and type Created on, colon, space close quote, then type todate with $now enclosed by brackets. Add another comma, reopen single quotes, space, type with QGIS, space, close quote and finally double-click @qgis_short_version in the variables drop-down.
So with map 1 formatted and all mandatory components added, we will select the main map and Lock the Layers and Layer Styles in the Item Properties panel so that changes in the main interface do not affect its format or scale in the Layout window. Now we can click the Save icons at the top. The first will save both the project in the main interface as well as the Layout, which we can then access with the Layout Manager for further edits. Conversely the second tool can be used to save a particular Layout as a Template for repeated use, which we'll cover in a follow up demo. So click the first Save Icon, which we'll then access to discuss optional map items in Part II.
The Agriculture Statistics Program (ASP) is comprised of an integrated set of components including crop and livestock surveys, farm economic statistics, agri-environmental statistics, tax and other administrative data, research and analysis, remote sensing and the Census of Agriculture (CEAG). The statistical information produced by the CEAG is unique in its ability to provide a comprehensive snapshot of the industry and its people, as well as small area data, both of which are instrumental not only to the agricultural industry, but also for meeting the data requirements of environmental programs, health programs, trade and crisis management. ASP statistical information is used by a wide range of organizations, including different levels of government, not-for-profit and private organizations, academic institutions, and individual Canadians.
This evaluation was conducted by Statistics Canada in accordance with the Treasury Board Secretariat's Policy on Results (2016) and Statistics Canada's Risk-Based Audit and Evaluation Plan (2019/2020 to 2023/2024). The main objective of the evaluation was to provide a neutral, evidence-based assessment of the 2016 CEAG dissemination strategy, and of the design and delivery of the CEAG migration to the Integrated Business Statistics Program (IBSP). The evaluation also assessed projects in the broader ASP, with a focus on projects supporting Statistics Canada's modernization initiative.
The evaluation methodology consisted of a document review, administrative reviews and key informant interviews with Statistics Canada professionals working in the Agriculture Division, and other relevant divisions. Additionally, interviews were conducted with key users and partners external to Statistics Canada. The triangulation of these data collection methods was used to arrive at the overall evaluation findings.
Key findings and recommendations
Census of Agriculture dissemination strategy
CEAG data are used by a wide range of organizations to understand and monitor trends, formulate advice on policies and programs, and address requests from stakeholders. The majority of interviewees were satisfied with the dissemination of the 2016 CEAG and noted it was an improvement over 2011. Data tables were identified as the most used product while other products were relevant but less useful. In terms of timeliness, interviewees were satisfied with the release of the first set of tables (farm operator data - one year after Census Day); however, the timeframe for releasing the remaining two sets of data tables affected their usefulness (2.5 years after Census Day for the last data table release with socioeconomic data). They also noted that there were gaps in cross-analysis with non-agricultural sectors and in emerging sectors. Finally, web tools were not being used because of a lack of guidance on how to use them and how to interpret the data.
The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure that:
Recommendation 1
For the 2021 CEAG, the Agriculture Division explore ways to improve the timeliness of the last two sets of data tables (historical data, and socio-economic data) and increase cross-analysis with non-agricultural sectors.
Recommendation 2
Web tools include guidance on how to use them and how to interpret data from them. A proactive approach to launching new tools should be taken. Webinars were identified as an effective channel and the use of other channels would allow for even a wider coverage.
Census of Agriculture migration to the Integrated Business Statistics Program
The CEAG migration to the IBSP was proceeding as planned at the time of the evaluation. The transition phase was complete and the integration phase was well underway. Governance structures were in place and deliverables and schedules were being managed effectively. Efforts to resolve issues, such as those related to compatibilities between the Collection Management Portal (CMP) and the IBSP, and the availability of tools and capacity to support data quality assessments, were continuing. The start of the production phase will bring additional risks as new resources become involved and time pressures increase.
The ACS, Field 5, should ensure that:
Recommendation 3
Unresolved issues for the migration to the IBSP, including incompatibilities between the IBSP and the CMP as well as the IBSP processing capacity, are addressed prior to the production phase.
Recommendation 4
Significant risks during the production phase, particularly with regard to data quality assessments and the exercising of roles and responsibilities, are monitored and mitigated.
Projects supporting the modernization initiative
All five projects reviewed were aligned with the modernization pillars and expected results. Most of the projects focussed on increasing the use of data from alternative sources and integrating data. The evaluation found that while governance structures existed and regular monitoring was taking place, project management practices could be strengthened. For example, clearly defined measurable outcomes were often missing, best practices were not being systematically documented, shared or leveraged, and risk management was ad-hoc in some cases. Project management is perceived to be time and resource consuming in an environment focussed on expediency.
The ACS, Field 5, should ensure that:
Recommendation 5
Planning processes for future projects falling outside the scope of the Departmental Project Management FrameworkFootnote 1 include an initial assessment that takes into account elements such as risk, materiality, public visibility and interdependencies. The assessment should then be used to determine the appropriate level of oversight and project management.
Recommendation 6
Processes and tools for documenting and sharing of best practices are implemented and lessons learned from other organizations (internal and external) are leveraged.
What is covered
The evaluation was conducted in accordance with the Treasury Board Secretariat's Policy on Results (2016) and Statistics Canada's Integrated Risk-Based Audit and Evaluation Plan (2019/2020 to 2023/2024). In support of decision making, accountability, and improvement, the main objective of the evaluation was to provide a neutral, evidence-based assessment of the 2016 Census of Agriculture (CEAG) dissemination strategy, and of the design and delivery of the CEAG migration to the Integrated Business Statistics Program (IBSP)Footnote 2. The evaluation also assessed projects in the broader Agriculture Statistics Program (ASP), with a focus on projects supporting Statistics Canada's modernization initiative.
The Agriculture Statistics Program
The mandate of the ASP is to provide economic and social statistics pertaining to the characteristics and performance of the Canadian agriculture sector and its people. It aligns with section 22 of the Statistics Act, which stipulates that Statistics Canada shall "collect, compile, analyse, abstract and publish statistics in relation to all or any of the following matters in Canada: (a) population, (b) agriculture." It also aligns with section 20Footnote 3 of the Statistics Act, which requires Statistics Canada to conduct a CEAG. A CEAG has been conducted nationally and concurrently with the Census of Population since 1951Footnote 4.
According to the ASP Performance Information Profile, the ASP provides data to support and evaluate the fulfillment of requirements or objectives contained in other legislation such as the Farm Products Agencies Act, the Agricultural Products Marketing Act, and the Pest Control Products Act. The ASP also supplies the Canadian System of Macroeconomic Accounts with data required under the Federal-Provincial Fiscal Arrangements Regulations and the International Monetary Fund's Special Data Dissemination Standard.
The ASP includes an integrated set of components that includes crop and livestock surveys, farm economic statistics, agri-environmental statistics, tax and other administrative data, research and analysis, remote sensing and, the CEAG.
The Census of Agriculture
The CEAG collects data on the state of all agricultural operations in CanadaFootnote 5 including: farms, ranches, dairies, greenhouses, and orchards. The information is used to develop a statistical portrait of Canada's farms and agricultural operators. Typically, data are collected on: size of agricultural operation, land tenure, land use, crop area harvested, irrigation, livestock numbers, labour, and other agricultural inputs. Its "whole farm" approach to capturing data directly from agricultural producers provides a comprehensive count of the major commodities of the industry and its people, and a range of information on emerging crops, farm finances, and uses of technologies in agricultural operations.
The objectives of the CEAG are
to maintain an accurate and complete list of all farms and types of farms for the purpose of ensuring optimal survey sampling - at the lowest cost and response burden - through categorization of farms by type and sizeFootnote 6
to provide comprehensive agriculture information for detailed geographic areas such as counties - information for which there is no other source and that is critical to formulating and monitoring programs and policies related to the environment, health, and crisis management for all levels of government
to provide measurement of rare or emerging commodities, which is essential for disease control and trade issues
to provide critical input for managing federal and provincial government expenditures in the agriculture sector.
The Agriculture Division of the Agriculture, Energy and Environment Statistics Branch is responsible for the ASP. The division has many long-standing strategic partnerships with key stakeholders and data users, including federal departments and agencies, provincial and territorial agriculture ministries, local and regional governments, farmers' associations, the agriculture industry, universities, and researchers. The division has established forums to obtain feedback on emerging issues and needs. These include the Advisory Committee on Agriculture and Agri-Food Statistics and the Federal-Provincial-Territorial Committee on Agriculture Statistics. Internal governance bodies such as the CEAG Steering Committee are also in place to help direct and monitor implementation.
The Evaluation
The scope of the evaluation was established based on meetings and interviews with divisions involved in the ASP. The following areas were identified for review:
Evaluation issues, Evaluation questions
Evaluation issues
Evaluation questions
2016 CEAG dissemination strategy
To what extent did the 2016 CEAG dissemination strategy address the needs of key users in the following areas?
Timeframe of releases (i.e., for all releases, between each release)
Coverage and level of detail
Types and formats of products
Cross-analysis with non-agricultural sectors
Access to data
Design and delivery: CEAG migration to the IBSP
To what extent are governance structures for collection and processing (migration to the IBSP) designed to contribute to an effective and efficient delivery of the 2021 CEAG?
ASP projectsFootnote 7 supporting the modernization initiative
To what extent are there effective governance, planning and project management practices in place to support modernization projects within the ASP?
Guided by a utilization-focused evaluation approach, the following quantitative and qualitative collection methods were used:
Administrative reviews
Review of ASP administrative data on activities, outputs and results.
Document review
Review of internal agency strategic documents.
Key informant interviews (external) n=28
Semi-structured interviews with key users from federal departments, provincial and local governments, farm associations, private sector organizations and research institutions.
Key informant interviews (internal) n=14
Semi-structured interviews with individuals working in the Agriculture Division and partner divisions.
Four main limitations were identified, and mitigation strategies were employed:
Limitations, Mitigation strategies
Limitations
Mitigation strategies
Because of the large number of users and partners using data, the perspectives gathered through external interviews may not be fully representative.
External interviewees were selected using specific criteria to maximize a strategic reach for the interviews. Different types of organizations from a wide range of locations across Canada, and that use CEAG data extensively were selected. Evaluators were able to find consistent overall patterns.
Key informant interviews have the possibility of self-reported bias, which occurs when individuals who are reporting on their own activities portray themselves in a more positive light.
By seeking information from a maximized circle of stakeholders involved in the ASP, including the CEAG migration to the IBSP (e.g. the main groups involved, multiple levels within groups), evaluators were able to find consistent overall patterns.
Limited documentation was available on the projects sampled for the evaluation.
Key staff working on ASP projects were interviewed and a strategy to gather additional documents during the interview sessions was put in place. Additional interviews were conducted, as needed, to fill the gaps.
The scope of the evaluation related to innovation reflected only a select number of topics (i.e., alignment, project management) rather than the full spectrum of factors which may have an impact.
The evaluation methodology was conducted in such a way that other topics related to innovation could be identified and considered.
What we learned
1.1 2016 Census of Agriculture dissemination strategy
Evaluation question
To what extent did the 2016 CEAG dissemination strategy address the needs of key users in the following areas?
Timeframe of releases (i.e., for all releases, between each release)
Coverage and level of detail
Types and formats of products
Cross-analysis with non-agricultural sectors
Access to data
Summary
To inform the 2021 CEAG dissemination strategy the evaluation assessed the extent to which the 2016 dissemination strategy addressed the needs of key users in different areas. The majority of users considered the 2016 CEAG an improvement compared with the 2011 CEAG and were satisfied with the overall approach taken. However, the evaluation found some areas for improvement, particularly with regard to the timeframe of releases, coverage, and guidance on web tools.
Census of Agriculture data are used for multiple purposes with data tables being the product of choice
CEAG data are used by organizations to portray the agriculture sector in their jurisdiction or sector of the economy. For provincial government departments, their portrait allows them to understand trends within their province and to compare them with other jurisdictions. Subprovincial data are also available for analysis of smaller geographic areas. For farm associations, data allow them to monitor trends within their area of interest. Overall, CEAG statistical information is used for identifying and monitoring trends, providing advice on policies and programs, addressing requests or questions from various stakeholders, and informing internal or public communications.
A large majority of external interviewees mentioned that, in general, the 2016 CEAG products and statistical information shed light on the issues that were important for their organization. The evaluation found that the data tables from Statistics Canada's website were the products of greatest utility to users. In particular, the Farm and Farm Operator Data tables were identified as the products most used. This was especially true for organizations that had internal capacities to conduct their own analysis. The analytical products and The Daily releases were identified as being less useful, but still relevant since they provided a different and objective perspective on specific topics. This was true for other products as well (e.g., maps, infographics) - interviewees responded that they used them only occasionally or rarely but still believed they were useful. Finally, a number of provincial users also mentioned that they received a file containing CEAG statistical information, which helped facilitate their ability to conduct their own analyses.
Table 1: Use of Census of Agriculture products
Products
Extensively
Occasionally
Rarely
Don't know
Data tables from the website
17
6
1
0
The Daily releases
9
5
10
0
Boundary files
7
6
11
0
Analytical products
5
12
7
0
Thematic maps
4
8
12
0
Infographics
3
9
12
0
Dynamic web application
3
6
14
1
Besides CEAG statistical information, a majority of users mentioned that they consulted additional sources of information, either from Statistics Canada or other national and international organizations, to fill gaps. This included information on commodity prices, imports and exports of agricultural products, land values, and interest rates. Users consulted international sources to compare data with other countries (e.g., United States and Australia) or to assess global market demand for certain agricultural commodities (e.g., livestock, crops, etc.).
Historical and socioeconomic data tables wanted sooner
Three data table releases took place for the 2016 CEAG: Farm and Farm Operator Data (May 10, 2017 - one year after Census Day); select historical data (December 11, 2017 – approximately one and a half years after Census Day); and a socioeconomic portrait of the farm population (November 27, 2018 – approximately two and a half years after Census Day). It should be noted that the tool used to create the socioeconomic portrait of the farm population was not part of the original scope for the 2016 CEAG but was added later - thus the reason for the relatively late release.
The majority of interviewees believed the time lapse to receive the first set of data tables was satisfactory given the quality of information they received. While they would have welcomed an earlier release, they recognized the level of effort required to produce the information and felt the time lapse was reasonable given the quality of information they received. However, overall, interviewees believed that the time lapse between Census Day and the final data table releases, specifically for the socioeconomic tables, affected the usefulness of the statistical information. In particular, organizations developing policies or programs targeting young farmers, specific population groups, or educational advancements would have benefited from timelier data.
Figure 1: Dissemination schedule (refer to Appendix B for additional details)
May 10, 2016
Launch of the 2016 Cencus of Agriculture and the 2016 Census of Population.
May 10, 2017
The first set of products for the 2016 CEAG - including a Daily release, farm and farm operator data (47 data tables), provincial and territorial trends (11 analytical products) and provincial reference maps (34 maps).
May - June 2017
A series of weekly analytical articles were released covering different topics, including an infographic titled 150 Years of Canadian Agriculture.
September - November 2017
A boundary file and analytical products were released.
December 11, 2017
Select historical data were released.
December 2017 – April 2018
A number of maps along with an analytical article were released.
November 27, 2018
The Agricultural Stats Hub, a dynamic web application, was released as well as a Daily article, 13 data tables and 3 infographics. The application provided a socioeconomic overview of the farm population by linking agricultural and population data.
December 2018 – March 2019
A number of analytical articles were released
July 3, 2019
Last release from the 2016 CEAG
Table 2: Satisfaction with releases
How satisfied are you with the following?
Satisfied
Somewhat satisfied
Not satisfied
Unsure
Time lapse between Census Day and first release
15
5
3
1
Time lapse between each release
13
5
2
4
Time lapse between Census Day and release of all data
6
11
4
3
Some interest in preliminary estimates, so long as differences are small
Users were asked about the possibility of releasing preliminary estimates for specific high-level variables. The estimates would differ from the final data released, however no specific examples of variables were provided to interviewees for consideration.
Half of the interviewees were not interested, with a large proportion advising against it. Several explained that the release of preliminary estimates would create confusion within their organizations and they would be required to explain the differences between the preliminary and final data. Most interviewees noted that any policy decision-making and trend analysis would continue to be based solely on final data.
Those who were either "very interested" or "slightly interested" indicated that the difference between the estimates and the final data would need to be small, otherwise they would prefer the status quo.
Some gaps remain
The Agriculture Division has several mechanisms in place to identify information gaps including: regular pre-census cycle consultations, the Federal-Provincial-Territorial Committee on Agriculture Statistics, the Advisory Committee on Agriculture Statistics, and engagement with national farm organizations. Based on these mechanisms, the CEAG builds on the content approved for the previous cycle to better address new and emerging agricultural activities. In addition, projects recently implemented by the Agriculture Division, particularly the Agriculture-Zero project, have filled several gaps (e.g., temporary foreign workers data).
The majority of interviewees were satisfied with the diversity of topics and themes covered. However, a number of information gaps were identified, particularly regarding emerging operations and fast-growing sectors such as organic farming. Additional statistical information and further analysis were also identified related to farm succession, labour (e.g., foreign and contract workers), pesticide use, and new land use categories (e.g., loss of land to urbanization). Additional variables covered over time (i.e., historical data) was also identified as a need. Finally, all interviewees wanted more granular data, although they recognized there are limitations related to confidentiality.
Table 3: Coverage
How satisfied are you with the following?
Satisfied
Somewhat satisfied
Not satisfied
Unsure
Types of agricultural operations covered
15
9
0
0
Number of topics or themes covered in each release
15
5
0
4
Cross-analysis with other topics and other agricultural surveys
10
7
2
5
Interviewees also wanted additional cross-cutting analysis between the agricultural sector and other sectors. For the 2016 CEAG, analysis with non-agricultural data, such as technology, innovation, and socioeconomic issues was provided to users. This approach was highly regarded by those interviewed - but they wanted more. Evidence suggests that there is a growing appetite for cross-cutting analysis in areas such as technology, farm profitability, demographic shifts, transportation, and the environment.
Increased guidance on tools is needed
Two web tools were released for the 2016 CEAG: boundary files and the Agriculture Stats Hub. The evaluation found relatively low use of these two products when compared with other products such as data tables. Although some interviewees used the boundary files, the majority rarely did. Similarly, few interviewees used the Agriculture Stats Hub. A lack of guidance on how to use the tools and how to interpret the data were noted as key impediments. The lengthy timeframe for releasing socioeconomic data, which included the Agriculture Stats Hub, was also identified as a factor that limited the use of the Hub.
Although the use of existing web tools for the 2016 CEAG was somewhat limited, a majority of interviewees were interested in having additional web-based tools, such as interactive maps, custom table building, and query tools, which would allow for the increased customization of products. As data tables were the product most used, tools attached to the tables would greatly benefit users. However, guidance and support must accompany the tools, and a more active approach to launching the tools would be recommended.
More prominent communication of methodological information would be useful
Although methodological information is generally available, some interviewees noted that it would be useful to have it more prominently displayed in the products that are released, either in The Daily or as footnotes in the data tables. For example, since definitions used by Statistics Canada may differ from definitions used by farmer associations (e.g., how farm operator counts are calculated), information to explain the differences would be helpful.
Users were aware of releases, and data were accessible
The evaluation found a high level of satisfaction with the accessibility of statistical information even though Statistics Canada's website was identified as being a challenge. A high level of satisfaction was also reported for any custom data received. Interviewees were highly satisfied with the time lapse between first contact with Statistics Canada and the delivery of the product, the quality of the product, and the level of detail provided.
In terms of awareness of releases, the majority of interviewees stated that they were informed far enough in advance and were satisfied with the channels used. Most interviewees identified reminder emails as the most effective channel for being kept informed about releases.
Table 4: Notification of releases
Best way to be informed of releases
Number of Respondents
Reminder emails
20
Calendar invites
7
Webinars
5
Social media posts
3
In addition, those who participated in webinars were very satisfied since the webinars provided additional information on the data available and major trends observed. Webinars were identified as opportunities to raise awareness of the products and data that will be available and to facilitate interpretation of the data and the use of the web tools.
1.2 Design and delivery: Census of Agriculture migration to the Integrated Business Statistics Program
Evaluation question
To what extent are governance structures for collection and processing (migration to the IBSP) designed to contribute to an effective and efficient delivery of the 2021 CEAG?
Summary
The evaluation assessed whether the governance structures associated with the CEAG's migration to the IBSP - including roles and responsibilities, interdependencies, and project management practices - will contribute to an effective and efficient delivery of the 2021 CEAG. The evaluation found some areas of risk that could have a negative impact on the delivery of the 2021 CEAG.
Migrating the Census of Agriculture to the Integrated Business Statistics Program is expected to create benefits
At the time of the evaluation, the Agriculture Division had already successfully migrated all of its surveys to the IBSP. The last component to be migrated is the CEAG; migration work began in fiscal year 2018/2019 and is expected to continue until fiscal year 2022/2023. The IBSP migrations are conducted in three phases: transition (defining program-specific requirements), integration (development and testing activities), and production (collection and processing tasks are implemented through the IBSP).
Because of its five-year cycle, the CEAG is considered an ever-migrating component to the IBSP. Similar to the divisional surveys that have already been migrated, it is expected that migration of the CEAG to the IBSP will create specific benefits for the CEAG:
reduced number of systems for collection, processing and storage through the adoption of common tools and statistical methods
facilitated integration and harmonization of data with all programs in the IBSP, including agriculture surveys
increased corporate support for systems, particularly when significant changes occur (e.g., cloud technology)
a more targeted approach for collection (i.e., follow-up operations) through the IBSP's Quality Indicators and Measures of Impact (QIMI) feature.
Roles and responsibilities are a risk during the production phase
For previous CEAG cycles, the Agriculture Division was responsible for designing, planning, implementing, and managing all required tasks, such as content determination, collection, processingFootnote 8, data quality assessmentFootnote 9, and dissemination. The migration to the IBSP for the 2021 cycle will change the governance of collection and processing tasks (and associated roles and responsibilities) because the Enterprise Statistics Division (ESD) is responsible for managing the IBSP.Footnote 10
The shift of processing responsibilities to ESD affect the CEAG team since it will now act only in an advisory capacity for this task, rather than being fully responsible for it. The same structures for the overall management of the CEAG will remain within the Agriculture Division while the migration to the IBSP brings in governance structures already established within ESD.Footnote 11
The evaluation found that the early part of the transition phase was challenging for the CEAG team as they were not familiar with the implications of migrating the processing activities to a different system run by another division. The CEAG's management team and ESD were key in resolving early challenges in the transition. In particular, both groups showed leadership in explaining potential benefits and impacts of the migration while ensuring that roles and responsibilities were well communicated and understood. Governance structures are also adequate. The leadership demonstrated during the transition phase facilitated the start of the second phase of the project – the integration phase.
The evaluation found that there are concerns regarding roles and responsibilities during the production phase as new individuals, such as subject-matter experts within the Agriculture Division and the IBSP production team within ESD, become involved while others leave the project as the integration phase ends. Based on previous survey migrations to the IBSP, the roles and responsibilities during the production phase are typically less clear than during previous phases. To help with this, a good practice identified during interviews is the involvement of production staff during the integration phase to help build continuity and understanding – this took place when the surveys conducted by the ASP were migrated to the IBSP. With the CEAG, most of the divisional staff participating in the integration phase are also part of the team for the production phase.
ESD's Change Management Committee, which is responsible for the triage of required changes, will be involved during the production phase. Consideration of escalation processes is required when multiple committees (i.e., the CEAG Steering Committee, the IBSP Project Management Team and the Change Management Committee) are involved in the decision-making process, particularly during crunch periods typically observed in the production phase. Although the change management process has been defined and cross-membership within committees and working groups was identified as a mitigating factor, the risk of ineffective and inefficient decision-making because of an increased number of governing bodies remains.
Deliverables and associated schedules are well-managed
The migration of the CEAG to the IBSP is managed by the existing working groups. So far, for the transition and integration phases, effective practices were in place to manage deliverables, associated schedules, and outstanding issues. The transition phase, led by ESD in collaboration with the Agriculture Division, worked as planned. Activities for the integration phase, which were being implemented at the time of the evaluation, were also working as planned. The current IBSP integration schedule is seen as robust and includes the first set of processing activities. The schedule for the production phase is in place and is reviewed and updated regularly. While deliverables, schedules, and outstanding issues are being managed effectively, the differentiation between outstanding issues and risks inherent to the migration, particularly for the production phase, have yet to be clearly articulated.
In addition to the IBSP migration schedules, two other schedules come into play. As collection for the 2021 Census of Population and the 2021 CEAG are conducted in parallel, the Census of Population schedule is a crucial element for the development and implementation of the CEAG's internal schedule. All three schedules have varying levels of flexibility: the Census of Population schedule is inflexible and the CEAG schedule is flexible while the IBSP, given its focus on collection and processing, is considered to be moderately flexible. The Agriculture Division is the main conduit for the alignment of all schedules. Requirements from the Census of Population schedule are assessed on a continuous basis and discussions are held with ESD, as needed, to modify the IBSP schedule and the CEAG schedule. At the time of the evaluation, no major changes to the schedules were required, but it is expected that shifts will occur during the production phase. JIRA, which is the system used for change management (e.g., outstanding issues, schedules, deliverables) by the CEAG, the IBSP, and the Census of Population is seen as an effective tool.
Incompatibilities between the Collection Management Portal and the Integrated Business Statistics Program to be resolved
A unique approach for data collection will be used for the 2021 CEAG - different from the one used for the 2016 CEAG and different from other Statistics Canada surveys that have migrated to the IBSP. For the 2016 CEAG, collection was under the responsibility of the Agriculture Division and took place through the Collection Management Portal (CMP) – a shared collection platform with the Census of Population. The CEAG team was responsible for monitoring collection and the management of follow-up operations. Because of synchronicity with the Census of Population, the CMP will continue to be used for CEAG collection in 2021.
Surveys under the IBSP (which are business-focused in nature) are typically collected through a different platform, the Business Collection Portal. New and unique linkages between the CMP and the IBSP need to be designed, tested, and operationalized for the 2021 CEAG collection operations. Links were still under development at the time of the evaluation. Although some functionalities are now operational, there is still development work to be done. For example, paradata from the CMP (e.g., information related to the collection process, such as attempts to contact someone, comments provided to an interviewer, completion rate) were not compatible with the IBSP at the time of the evaluation. Although work is being done to resolve the issue, the incompatibility of CMP paradata would disable the IBSP's QIMI feature, which allows for a targeted process for follow-up operations (i.e., prioritizing follow-up operations to target units that have the most effect on the data). QIMI is an effective tool used to support data quality, the 2021 CEAG data quality assessment strategy would need to be adapted should it not be available. There is a risk that some of the relationships between the CMP and the IBSP will not be fully developed or tested in time for the production phase.
Integrated Business Statistics Program processing capacity and available tools will affect data quality assessment activities
Data quality assessment activities will remain under the responsibility of the Agriculture Division. While the IBSP is designed for data collection and processing, it also includes features supporting data quality assessments, such as QIMI, rolling estimates, inclusion of non-response variances, and values attributed by imputation. However, given the volume of data with the CEAG, some validation processes will not be usable, and alternative tools outside the IBSP will need to be developed and tested. The use of alternative tools will require a reconfiguration of the data quality assessment strategy.
Although the generation of rolling estimates is seen as an important step to ensuring data quality, concerns were raised about the IBSP's processing capacity. In previous cycles, the CEAG team was able to impute, run, and analyze data at a higher rate than is currently possible under the IBSP. As data change from the generation of a rolling estimate and its completion, there is a concern that subject-matter experts will be validating outdated data. Although the IBSP's processing capacity has improved since the start of the CEAG migration and further improvements are expected, concerns remain.
Lessons learned from past migrations to the IBSP suggest that the level of effort required for data quality assessments is either similar to or greater than what is typical. A number of large surveys that have migrated to the IBSP in the past have encountered delays during the production phase because of challenges associated with the data quality assessment task.Footnote 12 At the time of the evaluation, the data quality assessment strategy was being developed.
Migration will benefit from an extended timeframe and experience
Migration activities started in fiscal year 2018/2019 and will continue until 2022/2023. Testing activities will also continue, as needed, during collection. The extended timeframe available for testing (because of the CEAG's five-year cycle) will allow for additional testing activitiesFootnote 13 (i.e., with simulated data, actual data from the 2016 CEAG, and data from content tests conducted in 2019). However, the production phase will be implemented with real 2021 data, with no options available for parallel testing.
Finally, since approximately 30 surveys from the Agriculture Division have already migrated to the IBSP, expertise has been built within the division and ESD. Knowledge gained from previous experience will contribute to the successful migration of the CEAG.
Additional pressures may affect the migration
A risk that could affect the CEAG's migration to the IBSP is the move to cloud technology. Although there are no specific scheduled implementation dates for Statistics Canada programs, any move of IBSP components to the cloud technology during the production phase, where most testing will have been completed, would affect the migration. At the time of the evaluation, this topic was still under discussion.
Another element that is noted for every CEAG cycle is the timeliness of content approval. Any changes to content will affect various elements, including the questionnaire and systems.
1.3 Agriculture Statistics Program projects supporting the modernization initiative
Evaluation question
To what extent are there effective governance, planning and project management practices in place to support modernization projects within the ASP?
Summary
The evaluation reviewed a sample of ongoing and completed projects undertaken within the ASP to examine their relationship to Statistics Canada's modernization pillars and expected resultsFootnote 14, and to identify areas for improvement regarding governance, planning and project managementFootnote 15 practices. The evaluation found that the projects were aligned with the modernization initiative and that governance is in place, but project management practices could be improved.Footnote 16
Projects are aligned with the modernization pillars and expected results
Statistics Canada's modernization initiative supports a vision for a data-driven society and economy. The modernization of Statistics Canada's workplace culture and its approach to collecting and producing statistics will result in "greater and faster access to needed statistical products for Canadians."Footnote 17 Five modernization pillars along with expected results have been articulated to guide the modernization initiative (Figure 2).
Description for Figure 2: Statistics Canada modernization initiative
The Vision: A Data-driven Society and Economy
Modernizing Statistics Canada's workplace culture and its approach to collecting and producing statistics will result in greater and faster access to needed statistical products for Canadians. Specifically, the initiative and its projects will:
Ensure more timely and responsive statistics – Ensuring Canadians have the data they need when they need it!
Provide leadership in stewardship of the Government of Canada's data asset: Improve and increase alignment and collaboration with counterparts at all levels of government as well as private sector and regulatory bodies to create a whole of government, integrated approach to collection, sharing, analysis and use of data
Raise the awareness of Statistics Canada's data and provide seamless access
Develop and release more granular statistics to ensure Canadians have the detailed information they need to make the best possible decisions.
The Pillars:
User-Centric Delivery Service:
Users have the information/data they need, when they need it, in the way they want to access it, with the tools and knowledge to make full use of it.
User-centric focus is embedded in Statistics Canada’s culture.
Leading-edge Methods and Data Integration:
Access to new or untapped data modify the role of surveys.
Greater reliance on modelling and integration capacity through R&D environment.
Statistical Capacity Building and Leadership:
Whole of government, integrated approach to collection, sharing, analysis and use of data.
Statistics Canada is the leader identifying, building and fostering savvy information and critical analysis skills beyond our own perimeters.
Sharing and Collaboration:
Program and services are delivered taking a coordinated approach with partners and stakeholders.
Partnerships allow for open sharing of data, expertise and best practices.
Barriers to accessing data are removed.
Modern Workforce and Flexible Workplace:
Organization is agile, flexible and responsive to client needs.
Have the talent and environment required to fulfill our current business needs and be open and nimble to continue to position ourselves for the future.
Expected Outcome
Modern and Flexible Operations: Reduced costs to industry, streamlined internal processes and improved efficiency/support of existing and new activities.
Most of the projects examined focus on increasing the use of administrative data and integrating data into centralized systems. The evaluation selected a sample of projects through an objective methodology using the following criteria: level of priority for the ASP, budget, expected impact (e.g., data users, respondents, data quality, and costs) and the perceived contribution to modernization. Additional criteria, such as length, start date, and project stage were also considered. Based on this methodology, five projects were selected:
The Agriculture-Zero (Ag-Zero) project is a 7-year project which received funding commencing fiscal year 2019/2020. It is designed to reduce response burden by replacing survey data with data from other sources. The purpose of AG-Zero is to undertake multiple pilot projects involving the acquisition and the extensive use of satellite imagery, scanner and other administrative data, and models to serve as inputs to the ASP in place of direct data collection from farmers. The project aims to reduce response burden on farmers to as close to zero as possible by 2026, while maintaining the amount and quality of information available.
The project adopts a "collect once, use multiple times" approach. Administrative data will be used to directly replace survey data, to model estimates that are currently generated using survey data, and to produce value-added statistical products for stakeholders. Under the umbrella of Ag-Zero, a series of subprojects are planned to be implemented over the seven year periodFootnote 18; at the time of the evaluation, three had been initiated. The following two were selected for review:
Pig Traceability uses administrative data to model estimates of pig inventories and has the potential to replace biannual survey estimates with near real-time estimates. The source data are pig trace data collected under the Health of Animals Act.
In-season Weekly Yield Estimates uses a combination of satellite imagery, administrative data from crop insurance corporations, and modelling to create in-season estimates of crop yields and area.
The Agriculture Taxation Data Program (ATDP) is being redesigned to move from a survey-based to a census-based activity that uses tax records to estimate a range of financial variables including revenues, expenses, and income. The ATDP's redesign to a census-based activity will support replacement of financial data in the CEAG.
The Agriculture Data Integration (Ag-DI) project will integrate agriculture commodity surveys that require processing outside the IBSP into the existing Farm Income and Prices Section Data Integration (FIPS-DI) system. The system will combine data from over 100 sources to produce aggregate integrated data for the System of National Accounts. The project will involve the integration of a multitude of spreadsheets and other systems into one common divisional tool. The project will also update the formulas in FIPS-DI to accept the naming convention used by the IBSP or other data sources loaded directly to FIPS-DI. It is expected that cross-analysis between data-sets will be facilitated, particularly when the CEAG will be migrated to the IBSP.
Table 5: Overview of the innovative projects selected, and alignment with modernization pillars
Project
Timeframe
Alignment with modernization pillars
Ag-Zero
Budget - $ 2.8M
Start: 2019/2020
Length: 7 years
Stage: Planning
Leading-edge methods & data integration: This project involves the use of new sources of data and new methods for collecting data. Extensive use of modelling, machine learning, and data integration are also featured.
Sharing and collaboration: A key element of this project involves the establishment and maintenance of mutually beneficial partnerships with other federal departments and industry associations.
User-centric service delivery: This project is expected to yield improvements in data quality and timeliness of data releases, as well as offer opportunities for new products.
Pig Traceability (AG-Zero sub-project)
Start: 2018/2019
Length: 1 to 3 years
Stage: Execution
Start: 2019/2020
Length: 1 to 3 years
Stage: Initiation
Redesign of the ATDP
Budget - $ 1M (approx.)Footnote 19
Start: 2015/2016
Length: 3 to 5 years
Stage: Close-out
User-centric service delivery: Consultations were held with Agriculture and Agri-Food Canada (AAFC) on priorities for the project. AAFC is the primary client and sponsor of the project.
Leading-edge methods and data integration: This project relied heavily on modelling and the integration of agriculture data (CEAG) and tax data.
Ag-DI
Budget - $ 696K (approx.)
Start: 2015/2016
Length: 3 to 5 years
Stage: Execution
Leading-edge methods and data integration: This project features data integration from an operational point of view. The integration will affect efficiency, data quality, and coherence of the data. It will also enable further cross-analysis opportunities.
Governance is in place
Overall, the evaluation found that governance structures are in place to support the projects. Similarly, schedules are developed and regular meetings take place to monitor progress, budgets and outstanding issues.
The projects employ different governance structures. AG-Zero is monitored under the Departmental Project Management Framework (DPMF)Footnote 20 and started on April 1, 2019. The first three years of the project are funded by a modernization investment submission while the final four years will be self-funded with savings realized through the first set of subprojects. Some of the subprojects under the AG-Zero umbrella have additional dedicated funding.
For AG-Zero, as required under the DPMF guidelines, detailed project planning documentation is in place including a project charter, Project Complexity and Risk Assessment, and an IT Development Plan. Monthly dashboards are provided to the Departmental Project Management Office (DPMO), reporting on various aspects including timelines, deliverables, expenditures, and risks. Within the division, a governance structure exists that includes working groups, divisional management, and the CEAG Steering Committee. AG-Zero subprojects are managed through the same governance structure. They are discussed within the division, and updates on elements such as deliverables, risks, and schedules are rolled-up into the AG-Zero monthly dashboard, as needed.
The Ag-DI project is also a DPMF project. It is small in scope with one resource working fulltime and no non-salary investment. Oversight and reporting are via the standard governance structure for the Agriculture Division and the DPMF.
Project management for the ATDP takes place via the regular divisional governance structure; it is not a DPMF project. Evidence indicates that project management has improved over time (e.g., budget planning, schedules, assumptions, governance, roles and responsibilities) and that at the time of the evaluation, adequate governance was in place and the project was on track to meet its overall objectives.
Risk assessments are conducted on an ad hoc basis
Risks for AG-Zero as a whole were identified at the outset of the project and are monitored every month as per DPMF requirements. Risks at the subproject level are meant to be rolled-up to inform risk management at the AG-Zero project level. While project-specific risks are identified and entered into JIRA during regular team lead meetings, there is little evidence that initial risk assessments were conducted for the subprojects. As AG-Zero is the sum of its subprojects, informal risk management at the sub-project level limits the effectiveness of risk management.
For example, the interruption of reliable access to administrative data (short-term or long-term) has been identified as a risk for the AG-Zero project overall. The division has developed mitigation and contingency options, including the feasibility or practicality of remaining "survey ready" in the case that this risk materializes. Because the risk has not been fully assessed at the subproject level, the management of this risk is limited. Similarly, risk management for other non-DPMF activities is taking place on an ad hoc informal basis.
Quantifiable objectives and targets are missing
The projects examined have the potential to advance innovation in important areas such as data collection, processing, analysis, and dissemination. The evaluation found that clearly defined, quantifiable expected outcomes have not been articulated in most cases. There is a general understanding of what types of positive effects these projects "might" generate, but there are few specific objectives that quantify the expected level of improvement in areas such as data quality, cost efficiency, response burden, timeliness, or relevance.
For example, while it is generally assumed that the integration of data from alternative sources will eventually lead to savings in data collection costs, there are no documented expectations for what the level of savings will be and when they will be realized. This is especially true for the subprojects under the AG-Zero umbrella. The AG-Zero project, which has a hybrid funding scheme (i.e., approved funding during the first three years, and self-funding for the remaining four years), does not have a clear plan to identify and measure returns on investment.Footnote 21
Finally, the measurement of returns on investment should be thorough and comprehensive. For example, as data from alternative sources are acquired from external sources in exchange for some type of service (such as data cleaning or preparation), the associated cost of the service must be considered. Non-payment for the administrative data does not mean they are free; there is still a cost for the "quid pro quo" service that must be accounted for. Similarly, associated costs for remaining "survey ready" while using administrative data (i.e., the mitigation strategy implemented for the risk associated with the accessibility of administrative data) should be accounted for.
The establishment of overall performance indicators for projects and for key milestones during the timeline of the project is critical for monitoring the progress of the work and, ultimately, for measuring the return to the agency for the initiative. The return can be in the form of data quality improvements, cost reduction, reduction of response burden, improvements to data access and availability, or any other improvement realized by the agency.
Best practices could be better leveraged
The evaluation found little evidence that best practices and lessons learned from the projects are being shared (or were planned to be shared) outside of the division; nor did it appear that the projects took advantage of experiences acquired by other divisions.Footnote 22 Lessons learned and best practices were not being documented. Instead, they were being deferred until there was "more time."
While minimal effort was made in this regard, staff recognized the importance of sharing and benefiting from others and that sharing and using best practices could be improved. Staff were also aware of channels for this purpose, such as the Innovation Radar and the Economic Statistics Forum. In November 2019, the division provided an overview of the Geospatial Statistics Framework (a system built to view and analyze geospatial data) at the Economics Statistics Forum.
When asked about ways to enhance information sharing, a number of suggestions were provided: encourage the use of existing corporate mechanisms such as the Innovation Radar; develop a user-friendly open corporate platform where more detailed information about initiatives organized by themes, including contact information, could be housed; involve partner areas such as the Finance, Planning and Procurement Branch, the Informatics Branch, and the Modern Statistical Methods and Data Science Branch (which support different projects for sound statistical approaches) at the outset of a new initiative since these groups have a corporate perspective of innovative projects.
Focus is on expediency
The level of project management typically reflects several factors including risk, materiality and interdependencies. The evidence suggests that timeliness for delivering results is given the highest priority for the projects and that project management is viewed as being a time consuming onerous task that slows things down. As such, minimal effort is placed on activities such as conducting formal risk assessments, identifying quantifiable goals, undertaking cost-benefit analyses, and sharing best practices (as well as learning from experiences of other divisions). An appropriate balance is missing.
How to improve the program
2016 Census of Agriculture dissemination strategy
The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure that:
Recommendation 1:
For the 2021 CEAG, the Agriculture Division explore ways to improve the timeliness of the last two sets of data tables (historical data, and socio-economic data) and increase cross-analysis with non-agricultural sectors.
Recommendation 2:
Web tools include guidance on how to use them and how to interpret data from them. A proactive approach to launching new tools should be taken. Webinars were identified as an effective channel and the use of other channels would allow for even a wider coverage.
Design and delivery: Census of Agriculture migration to the Integrated Business Statistics Program
The ACS, Field 5, should ensure that:
Recommendation 3:
Unresolved issues for the migration to the IBSP, including incompatibilities between the IBSP and the CMP as well as the IBSP processing capacity, are addressed prior to the production phase.
Recommendation 4:
Significant risks during the production phase, particularly with regard to data quality assessments and the exercising of roles and responsibilities, are monitored and mitigated.
Agriculture Statistics Program projects supporting the modernization initiative
The ACS, Field 5, should ensure that:
Recommendation 5:
Planning processes for future projects falling outside the scope of the Departmental Project Management Framework include an initial assessment that takes into account elements such as risk, materiality, public visibility and interdependencies. The assessment should then be used to determine the appropriate level of oversight and project management.
Recommendation 6:
Processes and tools for documenting and sharing of best practices are implemented and lessons learned from other organizations (internal and external) are leveraged.
Management response and action plan
Recommendation 1:
For the 2021 CEAG, the Agriculture Division explore ways to improve the timeliness of the last two sets of data tables (historical data, and socio-economic data) and increase cross-analysis with non-agricultural sectors.
Management response
Management agrees with the recommendation.
For the 2016 Census of Agriculture, no funding was provided for the creation and release of the socioeconomic portrait of the farm population; as such, it was not part of the original scope for the 2016 dissemination plan but was added later. The tool used to create the socio-economic dataset from the 2016 CEAG (dealing specifically with the linkage between the Censuses of Agriculture and Population) is specifically in-scope as a deliverable for the 2021 Census of Agriculture.
The 2021 CEAG dissemination strategy and release schedule will be presented to the CEAG steering committee for review and approval. Related processes for the release of selected historical farm and farm operator data will also be reviewed and the timeline for releases will be adjusted based on feedback from the Federal Provincial Territorial partners (key users of the data).
Agriculture Division has already taken steps to increase cross-sectoral analysis with non-agricultural sectors, including the infographics on:
Which came first: The chicken or the egg? Poultry and eggs in Canada
Thanksgiving: Around the Harvest Table.
The CEAG will continue to build on this initiative by developing cross-sectoral infographics, analytical studies, Daily releases and interactive data visualization for the 2021 CEAG data release.
Deliverables and timelines
The Assistant Chief Statistician, Economic Statistics (Field 5) will ensure the delivery of
The approved Dissemination strategy (December 2020)
A proposal for cross-analysis such as Infographics, analytical studies, and Daily releases integrating CEAG data with data from other sectors (March 2021)
A proposal for new interactive visualization tools within the Agriculture Stats Hub (March 2021).
Recommendation 2:
Web tools include guidance on how to use them and how to interpret data from them. A proactive approach to launching new tools should be taken. Webinars were identified as an effective channel and the use of other channels would allow for even a wider coverage.
Management response
Management agrees with the recommendation.
YouTube tutorial videos on how to use Statistics Canada geographic boundary files using open source GIS software (QGIS) have been produced and added to the Agriculture and Food portal.
The CEAG will create "How to" instructions and demos on how to use the interactive visualization web tools. The "How to" instructions will be available within each tool and the demos will be presented to data users in a series of webinars planned for the 2021 CEAG releases.
Deliverables and timelines
The Assistant Chief Statistician, Economic Statistics (Field 5) will ensure the delivery of
A proposal for new interactive visualization tools within the Agriculture Stats Hub, with integral "How to use" instructions and webinar demos (March 2021).
Recommendation 3:
Unresolved issues for the migration to the IBSP, including incompatibilities between the IBSP and the CMP as well as the IBSP processing capacity, are addressed prior to the production phase.
Management response
Management agrees with the recommendation.
The CEAG will continue to work with partners to identify relevant and emerging issues related to the migration to the IBSP during the integrated testing commencing June 2020. Issues will be captured in JIRA and major risks entered in the CEAG risk register. Consolidated risks and issues will be tracked and actioned in project plan documentation.
The integrated testing will take place over several months. All relevant and emerging issues must be resolved by December 2020 to ensure the readiness of production activities.
Issues and risks to be monitored through the CEAG Steering Committee.
Deliverables and timelines
The Assistant Chief Statistician, Economic Statistics (Field 5) will ensure that relevant and emerging IBSP issues and risks are tracked consistent with the DPMF (December 2020).
Recommendation 4:
Significant risks during the production phase, particularly with regard to data quality assessments and the exercising of roles and responsibilities, are monitored and mitigated.
Management response
Management agrees with the recommendation.
A table top exercise will be conducted to identify potential gaps in the processes in place (including risk management) for the production phase. Information gathered during the exercise will be used to inform plans and develop potential contingencies. Results will be presented to the CEAG Steering Committee.
The CEAG will engage the IBSP and all its stakeholders ("SWAT" team) in convening meetings to communicate relevant and emerging issues and risks during the production phase and to find resolutions. Roles and responsibilities will be formally documented and presented at the CEAG Steering committee.
The SWAT team will be ready for the production phase.
Deliverables and timelines
The Assistant Chief Statistician, Economic Statistics (Field 5) will ensure the delivery of
The results from table top exercise (December 2020)
The CEAG "SWAT" team with documented roles and responsibilities (March 2021).
Recommendation 5:
Planning processes for future projects falling outside the scope of the Departmental Project Management Framework include an initial assessment that takes into account elements such as risk, materiality, public visibility and interdependencies. The assessment should then be used to determine the appropriate level of oversight and project management.
Management response
Management agrees with the recommendation.
A new process will be implemented (for both subprojects under AG-Zero and non-DPMF projects) that will require the development of a project plan prior to the launching of a new project. The plan will include among other things: an initial assessment of the issues and risks (and mitigation strategies); a description of the methodology and assumptions; the identification of interdependencies and expected outcomes; and communication plans. The monitoring of projects will take place through existing governance mechanisms. Finally, existing projects already underway will be subject retroactively to the new process.
Where relevant, the plans will be used to update the DPMF project issues and risks register and the DPMF Project Plan.
Deliverables and timelines
The Assistant Chief Statistician, Economic Statistics (Field 5) will ensure the delivery of a new project plan process (June 2020).
Recommendation 6:
Processes and tools for documenting and sharing of best practices are implemented and lessons learned from other organizations (internal and external) are leveraged.
Management response
Management agrees with the recommendation.
The Agriculture Division has already shared lessons learned and best practices through various mechanisms including:
a presentation at AAFC on producing crop yield estimates using earth observation and administrative data on March 14, 2019
a presentation at the Economic Statistics Forum on November 12th, 2019
a presentation at AAFC on February 7th, 2020, on Predicting the Number of Employees using Tax Data.
As part of the new project plan process outlined previously, the Agriculture Division will leverage lessons learned from other organizations where applicable. In addition, as part of ongoing monitoring, lessons learned and best practices from projects will be documented.
Deliverables and timelines
The Assistant Chief Statistician, Economic Statistics (Field 5) will ensure the delivery of:
A systematic approach to share and document lessons learned (December 2020)
A presentation(s) at conferences such as the Economic Statistics Forum (March 2021)
An article(s) in @StatCan or the Modernization bulletin (March 2021)
A presentation(s) at AAFC (March 2021).
Appendix A: Integrated Business Statistics Program (IBSP)
The IBSP provides a standardized framework for surveys with common methodologies for collection and processing. Through standardization and use of corporate services and generalized systems, the program optimizes the processes involved in the production of statistical outputs; improves governance across all areas involved in statistical data output, particularly for change management; and modernizes the data processing infrastructure. This is achieved by balancing the development of a coherent standardized model with the maintenance of flexible program-specific requirements. It is expected that the IBSP surveys will use:
the Business Register (BR) as a common frame;
harmonized concepts and content for questionnaires;
electronic data collection as the principal mode of collection;
shared common sampling, collection and processing methodologies;
common tools for data editing and analysis; and
the tax data universe for estimating financial information.
Appendix B: List of products released (2016 CEAG)
Table 6: List of products released (2016 CEAG)
Date of release
Title of product (including link)
Type of product
Timeliness
Time lapse between collection and release:
Days (years)
Appendix C: Governance and management structures (Census of Agriculture and the Integrated Business Statistics Program)
Overall management of the Census of Agriculture:
CEAG Working Group (WG) for overall management of the CEAG (monthly meetings): chaired by the Assistant Director (AD) and Chief, and includes Chiefs from other relevant areas (e.g., methodology, IT and unit heads)
CEAG Management Team for day-to-day management of the CEAG (weekly meetings): includes the same members as the CEAG WG, but is also extended to other staff involved
CEAG Steering Committee:Footnote 23 an overarching advisory and decision-making function (monthly meetings)
Other WGs and committees for various functions (e.g., Census of Population/CEAG WG, Collection WG, Advisory Committee on Agriculture and Agri-Food Statistics, Federal-Provincial-Territorial Committee on Agriculture Statistics.)
Governance structures already established within the Enterprise Statistics Division (ESD) for the IBSP:
IBSP Transition/Integration/Production WGs: chaired by ESD, and includes the CEAG and all other partners such as the Operations and Integration Division (OID), the Collection Planning and Research Division (CPRD), as well as methodology and IT (bi-weekly meetings), to support the transition, integration, and production phases;
IBSP Project Management Team:Footnote 24 an overarching advisory and decision-making function that includes directors general, directors and ADs involved in the IBSP migrations
Change Management Committee: involved only during the production phase, it will be responsible for overseeing change management during production (e.g., if the schedule needs to be changed, the Committee will triage the request to the different stakeholders involved.)
Appendix D: Innovation Maturity Survey
In 2018, Statistics Canada conducted a survey to measure the innovation maturity level of the agency across 6 attributes:Footnote 25
Client expectations - incorporating the expectations and needs of clients in the design and development of innovative services and policies
Strategic alignment - articulating clear innovation strategies that are aligned with the organization's priorities and mandate
Internal activities - building the right capabilities aligned with the innovation strategies
External activities - collaborating across the whole of government and with external partners to co-innovate policies, services and programs
Organization - fostering the right organizational elements to drive innovation performance at optimal cost
Culture - aligning the innovation goals, cultural attributes, and behaviours with the innovation strategies
The Agriculture Division had maturity levels higher than those for all of Statistics Canada and compared with the Economic Statistics Field as a whole.
Figure 3: Results from the Innovation Maturity Survey (5 point scale)Footnote 26Description for Figure 3 - Results from the Innovation Maturity Survey (5 point scale)
The figure depicts the results of Statistics Canada Innovation Maturity Survey level for 4 different groups (Statistics Canada; Economic Statistics Field; Agriculture, Energy and Environment Statistics Branch; and, Agriculture Division. Six different attributes were used: Client expectations; Strategic alignment; Internal activities; External activities; Organization; and, Culture. Overall maturity was also assessed.
Results from the Innovation Maturity Survey (5 point scale)
Attribute
Statistics Canada
Economic Statistics Field
Agriculture, Energy and Environment Statistics Branch
Join by Attributes (Part 2) - One to Many Joins - Video transcript
(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Join by Attributes (Part 2) - One to Many Joins")
Hello everyone, following up from the previous tutorial on one-to-one join by attributes, today we'll discuss the second type – the one-to-many join, where there are many rows or features for one corresponding feature geometry. We'll demonstrate these procedures using the Farms Classified by Total Farm Capital table, which includes the numbers of farms reporting in six separate capital categories for each corresponding boundary. The table also requires some formatting in an external editor, including extracting the join information which is not unusual for performing joins or integrating Statistics Canada tables with vector data presently.
So within the table we'll begin by moving the first Census Agricultural Region name down to align with the first capital category and delete the row it was in. Then we'll copy the main table to a new workbook to continue editing. So now we must fill the Census Agricultural Region names for each capital category. So under Find and Select, we can click Go to Special and check Blanks. In the Formula Bar at the top we'll then define the cell used to fill the first blank cell and hit Ctrl and Enter to auto-fill the formula down the entire column.
Next we want to extract the Census Agricultural Region Unique Identifiers found within the square brackets of the "Geography" column. So to do so we can apply this formula, which will find and extract characters within square brackets. I've included this and other helpful formulas for extracting join information within the video description. So after pasting the formula we can use the tab in the bottom-right corner of the cell to apply the formula for the entire column.
Now we want to remove the CAR prefix so we'll use the RIGHT formula, specifying the cell and the number of characters to take, in this case 9.
The final procedure to isolate the Census Agricultural Region Unique Identifiers is to take the first four values from the LEFT.
To extract the join information for finer census geographies such as subdivisions, we would actually take the first three digits on the left and the last four digits on the right and concatenate them together, with the procedures otherwise being the same.
Now we'll copy the extracted IDs and paste them as values in the adjacent column. We can then delete the previous three columns with formulas. And the final step is to provide abbreviated field names less than 10 characters in length as we learned in the previous video. I'm adding FN in front of 2011 and 2016 to distinguish that these are farm numbers being reported within these columns. Then we'll save our table first as an excel workbook, which we'll call FarmCap and then we'll resave as a comma-separated values file selected from the Save As Type drop-down. Saving directly to .csv could've adversely affected our table formatting. So now we can close the program, and click Don't Save.
Now within QGIS we can refresh our browser panel and load in the formatted table along with our Census Agricultural Regions boundary file.
Rather than using the Joins tab which results in a more complex workflow, we'll go to the Processing Toolbox, and search Join Value - opening the Join Attributes by Field Value tool. Despite the slightly different appearance and format the tool contains the exact same parameters as the Joins tab. We specify the two layers that we'd like to join, here the Census Agricultural Regions boundary file and the formatted farm capital table – and the two fields with common entries used to link the datasets together – in this case both being CARUID. We can also specify which fields to add from our second layer. The main distinguishing feature of the tool is the ability to select the One-To-Many from the Join Type drop-down.
So we'll now save it to a permanent file – calling it JFarmCap – in our Joins folder. Run the tool and once complete refresh the Browser Panel and load the layer.
So we'll isolate the Census Agricultural Regions in Manitoba for a smaller area of analysis. With our selection we can then calculate the difference in the farm numbers reporting in each capital category between the two census years using the Field Calculator. So I'll call the field D-Farm-16-11 and expanding the Fields and Values drop-down we'll subtract the number of farms in 2016 from those in 2011.
So now we can apply a graduated symbology to our calculated difference field. I created a symbology file earlier for this purpose which I'll load from File. So Natural Jenks was the Mode to establish the ranges for visualization and the same field was also used to label the features . It's important that we are applying the visualization to this layer, as it contains the full range of values which might not be replicated by individual capital categories.
Now within the Processing Toolbox, the Split Vector Layer tool can be used to create many separate layers from a vector according to a unique ID. So if we left it as is, using the CARUID – each census agricultural region would be output as a separate file. However, since we're interested in examining the changing number of farms in each capital category - we'll apply it to the Farm Capital Class field. We'll create a new directory within our Joins folder for the output files, calling it Split Farm Capital. Ensuring it's only applied to the selected features we can then click Run.
Once complete, we can now expand the directory and for a selection of layers in the Browser Panel we can right-click and select Add Layers.
So with the symbology already applied to our original joined vector, we can simply right-click copy style, and selecting all style categories. Now we can select and right-click the split layers and click paste style. So this enables a uniform visualization, with the complete value ranges, to be rapidly applied across multiple layers.
I've loaded and ordered the split layers in ascending capital ranges in the Prepared Layers group. So now if we toggle through the separate Farm Capital Categories we can see a broad trend. In general there's a decline in the numbers of farms reporting in smaller capital categories – and as we approach larger capital categories, there's a general increase in the number of farms reporting. This is an established trend within the agricultural sector.
So joining table and vector data is a powerful tool for integrating tabulated variables in a geospatial format – facilitating analyzing and visualizing spatial and temporal variations in a wide-range of reported variables. With the skills developed in this demo, users should be able to: Extract or match join information from datasets, Perform one-to-one and one-to-many joins between tables and vector datasets, and visualize joined variables and examine the relations with the graphics tools.
Although we only joined one table to the vector datasets for each of these case-uses, these procedures can be repeated to combine multiple tables and explore relations between variables at a common level. Apply these skills to datasets of interest to you. So stay tuned for the next tutorial in which we'll cover creating Maps in QGIS.
And on one final note - Statistics Canada is increasingly releasing datasets in table and spatial formats on the Federal Geospatial Platform, such as Median Income after Tax and Farm Operator datasets. They can be readily loaded into QGIS and used to examine trends at multiple scales. So download these datasets when available to facilitate using Statistics Canada data in QGIS.
CVs for Total sales by geography - July 2020
Table summary
This table displays the results of Annual Retail Trade Survey: CVs for Total sales by geography - July 2020. The information is grouped by Geography (appearing as row headers), Month and Percent (appearing as column headers).
Join by Attributes (Part 1) - One-to-One Joins - Video transcript
(The Statistics Canada symbol and Canada wordmark appear on screen with the title: "Join by Attributes (Part 1) - One-to-One Joins")
So in this tutorial we'll introduce joining datasets by attributes, specifically linking tables to vector datasets for analysis and visualization.This is a powerful way to examine tabulated variables, linking them to vector geometries via common entries - in this case a column in the table and a matching field within the vector.There are two types of attribute joins, with slightly different procedures. Today we'll cover the first, the one-to-one join, where there is one row for each corresponding feature or geometry.For a successful join the entries must match perfectly. Thus, numeric identifiers are best due to complications with text such as special characters, spacing and case-sensitivities.Copying and matching entries between datasets is another method to improve the likelihood of a successful join.
For the tutorial we'll use the Population and Dwelling Highlight tables - downloaded previously.They are ideally formatted, as the join information is readily available and boundary changes between census collections have been accounted for – requiring no external formatting.
So we can load table datasets in to QGIS using the established procedures, double left clicking or dragging and dropping into the Layers Panel.So with the tables and corresponding boundaries loaded we'll open the Layer Property box of the Census Division layer which we'll use to demonstrate the procedures.We can use the Joins tab to perform one-to-one joins and to create our join click the Plus icon.
So the Join layer is the dataset that we'd like to join. And the join field is the specific column or field used to link the datasets.While the target field is the corresponding field within the vector containing matching entries, in this case the Unique Census Division Identifier.
We'll also check the following boxes, and specify the fields to join – as otherwise all fields are joined by default.Specifically we'll add the population and total private dwelling counts and percent change fields,as well as the land area, population density and two population rank fields.We'll remove the custom prefix to retain the original column names.
Now in the Source Fields tab we can see that the columns are joined temporarily and have been misattributed as text field types – a default in QGIS.Additionally, the joined field names exceed the limits of the file format we're using.So to permanently join the datasets we would have to export the dataset to a new layer.But to accomplish all three tasks at once we'll use the Refactor Fields tool.
In the Processing toolbox we can search the tool and double-left click to open it.So we'll close the tool description. And using the drop-downs we can change the field types for the joined columns.Specifically we'll use integer for the count variables, which will be whole numbers, while for the percent change field, land area and population density fields we'll use double - the equivalent for the decimal number field type.Then we need to specify the parameters – using a length of 12 and, for our double columns, a precision of 2.Finally we'll rename our fields to abbreviated headers less than 10 characters in length – the limit of the shapefile format we're using.So feel free to use abbreviations that are most interpretable to you.
Now we'll save the file - providing it an output directory and name.I am storing it in the Joins demo folder and calling it JPopCD for Joined Population - Census Divisions.Once complete, refresh the Browser panel and load the joined dataset into the Layers Panel.
So with our fields correctly attributed - we can now visualize the joined variables, applying a graduated symbology to the numeric fields – specifically in this case we'll use the percent population changes field.We'll select an appropriate colour ramp for visualization and for the Census Division layer we can use Pretty Breaks to establish the value ranges for visualization – clicking OK.
Now we can examine the spatial variations in the joined variables across Divisions within Canada.
We recommend practicing these procedures on your own by repeating with the Census Subdivisions layer to familiarize yourself with the workflow.Once again specify the Join layer and the join information in this case using the Census Subdivision unique identifier to link the datasets.
For the demo, I've loaded a Joined Subdivision layer I created earlier.Due to the data distribution at this level Quantile (Equal Count) was the method used to establish the break values for visualization.But toggling back and forth between the layers demonstrates how these procedures enable variables of interest to be assessed at multiple scales relatively quickly and easily.
And on that note - if we want to determine trends at a broader level we can use the Aggregate tool.So we specify the layer to be aggregated, and then the Group by Expression drop-down enables us to select a field to use in aggregating both the geometries and attributes of the layer.Since we are gonna use the Provin…Unique Provincial Identifier, the first three Census Division fields are redundant – so we'll remove them.We can specify the operator applied in aggregating the fields from the drop-down. So we'll use First Value for the first two text fields and for the percent changes we'll use Mean.Finally we'll remove the Population Rank fields. And we'll save this layer in the same directory calling it JAgPrPop for Joined Aggregated Provincial Populations.Once it's complete we'll refresh the Browser Panel once again loading the layer.
And now we'll recalculate the percent population change field with the Field Calculator.So we'll add a bracket, and then subtract the Population in 2016 from that in 2011, close the bracket and divide it by the baseline in this case being the population in 2011 multiplied by 100.Once again we can use the symbology tab to visualize the variables at three separate levels.
So the final item I'd like to discuss is the Graphics drop-down in the Processing Toolbox, which can be used to quickly assess data distributions,such as the feature counts in different categories using histograms and boxplots, or variable relations between joined variables – using the Scatterplot tool.So we could specify the 2016 Population as our independent variable and the Total Private Dwellings in the same year as the dependent.Once run, we can click on the Hyperlink in the Results Viewer to examine the relation.Unsurprisingly there is a very strong positive relationship between the total population and number of private dwellings.
So that concludes the one-to-one join - when there is one entry for each corresponding feature - using the Joins tab.The procedures can be applied to join tables or link two or more vector datasets together - enabling the visualization of variables of interest.We also learned how to apply the Refactor Fields tool to alter field types of joined data and use the Aggregate tool to examine trends at broader levels.We could iterate these procedures to combine multiple variables for examining changing variable relations between locations or over time.In the next demo we'll examine how to perform a one-to-many join.
