Data Processing for Social Scientists
About
Data rarely arrives ready to analyze. This course teaches the everyday work between collecting data and analyzing it: cleaning raw tables and layers, geoprocessing (buffers, overlays, distances and areal weighting), raster processing (map algebra, reclassification and zonal statistics) and automating the steps you repeat. It is written for social scientists and other beginners who have finished the Foundations course.
Each topic is taught twice, once in QGIS (point and click) and once in Python (Jupyter notebooks), with the same data and the same results. Follow either one, or both. All examples use Tippecanoe County, Indiana (Lafayette and West Lafayette) and Indiana as a whole.
| Module | Question it answers | Tools |
|---|---|---|
| Introduction to Data Processing | What happens between download and analysis, and how do I keep it repeatable? | Concepts |
| Cleaning and Preparing Data (Python / QGIS) | How do I fix identifiers, types, special codes, duplicates and geometry problems? | pandas, geopandas / Refactor Fields, Check Validity |
| Geoprocessing (Python / QGIS) | What lies within a distance, inside a boundary, or nearest? | geopandas / QGIS Vector menu |
| Raster Processing (Python / QGIS) | How do rasters combine, and what do they say about each tract? | rasterio, rasterstats / Raster Calculator, Zonal statistics |
| Automating Repeated Steps (Python / QGIS) | How do I run the same workflow for many inputs without mistakes? | Functions and loops / Model Designer, batch processing |
Start with the Introduction, then take the modules in order: each one uses results or ideas from the one before. The advanced methods (network analysis, spatial autocorrelation, clustering and NDVI) follow in the Data Analysis course.
Prerequisites
- For the Python modules: a free CyberFaCES account, and basic Python,
pandasand GeoPandas (Foundations notebooks 2a to 4c) - For the QGIS modules: QGIS installed, and the Foundations QGIS modules (Getting Started with QGIS; Tables, Joins and Selections)
- No Census API key is needed: the Census tables are provided as downloaded
New to Python, Jupyter notebooks or QGIS? Start with the Foundations course, then read the Setup page.
This course is part of Enabling the Geospatial Turn in the Social Sciences, a set of training modules developed at Purdue University with support from the National Science Foundation.
| Type | Course |
|---|---|
| Level | Expert |
| Length | 11 hours |
| Topics |
python QGIS Concepts Community: Spatial Turn |
Modules
-
Introduction to Data Processing
Module 1 • 35 minutes
What happens between download and analysis, and how to keep processing repeatable
-
Cleaning and Preparing Data in Python
Module 2 • 1 hour
Fix identifiers, types, special codes, duplicates and geometry problems with pandas and GeoPandas
-
Cleaning and Preparing Data in QGIS
Module 3 • 1 hour
Fix identifiers, types, special codes, duplicates and geometry problems with Refactor Fields and Check Validity
-
Geoprocessing in Python
Module 4 • 1 hour
Buffers, selecting by location, clip, intersect, dissolve, areal weighting and nearest features in GeoPandas
-
Geoprocessing in QGIS
Module 5 • 1 hour
Buffers, selecting by location, clip, intersect, dissolve, areal weighting and nearest features in QGIS
-
Raster Processing in Python
Module 6 • 1 hour
Map algebra, reclassification and zonal statistics with rasterio and rasterstats
-
Raster Processing in QGIS
Module 7 • 1 hour
Map algebra, reclassification and zonal statistics with the Raster Calculator and Zonal statistics
-
Automating Repeated Steps in Python
Module 8 • 1 hour
Functions and loops that run one workflow for many distances, layers and counties
-
Automating Repeated Steps in QGIS
Module 9 • 1 hour
The Processing History, the Model Designer and batch processing
Instructor
Aniket Dev Roy
Discussions
Please login to view discussions.
