Module 2: Wrangle ML Datasets

Module Overview

In this module, you'll learn essential techniques for wrangling datasets for machine learning. Data preparation is a critical step in the machine learning workflow, often taking up to 80% of a data scientist's time. You'll explore methods for data cleaning, exploration, and joining relational data to create meaningful feature sets for your models.

Learning Objectives

• explore tabular data for supervised machine learning
• join relational data for supervised machine learning

Objective 01 - Explore tabular data for supervised machine learning

Overview

In this module, we'll continue with our previous theme of learning to better understand our data in terms of preparation for modeling. In this objective, we'll explore a small data set and get some additional practice in basic cleaning; identifying columns that might be suitable for feature engineering and generally preparing the data for machine learning tasks.

Follow Along

The data we'll use in the following example is from the USGS Earthquake Catalog and the PNSN Earthquake catalog. We'll load in this data and go through a few processing steps to prepare it for some further analysis.

# Load in earthquake data sets
import pandas as pd

cols_set1 = ['Evid', 'Magnitude', 'Magnitude Type', 'Time UTC', 'Lat', 'Lon', 'Depth Km']
eq_set1 = pd.read_csv('pnsn_eqlist.csv', usecols=cols_set1)

cols_set2 = ['time', 'latitude', 'longitude', 'depth', 'mag', 'magType', 'net', 'id']
eq_set2 = pd.read_csv('usgs_eqlist.csv', usecols=cols_set2)

display(eq_set1.head())
display(eq_set2.head())

# Clean up eq_set2 id column
# (strip the 'uw' or 'us' from the number)
eq_set2['id'] = eq_set2['id'].map(lambda x: x.lstrip('uws'))

# Add column with the network code
eq_set1['net'] = 'uw'

# Rename columns
new_cols = ['id', 'mag','magType','time','latitude','longitude','depth','net']
eq_set1.columns = new_cols

eq_set1.head()

# Check data types
print('Data type for eq_set1: ', eq_set1['id'].dtype)
print('Data type for eq_set2: ', eq_set2['id'].dtype)

# Change data type
eq_set1['id'] = eq_set1['id'].astype('str')
print('Data type for eq_set1: ', eq_set1['id'].dtype)

Data type for eq_set1: int64
Data type for eq_set2: object
Data type for eq_set1: object

Objective 02 - Join relational data for supervised machine learning

Overview

At this point, we have explored the earthquake data set, cleaned it up a little bit, and prepared to combine it into one file. In the next section we'll combine the two DataFrames, and in the process review some of the techniques we learned previous Sprints.

Follow Along

In the following section, we'll reproduce the cleaning and preparation process from the previous objective and continue on with combining our DataFrames. After they're combined, we'll take a look and make sure there are no duplicates.

# Load in earthquake data sets
import pandas as pd

cols_set1 = ['Evid', 'Magnitude', 'Magnitude Type', 'Time UTC', 'Lat', 'Lon', 'Depth Km']
eq_set1 = pd.read_csv('pnsn_eqlist.csv', usecols=cols_set1)

cols_set2 = ['time', 'latitude', 'longitude', 'depth', 'mag', 'magType', 'net', 'id']
eq_set2 = pd.read_csv('usgs_eqlist.csv', usecols=cols_set2)

# Clean up eq_set2 id column
# (strip the 'uw' or 'us' from the number)
eq_set2['id'] = eq_set2['id'].map(lambda x: x.lstrip('uws'))

# Add column with the network code
eq_set1['net'] = 'uw'

# Rename columns
new_cols = ['id', 'mag','magType','time','latitude','longitude','depth','net']
eq_set1.columns = new_cols

# Change data type
eq_set1['id'] = eq_set1['id'].astype('str')

# Take a look at our work
display(eq_set1.head())
display(eq_set2.head())

Now we can combine the two sets. First, we need to think about how to join the DataFrames. In general, we should keep all of the data if possible. In this case, the columns in each DataFrame are the same, so we're essentially concatenating the two DataFrames. After we do that, we'll check for duplication in the event id and remove any duplicates.

# Concatenate eq_set1 and eq_set2
eq_set_all = pd.concat([eq_set1, eq_set2])
print('Size of DataFrame before dropping duplicates: ', eq_set_all.shape)

# Remove duplicate event id numbers
eq_set_all.drop_duplicates(subset='id', inplace=True)
print('Size of DataFrame after dropping duplicates: ', eq_set_all.shape)

Size of DataFrame before dropping duplicates: (68, 8)
Size of DataFrame after dropping duplicates: (50, 8)

Challenge

It's your turn to find some data that you need to combine or join in some way. Dig up two (or more) data sets and identify what you would need to do to combine them into one DataFrame. For more of a challenge, you can load, clean, and combine your data set. To make this a little easier, you could use a small subset of each to practice on.

Additional Resources

• USGS Earthquakes
• USGS Earthquakes Data
• PNSN Earthquakes
• PNSN Earthquakes Data

Guided Project

Open DS_232_guided_project.ipynb in the GitHub repository below to follow along with the guided project:

Guided Project Video

Module Assignment

For this assignment, you'll continue working with your portfolio dataset from Module 1. You'll apply what you've learned to clean, explore, and prepare your data for modeling.

Note: There is no video for this assignment as you will be working with your own dataset and defining your own machine learning problem.

Additional Resources