Practical Machine Learning - Day 2 - Housing¶
VMware Bangalore
June 18-20, 2018
Amit kapoor • Anand Chitipothu • Bargava Subramanian
Notes of this workshop are available online at: https://bit.ly/vmware-ml
Home | Day 1 | Day 2 - Iris | Day 2 - Housing | Day 3
Housing¶
In [2]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['figure.subplot.bottom'] = 0.15
%matplotlib inline
FRAME¶
Find if a retail listing is going generate interest from the protential customers?
ACQUIRE¶
In [3]:
url = "https://notes.pipal.in/2018/vmware-ml/rent-data.json.zip"
# url = "https://notes.pipal.in/2018/vmware-ml/rent-data.json"
In [4]:
df = pd.read_json(url)
In [5]:
df.columns
Out[5]:
In [8]:
df.shape
Out[8]:
In [9]:
df.dtypes
Out[9]:
In [6]:
df.head()
Out[6]:
REFINE¶
Prediction Variable: interest_level
In [11]:
df.interest_level.value_counts()
Out[11]:
What are my Y feature set
In [13]:
df.columns
Out[13]:
Refinement¶
- Domain Check: Is the type of data match what it should be?
- Quality of the Data Is there any missing values in the column?
- Does the distribution fit the expectation about the data: Are their any outliers in that column
In [19]:
# What are the unique values in the column
df.bathrooms.unique()
Out[19]:
In [20]:
# Is the type of bathrooms correct?
df.bathrooms.dtypes
Out[20]:
In [21]:
# Are there any null values
df.bathrooms.isnull().sum()
Out[21]:
In [24]:
# Are there any outliers in this data
df.bathrooms.hist()
Out[24]:
In [29]:
df.bathrooms.value_counts(sort=True)
Out[29]:
Bedrooms¶
In [32]:
# Is there null values
df.bedrooms.isnull().sum()
Out[32]:
In [33]:
df.bedrooms.value_counts(sort=True)
Out[33]:
In [35]:
# Cross Tabulation
pd.crosstab(df.bedrooms, df.bathrooms)
Out[35]:
Outlier / Null Identification & Redressal System¶
Identification
- Domain Knowledge: Is it even possible
- Correlate with other feature: Bathrooms vs Bedrooms, Check with the size of the datasets
- Visual inspection: Literally send a person, Look at photos associated to find it
- Statistial Measure: Box-Plot, ...
Redressal system for Outliers & Null Values
- REMOVE: Drop that row or colum from the dataset
- IMPUTATION:
- Domain Knowledge: Bathroom = Bedrooms - 0.5
- Calculate a figure: mean, median, mode
- Build a cluster model to impute value
- Conversion: Convert from Continuous to Categorical
In [37]:
df[df.bathrooms == 10]
Out[37]:
In [38]:
## Check for dtypes for all
df.dtypes
Out[38]:
- bathrooms ratio
- bedrooms ratio
- building_id interval
- created datatime
- description text
- display_address text
- features list of text
- interest_level ordinal
- latitude interval
- listing_id interval
- longitude interval
- manager_id nominal
- photos list but -> photos*
- price ratio
- street_address text
In [41]:
# Fix the dtypes
df.created = pd.to_datetime(df.created)
Null values together¶
If you want to visalise missing nos. - then you can also use missingno library
In [45]:
df.isnull().sum()
Out[45]:
In [54]:
## Outlier for Price
df.price.hist()
Out[54]:
In [59]:
df[df.price > 200000]
Out[59]:
Outlier treatment¶
- Drop Price > 200000
- Drop Bathrooms > 7
In [71]:
df.drop?
In [75]:
# Need to run this !
df.drop(df[df.price > 200000].index, inplace=True)
In [76]:
df.drop(df[df.bathrooms > 7].index, inplace=True)
In [77]:
df.shape
Out[77]:
Explore the data¶
Lets looks at the possible input features
- Bathrooms
- Bedrooms
- Price
- Latitude
- Longitude
- Features --> # len of features as a variable
- Created
In [78]:
# Price
df.price.hist()
Out[78]:
In [83]:
df.price.plot(kind="hist", logx=True, bins=100)
Out[83]:
In [143]:
plt.scatter(df.longitude, df.latitude, alpha=0.1)
plt.xlim(-70,-74)
plt.ylim(38,45)
Out[143]:
In [155]:
df.head()
Out[155]:
In [146]:
from plotnine import *
In [163]:
(ggplot(df) + aes('longitude', 'latitude', color='interest_level')
+ geom_point(alpha=0.01) + ylim(40.6,40.8) +xlim(-73.5,-74.5) +
facet_wrap("interest_level")
)
Out[163]:
Transfom¶
- Shape Transform: Change
priceto log numbers - Encoding: Convert *categorical to numbers
In [84]:
df["priceLog"] = np.log(df.price)
In [89]:
df.columns
Out[89]:
In [93]:
X = df[["bathrooms", "bedrooms", "latitude", "longitude", "priceLog"]]
X.head()
Out[93]:
In [95]:
y = df['interest_level']
y.head()
Out[95]:
In [96]:
type(y)
Out[96]:
In [97]:
from sklearn.preprocessing import LabelEncoder
In [98]:
le_y = LabelEncoder()
In [99]:
le_y.fit(y)
Out[99]:
In [109]:
le_y.classes_
Out[109]:
In [100]:
y_encoded = le_y.transform(y)
In [101]:
y_encoded
Out[101]:
In [108]:
print("Original Values: \n" , y.iloc[:5], "\n\n Encoded values: \n", y_encoded[:5])
First model - Decision Tree¶
In [113]:
from sklearn.tree import DecisionTreeClassifier
In [114]:
model_dt = DecisionTreeClassifier(max_depth=2)
In [115]:
model_dt.fit(X, y_encoded)
Out[115]:
Cross-validation¶
In [116]:
#from sklearn.model_selection import StratifiedKFold
In [119]:
from sklearn.model_selection import cross_val_score
In [120]:
score = cross_val_score(model_dt, X, y_encoded, scoring="accuracy", cv=5, n_jobs=-1)
In [121]:
np.mean(score)
Out[121]:
Problem Try a different error metric (precision, recall)
Problem Change model depth and see how the model performance changes
In [135]:
def depthDT(depth):
model = DecisionTreeClassifier(max_depth=depth)
model.fit(X, y_encoded)
score = cross_val_score(model, X, y_encoded, scoring="accuracy", cv=5, n_jobs=-1)
return np.mean(score)
In [139]:
scores = []
for i in range(1,20,1):
score = depthDT(i)
scores.append(score)
In [140]:
plt.plot(scores)
Out[140]:
