Practical Machine Learning - Day 1¶
Airwatch Bangalore
May 7-9, 2018
In [1]:
1 + 2
Out[1]:
This is a simple text.
You can execute any cell by pressing Shift+Enter.
You can convert a cell to markdown cell by pressing Esc + M. You may also be able to do that using the dropdown on the top.
To convert a markdown cell to code cell press Esc + Y.
Data Science Libaries¶
For Data Science we are going to use the following libraries:
- numpy - numerical computation
- pandas - spreadhseet for hackers
- matplotlib - ploting
- scikit-learn (sklearn) - machine learning library
numpy¶
In [2]:
import numpy as np
In [3]:
x = np.array([1, 2, 3])
In [4]:
x.shape
Out[4]:
In [5]:
x.dtype
Out[5]:
In [6]:
x.sum()
Out[6]:
In [7]:
x + 1
Out[7]:
In [8]:
d = np.array([[0.1, 0.2, 0.3], [1.1, 1.2, 1.3]])
In [9]:
d
Out[9]:
In [10]:
d.shape
Out[10]:
In [11]:
d * 10
Out[11]:
In [12]:
d
Out[12]:
In [13]:
d > 1
Out[13]:
In [14]:
np.sum(d > 1)
Out[14]:
Set all elements less than 1 to 0.
In [15]:
d[d < 1] = 0
In [16]:
d
Out[16]:
pandas¶
In [1]:
import pandas as pd
Pandas has two important data structures. Series and DataFrame.
Series is a column and DataFrame is a spreadsheet.
In [18]:
x = pd.Series([1.1, 2.2, 3.3, 4.4])
In [19]:
x
Out[19]:
In [20]:
x = pd.Series([1.1, 2.2, 3.3, 4.4], ["a", "b", "c", "d"])
In [21]:
x
Out[21]:
DataFrame¶
In [22]:
data = [[1, 1], [2, 4], [3, 9], [4, 16]]
d = pd.DataFrame(data)
In [23]:
d
Out[23]:
In [24]:
data = [[1, 1], [2, 4], [3, 9], [4, 16]]
d = pd.DataFrame(data, columns=["x", "y"])
In [25]:
d
Out[25]:
In [26]:
d.x
Out[26]:
In [27]:
d[d.x > 2]
Out[27]:
Weather Data¶
In [ ]:
url = "https://archive.org/download/www.imdaws.com-2012/daily.zip/daily%2FARG-2012-07-01.csv"
In [36]:
!wget -nv -O ARG-2012-07-01.csv $url
In [3]:
url = "https://notes.pipal.in/2018/airwatch-ml/ARG-2012-07-01.csv"
url = "ARG-2012-07-01.csv"
In [59]:
df = pd.read_csv(url)
In [60]:
df.head()
Out[60]:
In [61]:
df.shape
Out[61]:
In [62]:
df.dtypes
Out[62]:
In [63]:
df["STATION ID"].nunique()
Out[63]:
In [64]:
df.columns
Out[64]:
Let us simplify the column names.
In [65]:
df.columns = ["srno", "station", "date", "time",
"lat", "lon", "rainfall",
"temp", "tmax", "tmin"]
In [66]:
df.head()
Out[66]:
If you want to save it to the disk, then use:
In [12]:
df.to_csv("rainfall.csv")
In [13]:
df.date.unique()
Out[13]:
In [14]:
df.date.nunique()
Out[14]:
In [15]:
df.time.nunique()
Out[15]:
In [16]:
df.time.unique()
Out[16]:
In [17]:
df.dtypes
Out[17]:
How to get all entries for station "AGAR"?
In [18]:
df[df.station == "AGAR"]
Out[18]:
In [19]:
c = df[df.station == "AGAR"].rainfall
In [20]:
c2 = c.astype(np.float)
In [21]:
c2
Out[21]:
The rainfall column is not numeric. There may be an invalid value. Let us find that out.
In [22]:
df.rainfall.unique()
Out[22]:
In [71]:
df.rainfall = df.rainfall.replace(" ", "0").astype(np.float)
In [24]:
df.head()
Out[24]:
How many stations had atleast some rainfall?
Which station got the most rainfall during that day?
In [25]:
df[df.station == 'SATHYABAMA UNIVERSITY']
Out[25]:
In [26]:
df.rainfall.sum()
Out[26]:
In [27]:
df_station = df.groupby(["station", "lat", "lon"]).sum()
In [28]:
# delete the srno column
del df_station['srno']
In [29]:
df_station.head()
Out[29]:
In [30]:
df_station.reset_index(inplace=True)
In [31]:
df_station.head()
Out[31]:
In [32]:
np.sum(df_station.rainfall > 0)
Out[32]:
In [33]:
df_station.sort_values('rainfall', ascending=False).head()
Out[33]:
In [34]:
x = df.groupby('station').count()['rainfall'].value_counts()
In [35]:
x
Out[35]:
In [36]:
%matplotlib inline
In [37]:
(df.groupby('station')
.count()
.rainfall
.value_counts()
.sort_index()
.plot(kind="bar"))
Out[37]:
Visualization¶
In [38]:
data = [
["N", 25, 8],
["E", 5, 2],
["W", 10, 3],
["S", 15, 5],
["C", 20, 6]]
columns = ["Area", "Sales", "Profit"]
df = pd.DataFrame(data, columns=columns)
In [39]:
df
Out[39]:
Lets get back to the rainfall data.
Where is more rain fall happening in the country? Can you show it visually?
In [41]:
df_station.head()
Out[41]:
In [43]:
# Add this line to the beginning of your notebook
%matplotlib inline
In [67]:
df.columns
Out[67]:
In [74]:
df_station = df.groupby(["station", "lat", "lon"]).sum().reset_index()
In [75]:
df_station.head(50).plot(kind="bar", x="station", y="rainfall")
Out[75]:
In [76]:
(df_station
.sort_values("rainfall")
.tail(50)
.plot(kind="barh", x="station", y="rainfall", figsize=(5, 20)))
Out[76]:
In [54]:
df_station.rainfall.hist(bins=50)
Out[54]:
In [84]:
df_station.plot(kind="scatter",
x="lon", y="lat",
s=df_station.rainfall,
figsize=(4, 4),
alpha=0.3)
Out[84]:
In [99]:
import matplotlib.pyplot as plt
plt.style.use("default")
In [100]:
df_station.plot(kind="scatter",
x="lon", y="lat",
c="rainfall",
figsize=(4, 4),
alpha=0.3,
cmap="Reds")
Out[100]:
In [ ]: