Advanced Python Training at Arcesium - Day 2¶
Nov 15-17, 2017 Vikrant Patil
These notes are available online at http://notes.pipal.in/2017/arcesium-oct-advpython/day2.html
© Pipal Academy LLP
Iterators and Generators¶
In [1]:
nums = list(range(5))
In [2]:
for n in nums:
print(n)
In [3]:
for c in "string":
print(c)
In [4]:
for key in {"x":1,"y":2}:
print(key)
In [5]:
for line in open("data.csv"):
print(repr(line))
The Iteration protocol¶
In [6]:
items = [1,2,3]
In [7]:
itr = iter(items)
In [8]:
next(itr)
Out[8]:
In [9]:
next(itr)
Out[9]:
In [10]:
next(itr)
Out[10]:
In [11]:
next(itr)
Generators¶
In [12]:
def square(numbers):
for n in numbers:
yield n*n
In [13]:
sq5 = square(range(1,6))
In [14]:
sq5
Out[14]:
In [15]:
for i in sq5:
print(i)
In [16]:
sq4 = square(range(1,4))
In [17]:
next(sq4)
Out[17]:
In [18]:
range(5)
Out[18]:
In [22]:
def square(numbers):
print("Begin squares")
for i in numbers:
print("Computing square of ",i)
yield i*i
print("After yield")
print("Finish square")
In [23]:
sq4 = square(range(1,4))
In [24]:
next(sq4)
Out[24]:
In [25]:
next(sq4)
Out[25]:
In [26]:
next(sq4)
Out[26]:
In [27]:
next(sq4)
In [28]:
for s in square(range(1,5)):
print(s)
In [32]:
def f():
for i in range(1000):
if i ==13:
return
yield i*i
In [33]:
for s in f():
print(s)
In [34]:
def f():
for i in range(1000):
if i ==3:
return
yield i*i
In [35]:
g = f()
In [36]:
next(g)
Out[36]:
In [37]:
next(g)
Out[37]:
In [38]:
next(g)
Out[38]:
In [39]:
next(g)
problem: Write a generators function countdown that takes a number n as argument and generates all numbers down to 0 starting from n
>>> for i in countdown(3):
... print(i)
3
2
1
0 problem: Write a generator triangular that takes number n as argument and generates sequence of first n triangular numbers. nth triangular number is sum of fisrt n natural numbers.
>>> for t in triangular(5):
... print(t, end=",")
1,3,6,10,15Bonus problem: Remove duplicates from a sequence while maintianing order. Can same function be used to remove duplicate lines from a file?
>>> for item in consumedup([3,5,3,4,5,6,7,8,8,9]):
... print(item, end=",")
2,5,4,6,7,8,9In [40]:
x = set()
In [41]:
def countdown(n):
while n>=0:
yield n
n -= 1
In [42]:
for i in countdown(3):
print(i)
In [43]:
def triangular(n):
for i in range(1, n+1):
yield sum(range(1,i+1))
In [44]:
for t in triangular(5):
print(t, end=",")
In [45]:
def consumedup(seq):
seen = set()
for item in seq:
if item not in seen:
yield item
seen.add(item)
In [47]:
g = consumedup([3,5,3,4,5,6,7,8,8,9])
In [48]:
for item in g:
print(item, end=",")
In [51]:
%%file duplicatelines.txt
Saving file at /day2.ipynb
Saving file at /day2.ipynb
Saving file at /day2.ipynb
hello
hello
x
In [52]:
for line in consumedup(open("duplicatelines.txt")):
print(line, end="")
Generator Expressions¶
In [53]:
[n*n for n in range(1,11)] # list comprehension
Out[53]:
In [56]:
s = (n*n for n in range(1,11)) # generator expression
In [55]:
s
Out[55]:
In [57]:
sum(s)
Out[57]:
In [58]:
max(s)
In [59]:
sum((x*x for x in range(1000000)))
Out[59]:
In [64]:
sum(x*x for x in range(1000000))# when generator expression is the only argument
# to function then you can skeep parenthesis
Out[64]:
In [62]:
g = consumedup(x*x for x in range(1,5))
In [63]:
for i in g:
print(i, end=",")
In [66]:
def ones():
count = 0
while True:
if count >=3:
DOOM
yield 1
count += 1
In [67]:
one = ones()
next(one)
next(one)
next(one)
next(one)
next(one)
In [68]:
next(one)
What is the advantage of using generators/itereators¶
- Very usefull for huge data, only part of data is loded in memory
- you can build lazy pipelines of dataprocessing
Example: Building data pipelines¶
In [69]:
import os
def find(root):
for path, dirnames, filenames in os.walk(root):
for f in filenames:
yield os.path.join(path, f)
In [70]:
def take(n, seq):
it = iter(seq)
return list(next(it) for i in range(n))
In [71]:
def integers():
"""
generates infinite sequence of natural numbers
"""
i = 1
while True:
yield i
i += 1
def squares(numbers):
return (n*n for n in numbers)
In [72]:
take(10, squares(integers()))
Out[72]:
In [76]:
def grep(pattern, seq):
return (x for x in seq if pattern in x)
In [77]:
files = find(".")
pyfiles = grep(".py", files)
print(take(10, pyfiles))
In [80]:
def count(seq):
return sum(1 for item in seq)
In [81]:
count(range(100))
Out[81]:
In [82]:
def readlines(filenames):
"""
returns iterator over lines of all files
"""
for f in filenames:
for line in open(f):
yield line
How many line of python code we have written during this course
In [83]:
files = find(".")
pyfiles = grep(".py", files)
lines = readlines(pyfiles)
print(count(lines))
How many pythong function we have written?
In [85]:
files = find(".")
pyfiles = grep(".py", files)
lines = readlines(pyfiles)
functions = grep("def " ,lines)
print(count(functions))
problem: Write a function get_paragraphs to split given text into paragraphs. Paragraphs are seperated by empty line. The function should take a sequence of lines as argument and return a sequence of paragraphs. For sample input, see http://anandology.com/tmp/pg1342.txt once the function is there, we should be able to find:
- number of paragraphs
- The longest paragraph
In [86]:
x = [1,2,3,4]
In [87]:
itr = iter(x)
In [88]:
type(itr)
Out[88]:
In [89]:
y = (i for i in range(5))
In [90]:
type(y)
Out[90]:
In [91]:
type(range(1,2))
Out[91]:
In [92]:
r = range(1,2)
In [95]:
type(range)
Out[95]:
In [96]:
range.__class__
Out[96]:
In [99]:
next(y)
Out[99]:
In [100]:
if "":
print("x")
In [101]:
def get_paragraphs(seq):
paragraphs = []
for line in seq:
if line.strip()=="" and paragraphs:
yield "".join(paragraphs)
paragraphs = []
paragraphs.append(line)
if paragraphs:
yield "".join(paragraphs)
In [103]:
count(get_paragraphs(["A\nB", "\n", "A\n","B\n","\n" ,"A\n", "A\n"]))
Out[103]:
In [104]:
max(get_paragraphs(["A\nB", "\n", "A\n","B\n","\n" ,"A\n", "A\n"]), key=len)
Out[104]:
In [107]:
g = get_paragraphs(open("pg1342.txt"))
In [108]:
count(g)
Out[108]:
In [109]:
g = get_paragraphs(open("pg1342.txt"))
In [110]:
max(g, key=len)
Out[110]:
Working with XML¶
In [111]:
import requests
url = "http://www.thehindu.com/"
response = requests.get(url, params = {"service":"rss"})
In [112]:
xmltext = response.text
In [113]:
xmltext[:100]
Out[113]:
In [115]:
from xml.etree import ElementTree as et
In [116]:
root = et.fromstring(xmltext)
In [118]:
items = root.findall(".//item")
In [119]:
len(items)
Out[119]:
In [120]:
items[0]
Out[120]:
In [121]:
print(et.tostring(items[0]).decode())
In [122]:
for item in items[:10]:
print(item.findtext("title"))
print(item.findtext("link"))
print("-"*50)
In [123]:
from xml.dom.minidom import parseString
In [124]:
root = parseString(xmltext)
In [125]:
items = root.getElementsByTagName("item")
In [126]:
len(items)
Out[126]:
In [127]:
item = items[0]
In [130]:
title = item.getElementsByTagName("title")[0]
In [131]:
title.firstChild.data
Out[131]:
JSON¶
In [132]:
import json
In [136]:
j = json.loads('{"a":2,"l":["a","b","c"]}')
In [137]:
type(j)
Out[137]:
In [138]:
j['a']
Out[138]:
In [139]:
j['l']
Out[139]:
In [140]:
d = {"service":"rss", "x":[1,2,3,4,5]}
In [141]:
json.dumps(d)
Out[141]:
Find distance between two cities using google API
In [144]:
import requests
def distance(origin, dest):
url = "https://maps.googleapis.com/maps/api/distancematrix/json"
response = requests.get(url, params={"units":"metric",
"origins":origin,
"destinations":dest
})
data = response.json()
return data['rows'][0]['elements'][0]['distance']['text']
In [143]:
distance("hyderabad", "mumbai")
Out[143]:
In [145]:
distance("hyderabad", "mumbai")
Out[145]:
Numpy¶
In [146]:
import numpy as np
In [147]:
x = np.arange(32)
In [148]:
x
Out[148]:
In [149]:
x.reshape(4,8)
Out[149]:
In [150]:
y = np.arange(64).reshape(4,2,8)
In [151]:
y
Out[151]:
In [152]:
len(y[-1][-1])
Out[152]:
In [153]:
len(y[-1])
Out[153]:
In [154]:
len(y)
Out[154]:
In [155]:
y.shape
Out[155]:
In [156]:
l,w,h = y.shape
In [157]:
l,w,h
Out[157]:
In [158]:
y.dtype
Out[158]:
In [159]:
y.size
Out[159]:
In [160]:
y.itemsize
Out[160]:
problem: create a 2D array of size 5x6
other ways of creating arrays¶
In [161]:
np.random.random(50).reshape(5,10)
Out[161]:
In [162]:
np.linspace(1.0, 10, 15)
Out[162]:
In [163]:
np.zeros(20).reshape(4,5)
Out[163]:
vector operations¶
In [164]:
x = np.arange(10)
In [165]:
x + 10
Out[165]:
In [166]:
x * 2
Out[166]:
In [167]:
x + x
Out[167]:
In [168]:
x * x
Out[168]:
In [169]:
x ** 3
Out[169]:
Playing with images¶
In [170]:
from scipy import misc
In [171]:
face = misc.face(gray=True)
In [172]:
type(face)
Out[172]:
In [173]:
face.ndim
Out[173]:
In [174]:
face.shape
Out[174]:
In [175]:
face.dtype
Out[175]:
In [176]:
face[1][:10]
Out[176]:
In [177]:
#show images from matplotlib in the same HTML page
%matplotlib inline
In [178]:
import matplotlib.pyplot as plt
In [180]:
plt.imshow(face, cmap=plt.cm.gray)
Out[180]:
In [181]:
import matplotlib.pyplot as plt
def imshow(img):
plt.imshow(img, cmap=plt.cm.gray)
plt.show()
In [182]:
negface = 255 - face
In [183]:
imshow(negface)
In [184]:
face[-1][:10]
Out[184]:
In [185]:
negface[-1][:10]
Out[185]:
Transpose¶
In [186]:
x = np.arange(20).reshape(4,5)
In [187]:
x[1][2]
Out[187]:
In [188]:
x[1,2]
Out[188]:
In [189]:
x[1,:] # 1st row
Out[189]:
In [193]:
x[:,0] # 0th column
Out[193]:
In [194]:
x[:,:2] # first two columns
Out[194]:
In [195]:
x.transpose()
Out[195]:
In [198]:
x.transpose().shape
Out[198]:
In [199]:
x.shape
Out[199]:
In [200]:
facet = face.transpose()
In [201]:
imshow(facet)
In [202]:
face.mean()
Out[202]:
In [203]:
x = np.arange(10)
In [204]:
x < 5
Out[204]:
In [205]:
x[x<5]
Out[205]:
In [206]:
a = x < 5
In [207]:
a
Out[207]:
In [208]:
a.sum()
Out[208]:
problem: Convert the face image to black and white image (instead of gray scale)
In [209]:
facebw = face > 127
In [210]:
imshow(facebw)
In [211]:
x = np.arange(10000).reshape(100,100)
In [212]:
imshow(x)
In [213]:
x[:,50] = 9999
x[50,:] = 9999
In [215]:
x = np.zeros_like(face)
In [216]:
imshow(x)
In [217]:
x[::10,:] = 255
In [218]:
imshow(x)
In [219]:
x = np.zeros(10000).reshape(100,100)
In [220]:
x[::5,:] = 255
In [221]:
imshow(x)
In [222]:
x[:,::5] = 255
In [223]:
imshow(x)
In [224]:
mesh = np.zeros_like(face)
mesh[::50,:]= 255
mesh[:,::50]= 255
In [225]:
imshow(mesh)
In [226]:
imshow(0.5*face + 0.5*mesh)
In [227]:
x = list(range(10))
In [228]:
x
Out[228]:
In [229]:
x[2:]
Out[229]:
In [230]:
x[:3]
Out[230]:
In [231]:
x[::2]
Out[231]:
In [232]:
x[::3]
Out[232]:
In [233]:
x[::4]
Out[233]:
In [235]:
x = np.zeros(10000).reshape(100,100)
In [236]:
x[:,::10] = 255 #columns at interval of 10
In [237]:
x[::10,:] = 255 #rows at interval of 10
In [238]:
imshow(x)
In [239]:
face2 = face + mesh
imshow(face2)
In [240]:
imshow(face)
In [241]:
face2[:,50]
Out[241]:
In [242]:
face[:,50]
Out[242]:
In [243]:
x = face2 - face
In [244]:
imshow(x)
problem: Try to swap parts of image. split the image in four parts like
AB
CDit should become
AC
BDIn [245]:
face2 = face.copy()
In [246]:
h, w = face2.shape
In [247]:
TR = face2[:h//2,w//2:].copy()
BL = face2[h//2:,:w//2].copy()
In [248]:
imshow(face2)
imshow(TR)
imshow(BL)
In [249]:
face2[:h//2,w//2:] = BL
face2[h//2:,:w//2] = TR
imshow(face2)
In [250]:
f2 = np.rot90(face)
imshow(f2)
In [251]:
imshow(np.roll(face, 400))
In [252]:
imshow(np.flip(face, 1))
In [255]:
thumb = face[::4,::4]
In [256]:
imshow(thumb)
In [257]:
thumb.shape
Out[257]:
In [258]:
t = np.hstack([thumb, thumb, thumb, thumb])
v = np.vstack([t,t,t,t])
imshow(v)
Matplotlib¶
In [259]:
import numpy as np
import matplotlib.pyplot as plt
In [260]:
X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
In [261]:
X.shape
Out[261]:
In [262]:
C = np.cos(X)
In [263]:
S = np.sin(X)
In [264]:
plt.plot(X,C, label="cos(x)")
plt.plot(X,S, label="sin(x)")
plt.legend()
plt.show()
In [266]:
T = np.tan(X)
T
Out[266]:
In [267]:
plt.plot(X,T, label="tan(x)")
plt.legend()
plt.show()
In [268]:
n = 1024
X = np.random.normal(0,1, n)
Y = np.random.normal(0,1, n)
plt.scatter(X,Y)
plt.show()
Example: temperature and rainfall data¶
Download data from http://notes.pipal.in/2017/arcesium-oct-advpython/HYDERABAD-weather.csv
In [269]:
import csv
In [270]:
data = list(csv.reader(open("HYDERABAD-weather.csv")))
In [271]:
data[:3]
Out[271]:
In [272]:
data = data[1:] # skip header
In [273]:
tmax = [float(row[4]) for row in data]
tmin = [float(row[5]) for row in data]
In [274]:
plt.scatter(tmin, tmax)
Out[274]:
In [275]:
rainfall = [float(row[6]) for row in data]
In [276]:
def safefloat(value):
try:
return float(value)
except ValueError:
print("bad value: %r"% value)
return 0.0
rainfall = [safefloat(row[6]) for row in data]
In [277]:
plt.scatter(tmax, rainfall)
Out[277]:
In [278]:
n = 10
X = np.arange(n)
Y = np.random.normal(0,100,n)
plt.bar(X,Y)
Out[278]:
problem: Using above dataset, plot a bar chart of average rainfall per month.
In [279]:
data[:2]
Out[279]:
In [280]:
data[:10]
Out[280]:
In [281]:
months = np.array([row[2] for row in data])
In [283]:
rainfall = np.array([safefloat(row[-1]) for row in data])
In [284]:
rainfall[months == "January"].mean()
Out[284]:
In [285]:
import datetime
list_of_months = [datetime.date(2000, i+1, 1).strftime("%B") for i in range(12)]
In [286]:
list_of_months
Out[286]:
In [287]:
def get_mean_rainfall(month):
return rainfall[months == month].mean()
In [288]:
mean_rainfall = [get_mean_rainfall(m) for m in list_of_months]
In [289]:
mean_rainfall
Out[289]:
In [290]:
plt.bar(range(12), mean_rainfall)
Out[290]:
In [292]:
x = np.arange(10)
In [293]:
x > 5
Out[293]:
In [294]:
x = np.arange(3)
In [295]:
x
Out[295]:
In [297]:
x[np.array([True, False, True, True])]
Pandas¶
In [299]:
import pandas as pd
import numpy as np
%matplotlib inline
In [300]:
x = pd.Series(range(10))
In [301]:
x
Out[301]:
In [302]:
s = pd.Series(np.random.randn(5), index=['a','b','c','d','e'])
In [303]:
s
Out[303]:
In [304]:
d = {'a':0, 'b':1, 'c':2}
s = pd.Series(d)
In [305]:
s
Out[305]:
In [306]:
pd.Series(d, index=['b','c','d','a'])
Out[306]:
In [307]:
s = pd.Series(np.random.randn(5), index=['a','b','c','d','e'])
In [308]:
s[0]
Out[308]:
In [309]:
s[:3]
Out[309]:
In [310]:
s['a']
Out[310]:
In [311]:
s[1:4:2]
Out[311]:
In [312]:
s[s > s.median()]
Out[312]:
In [313]:
s.mean()
Out[313]:
In [314]:
np.exp(s)
Out[314]:
In [315]:
'e' in s
Out[315]:
In [316]:
s['a']
Out[316]:
In [317]:
'z' in s
Out[317]:
In [318]:
s + s
Out[318]:
In [319]:
s * s
Out[319]:
In [320]:
data = [["A",1], ["B", 2], ["c",3], ["D",4]]
In [321]:
pd.DataFrame(data)
Out[321]:
In [322]:
d = {"one":[1. , 2. , 3., 4.],
"two":[4. ,3., 2. , 1.]
}
df = pd.DataFrame(d, index=['a','b','c','d'])
In [323]:
df
Out[323]:
In [324]:
df['one']
Out[324]:
In [325]:
df['one']['a']
Out[325]:
In [326]:
df.columns
Out[326]:
In [327]:
df.columns = ["column1", "column2"]
In [328]:
df
Out[328]:
In [329]:
df2 = df.set_index("column2")
In [330]:
df2
Out[330]:
In [331]:
df2['column1'][4.0]
Out[331]:
In [332]:
df.to_csv("df.csv")
In [334]:
!cat df.csv
Playing with weather dataset¶
In [335]:
df = pd.read_csv("HYDERABAD-weather.csv", index_col=0)
In [336]:
df
Out[336]:
In [337]:
df.head()
Out[337]:
In [338]:
df.tail()
Out[338]:
In [339]:
df.plot("maxtemp", "mintemp", kind="scatter")
Out[339]:
In [340]:
mean = df.groupby("year").mean()
In [341]:
mean
Out[341]:
In [342]:
mean.plot()
Out[342]:
In [343]:
bymonth = df.groupby("month").mean()
In [344]:
bymonth
Out[344]:
In [345]:
del bymonth['year']
In [346]:
bymonth
Out[346]:
In [347]:
bymonth.index
Out[347]:
In [348]:
newindex = [list_of_months.index(month) for month in bymonth.index]
In [349]:
newindex
Out[349]:
In [350]:
bymonth['m'] = newindex
In [351]:
bymonth
Out[351]:
In [352]:
bymonth2 = bymonth.set_index("m")
In [353]:
bymonth2
Out[353]:
In [354]:
bymonth
Out[354]:
In [355]:
bymonth2.sort_index().plot()
Out[355]:
In [ ]: