Advanced Python Training at Arcesium - Day 2¶
Sep 25-27, 2019 Vikrant Patil
These notes are available online at http://notes.pipal.in/2020/arcesium_advanced_feb/day2.html
© Pipal Academy LLP
We will be using python 3.7 from anaconda for this training. You can download it from
Understanding iterations¶
In [1]:
for n in [2, 3, 4, 5, 6, 7]:
print(n)
In [2]:
for c in "This is a string to test for loop":
print(c, end=",")
In [3]:
for item in {"a":True, "b":False}:
print(item)
The iteration protocol¶
In [6]:
items = [1, 2, 3, 4, 5]
In [7]:
itr_items = iter(items)
In [8]:
itr_items
Out[8]:
In [9]:
next(itr_items)
Out[9]:
In [10]:
next(itr_items)
Out[10]:
In [11]:
next(itr_items)
Out[11]:
In [12]:
next(itr_items)
Out[12]:
In [13]:
next(itr_items)
Out[13]:
In [14]:
next(itr_items)
generators¶
In [15]:
def squares(numbers):
for n in numbers:
yield n*n
In [16]:
squares
Out[16]:
In [17]:
s =squares([2,3,4])
In [18]:
s
Out[18]:
In [19]:
for i in s:
print(i)
In [20]:
def squares(numbers):
print("Begin squares")
for n in numbers:
print("Computing square of ", n)
yield n*n
print("Back to squares")
print("Finished squares")
In [21]:
sqrs = squares([4,5,6])
In [22]:
sqrs
Out[22]:
In [23]:
next(sqrs)
Out[23]:
In [24]:
next(sqrs)
Out[24]:
In [25]:
next(sqrs)
Out[25]:
In [26]:
next(sqrs)
problems
- Write a generator
coutdownwhich works exactly opposite of range! - Can we write a term generator for y'days piseries? How can we use this generator to sum pi?
- is it possible to write infinite sequence generator? write infinite fib series generator.
- How will we work with inifinite sequences? can you write a function called
takewhich takes only n items from given sequence.>>> ones = infiniteones() >>> take(ones, 5) [1, 1, 1, 1, 1]
In [27]:
def hold():
print("Enter ....")
yield 1
print("After 1")
print("Going 2")
yield 2
print("After 2")
print("Going 3")
yield 3
print("After 3")
print("Stopping....")
In [28]:
h = hold()
In [29]:
next(h)
Out[29]:
In [30]:
next(h)
Out[30]:
In [31]:
next(h)
Out[31]:
In [32]:
next(h)
In [33]:
def foo(x):
if x:
return 1
else:
yield 0
In [39]:
f = foo(True)
In [40]:
f
Out[40]:
In [41]:
next(f)
In [42]:
f = foo(True)
In [43]:
x = next(f)
In [44]:
print(x)
In [45]:
def loop():
n = 0
while True:
yield n
if n == 3:
return
n += 1
In [46]:
l = loop()
In [47]:
next(l)
Out[47]:
In [48]:
next(l)
Out[48]:
In [49]:
next(l)
Out[49]:
In [50]:
next(l)
Out[50]:
In [51]:
next(l)
In [52]:
help(max)
In [53]:
def countdown(n):
while n > 0:
yield n
n -= 1
In [54]:
for i in countdown(4):
print(i, end=",")
In [55]:
def piseries():
n = 1
while True:
yield 8/((4*n-3)*(4*n-1))
n += 1
In [56]:
def take(seq, n):
return [next(seq) for _ in range(n)]
In [57]:
take(piseries(), 5)
Out[57]:
In [58]:
sum(take(piseries(), 1000))
Out[58]:
In [59]:
def fibseries():
cur, next_ = 1, 1
while True:
yield cur
cur , next_ = next_, cur+next_
In [60]:
take(fibseries(), 10)
Out[60]:
In [61]:
cntd3 = countdown(3)
In [63]:
for i in cntd3:
print(i, end=",")
In [64]:
for i in cntd3:
print(i, end=",")
In [65]:
c3 = countdown(3)
In [66]:
copyc3 = c3
In [67]:
for i in c3:
print(i, end=",")
In [68]:
for i in copyc3:
print(i, end=",")
Building data pipeline using generators¶
In [73]:
import os
def take(seq, n):
return [next(seq) for _ in range(n)]
def find(root):
for path, dirnames, filenames in os.walk(root):
for f in filenames:
yield os.path.join(path, f)
def grep(pattern, seq):
return (x for x in seq if pattern in x) # this called generator expression
In [74]:
files = find("/home/vikrant/trainings")
pyfiles = grep(".py", files)
print(take(pyfiles, 5))
In [78]:
def readlines(filenames):
for file in filenames:
with open(file) as f:
yield from f
def count(seq):
return sum(1 for item in seq) # this is also generator expression
In [81]:
files = find("/home/vikrant/trainings/")
csvfiles = grep(".csv", files)
lines = readlines(csvfiles)
count(lines)
Out[81]:
In [82]:
import re
def grep(pattern, seq):
p = re.compile(pattern)
return (x for x in seq if p.match(x)) # this called generator expression
In [88]:
files = find("/home/vikrant/trainings/")
pyfiles = grep(r"[\w\/]+\.py", files)
take(pyfiles, 5)
Out[88]:
In [85]:
import re
In [87]:
pattern = re.compile(r"\w+.py")
pattern.match("/vikrant/trainings/hello.py")
In [89]:
files = find("/home/vikrant/trainings/")
pyfiles = grep(r"[\w\/]+\.py", files)
lines = readlines(pyfiles)
funcs = grep(r"def .*", lines)
count(funcs)
Out[89]:
In [90]:
files = find("/home/vikrant/trainings/")
pyfiles = grep(r"[\w\/]+\.py", files)
lines = readlines(pyfiles)
funcs = grep(r"def .*", lines)
In [91]:
next(funcs)
Out[91]:
In [93]:
count(funcs)
Out[93]:
problem
https://ia802902.us.archive.org/4/items/prideandprejudic01342gut/pandp12.txt
- Write a function
get_paragraphsto split text in above text text file into paragraphs. When an empty line comes, thats end of paragraph.- How many paragraphs are there?
- Which is longest paragraph?
In [94]:
import requests
def wget(url, filename):
resp = requests.get(url)
with open(filename, "w") as f:
f.write(resp.text)
In [95]:
novelurl = "https://ia802902.us.archive.org/4/items/prideandprejudic01342gut/pandp12.txt"
wget(novelurl, "pandp.txt")
In [96]:
!tail pandp.txt
In [97]:
!head pandp.txt
In [119]:
def get_paragraphs(lines):
para = []
for line in lines:
if line.strip() !="":
para.append(line.strip())
elif para:
yield "\n".join(para)
para = []
if para:
yield "\n".join(para)
def get_paragraphs_(lines):
para = ""
for line in lines:
line = line.strip()
if line=="":
if para=="":
continue
else:
yield para
para = ""
else:
para = para + "\n" + line
if para:
yield para
In [120]:
lines = readlines(["pandp.txt"])
count(get_paragraphs(lines))
Out[120]:
In [121]:
!wc pandp.txt
In [123]:
def test_get_paragraphs(func):
def append_newl(items):
return [item+"\n" for item in items]
lines = [""]
assert count(func(append_newl(lines))) == 0
lines = ["A","B","","","C"]
assert count(func(append_newl(lines))) == 2
lines = ["A","","B","","","C","D"]
assert count(func(append_newl(lines))) == 3
assert max(func(append_newl(lines)), key=len)=="C\nD"
#test_get_paragraphs(get_paragraphs_)
test_get_paragraphs(get_paragraphs)
numpy¶
!pyhton -m pip install numpy
In [124]:
import numpy as np
In [125]:
a = np.array([1,2,3,4,5])
In [126]:
a
Out[126]:
In [127]:
a.shape
Out[127]:
In [128]:
a.ndim
Out[128]:
In [129]:
a100 = np.arange(100).reshape(10,10)
In [130]:
a100
Out[130]:
In [131]:
a100.shape
Out[131]:
In [132]:
a100.ndim
Out[132]:
In [133]:
a100.dtype
Out[133]:
In [134]:
a100[0]
Out[134]:
In [135]:
a100[-1]
Out[135]:
In [137]:
a100[:,0]
Out[137]:
In [138]:
a100[1,:]
Out[138]:
In [139]:
a100
Out[139]:
various ways to create arrays¶
In [140]:
np.zeros(100).reshape(20,5)
Out[140]:
In [141]:
z = _
In [143]:
z.dtype
Out[143]:
In [145]:
help(np.zeros)
In [146]:
np.zeros(10, dtype=np.int16)
Out[146]:
In [147]:
np.zeros_like(a100)
Out[147]:
In [148]:
np.ones_like(a100)
Out[148]:
In [149]:
np.asarray([1, 2, 3, 4, 5])
Out[149]:
In [150]:
np.empty(100).reshape(25,4)
Out[150]:
In [152]:
np.empty_like(range(10))
Out[152]:
Access patterns¶
In [153]:
a100 = np.arange(100).reshape(10,10)
In [154]:
a100[:5, :5]
Out[154]:
In [155]:
a100[5:,5:]
Out[155]:
In [156]:
a100[:5,5:]
Out[156]:
In [157]:
subview = a100[:5, :5]
In [158]:
subview
Out[158]:
In [159]:
subview.shape
Out[159]:
In [160]:
subview[0,0]= -1
In [161]:
subview
Out[161]:
In [162]:
type(subview)
Out[162]:
In [163]:
a100
Out[163]:
In [164]:
type(a100)
Out[164]:
In [165]:
copy_subview = subview.copy()
In [166]:
copy_subview
Out[166]:
In [167]:
copy_subview[0,0] = 0
In [168]:
copy_subview
Out[168]:
In [169]:
subview
Out[169]:
In [170]:
a100
Out[170]:
operations¶
In [171]:
a = np.array(range(10))
In [172]:
a
Out[172]:
In [173]:
a > 4
Out[173]:
In [174]:
a[a>3]
Out[174]:
In [175]:
a + 4
Out[175]:
In [176]:
a - 5
Out[176]:
In [177]:
a * 2
Out[177]:
In [178]:
a **2
Out[178]:
In [180]:
a + a
Out[180]:
In [181]:
a*a
Out[181]:
In [182]:
np.exp(a)
Out[182]:
In [183]:
a100
Out[183]:
In [184]:
a100.max()
Out[184]:
In [185]:
a100.min()
Out[185]:
In [186]:
a100.std()
Out[186]:
In [187]:
a100.sum()
Out[187]:
In [188]:
a100.cumsum()
Out[188]:
In [189]:
from scipy.misc import face
In [190]:
image = face(gray=True)
In [191]:
image
Out[191]:
In [192]:
from matplotlib import pyplot as plt
In [193]:
def imshow(img):
plt.imshow(img, cmap=plt.cm.gray)
plt.show()
In [194]:
%matplotlib inline
In [195]:
imshow(a100)
In [196]:
imshow(image)
In [197]:
image
Out[197]:
In [198]:
negate = 255 - image
In [199]:
imshow(negate)
In [200]:
thumbnail = image[::3,::3]
In [201]:
imshow(thumbnail)
In [202]:
imshow(image[::5,::5])
In [203]:
imshow(image[::20,::20])
In [204]:
plain = np.zeros_like(thumbnail)
In [205]:
imshow(plain)
In [206]:
plain[::10,:] = 255
plain[:, ::10] = 255
In [207]:
imshow(plain)
In [209]:
plain[:12,:12]
Out[209]:
In [210]:
small = np.zeros(100).reshape(10,10)
In [211]:
small[::3,:] = 255
small[:,::3] = 255
imshow(small)
In [212]:
imshow(thumbnail + plain)
In [220]:
imshow(0.75*thumbnail + 0.25*plain)
In [219]:
imshow((0.5*thumbnail + 0.5*plain)*2)
In [217]:
imshow(a100+200)
In [221]:
imshow(np.maximum(thumbnail, plain))
problem
- SWap topleft and bottm right corner of rhis image
In [224]:
def swapcorners(img):
imglike = img.copy()
h, w = img.shape
q1 = img[:h//2, :w//2].copy()
q2 = img[h//2:,w//2:].copy()
imglike[:h//2, :w//2] = q2
imglike[h//2:, w//2:] = q1
return imglike
In [225]:
imshow(swapcorners(thumbnail))
In [226]:
thumb = image[::10, ::10]
In [227]:
hthumb = np.hstack([thumb, thumb, thumb])
vthump = np.vstack([hthumb, hthumb, hthumb])
imshow(vthump)
In [228]:
imshow(np.flip(thumb))
In [229]:
url = "https://notes.pipal.in/2020/arcesium_advanced_feb/HYDERABAD-weather.csv"
wget(url, "HYDERABAD-weather.csv")
In [230]:
!tail -n 5 HYDERABAD-weather.csv
In [231]:
import csv
In [232]:
with open("HYDERABAD-weather.csv") as f:
data = list(csv.reader(f))
In [233]:
type(data)
Out[233]:
In [234]:
data[0]
Out[234]:
In [235]:
numeric_data = data[1:]
In [236]:
data[0]
Out[236]:
In [237]:
numeric_data[:3]
Out[237]:
In [238]:
def floatcolumn(matrix, colnum):
return [float(row[colnum]) for row in matrix]
In [239]:
maxtemp = floatcolumn(numeric_data, 4)
In [240]:
mintemp = floatcolumn(numeric_data, 5)
In [241]:
rainfall = floatcolumn(numeric_data, 6)
In [242]:
def parsefloat(sf):
try:
return float(sf)
except ValueError as v:
print(v)
return 0
def floatcolumn(matrix, colnum):
return [parsefloat(row[colnum]) for row in matrix]
In [243]:
rainfall = floatcolumn(numeric_data, 6)
In [244]:
plt.scatter(rainfall, maxtemp)
Out[244]:
In [245]:
plt.scatter(rainfall, mintemp)
Out[245]:
In [248]:
year = [int(row[3]) for row in numeric_data]
In [249]:
numeric_data[:6]
Out[249]:
In [250]:
numeric_data[-6:]
Out[250]:
In [252]:
plt.plot(year, rainfall)
Out[252]:
In [253]:
sorted_data = sorted(numeric_data, key= lambda r:r[3])
In [254]:
year = [int(row[3]) for row in sorted_data]
In [255]:
rainfall = floatcolumn(sorted_data, 6)
In [256]:
plt.plot(year, rainfall)
Out[256]:
In [258]:
a100.mean()
Out[258]:
In [272]:
months = np.array([row[2] for row in numeric_data])
In [273]:
months
Out[273]:
In [274]:
rainfall = np.array(floatcolumn(numeric_data, 6))
In [275]:
rainfall[:5]
Out[275]:
In [276]:
a = np.array(range(5))
In [277]:
b = np.array(['a','b','a','b','c'])
In [278]:
b=="a"
Out[278]:
In [279]:
a[b=="a"]
Out[279]:
In [280]:
rainfall[months=="March"].mean()
Out[280]:
In [271]:
months=="March"
Out[271]:
In [281]:
def get_mean_rainfall(rainfall, months, month):
return rainfall[months==month].mean()
In [282]:
import datetime
In [283]:
set(months)
Out[283]:
In [285]:
uniqmonths = list(set(months))
rainfall_ = [get_mean_rainfall(rainfall, months, month) for month in uniqmonths]
In [286]:
plt.bar(uniqmonths, rainfall_)
Out[286]:
In [287]:
import altair as alt
import pandas as pd
In [288]:
!python -m pip install altair
In [289]:
import altair as alt
import pandas as pd
In [290]:
%%file sales.txt
area,sales,profit
North,5,2
East,25,8
West,15,6
South,20,5
Central,10,3
In [291]:
sales = pd.read_csv("sales.txt")
In [292]:
sales
Out[292]:
In [293]:
alt.Chart(sales).mark_point()
Out[293]:
In [295]:
alt.Chart(sales).mark_point().encode(y="area")
Out[295]:
In [296]:
alt.Chart(sales).mark_point().encode(
x="sales",
y="area")
Out[296]:
In [297]:
alt.Chart(sales).mark_bar().encode(
x="sales",
y="area")
Out[297]:
In [298]:
alt.Chart(sales).mark_line().encode(
x="sales",
y="area")
Out[298]:
In [299]:
base = alt.Chart(sales).mark_bar().encode(
x="sales",
y="area")
In [300]:
base.mark_circle()
Out[300]:
In [301]:
base.encode(color="area")
Out[301]:
In [302]:
base.encode(color="area", size="profit")
Out[302]:
In [303]:
base.encode(color="area", size="profit").mark_circle()
Out[303]:
In [305]:
base.to_json()
Out[305]:
JSON¶
In [306]:
import json
In [307]:
d = {"a":28.565,
"b": 30,
"c" : [1,2,3]}
In [308]:
d
Out[308]:
In [309]:
json.dumps(d)
Out[309]:
In [310]:
jsondata = '{"a": 28.565, "b": 30, "c": [1, 2, 3]}'
In [311]:
json.loads(jsondata)
Out[311]:
In [312]:
url = "https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=MSFT&interval=5min&outputsize=full&apikey=demo"
In [313]:
resp = requests.get(url)
data = resp.json()
In [314]:
type(data)
Out[314]:
In [315]:
data.keys()
Out[315]:
In [316]:
data['Meta Data']
Out[316]:
In [317]:
pd.DataFrame(data['Time Series (5min)'])
Out[317]:
In [318]:
pd.DataFrame(data['Time Series (5min)']).transpose()
Out[318]:
XML¶
In [319]:
url = "http://www.thehindu.com"
In [320]:
resp = requests.get(url, params={"service":"rss"})
In [321]:
xmltext = resp.text
In [323]:
print(xmltext[:1300])
In [324]:
from xml.etree import ElementTree as et
In [325]:
root = et.fromstring(xmltext)
In [326]:
items = root.findall(".//item")
In [327]:
type(items)
Out[327]:
In [328]:
len(items)
Out[328]:
In [329]:
items[0]
Out[329]:
In [331]:
print(et.tostring(items[0]).decode())
In [332]:
for item in items[:10]:
print(item.findtext("title"))
print(item.findtext("link"))
print("*"*30)
In [333]:
from xml.dom.minidom import parseString
In [334]:
root = parseString(xmltext)
items = root.getElementsByTagName("item")
In [336]:
for item in items[:10]:
title = item.getElementsByTagName("title")[0]
link = item.getElementsByTagName("link")[0]
print(title.firstChild.data)
print(link.firstChild.data)
print("*"*30)
Database¶
In [337]:
import sqlite3
In [338]:
conn = sqlite3.connect("data.db")
In [339]:
cur = conn.cursor()
In [340]:
cur.execute("create table person (name varchar(100), email varchar(100))")
Out[340]:
In [341]:
cur.execute("insert into person (name, email) values('alice', 'alice@wonder.land')")
Out[341]:
In [342]:
cur = cur.execute("select * from person")
In [343]:
cur.fetchall()
Out[343]:
In [352]:
def find(conn , email):
q = "select * from person where email='{}'".format(email)
print(q)
cur = conn.cursor()
return cur.execute(q).fetchall()
In [353]:
find(conn, "alice@wonder.land")
Out[353]:
In [350]:
def find_(conn, email):
q = "select * from person where email=?"
cur = conn.cursor()
return cur.execute(q, (email,)).fetchall()
In [351]:
find_(conn, "alice@wonder.land")
Out[351]:
In [354]:
conn.commit()
In [355]:
conn.close()
In [356]:
conn = sqlite3.connect("data.db")
In [357]:
find(conn, "*")
Out[357]:
In [358]:
find_(conn, "alice@wonder.land")
Out[358]:
In [359]:
records = [
("alex", "alex@zoo.in"),
("Elsa", "elsa@frozen.mov"),
("ELisa", "elisa@hacker.hack")
]
In [360]:
cur = conn.cursor()
cur.executemany("insert into person values(?,?)", records)
Out[360]:
In [361]:
cur.execute("select * from person").fetchall()
Out[361]:
To manage database tables as classes, one should use ORM (Object Relational Mapping) which can be done using library sqlalchemy. More details can ne seen at library homepage
In [ ]: