Advanced Python Training at Arcesium - Day 2¶
Sep 25-27, 2019 Vikrant Patil
These notes are available online at http://notes.pipal.in/2019/arcesium_advanced_sep/day2.html
© Pipal Academy LLP
We will be using python 3 (>= 3.0) from anaconda for this training. You can download it from
Understanding Iterations¶
In [1]:
for i in [1, 2, 3, 4]:
print(i)
In [2]:
for i in range(3):
print(i)
In [4]:
for key in {"one":1, "two":2}:
print(key)
The iteration protocol¶
In [5]:
items = [1, 2, 3, 4, 5]
In [6]:
itr_items = iter(items)
In [7]:
itr_items
Out[7]:
In [8]:
next(itr_items)
Out[8]:
In [9]:
next(itr_items)
Out[9]:
In [10]:
next(itr_items)
Out[10]:
In [11]:
next(itr_items)
Out[11]:
In [12]:
next(itr_items)
Out[12]:
In [13]:
next(itr_items)
generators¶
In [14]:
def squares(numbers):
for n in numbers:
yield n*n
In [15]:
squares
Out[15]:
In [17]:
sqr = squares(range(5))
In [18]:
sqr
Out[18]:
In [19]:
for s in sqr:
print(s)
In [20]:
for s in sqr:
print(s)
In [21]:
def squares(numbers):
print("Begin squares")
for n in numbers:
print("Computing square of", n)
yield n*n
print("Back to squares")
print("Finished squares")
In [22]:
sq4 = squares(range(1, 5))
In [23]:
sq4
Out[23]:
In [24]:
next(sq4)
Out[24]:
In [25]:
next(sq4)
Out[25]:
In [26]:
next(sq4)
Out[26]:
In [27]:
next(sq4)
Out[27]:
In [28]:
next(sq4)
problems
- Write a generator which can count number from n to 1.
>>> for i in countdown(4): print(i, end=",") 4,3,2,1 - Is it possible to know lenghth of sequence generated through generator object?
- Can we write a generator
oneswhich can generate infinite sequence of ones? - How do we work with infinite sequences?
- Write a term generator for y'days piseries. how can we use this generator to sum?
In [51]:
def foo(n):
i = 1
while True:
print("Before yield")
yield i
print("after yield")
if i ==4:
return i+1
i += 1
In [34]:
for i in foo(50):
print(i, "="*5)
In [35]:
def ones():
while True:
yield 1
In [36]:
def take(seq, n):
return [next(seq) for i in range(n)]
In [37]:
infinite = ones()
In [38]:
take(infinite, 10)
Out[38]:
In [39]:
In [52]:
f = foo(4)
In [53]:
x = next(f)
In [54]:
x
Out[54]:
In [55]:
x=next(f)
In [56]:
x
Out[56]:
In [57]:
x = next(f)
In [58]:
x
Out[58]:
In [59]:
x = next(f)
In [60]:
x
Out[60]:
In [61]:
x = next(f)
In [62]:
x
Out[62]:
In [64]:
f = foo(3)
In [65]:
next(f)
Out[65]:
In [66]:
next(f)
Out[66]:
In [67]:
next(f)
Out[67]:
In [68]:
f()
In [70]:
try:
next(f)
except StopIteration as s:
In [73]:
def piseries():
n = 1
while True:
yield 8/((4*n-3)*(4*n-1))
n += 1
def pi(n):
series = piseries()
return sum(take(series, n))
In [75]:
pi(10000)
Out[75]:
In [76]:
len(piseries())
In [78]:
take(piseries(), 10)
Out[78]:
Building data pipeline¶
In [79]:
import os
In [80]:
def find(root):
for path, direnames, filenames in os.walk(root):
for f in filenames:
yield os.path.join(path, f)
In [81]:
def grep(pattern , seq):
return (x for x in seq if pattern in x)
In [94]:
files = find("/home/vikrant/trainings")
pyfiles = grep(".py", files)
print(take(pyfiles, 5))
In [88]:
x = (i*i for i in range(5))
In [89]:
x
Out[89]:
In [90]:
next(x)
Out[90]:
In [91]:
next(x)
Out[91]:
In [92]:
for i in x:
print(i)
In [93]:
def count(seq):
return sum(1 for i in seq)
def count(seq):
i = 0
for x in seq:
i += 1
return i
def count(seq):
return sum((1 for i in seq))
In [97]:
files = find("/home/vikrant/trainings")
pyfiles = grep(".py", files)
print(count(pyfiles))
In [129]:
def readlines(filenames):
for file in filenames:
with open(file) as f:
for line in f:
yield line
In [130]:
f = open("day1.html")
In [131]:
f. close()
In [132]:
files = find(".")
pyfiles = grep(".", files)
lines = readlines(pyfiles)
funcs = grep("def", lines)
count(funcs)
In [133]:
def antigrep(pattern, seq):
return (x for x in seq if pattern not in x)
In [134]:
files = find("/home/vikrant/trainings/")
pyfiles = grep(".py", files)
pyfiles = antigrep(".pyc", pyfiles)
pyfiles = antigrep("~", pyfiles)
#print(take(pyfiles, 50))
lines = readlines(pyfiles)
funcs = grep("def", lines)
count(funcs)
Out[134]:
problem
Write a function get_paragraphs to split given text into paragraphs. The function should take a sequence of lines and should return a sequence of paragraphs. Whenever you encounter empty line, it is end of previous paragraph.
- How many paragraphs are there?
- longest paragraph
https://ia802902.us.archive.org/4/items/prideandprejudic01342gut/pandp12.txt
In [135]:
import requests
def wget(url, filename):
resp = requests.get(url)
with open(filename, "w") as f:
f.write(resp.text)
In [137]:
url="https://ia802902.us.archive.org/4/items/prideandprejudic01342gut/pandp12.txt"
wget(url, "pandp.txt")
In [138]:
!tail pandp.txt
In [139]:
def get_paragraphs(lines):
paragraph = []
for line in lines:
if line.strip() !="":
paragraph.append(line.strip())
elif paragraph:
yield "\n".join(paragraph)
paragraph = []
if paragraph:
yield "\n".join(paragraph)
In [147]:
lines = readlines(["pandp.txt"])
paras = get_paragraphs(lines)
count(paras)
Out[147]:
In [148]:
lines = readlines(["pandp.txt"])
paras = get_paragraphs(lines)
max(paras, key=len)
Out[148]:
In [145]:
Out[145]:
In [149]:
!wc pandp.txt
Numpy¶
In [150]:
import numpy as np
In [151]:
a = np.array([1, 2, 3, 4, 5, 6, 7, 8])
In [152]:
a
Out[152]:
In [153]:
a.shape
Out[153]:
In [154]:
a.ndim
Out[154]:
In [155]:
a100 = np.arange(100).reshape(10, 10)
In [156]:
a100
Out[156]:
In [157]:
a100.shape
Out[157]:
In [158]:
a100.ndim
Out[158]:
In [159]:
a100[0]
Out[159]:
In [160]:
a100[0][0]
Out[160]:
In [161]:
a100[:,0]
Out[161]:
In [162]:
a100[0,:]
Out[162]:
In [164]:
a100[1,:]
Out[164]:
In [165]:
np.zeros(100).reshape(5,20)
Out[165]:
In [166]:
np.zeros_like(a100)
Out[166]:
In [167]:
np.ones_like(a100)
Out[167]:
In [168]:
np.asarray([1,2,3,4,5,6,7])
Out[168]:
In [169]:
np.asarray(np.zeros(20).reshape(5,4))
Out[169]:
In [172]:
empty = np.empty(1000).reshape(10,10,10)
In [173]:
empty.ndim
Out[173]:
In [174]:
empty.shape
Out[174]:
In [175]:
np.empty_like(a100)
Out[175]:
In [176]:
a100[:5, :5]
Out[176]:
In [177]:
a100[5:,5:]
Out[177]:
In [178]:
subview = a100[:5,:5]
In [179]:
subview
Out[179]:
In [180]:
subview[0,0] = -1
In [181]:
a100
Out[181]:
In [182]:
scopy = subview.copy()
In [183]:
scopy[0,0]=0
In [184]:
subview
Out[184]:
In [185]:
scopy
Out[185]:
In [186]:
d = np.array(range(10))
In [187]:
d
Out[187]:
In [188]:
d > 3
Out[188]:
In [189]:
d[d>3]
Out[189]:
In [190]:
d - 2
Out[190]:
In [191]:
d2 = d*2
In [192]:
d2
Out[192]:
In [193]:
d + d2
Out[193]:
In [194]:
d * d2
Out[194]:
In [195]:
np.exp(d)
Out[195]:
In [196]:
a100.max()
Out[196]:
In [197]:
a100.std()
Out[197]:
In [198]:
a100.cumsum()
Out[198]:
In [199]:
help(np.arange)
In [200]:
from scipy.misc import face
In [201]:
image = face(gray=True)
In [202]:
image
Out[202]:
In [203]:
from matplotlib import pyplot as plt
In [204]:
%matplotlib inline
In [205]:
def imshow(img):
plt.imshow(img, cmap=plt.cm.gray)
plt.show()
In [206]:
imshow(image)
In [207]:
negate = 255 - image
In [208]:
imshow(negate)
In [209]:
thumb = image[::3, ::3]
In [210]:
thumb.shape
Out[210]:
In [211]:
image.shape
Out[211]:
In [212]:
imshow(thumb)
In [213]:
plain = np.zeros_like(thumb)
In [214]:
imshow(plain)
In [215]:
plain[::10, :] = 255
plain[:,::10] = 255
In [216]:
imshow(plain)
In [217]:
plain
Out[217]:
In [218]:
plain[:21,:21]
Out[218]:
In [219]:
p = np.zeros(100).reshape(10,10)
In [220]:
p[::3, :] = 255
p[:,::3] = 255
imshow(p)
In [221]:
imshow(thumb*0.5 + plain*0.5)
In [222]:
imshow(thumb)
In [227]:
def swapcorners(img):
imglike = img.copy()
h, w = img.shape
q1 = img[:h//2, :w//2].copy()
q4 = img[h//2:, w//2:].copy()
imglike[:h//2, :w//2] = q4
imglike[h//2:, w//2:] = q1
return imglike
In [228]:
imshow(swapcorners(thumb))
In [229]:
5/3
Out[229]:
In [230]:
5//3
Out[230]:
In [231]:
thumb = image[::10, ::10]
In [232]:
hthumb = np.hstack([thumb, thumb, thumb])
vthumb = np.vstack([hthumb, hthumb, hthumb])
imshow(vthumb)
In [233]:
imshow(np.flip(thumb))
In [234]:
np.roll?
In [236]:
imshow(np.roll(image, 300, 0))
Matplotlib¶
Download data from http://notes.pipal.in/2019/arcesium_advanced_sep/HYDERABAD-weather.csv
In [239]:
url = "http://notes.pipal.in/2019/arcesium_advanced_sep/HYDERABAD-weather.csv"
wget(url, "HYDERABAD-weather.csv")
In [240]:
!tail HYDERABAD-weather.csv
In [241]:
import csv
In [245]:
with open("HYDERABAD-weather.csv") as f:
data = list(csv.reader(f))
In [246]:
type(data)
Out[246]:
In [247]:
data[0]
Out[247]:
In [248]:
d = data[1:]
In [250]:
d[:3]
Out[250]:
In [251]:
def floatcolumn(data, n):
return [float(row[n]) for row in data]
In [252]:
maxtemp = floatcolumn(d, 4)
In [253]:
mintemp = floatcolumn(d, 5)
In [254]:
rainfall = floatcolumn(d, 6)
In [255]:
def float_(sf):
try:
return float(sf)
except Exception as e:
print(e)
return 0
def floatcolumn(data, n):
return [float_(row[n]) for row in data]
In [256]:
rainfall = floatcolumn(d, 6)
In [257]:
plt.scatter(rainfall, maxtemp)
Out[257]:
In [258]:
plt.scatter(rainfall, mintemp)
Out[258]:
In [259]:
data[0]
Out[259]:
In [260]:
d[:2]
Out[260]:
In [261]:
year = [int(row[3]) for row in d ]
In [262]:
d[:6]
Out[262]:
In [264]:
d[-6:]
Out[264]:
In [266]:
len(set(year))
Out[266]:
In [267]:
ra = np.array(rainfall)
In [268]:
sorteddata = sorted(d, key=lambda r:r[3])
In [269]:
rainfall = floatcolumn(sorteddata, 6)
In [270]:
year = [int(row[3]) for row in sorteddata]
In [271]:
plt.plot(year, rainfall)
Out[271]:
In [272]:
import random
plt.bar(range(12), [random.random() for i in range(12)])
Out[272]:
In [273]:
months = np.array([row[2] for row in d])
In [274]:
rainfall = np.array(floatcolumn(d, 6))
In [275]:
rainfall[months=="January"].mean()
Out[275]:
In [276]:
import datetime
In [277]:
def get_mean_rainfall(rainfall, months, month):
return rainfall[months==month].mean()
In [279]:
d = datetime.datetime(2019, 9, 26)
In [280]:
d.strftime("%B")
Out[280]:
In [281]:
help(d.strftime)
In [282]:
mnames = [datetime.datetime(2010, i+1, 1).strftime("%B") for i in range(12)]
In [283]:
mnames
Out[283]:
In [284]:
rainfall_ = [get_mean_rainfall(rainfall, months, m) for m in mnames]
In [285]:
rainfall_
Out[285]:
In [286]:
plt.bar(mnames, rainfall_)
Out[286]:
In [287]:
plt.bar(range(12), rainfall_)
Out[287]:
In [288]:
import altair as alt
In [289]:
%%file sample.txt
area,sales,profit
North,5,2
East,25,8
West,15,6
South,20,5
Central,10,3
In [290]:
import pandas as pd
In [291]:
sample = pd.read_csv("sample.txt")
In [292]:
alt.Chart(sample).mark_point()
Out[292]:
In [293]:
alt.renderers.enable('notebook')
Out[293]:
In [294]:
alt.Chart(sample).mark_point()
Out[294]:
In [295]:
alt.Chart(sample).mark_point().encode(y="area")
Out[295]:
In [297]:
alt.Chart(sample).mark_point().encode(
y = "area",
x = "sales"
)
Out[297]:
In [298]:
base = alt.Chart(sample).mark_point().encode(
y = "area",
x = "sales"
)
In [299]:
base.mark_bar()
Out[299]:
In [300]:
base.mark_line()
Out[300]:
In [301]:
base.mark_circle()
Out[301]:
In [303]:
base = alt.Chart(sample).mark_bar().encode(
y = "area",
x = "sales",
color = "area"
)
In [304]:
base
Out[304]:
In [305]:
base.encode(size="profit")
Out[305]:
In [306]:
base.encode(size="profit").mark_circle()
Out[306]:
In [307]:
print(base.to_json())
In [308]:
s = base.encode(size="profit").mark_circle()
In [309]:
s.save("sample.html")
In [310]:
!less sample.html
In [312]:
sales = alt.Chart(sample).mark_bar().encode(
alt.Y("area"),
alt.X("sales"),
alt.Color(value="green")
)
sales
Out[312]:
In [313]:
profit = alt.Chart(sample).mark_bar().encode(
alt.Y("area"),
alt.X("profit"),
alt.Color(value="firebrick")
)
profit
Out[313]:
In [314]:
sales + profit
Out[314]:
pandas¶
In [315]:
sample
Out[315]:
Series¶
In [317]:
area = pd.Series(['North','East','West','South','Central'])
In [318]:
area
Out[318]:
In [319]:
sales = pd.Series([5,25,15,20,10], index=area)
In [320]:
sales
Out[320]:
In [321]:
sales['North']
Out[321]:
In [322]:
sales[0]
Out[322]:
In [327]:
sales.reindex(index=sorted(area))
Out[327]:
In [324]:
sales
Out[324]:
In [325]:
sales[sales > 10]
Out[325]:
In [326]:
sales
Out[326]:
In [328]:
sales[-1]
Out[328]:
In [329]:
profit = pd.Series([2, 8, 6, 5, 3])
In [330]:
profit[-1]
In [331]:
profit.mean()
Out[331]:
In [332]:
profit.std()
Out[332]:
In [333]:
df = pd.DataFrame({"sales":[20,23,12,6,25],
"profit":[5,2,7,1,8]
}, index=['North','East','West','South','Central']
)
In [334]:
df
Out[334]:
In [335]:
df['sales']
Out[335]:
In [336]:
df.sales
Out[336]:
In [337]:
df.head()
Out[337]:
In [340]:
weather = pd.read_csv("HYDERABAD-weather.csv")
In [341]:
weather.head()
Out[341]:
In [342]:
df.loc['North']
Out[342]:
In [343]:
df.iloc[3]
Out[343]:
In [344]:
weather.info()
In [346]:
weather.plot("maxtemp", "rainfall", kind="scatter")
Out[346]:
In [347]:
weather.groupby('year').mean()
Out[347]:
In [348]:
weather.groupby("month").mean()
Out[348]:
In [349]:
weather
Out[349]:
In [350]:
groupbymonth = weather.groupby('month').mean()
In [351]:
groupbymonth
Out[351]:
In [352]:
del groupbymonth['Unnamed: 0']
del groupbymonth['year']
In [353]:
groupbymonth
Out[353]:
In [354]:
groupbymonth.plot()
Out[354]:
In [355]:
groupbymonth
Out[355]:
In [356]:
groupbymonth.index
Out[356]:
In [357]:
Out[357]:
In [358]:
mnames
Out[358]:
In [360]:
groupbymonth.reindex(index=mnames).plot()
Out[360]:
In [366]:
decdata = groupbymonth.loc['December']
In [367]:
decdata
Out[367]:
In [369]:
decdata['maxtemp']
Out[369]:
In [370]:
dict(decdata)
Out[370]:
In [379]:
groupbymonth.plot(kind='bar')
Out[379]:
XML¶
In [380]:
url = "http://www.thehindu.com/"
In [381]:
response = requests.get(url, params={"service":"rss"})
In [382]:
xmltext = response.text
In [390]:
print(xmltext[:1200])
In [384]:
from xml.etree import ElementTree as et
In [385]:
root = et.fromstring(xmltext)
In [391]:
items = root.findall(".//item")
In [392]:
len(items)
Out[392]:
In [393]:
type(items)
Out[393]:
In [394]:
items[0]
Out[394]:
In [395]:
print(et.tostring(items[0]).decode())
In [396]:
for item in items[:10]:
print(item.findtext("title"))
print(item.findtext("link"))
print("-"*30)
In [398]:
from xml.dom.minidom import parseString
In [399]:
root = parseString(xmltext)
In [400]:
root
Out[400]:
In [402]:
items = root.getElementsByTagName("item")
In [403]:
type(items)
Out[403]:
In [404]:
for item in items[:10]:
title = item.getElementsByTagName("title")[0]
link = item.getElementsByTagName("link")[0]
print(title.firstChild.data)
print(link.firstChild.data)
print("-"*30)
json¶
In [405]:
import json
In [406]:
decdata = dict(decdata)
In [407]:
decdata
Out[407]:
In [409]:
s = json.dumps(decdata)
In [410]:
s
Out[410]:
In [411]:
json.loads(s)
Out[411]:
In [412]:
url = "https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=MSFT&interval=5min&outputsize=full&apikey=demo"
In [413]:
print(url)
In [414]:
resp = requests.get(url)
data = resp.json()
In [431]:
pd.DataFrame(data['Time Series (5min)']).transpose()
Out[431]:
In [416]:
url = "https://api.github.com/orgs/{}/repos".format("google")
In [417]:
url
Out[417]:
In [418]:
repos = requests.get(url).json()
In [419]:
type(repos)
Out[419]:
In [420]:
repos[0]
Out[420]:
In [421]:
r = repos[0]
In [422]:
r['forks']
Out[422]:
In [423]:
r['owner']['id']
Out[423]:
In [425]:
for r in sorted(repos, key=lambda r:r['owner']['id'], reverse=True)[:20]:
print(r['owner']['id'], r['forks'])
In [426]:
for r in sorted(repos, key=lambda r:r['forks'], reverse=True)[:10]:
print(r['full_name'], r['forks'])
In [427]:
pd.read_json("https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=MSFT&interval=5min&outputsize=full&apikey=demo")
Out[427]:
Flask¶
In [444]:
%%file flask_app.py
from flask import Flask, render_template
app = Flask(__name__)
@app.route("/hello/<name>")
def hellourl(name="Flask"):
return render_template("hello.html", name=name)
@app.route("/")
def index():
return "This is index page of flask app"
if __name__ == "__main__":
app.run()
In [445]:
!python flask_app.py
In [441]:
!mkdir templates
In [442]:
%%file templates/hello.html
<!doctype html>
<title>Hello from Flask</title>
{% if name %}
<h1>Hello {{ name }} </h1>
{% else %}
<h1>Hello, World!</h1>
{% endif %}
In [ ]: