Python Training at VMWare Bangalore - Day 1¶

Aug 16-18, 2017
Anand Chitipothu & Vikrant Patil

These notes are available online at http://notes.pipal.in/2017/vmware-advpy

© Pipal Academy LLP

Day 1 | Day 2 | Day 3

In [3]:
1 + 2
Out[3]:
3
In [4]:
2 ** 1000
Out[4]:
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376
In [5]:
%%file hello.py
print("Hello, world!")

Writing hello.py
In [6]:
!python hello.py

Hello, world!

Quick Introduction to Python

In [7]:
1 + 2
Out[7]:
3
In [8]:
x = 2
In [9]:
x = "hello"
In [10]:
x = 2
In [11]:
y = "3"
In [12]:
x + y

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-b50c5120e24b> in <module>()
----> 1 x + y

TypeError: unsupported operand type(s) for +: 'int' and 'str'
In [13]:
# this is comment
print(x*x)

4

Datatypes

Python has integers.

In [14]:
1 + 2
Out[14]:
3
In [15]:
2 ** 1000
Out[15]:
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376

Python as floating point numbers.

In [16]:
1.2 + 3.4
Out[16]:
4.6

Python has strings.

In [17]:
print("hello")

hello
In [18]:
"hello"
Out[18]:
'hello'

Strings can be enclosed in single quotes or double quotes. Both mean exactly the same.

In [19]:
"hello" + 'world'
Out[19]:
'helloworld'
In [20]:
"Alice's book"
Out[20]:
"Alice's book"
In [21]:
x = """This is a multi-line string
line two
line three
and the last line.
"""
In [22]:
x
Out[22]:
'This is a multi-line string\nline two\nline three\nand the last line.\n'
In [23]:
print(x)

This is a multi-line string
line two
line three
and the last line.

Python has lists.

In [24]:
x = ["a", "b", "c"]
In [25]:
len(x)
Out[25]:
3
In [26]:
x[0]
Out[26]:
'a'
In [28]:
x[:2] # first two elements of a list
Out[28]:
['a', 'b']

How to get the last element?

In [29]:
x[-1]
Out[29]:
'c'
In [30]:
"when in doubt, use brute force".split()[-1]
Out[30]:
'force'

Python has another datatype called tuple to represent fixed-width records.

In [31]:
point = (3, 4)
In [32]:
x, y = point
In [33]:
x
Out[33]:
3
In [34]:
y
Out[34]:
4
In [35]:
person = ("Alice", "alice@example.com", 45, ["Admin", "Staff"])
In [36]:
person
Out[36]:
('Alice', 'alice@example.com', 45, ['Admin', 'Staff'])

Python has dictionaries for representing key-value pairs.

In [37]:
person = {
    "name": "Alice",
    "email": "alice@example.com",
    "roles": ["Admin", "Staff"]
}
In [38]:
person["name"]
Out[38]:
'Alice'
In [39]:
person["roles"]
Out[39]:
['Admin', 'Staff']

Python also has a boolean type as well.

In [40]:
True
Out[40]:
True
In [41]:
False
Out[41]:
False
In [42]:
4 > 3
Out[42]:
True
In [43]:
"hell" in "hello"
Out[43]:
True
In [44]:
"yell" in "hello"
Out[44]:
False

Python has a special type called None.

In [45]:
x = None
In [46]:
print(x)

None

Any function that doesn't return a value, returns None.

In [48]:
x = print("hello")

hello
In [49]:
print(x)

None
In [50]:
def f():
    pass
In [51]:
print(f())

None

Example: CSV Parser

A small example to demonstrate the elegance of Python.

In [52]:
%%file a.csv
A1,B1,C1
A2,B2,C2
A3,B3,C3
A4,B4,C4

Writing a.csv
In [53]:
open("a.csv").readlines()
Out[53]:
['A1,B1,C1\n', 'A2,B2,C2\n', 'A3,B3,C3\n', 'A4,B4,C4']
In [54]:
# take every line
[line for line in open("a.csv")]
Out[54]:
['A1,B1,C1\n', 'A2,B2,C2\n', 'A3,B3,C3\n', 'A4,B4,C4']
In [56]:
# take every line after stripping the newline char
[line.strip("\n") for line in open("a.csv")]
Out[56]:
['A1,B1,C1', 'A2,B2,C2', 'A3,B3,C3', 'A4,B4,C4']
In [57]:
# split every line after stripping the newline char
[line.strip("\n").split(",") for line in open("a.csv")]
Out[57]:
[['A1', 'B1', 'C1'],
 ['A2', 'B2', 'C2'],
 ['A3', 'B3', 'C3'],
 ['A4', 'B4', 'C4']]

Functions

In [58]:
def square(x):
    return x*x
In [59]:
square(4)
Out[59]:
16
In [60]:
print(square(4))

16
In [61]:
print(square)

<function square at 0x1067cabf8>
In [62]:
f = square
In [63]:
print(f)

<function square at 0x1067cabf8>
In [64]:
f is square
Out[64]:
True

Let us see why having functions as first class objects is important.

In [66]:
def square(x):
    return x*x

def sum_of_squares(x, y):
    return square(x) + square(y)
In [67]:
sum_of_squares(3, 4)
Out[67]:
25
In [68]:
def cube(x):
    return x*x*x

def sum_of_cubes(x, y):
    return cube(x) + cube(y)
In [69]:
sum_of_cubes(3, 4)
Out[69]:
91
In [70]:
def sumof(f, x, y):
    return f(x) + f(y)
In [71]:
sumof(square, 3, 4)
Out[71]:
25
In [72]:
sumof(cube, 3, 4)
Out[72]:
91
In [73]:
sumof(len, "hello", "everyone")
Out[73]:
13

Passing functions as arguments is so useful and common in Python that there are even some built-in functions that take functions as arguments.

In [74]:
words = ["one", "two", "three", "four", "five"]
In [75]:
max(words)
Out[75]:
'two'

How about finding the longest word?

In [76]:
max(words, key=len)
Out[76]:
'three'
In [77]:
def mylen(x):
    print("mylen", x)
    return len(x)

max(words, key=mylen)

mylen one
mylen two
mylen three
mylen four
mylen five
Out[77]:
'three'

Lets say we have records of students with name and marks.

In [87]:
records = [
    ("A", 67),
    ("B", 98),
    ("C", 56)
]

How to find the name of the person who got the maximum marks?

In [88]:
def get_marks(record):
    return record[1]

max(records, key=get_marks)
Out[88]:
('B', 98)
In [86]:
max(records, key=get_marks)[0]
Out[86]:
'B'

Q: How to sort in descending order?

In [89]:
min(records, key=get_marks)
Out[89]:
('C', 56)
In [91]:
sorted(records)
Out[91]:
[('A', 67), ('B', 98), ('C', 56)]
In [92]:
sorted(records, key=get_marks)
Out[92]:
[('C', 56), ('A', 67), ('B', 98)]
In [93]:
sorted(records, key=get_marks, reverse=True)
Out[93]:
[('B', 98), ('A', 67), ('C', 56)]

Problem: Write a function imax that takes a list of words and finds the maximum out of them ignoring the case.

>>> imax(["a", "B", "c", "D"])
'D'
>>> imax(["a", "B", "c"])
'c'

Hint:

In [94]:
"HelloWorld".lower()
Out[94]:
'helloworld'
In [95]:
"HelloWorld".upper()
Out[95]:
'HELLOWORLD'
In [96]:
def imax(words):
    return max(words, key=ignore_case)

def ignore_case(s):
    print("ignore_case", s)
    return s.lower()
In [98]:
imax(["a", "B", "c"])

ignore_case a
ignore_case B
ignore_case c
Out[98]:
'c'

The lambda expression

In [99]:
square = lambda x: x*x
In [100]:
words = ["a", "B", "c", "D"]
max(words, key=lambda w: w.lower())
Out[100]:
'D'
In [101]:
records = [
    ("A", 67),
    ("B", 98),
    ("C", 56)
]
In [102]:
max(records)
Out[102]:
('C', 56)
In [103]:
# find the record with maximum marks
max(records, key=lambda rec: rec[1])
Out[103]:
('B', 98)

Problem: Implement a function maximum that takes 2 values x and y and a key function as argument and finds the maximum of the values based on the key. Can you do this without using the built-in max function?

>>> maximum(3, -4, abs)
-4
>>> maximum("ten", "seven", len)
'seven'
>>> maximum("a", "B", lambda x: x.lower())
'B'

Passing arguments by position and name

In [104]:
def diff(x, y):
    return x-y
In [105]:
diff(5, 3)
Out[105]:
2
In [106]:
diff(x=5, y=3)
Out[106]:
2
In [107]:
diff(5, y=3)
Out[107]:
2
In [108]:
diff(y=3, x=5)
Out[108]:
2
In [109]:
def incr(x, amount=1):
    return x+amount
In [110]:
incr(4, 2)
Out[110]:
6
In [111]:
incr(4)
Out[111]:
5

In python 3, you can enforce that some arguments can be passed only by name.

In [112]:
# python3 only
def incr(x, *, amount=1):
    return x+amount
In [114]:
incr(4, amount=2)
Out[114]:
6

Functions with variable number of arguments and keyword arguments

In [115]:
print("hello")

hello
In [116]:
print("hello", 1)

hello 1
In [117]:
print("hello", 1, 2, 3, 4)

hello 1 2 3 4

The print function can take any number of arguments.

How can we write a function that works like that?

In [119]:
def xprint(label, *args):
    for a in args:
        print(label, a)
In [120]:
xprint("[INFO]", 1, 2, 3, 4)

[INFO] 1
[INFO] 2
[INFO] 3
[INFO] 4

Problem: Write a function add that takes variable number of arguments and returns their sum.

>>> add(1, 2, 3)
6
>>> add(1, 2, 3, 4)
10
In [122]:
# Hint
sum([1, 2, 3, 4])
Out[122]:
10

Problem: Write a function strjoin that takes a separator as first argument followed by variable number of string to join with that separator.

>>> strjoin("-", "a", "b", "c")
'a-b-c'
In [123]:
# Hint
"-".join(['a', 'b', 'c'])
Out[123]:
'a-b-c'

We've seen how to handle variable number of positional arguments. How about taking variable number of keyword arguments?

In [124]:
def f(**kwargs):
    print(kwargs)
In [125]:
f(x=1, y=2)

{'x': 1, 'y': 2}
In [131]:
def render_tag(tagname, **attrs):
    pairs = ['{}="{}"'.format(k, v) for k, v in attrs.items()]
    pairs_str = " ".join(pairs)
    return "<{} {}>".format(tagname, pairs_str)

print(render_tag("a", href="http://google.com"))
print(render_tag("input", 
                 type="text", 
                 name="email", 
                 value="test@example.com"))

<a href="http://google.com">
<input type="text" name="email" value="test@example.com">

Now that we know how to pack variable number of positional arguments and variable number of keyword arguments, can you think of the reverse?

In [132]:
def f(a, b, c):
    return a*b-c
In [133]:
args = [3, 2, 1]

Write a function call_func that takes a function and a list of arguments as arguments and calls that function with those arguments.

>>> call_func(square, [4])
16
>>> call_func(sum_of_squares, [3, 4])
25
>>> call_func(f, [3, 2, 1])
5
In [134]:
def call_func(f, args):
    # unpack args when calling f
    return f(*args)
In [135]:
call_func(square, [4])
Out[135]:
16
In [136]:
call_func(f, [3, 2, 1])
Out[136]:
5

The packing and unpacking are also used when doing multiple assignments.

In [137]:
line = "A,1,2,3,4,5,pass"
In [140]:
# python3 only
name, *marks, status = line.split(",")
In [139]:
name
Out[139]:
'A'
In [141]:
marks
Out[141]:
['1', '2', '3', '4', '5']
In [142]:
status
Out[142]:
'pass'

Just like we unpacked a list of arguments when calling a function, we can do the same for keyword arguments as well.

In [143]:
def diff(x, y):
    return x-y
In [144]:
kwargs = {"x": 4, "y": 2}

How to call diff with kwargs?

In [146]:
diff(**kwargs) # equivalant of diff(x=4, y=2)
Out[146]:
2

Default Arguments

In [147]:
def incr(x, amount=1):
    return x + amount
In [148]:
incr(5)
Out[148]:
6
In [149]:
incr(5, 3)
Out[149]:
8

Problem: Write a function centeralign that takes a string and optional width as arguments and center aligns that string in that width. If the width is not specified, it should take 10 as the default value.

>>> centeralign("abcd", 6)
' abcd '
>>> centeralign("abcd")
'   abcd   '
In [150]:
# Hint:
"hello".center(7)
Out[150]:
' hello '

Q: How does it show the single quotes?

Python has two ways to convert an object into a string. str and repr.

In [151]:
print(str("a"), repr("a"))

a 'a'
In [152]:
print(str(1), str("1"))

1 1
In [153]:
print(repr(1), repr("1"))

1 '1'

Pitfalls of default arguments

In [154]:
def append(value, result=[]):
    result.append(value)
    return result
In [155]:
x = [1, 2, 3]
print(append(4, x))
print(x)

[1, 2, 3, 4]
[1, 2, 3, 4]
In [156]:
y = [1, 2]
print(append(3, y))

[1, 2, 3]
In [157]:
z = append(1)
print(z)

[1]
In [158]:
z2 = append(2)
print(z2)

[1, 2]

That is confusing, isn't it?

Let us try to understand what happens when default value is defined.

In [159]:
def peep(x):
    print("peep", x)
    return x
In [160]:
print("before defining incr")
def incr(x, amount=peep(1)):
    return x+amount
print("after defining incr")

incr(4)
print("after call to incr(4)")

incr(5)
print("after call to incr(5)")

before defining incr
peep 1
after defining incr
after call to incr(4)
after call to incr(5)

How to handle this kind issues? Simple! Make sure the default value is defined everytime the function is called.

In [161]:
def append(value, result=None):
    if result is None:
        # the result is initialized everytime the func is called
        result = []
    result.append(value)
    return result
In [162]:
append(1)
Out[162]:
[1]
In [163]:
append(2)
Out[163]:
[2]

Quiz

In [165]:
%%file q1.py
x = 1
y = x
x = 2
print(x, y)

Writing q1.py
In [166]:
!python q1.py

2 1
In [167]:
%%file q2.py
x = [1]
y = x
x = [2]
print(x, y)

Writing q2.py
In [168]:
!python q2.py

[2] [1]
In [169]:
%%file q3.py
x = [1]
y = x
x.append(2)
print(x, y)

Writing q3.py
In [170]:
!python q3.py

[1, 2] [1, 2]

Functions as return values

In [171]:
def make_adder(x):
    def adder(y):
        return x + y
    return adder
In [172]:
add5 = make_adder(5)
print(add5(3))

8
In [173]:
def make_adder(x):
    print("make_adder", x)
    def adder(y):
        print("adder x={}, y={}".format(x, y))
        return x + y
    return adder
In [174]:
add5 = make_adder(5)

make_adder 5
In [175]:
add5
Out[175]:
<function __main__.make_adder.<locals>.adder>
In [176]:
add5(3)

adder x=5, y=3
Out[176]:
8
In [177]:
def make_logger(prefix):
    def logger(*args):
        print(prefix, *args)
    return logger
In [178]:
info = make_logger("[INFO]")
warn = make_logger("[WARN]")
In [179]:
info("Hello, world!")

[INFO] Hello, world!
In [180]:
warn("something went wrong.")

[WARN] something went wrong.
In [181]:
records = [
    ["A", 40],
    ["B", 86],
    ["C", 48],
    ["D", 75]
]
In [182]:
# return a function to get a column
def column(colindex):
    def f(row):
        return row[colindex]
    return f
In [183]:
max(records, key=column(1))
Out[183]:
['B', 86]

Q: Can we give column name instead of column index?

Yes, if the data has column name.

In [184]:
records = [
    {"name": "A", "marks": 40},
    {"name": "B", "marks": 86},
    {"name": "C", "marks": 48},
    {"name": "D", "marks": 75},
]
In [185]:
def column(colname):
    def f(row):
        return row[colname]
    return f
In [186]:
max(records, key=column('marks'))
Out[186]:
{'marks': 86, 'name': 'B'}

Decorators

In [190]:
%%file sum.py

def square(x):
    print("square", x)
    return x*x

def sum_of_squares(x, y):
    print("sum_of_squares", x, y)
    return square(x) + square(y)

if __name__ == "__main__":
    print(sum_of_squares(3, 4))

Overwriting sum.py
In [191]:
!python sum.py

sum_of_squares 3 4
square 3
square 4
25

Can we get the prints without really adding prints manually inside each function?

In [195]:
%%file trace1.py

def trace(f):
    def g(*args):
        print(f.__name__, args)
        value = f(*args)
        print("return", value)
        return value
    return g

Overwriting trace1.py
In [196]:
%%file sum1.py

from trace1 import trace

@trace
def square(x):
    return x*x

#square = trace(square)

@trace
def sum_of_squares(x, y):
    return square(x) + square(y)

#sum_of_squares = trace(sum_of_squares)

if __name__ == "__main__":
    print(sum_of_squares(3, 4))

Overwriting sum1.py
In [197]:
!python sum1.py

sum_of_squares (3, 4)
square (3,)
return 9
square (4,)
return 16
return 25
25

Trace v2

In [224]:
%%file trace2.py

import os

level = 0

def log(*args):
    # print only if DEBUG is set to true
    if os.getenv("DEBUG") == "true":
        print(*args)

def trace(f):
    def g(*args):
        global level
        log("| " * level + "|-- " + f.__name__, args)
        level += 1
        value = f(*args)
        log("| " * level + "|-- " + "return", value)
        level -= 1
        return value
    return g

Overwriting trace2.py
In [220]:
%%file sum2.py

from trace2 import trace

@trace
def square(x):
    return x*x

#square = trace(square)

@trace
def sum_of_squares(x, y):
    return square(x) + square(y)

#sum_of_squares = trace(sum_of_squares)

if __name__ == "__main__":
    print(sum_of_squares(3, 4))

Overwriting sum2.py
In [221]:
!python sum2.py

|-- sum_of_squares (3, 4)
| |-- square (3,)
| | |-- return 9
| |-- square (4,)
| | |-- return 16
| |-- return 25
25
In [222]:
%%file fib.py
import sys
from trace2 import trace

@trace
def fib(n):
    if n == 0 or n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)
    
def main():
    n = int(sys.argv[1])
    print(fib(n))
    
if __name__ == "__main__":
    main()

Writing fib.py
In [225]:
!python fib.py 4

5
In [233]:
!DEBUG=true python fib.py 5

|-- fib (5,)
| |-- fib (4,)
| | |-- fib (3,)
| | | |-- fib (2,)
| | | | |-- fib (1,)
| | | | | |-- return 1
| | | | |-- fib (0,)
| | | | | |-- return 1
| | | | |-- return 2
| | | |-- fib (1,)
| | | | |-- return 1
| | | |-- return 3
| | |-- fib (2,)
| | | |-- fib (1,)
| | | | |-- return 1
| | | |-- fib (0,)
| | | | |-- return 1
| | | |-- return 2
| | |-- return 5
| |-- fib (3,)
| | |-- fib (2,)
| | | |-- fib (1,)
| | | | |-- return 1
| | | |-- fib (0,)
| | | | |-- return 1
| | | |-- return 2
| | |-- fib (1,)
| | | |-- return 1
| | |-- return 3
| |-- return 8
8

Problem: Write a function depricated that prints a warning message saying that the function is depricated everytime it is called.

@depricated
def square(x):
    return x*x

print(square(4))

Should produce:

WARNING: function square is depricated.
16

Problem: Write a decorator function with_retries that continue to retry for 5 times if there is any exception raised in calling the original function.

# py3
from urllib.request import urlopen

# py2
#from urllib2 import urlopen

@with_retries
def wget(url):
    return urlopen(url).read()

wget("http://google.com/no-such-page")

Should produce:

Failed to download, retrying...
Failed to download, retrying...
Failed to download, retrying...
Failed to download, retrying...
Failed to download, retrying...
Giving up!
In [234]:
from urllib.request import urlopen

def wget(url):
    return urlopen(url).read()
In [236]:
try:
    wget("http://google.com/no-such-page")
except Exception:
    print("Got error!")

Got error!

Let us look at the solution.

In [244]:
def with_retries(f):
    def g(*args):
        for i in range(5):
            try:
                return f(*args)
            except Exception as e:
                print(f.__name__, args, "failed:", e)
        print("Giving up...")
    return g
In [245]:
@with_retries
def foo(url):
    raise Exception("Doom!")
In [246]:
foo("http://google.com/no-such-page")

foo ('http://google.com/no-such-page',) failed: Doom!
foo ('http://google.com/no-such-page',) failed: Doom!
foo ('http://google.com/no-such-page',) failed: Doom!
foo ('http://google.com/no-such-page',) failed: Doom!
foo ('http://google.com/no-such-page',) failed: Doom!
Giving up...

Writing Custom Modules

In [247]:
%%file mymodule.py
print("BEGIN mymodule")

x = 1

def add(a, b):
    return a+b

print("END mymodule")

Writing mymodule.py
In [248]:
!python mymodule.py

BEGIN mymodule
END mymodule
In [251]:
%%file a.py

print("begin")
import mymodule
print("after first import")
import mymodule
print("after second import")

print(mymodule.x)
print(mymodule.add(3, 4))
print("end")

Overwriting a.py
In [252]:
!python a.py

begin
BEGIN mymodule
END mymodule
after first import
after second import
1
7
end

Q: How to reload a module?

In [257]:
%%file b.py
import mymodule

# python3
import importlib
importlib.reload(mymodule)

# python2
# reload(mymodule)

Overwriting b.py
In [256]:
!python b.py

BEGIN mymodule
END mymodule
BEGIN mymodule
END mymodule

The __name__ magic variable

In [258]:
%%file mymodule2.py

x = 1

def add(a, b):
    return a+b

print(add(3, 4))
print(__name__)

Writing mymodule2.py
In [259]:
!python mymodule2.py

7
__main__
In [260]:
!python -c "import mymodule2"

7
mymodule2

When the file is run as a script, the __name__ is set to "__main__", but when the file is imported as a module, it is set to the module name.

In [261]:
%%file mymodule3.py

x = 1

def add(a, b):
    return a+b

if __name__ == "__main__":
    # Run this block of code only when this file is executed
    # as a script. Ignore this when imported as a module.
    print(add(3, 4))

Writing mymodule3.py
In [262]:
!python mymodule3.py

7
In [263]:
!python -c "import mymodule3"

Docstrings

In [269]:
%%file sq.py
"""The square module.

Long description of the module.
"""

def square(x):
    """Computes square of a number.
    
        >>> square(4)
        16
    """
    return x*x

if __name__ == "__main__":
    print(square(3))

Overwriting sq.py
In [270]:
!python sq.py

9
In [271]:
import sq
In [272]:
help(sq)

Help on module sq:

NAME
    sq - The square module.

DESCRIPTION
    Long description of the module.

FUNCTIONS
    square(x)
        Computes square of a number.
        
        >>> square(4)
        16

FILE
    /Users/anand/trainings/2017/vmware-advpy/sq.py
In [273]:
In [ ]: