Python Virtual Training For Arcesium - Module II - Day 5¶

Feb 13-17, 2023 Vikrant Patil

All notes are available online at https://notes.pipal.in/2023/arcesium_finop_jan/

Please login to https://engage.pipal.in/ and launch jupyter lab

For today create a notebook with name module2-day5

notebook names are case sensitive. Make sure you give correct name

© Pipal Academy LLP

Virtual Environment¶

Many times it happens that different projects have requirements of python packages such that they conflict each other. In such cases how do you work on two different project on same machine? If we install python packages for one project, those packages will confict with other projects. Virtual environment is there to help us. Virtual environment allows us to have set of python packages seperately for each project. Also added advantage is, it won't affect system python's packages. The way to handle this is with help of venv module we create virtual environment for each project. All requirements for the proejct are installed in the virtual environment and not in system python's packages. Let's take some examples.

Conflicting Requirements¶

Suppose we have two projects, datascraping and analytics. For datascraping project requirements are following packages:

requests==2.24.0
openpyxl==2.4.8

and analytics project needs following packages:

pandas==1.1.2
openpyxl==3.0.5
requests==2.24.0

Now here is confiliting requirement, one project needs openpyxl verson 2.4.8 and other needs 3.0.5.

creating evenv¶

To create virutal environment on your system, what you need is python version > 3.5. Python comes with a package called venv (virtual environment). For older python, virtualenv was seperate application. We are going to work with virtual environment that comes with python 3. Easy steps to work with it are as given below. Open up terminal on linux/mac or cmd terminal on windows. on the prompt type following command to create virtual environment with name env1:

python -m venv env1

This will create a folder with name env1 in the current directory. On linux it will have following contents:

+-env1
  |
  +-bin
  +-include
  +-lib
  +-lib64
  +-pyenv.cfg

on windows system it will have following contents:

+-env1
  |
  +-Include
  +-Lib
  +-Scripts
  +-pyenv.cfg

To activate virtual environment on linux run following command on terminal.:

bash$ source env1/bin/activate
(env1) bash$ # you can see the env1 environment activated as change in prompt
To activate virtual environment on windows run following command on windows cmd terminal:
C:\Users\vik> env1\bin\activate.bat
(env1) C:\Users\vik>

Installing packages in virtual environment Once the virtul environment is created and activated, we are ready to use it. To install packages in this active virtual environment use pip install:

pip install typer
Collecting typer
  Using cached https://files.pythonhosted.org/packages/90/34/d138832f6945432c638f32137e6c79a3b682f06a63c488dcfaca6b166c64/typer-0.3.2-py3-none-any.whl
Collecting click<7.2.0,>=7.1.1 (from typer)
  Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Installing collected packages: click, typer
Successfully installed click-7.1.2 typer-0.3.2

to check packages installed

pip list
Package    Version
---------- -------
click      7.1.2
pip        19.2.3
setuptools 41.2.0
typer      0.3.2

requirements.txt¶

If we want to replicate exact same virtual environment on other machine we need list of packages that pip can understand. The format is called as requirements file. it can be generated using:

pip freeze
click==7.1.2
typer==0.3.2

we can also remove versions if we do want to be strict on versions

typer
click

The output can be saved to a file with name requirements.txt. This file can be used in other virtual env to recreate the same environment. For example, lets make use of above requirements to recreate another environment with name env1copy:

bash$ python -m venv env1copy
bash$ source env1copy/bin/activate
(env1copy) bash$ pip install -r requirements.txt
Collecting click==7.1.2 (from -r env1/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Collecting typer==0.3.2 (from -r env1/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/90/34/d138832f6945432c638f32137e6c79a3b682f06a63c488dcfaca6b166c64/typer-0.3.2-py3-none-any.whl
Installing collected packages: click, typer
Successfully installed click-7.1.2 typer-0.3.2

you can check the packages installed:

pip freeze
click==7.1.2
typer==0.3.2

Summary¶

Virtual environment can be created by any user. No admin privileges required. Every virtual environment is stored in seperate folder. Packages installed in a virtual environment are only in that particular virtual environment. With requirements.txt it is very easy to recreate the same replica of a particular virtul environment.

typer¶

In [76]:
%%file head2.py
import typer



def head(filename:str, n:int):
    pass


if __name__ == "__main__":
    typer.run(head)
    
Overwriting head2.py
In [77]:
!python head2.py --help
Usage: head2.py [OPTIONS] FILENAME N

Arguments:
  FILENAME  [required]
  N         [required]

Options:
  --help  Show this message and exit.
In [78]:
%%file head3.py
import typer



def head(filename:str, n:int=5):
    pass


if __name__ == "__main__":
    typer.run(head)
    
Overwriting head3.py
In [79]:
!python head3.py --help
Usage: head3.py [OPTIONS] FILENAME

Arguments:
  FILENAME  [required]

Options:
  --n INTEGER  [default: 5]
  --help       Show this message and exit.
In [80]:
%%file head4.py
"""head.py is python script which mimics unix head command
it shows first few lines of given file.
"""
import typer



def head(filename:str, n:int=5):
    """a function to print first n lines of file
    """
    pass


if __name__ == "__main__":
    typer.run(head)
    
Overwriting head4.py
In [81]:
!python head4.py --help
Usage: head4.py [OPTIONS] FILENAME

  a function to print first n lines of file

Arguments:
  FILENAME  [required]

Options:
  --n INTEGER  [default: 5]
  --help       Show this message and exit.
In [82]:
%%file head5.py
"""head.py is python script which mimics unix head command
it shows first few lines of given file.
"""
import typer



def head(filename:str, 
         n:int=typer.Option(default=5, help="Number of lines to be displyed")):
    """a function to print first n lines of file
    """
    pass


if __name__ == "__main__":
    typer.run(head)
    
Overwriting head5.py
In [83]:
!python head5.py --help
Usage: head5.py [OPTIONS] FILENAME

  a function to print first n lines of file

Arguments:
  FILENAME  [required]

Options:
  --n INTEGER  Number of lines to be displyed  [default: 5]
  --help       Show this message and exit.
In [84]:
def foo(arg1, arg2, arg3, arg4):
    pass
In [85]:
foo(2, 3, arg4=5, arg3=4)
In [86]:
foo(arg4=5, arg3=4, 2, 3)
  Cell In[86], line 1
    foo(arg4=5, arg3=4, 2, 3)
                            ^
SyntaxError: positional argument follows keyword argument

problem

  • implement a python script grep.py which takes a text filename and keyword and prints those lines which has that keyword
  • add option "--invert" such it prints all the lines which do not have that keyword
In [87]:
%%file grep.py
import typer


def grep():
    pass


if __name__ == "__main__":
    typer.run(grep)
Overwriting grep.py
In [88]:
def optional_sqr(a, flag=False):
    if flag:
        return a*a
    else:
        return a
In [89]:
optional_sqr(3)
Out[89]:
3
In [90]:
optional_sqr(3, True)
Out[90]:
9
In [91]:
"shdkjsahfa kjsahdkjsah sahsakjfh sa".find("XX")
Out[91]:
-1
In [92]:
"hello" in line
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[92], line 1
----> 1 "hello" in line

NameError: name 'line' is not defined
In [93]:
%%file grep.py
import typer


def grep(filename:str, keyword:str):
    with open(filename) as f:
        print("".join([line for line in f if keyword in line]), end="")


if __name__ == "__main__":
    typer.run(grep)
Overwriting grep.py
In [94]:
!python grep.py --help
Usage: grep.py [OPTIONS] FILENAME KEYWORD

Arguments:
  FILENAME  [required]
  KEYWORD   [required]

Options:
  --help  Show this message and exit.
In [95]:
!python grep.py poem.txt better
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Now is better than never.
Although never is often better than *right* now.
In [96]:
%%file grep.py
import typer


def grep(filename:str, 
         keyword:str, 
         invert:bool=typer.Option(default=False, help="invert the selection")):
    """selects line from file with given keyword. with invert option we can 
    invert the selection
    """
    with open(filename) as f:
        if invert:
            print("".join([line for line in f if keyword not in line]), end="")
        else:
            print("".join([line for line in f if keyword in line]), end="")


if __name__ == "__main__":
    typer.run(grep)
Overwriting grep.py
In [97]:
!python grep.py --help
Usage: grep.py [OPTIONS] FILENAME KEYWORD

  selects line from file with given keyword. with invert option we can  invert
  the selection

Arguments:
  FILENAME  [required]
  KEYWORD   [required]

Options:
  --invert / --no-invert  invert the selection  [default: no-invert]
  --help                  Show this message and exit.
In [98]:
!python grep.py poem.txt better
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Now is better than never.
Although never is often better than *right* now.
In [99]:
!python grep.py poem.txt better --invert
The Zen of Python, by Tim Peters

Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

a program that runs functionality after some particular time¶

In [100]:
%%file tick_automatic.py
import typer

import time

def tick(keyword):
    """selects line from file with given keyword. with invert option we can 
    invert the selection
    """
    while True:
        time.sleep(5) # this will wait for 5 seconds
        print("Tick... ", keyword)
        
if __name__ == "__main__":
    typer.run(tick)
Overwriting tick_automatic.py
In [101]:
import time
def schedule(seconds, func, args):
    while True:
        time.sleep(seconds)
        func(*args)
In [102]:
def say_hello(name, greeting):
    print(greeting, name)
    
schedule(2, say_hello, ("arcesium", "welcome"))
welcome arcesium
welcome arcesium
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[102], line 4
      1 def say_hello(name, greeting):
      2     print(greeting, name)
----> 4 schedule(2, say_hello, ("arcesium", "welcome"))

Cell In[101], line 4, in schedule(seconds, func, args)
      2 def schedule(seconds, func, args):
      3     while True:
----> 4         time.sleep(seconds)
      5         func(*args)

KeyboardInterrupt: 

Write a program paste.py that takes two files as command-line arguments and contacenates the corresponding lines in those two files with a tab character and prints it.

For example, of the first file files/a.txt has the following contents

A
B
C
D

and the second file files/b.txt has the following:

1
2
3
4

The output should be:

$ python paste.py files/a.txt files/b.txt
A       1
B       2
C       3
D       4

Note that the first line is "A\n".

For similicity assume that both the files have exactly same number of lines.

Hint:

You can use the strip method on a string to remove the new line character.

"a\n".strip("\n")

"a"

In [103]:
%%file file1.txt
A
B
C
D
E
Overwriting file1.txt
In [104]:
%%file file2.txt
1
2
3
4
5
6
Overwriting file2.txt
In [105]:
%%file paste.py
import typer

def paste(file1, file2):
    with open(file1) as f1:
        with open(file2) as f2:
            for line1, line2 in zip(f1, f2):
                print(line1.strip() + "\t" + line2.strip())
    


if __name__ == "__main__":
    typer.run(paste)
Overwriting paste.py
In [106]:
!python paste.py file1.txt file2.txt
A	1
B	2
C	3
D	4
E	5
In [ ]: