Oct 11-15, 2021 Vikrant Patil
These notes are available online at https://notes.pipal.in/2021/arcesium_finop_batch2/
© Pipal Academy LLP
Day 1 | Day 2 | Day 3 | Day 4 | Day 5
We will be using jupyter hub from https://lab2.pipal.in for this training.
create a notebook with name module3-day2
Many times it happens that different projects have requirements of python packages such that they conflict each other. In such cases how do you work on two different project on same machine? If we install python packages for one project, those packages will confict with other projects. Virtual environment is there to help us. Virtual environment allows us to have set of python packages seperately for each project. Also added advantage is, it won't affect system python's packages. The way to handle this is with help of venv module we create virtual environment for each project. All requirements for the proejct are installed in the virtual environment and not in system python's packages. Let's take some examples. Conflicting Requirements
Suppose we have two projects, datascraping and analytics. For datascraping project requirements are following packages:
requests==2.24.0
openpyxl==2.4.8
and analytics project needs following packages:
pandas==1.1.2
openpyxl==3.0.5
requests==2.24.0
Now here is confiliting requirement, one project needs openpyxl verson 2.4.8 and other needs 3.0.5. creating evenv
To create virutal environment on your system, what you need is python version > 3.5. Python comes with a package called venv (virtual environment). For older python, virtualenv was seperate application. We are going to work with virtual environment that comes with python 3. Easy steps to work with it are as given below. Open up terminal on linux/mac or cmd terminal on windows. on the prompt type following command to create virtual environment with name env1:
python -m venv env1
This will create a folder with name env1 in the current directory. On linux it will have following contents:
+-env1
|
+-bin
+-include
+-lib
+-lib64
+-pyenv.cfg
on windows system it will have following contents:
+-env1
|
+-Include
+-Lib
+-Scripts
+-pyenv.cfg
To activate virtual environment on linux run following command on terminal.:
bash$ source env1/bin/activate
(env1) bash$ # you can see the env1 environment activated as change in prompt
To activate virtual environment on windows run following command on windows cmd terminal:
C:\Users\vik> env1\bin\activate.bat
(env1) C:\Users\vik>
Installing packages in virtual environment
Once the virtul environment is created and activated, we are ready to use it. To install packages in this active virtual environment use pip install:
pip install typer
Collecting typer
Using cached https://files.pythonhosted.org/packages/90/34/d138832f6945432c638f32137e6c79a3b682f06a63c488dcfaca6b166c64/typer-0.3.2-py3-none-any.whl
Collecting click<7.2.0,>=7.1.1 (from typer)
Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Installing collected packages: click, typer
Successfully installed click-7.1.2 typer-0.3.2
to check packages installed
pip list
Package Version
---------- -------
click 7.1.2
pip 19.2.3
setuptools 41.2.0
typer 0.3.2
requirements.txt
If we want to replicate exact same virtual environment on other machine we need list of packages that pip can understand. The format is called as requirements file. it can be generated using:
pip freeze
click==7.1.2
typer==0.3.2
The output can be saved to a file with name requirements.txt. This file can be used in other virtual env to recreate the same environment. For example, lets make use of above requirements to recreate another environment with name env1copy:
bash$ python -m venv env1copy
bash$ source env1copy/bin/activate
(env1copy) bash$ pip install -r requirements.txt
Collecting click==7.1.2 (from -r env1/requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Collecting typer==0.3.2 (from -r env1/requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/90/34/d138832f6945432c638f32137e6c79a3b682f06a63c488dcfaca6b166c64/typer-0.3.2-py3-none-any.whl
Installing collected packages: click, typer
Successfully installed click-7.1.2 typer-0.3.2
you can check the packages installed:
pip freeze
click==7.1.2
typer==0.3.2
Virtual environment can be created by any user. No admin privileges required.
Every virtual environment is stored in seperate folder.
Packages installed in a virtual environment are only in that particular virtual environment.
With requirements.txt it is very easy to recreate the same replica of a particular virtul environment.
%%file outlook.py
"""all about outlook and python
create virtual environment with name pywinservices
python -m venv pywinservices
activate it using
\pywinservices\Scripts\activate.bat
after activation install following packages
pip install pypiwin32
following code makes use of MAPI api provided by microsoft
for more advanced cases refer
https://docs.microsoft.com/en-us/office/client-developer/outlook/outlook-home
"""
import datetime
import os
import win32com.client
def get_outlook():
return win32com.client.Dispatch("Outlook.Application").GetNamespace("MAPI")
def get_inbox(email=None):
outlook = get_outlook()
if email == None:
inbox = outlook.GetDefaultFolder(6) # this works
else:
inbox = outlook.Folders.Items(email).Folders.Items('Inbox')
# inbox1 = outlook.Folders.Items('123@abc.com').Folders.Items('Inbox') # To access 123@abc.com Inbox
# inbox2 = outlook.Folders.Items('456@def.com').Folders.Itmes('Inbox') # To access 456@def.com Inbox
return inbox
def get_default_inbox_messages():
inbox = get_inbox()
messages = inbox.Items
return messages
def saveattachemnts_from_email(subject):
"""saves attachment from email that matches with subject
"""
messages = get_default_inbox_messages()
path = os.path.expanduser("~/Desktop/Attachments")
today = datetime.date.today()
for message in messages:
if message.Subject == subject and message.Unread or message.Senton.date() == today:
# body_content = message.body
attachments = message.Attachments
attachment = attachments.Item(1)
for attachment in message.Attachments:
attachment.SaveAsFile(os.path.join(path, str(attachment)))
if message.Subject == subject and message.Unread:
message.Unread = False
break
# print first 3 emails from ...
def print_emails(folderindex: int = 6, emailcount: int = 3):
"""
folderindex 3, 4, 5, 6 ..Trash, Outbox, Sent, Inbox
""" # folders,and emailcount,
# try different numbers
messages = get_default_inbox_messages()
for i in range(emailcount):
message = messages[i]
print(message.Subject)
# this body of message can be parsed to extract table
print(message.Body)
print("="*30)
def send_email_with_attachment():
import win32com.client as win32
outlook = win32.Dispatch('outlook.application')
mail = outlook.CreateItem(0)
mail.To = 'To address'
mail.Subject = 'Message subject'
mail.Body = 'Message body'
mail.HTMLBody = '<h2>HTML Message body</h2>' # this field is optional
# To attach a file to the email (optional):
attachment = "Path to the attachment"
mail.Attachments.Add(attachment)
mail.Send()
Writing outlook.py
we will make use of requests library to download data from internet
http protocol (on which websites run) has four methods
Any website which has pubished web api to access or download data can be scraped using requests
lets have a look at data formats
import json
pydict = {"function":"CURRENCY_EXCHANGE",
"from_currency":"BTC",
"to_currency":"USD"}
pydict['function']
'CURRENCY_EXCHANGE'
json.dumps(pydict) # this is json data string
'{"function": "CURRENCY_EXCHANGE", "from_currency": "BTC", "to_currency": "USD"}'
d = {"a":{"a":34},
"b":[1, 2, 3, 4, 4],
"c":[{"one":1},{"two":2},{"three":3}]}
json.dumps(d)
'{"a": {"a": 34}, "b": [1, 2, 3, 4, 4], "c": [{"one": 1}, {"two": 2}, {"three": 3}]}'
jsonstring = '{"a": {"a": 34}, "b": [1, 2, 3, 4, 4], "c": [{"one": 1}, {"two": 2}, {"three": 3}]}'
json.loads(jsonstring)
{'a': {'a': 34},
'b': [1, 2, 3, 4, 4],
'c': [{'one': 1}, {'two': 2}, {'three': 3}]}
import requests
resp.status_code # 200 anything is ok, 400 or 500 anything is error
200
data = resp.json() # i know before hand from documenation that this url resturn json data
data
{'Realtime Currency Exchange Rate': {'1. From_Currency Code': 'BTC',
'2. From_Currency Name': 'Bitcoin',
'3. To_Currency Code': 'CNY',
'4. To_Currency Name': 'Chinese Yuan',
'5. Exchange Rate': '388162.05040200',
'6. Last Refreshed': '2021-10-27 06:24:07',
'7. Time Zone': 'UTC',
'8. Bid Price': '388162.05040200',
'9. Ask Price': '388162.11424900'}}
data['Realtime Currency Exchange Rate']
{'1. From_Currency Code': 'BTC',
'2. From_Currency Name': 'Bitcoin',
'3. To_Currency Code': 'CNY',
'4. To_Currency Name': 'Chinese Yuan',
'5. Exchange Rate': '388162.05040200',
'6. Last Refreshed': '2021-10-27 06:24:07',
'7. Time Zone': 'UTC',
'8. Bid Price': '388162.05040200',
'9. Ask Price': '388162.11424900'}
data['Realtime Currency Exchange Rate']['5. Exchange Rate']
'388162.05040200'
def exchange_rate(from_, to_):
url = "https://www.alphavantage.co/query"
params = {"function":"CURRENCY_EXCHANGE_RATE",
"from_currency":from_,
"to_currency":to_,
"apikey":"UKVEF0JLE0TBPDEF"}
resp = requests.get(url,params=params)
return resp.json()['Realtime Currency Exchange Rate']['5. Exchange Rate']
exchange_rate("BTC","USD")
'60920.01000000'
exchange_rate("BTC","INR")
'4562568.29632000'