[Python 文章收集] 在 Python 2.x 處理 Unicode 字串

Source From Here 
Preface 
寫過 Python 的人應該都遇過下面這個錯誤吧，這是 Python 2.x 典型的編碼錯誤訊息: 

UnicodeEncodeError: 'ascii' codec cant decode byte 0xe6 in position 0:
ordinal not in range(128)

相對於其他程式語言而言，Python 2.x 對於編碼的處理較不易讓新手理解，偏偏處理 CJK 一定得用 Unicode。本文用簡單的範例示範如何在 Python 2.x 處理 Unicode 字串。 

在 Python 2.x 處理 Unicode 字串 

程式碼內出現非 ascii 字元 
Python 2.x 預設的編碼是 ascii，如果程式碼(含註解)內出現中文的話，會在編譯時產生錯誤。在程式碼的檔案開頭加上下面這行就能成功編譯： 

view plaincopy to clipboardprint?

1. # -*- coding: utf-8 -*-  

Python 2.x 的「unicode 型態字串」與「str 型態字串」 
Python 2.x 中，字串分為「unicode 型態」與「str 型態」兩種， 

>>> str_name = '劉德華'
>>> print '1', str_name, type(str_name)
1 劉德華
>>> uni_name = u'劉德華' // 藉由在字串前面加上 u ，建立一個內容為 '金城武' 的 python 「unicode 物件」
>>> print '2', uni_name, type(uni_name)
2 劉德華

此時 uni_name 的資料型態是 python 的 「unicode 物件」，並非「str 物件」故當對 uni_name 這個變數做 「str 物件」的操作時會出現錯誤（例如與另一個「str 物件」相加）： 

>>> print str(uni_name)
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)
>>> print uni_name + "中文"
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)

我們可以對 uni_name 這個變數做「unicode 物件」操作（例如與另一個「unicode 物件」相加）： 

>>> print '3', uni_name + u'中文'
3 劉德華中文
// 相對的，str_name 是 python 「str 物件」，故做「str 物件」的操作時不會出現錯誤（例如與另一個「str 物件」相加）
>>> print '4', str_name + '中文'
4 劉德華中文

Python 2.x 的 encode([encoding]) 與 decode([encoding]) 
python 有個 method 叫做 encode([encoding_], [errors='strict']) 這個方法可以將「unicode 物件」轉換成以 encoding_ 方式編碼的「str 物件」: 

// 剛剛的 uni_name 變數原本是 「unicode 物件」
// 用 .encode('utf-8') 將其以 utf-8 編碼方式轉換為「str 物件」
>>> uni_name = u'劉德華'
>>> new_name = uni_name.encode('utf-8')
>>> print '5', new_name, type(new_name)
5 劉德華

// new_name 已經是「str 物件」，做「str 物件」的操作時不會出現錯誤（例如與另一個「str 物件」相加）
>>> print '6', new_name + '中文'
6 劉德華中文 

同樣的道理，我們也可以用 decode([encoding_]) 將「str 物件」還原成「unicode 物件」 

>>> original_unicode_form = new_name.decode('utf-8')
>>> print '7', original_unicode_form, type(original_unicode_form)
7 劉德華

# 之後就可對此變數「unicode 物件」操作（例如與另一個「unicode 物件」相加）
>>> print '7', original_unicode_form, type(original_unicode_form)
7 劉德華
>>> print '8', original_unicode_form + u'略懂略懂'
8 劉德華略懂略懂
>>> print '8', original_unicode_form + '略懂略懂'
8
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 0: ordinal not in range(128)

Python 2.x 字串操作 Unicode code print 
pyhton 的 「unicode 物件」除了在操作時不用擔心編碼問題外，也可以直接插入字元的 unicode code print，例如： 

# 註1. 在 python 中，以 "\uXXXX" 表示 unicode code print 的 U+XXXX
# 例如 '\u5566' 代表 U+5566
# 註2. http://www.charbase.com/5566-unicode-cjk-unified-ideograph
# 註3. \u6211 = 我, \u672C = 本, \u4EBA = 人, \u5566 = 啦
>>> print '9', original_unicode_form + u"\u6211\u672C\u4EBA\u5566"
9 劉德華我本人啦

在 Python 2.x 處理 Unicode 字串 - 檔案 I/O 

1. open(file) 
讀取檔案時，預設會以「str 型態」讀進資料: 

# python 預設的讀檔方式會將資料讀取成 python 的「str 物件」型態
>>> file_handler = open('test.txt', 'r')
>>> for line in file_handler: print("%s %s" % (line.rstrip(), type(line)))
...
出師表
諸葛亮
>>> file_handler.close()

2. codecs.open(file, encoding) 
用 codecs module 讀寫檔案時可指定 encoding，可以「unicode 型態」讀進資料 

# import codecs 後，可善用 codecs.open(encoding) 的 encoding 參數，
# 若設定正確，則 python 會自動在讀取資料時轉換成 python 的「unicode 物件」型態
>>> file_handler = codecs.open('test.txt', 'r', encoding='utf-8')
>>> for line in file_handler: print("%s\t%s" % (line.rstrip(), type(line)))
...
出師表
諸葛亮
>>> file_handler.close()

3. json.load(), json.loads() 
當使用 json.loads 讀取 json 資料時，回傳的結果會是「unicode 物件」型態: 

>>> import json
>>> file_handler = open('test_json.txt', 'r')
>>> data = json.loads(file_handler.read())
>>> title = data['title']
>>> author = data['author']
>>> print title, type(title)
出師表
>>> print author, type(author)
諸葛亮
>>> file_handler.close()

在 Python 2.x 處理 Unicode 字串 - 結論 

1. type() 看字串型態 
當出現亂碼時，用 type() 看看該變數是「unicode 物件」還是「str 物件」，然後用 encode() 或 decode() 將其轉成你要的型態。 

2. encode() 與 decode() 

Anyway, all you have to remember for your to-and-fro Unicode conversions is:
a Unicode string gets encoded to a Python 2.x string (actually, a sequence of bytes)
a Python 2.x string gets decoded to a Unicode string
In both cases, you need to specify the encoding that will be used. – tzot

*「unicode 物件」透過 encode(encoding) 變成「str 物件」(i.e. a sequence of bytes) 
*「str 物件」透過 decode(encoding) 變成「unicode 物件」 
* encode() 和 decode() 也能用來轉換其他編碼: 

>>> str_name = '金城武'
>>> print str_name, type(str_name)
金城武

>>> base64_name = str_name.encode('base64')
>>> print 'Base64 of', str_name, 'is', base64_name
Base64 of 金城武 is 6YeR5Z+O5q2m

>>> print base64_name.decode('base64')
金城武

3. I/O 輸入輸出 
如同 Unicode In Python, Completely Demystified 建議的，記住三個原則： 

* Decode early
* Unicode everywhere
* Encode late

並使用 codecs.open(file, encoding) 

Supplement 
* Python Doc - Unicode HOWTO

[ Python 常見問題 ] urllib3 - 解决Python爬取HTTPS网页时的错误

Source From Here
Question
因为想做一个爬虫定时领取淘宝的淘金币，无奈在使用 requests 获取页面内容时，收到了错误提示：

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)

How-To
根据 Google 到的结果，解决方案如下：

1. 在使用 requests 前加入：requests.packages.urllib3.disable_warnings()
2. 为 requests 添加 verify=False 参数，比如：r = requests.get('https://blog.bbzhh.com',verify=False)

如下是 urllib3 文档的说明:
- InsecurePlatformWarning

New in version 1.11.
Certain Python platforms (specifically, versions of Python earlier than 2.7.9) have restrictions in their ssl module that limit the configuration that urllib3 can apply. In particular, this can cause HTTPS requests that would succeed on more featureful platforms to fail, and can cause certain security features to be unavailable.
If you encounter this warning, it is strongly recommended you upgrade to a newer Python version, or that you use pyOpenSSL as described in theOpenSSL / PyOpenSSL section.
For info about disabling warnings, see Disabling Warnings.

2016年03月02日更新：
先附加一个官方的链接：
https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning
之所以更新是因为，我在virutalenv的环境下，使用pip安装第三方包时，也遇到了类似的提示：

/root/env/python27/debug/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, seehttps://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/root/env/python27/debug/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning

然而这种时候我是不可能去修改代码的，那么解决方案就在官方链接中的最后一部分：
Disabling Warnings
Making unverified HTTPS requests is strongly discouraged. ˙ ͜ʟ˙ But if you understand the ramifications and still want to do it...

Within the code
If you know what you’re doing and would like to disable all urllib3 warnings, you can use disable_warnings():

view plaincopy to clipboardprint?

1. import urllib3  
2. urllib3.disable_warnings()  

Alternatively, if you are using Python’s logging module, you can capture the warnings to your own log:

view plaincopy to clipboardprint?

1. logging.captureWarnings(True)  

Capturing the warnings to your own log is much preferred over simply disabling the warnings.

Without modifying code
If you are using a program that uses urllib3 and don’t want to change the code, you can suppress warnings by setting the PYTHONWARNINGS environment variable in Python 2.7+ or by using the -W flag with the Python interpreter (see docs), such as:

# PYTHONWARNINGS="ignore:Unverified HTTPS request" ./do-insecure-request.py

[Linux 常見問題] How to output a multiline string in Bash?

Source From Here
Question
How can I output a multipline string in Bash without using multiple echo calls like so:

view plaincopy to clipboardprint?

1. echo "usage: up [--level | -n ][--help][--version]"  
2. echo   
3. echo "Report bugs to: "  
4. echo "up home page: "  

I'm looking for a portable way to do this, using only Bash builtins.

How-To
Here documents are often used for this purpose.

view plaincopy to clipboardprint?

1. cat << EOF  
2. usage: up [--level | -n ][--help][--version]  
3.   
4. Report bugs to:   
5. up home page:  
6. EOF  

They are supported in all Bourne-derived shells including all versions of Bash. Another way is using bash built-in command read:

view plaincopy to clipboardprint?

1. # -d DELIM : The first character of DELIM is used to terminate the input line, rather than newline.  
2. read -d '' help <<- EOF  
3.   usage: up [--level | -n ][--help][--version]  
4.   
5.   Report bugs to:  
6.   up home page:  
7. EOF   
8.   
9. echo "$help"  

This message was edited 3 times. Last update was at 15/10/2016 12:23:02

[ Python 常見問題 ] How to remove a key from a python dictionary?

Source From Here 
Quesiton 
When trying to delete a key from a dictionary, I write: 

view plaincopy to clipboardprint?

1. if 'key' in myDict:  
2.     del myDict['key']  

Is there a one line way of doing this? 

How-To 
Use dict.pop(): 

>>> my_dict = {"last":"john", "first":"lee", "age":18, }
>>> my_dict.pop("last")
'john'
>>> my_dict.pop("not exist")
Traceback (most recent call last):
File "", line 1, in
KeyError: 'not exist'
>>> my_dict.pop("middle", "KC") # Second argument 'KC' is returned while the key doesn't exist
'KC'
>>> my_dict
{'age': 18, 'first': 'lee'}

For more usage of dict, please refer to "[Quick Python] 7. Dictionaries".

[ Python 常見問題 ] How to pass dictionary items as function arguments in python?

Source From Here 
Question 
As title. I have a function as below: 

view plaincopy to clipboardprint?

1. def my_function(first, last, age, middle=''):  
2.      print "My name is %s %s (%s) and I am %d years old" % (first, last, middle, age)  

How do I pass data as dict into my_function (With key as argument name and value as argument value). For example: 

view plaincopy to clipboardprint?

1. data_dict = {"first":"Lee", "last":"John", "middle":"KC", "age":18}  

How-To 
Check below usage: 

>>> data_dict = {"first":"Lee", "last":"John", "middle":"KC", "age":18}
>>> my_function(**data_dict)
My name is Lee John (KC) and I am 18 years old
>>> data_list = ["Lin", "Ken"]
>>> data_dict2 = {"age":33}
>>> my_function(*data_list, **data_dict2)
My name is Lin Ken () and I am 33 years old

More on how to use function in Python, please refer to "[Quick Python] 9. Functions"

[Wireshark 小技巧] Decrypting TLS Browser Traffic With Wireshark – The Easy Way!

Source From Here 
Intro 
Most IT people are somewhat familiar with Wireshark. It is a traffic analyzer, that helps you learn how networking works, diagnose problems and much more. 


One of the problems with the way Wireshark works is that it can’t easily analyze encrypted traffic, like TLS. It used to be if you had the private key(s) you could feed them into Wireshark and it would decrypt the traffic on the fly, but it only worked when using RSA for the key exchange mechanism. As people have started to embrace forward secrecy this broke, as having the private key is no longer enough derive the actual session key used to decrypt the data. The other problem with this is that a private key should not or can not leave the client, server, or HSM it is in. This lead me to coming up with very contrived ways of man-in-the-middling myself to decrypt the traffic(e.g. sslstrip ormitmproxy). 

Session Key Logging to the Rescue! 
Well my friends I’m here to tell you that there is an easier way! It turns out that Firefox and Chrome both support logging the symmetric session key used to encrypt TLS traffic to a file. You can then point Wireshark at said file and presto! decrypted TLS traffic. Read on to learn how to set this up. 

Setting up our Browsers 
We need to set an environmental variable. 

On Windows: 
Go into your computer properties, then click “Advance system settings” then “Environment Variables…” 

Add a new user variable called “SSLKEYLOGFILE” and point it at the location that you want the log file to be located at. 

On Linux or Mac OS X: 

$ export SSLKEYLOGFILE=~/path/to/sslkeylog.log

The next time that we launch Firefox or Chrome they will log your TLS keys to this file. 
Edit: If you are having trouble getting it to work on OS X take a look at the comments below. It seems that Apple has changed how environmental variables work in recent versions of OS X. Try launching firefox and wireshark within the same terminal window with, 

# export SSLKEYLOGFILE=/Users/username/sslkeylogs/output.log
# open -a firefox
# wireshark

Setting up Wireshark 
You need at least Wireshark 1.6 for this to work. We simply go into the preferences of Wireshark 

Expand the protocols section: 

Browse to the location of your log file 

The Results 
This is more along the lines of what we normally see when look at a TLS packet, 

This is what it looks like when you switch to the “Decrypted SSL Data” tab. Note that we can now see the request information in plain-text! Success! 

Conclusion 
I hope you learned something today, this makes capturing TLS communication so much more straightforward. One of the nice things about this setup is that the client/server machine that generates the TLS traffic doesn’t have to have Wireshark on it, so you don’t have to gum up a clients machine with stuff they won’t need, you can either have them dump the log to a network share or copy it off the machine and reunite it with the machine doing the packet capture later. Thanks for stopping by! 

Reference: 
* Mozilla Wiki 
* Imperial Violet 
* jSSLKeyLog