2021年5月7日 星期五

[Git 文章收集] GitLab -更新 Fork 出來的專案

 Source From Here

Question
在 GitLab 的介面上,可以透過 Fork 的功能,把別人的專案建立一個 fork 到自己的帳號底下,例如原始專案的網址是 http://gitlab/userA/project.git, Fork 出來的專案網址會是 http://gitlab/userB/project.git。 不過原始專案仍然會繼續更新,而自己 Fork 下來的專案則會停在執行 Fork 當時的狀況。

HowTo

1. git clone Fork 出來的專案路徑
$ git clone http://gitlab/userB/project.git

2. git remote 操作前後先看狀態
$ git remote -v
origin http://gitlab/userB/project.git (fetch)
origin http://gitlab/userB/project.git (push)

3. git remote add 新增 upstreamupstream 是 remote name,可以自己取名,不要重複就好
$ git remote add upstream http://gitlab/userA/project.git
$ git remote -v
origin http://gitlab/userB/project.git (fetch)
origin http://gitlab/userB/project.git (push)
upstream http://gitlab/userA/project.git (fetch)
upstream http://gitlab/userA/project.git (push)

有了原始專案的來源後我們就可以開始做更新了!

4. 切換回本地的 master
$ git checkout master


5. 接著把 upstream 的 master 更新給拉進來
$ git pull upstream master


6. 切換到開發用的分支 rebase 自己的 master
$ git checkout working_branch_name
$ git rebase master

7. 如果自己的 master 有 commit,也可以用 --rebase 來避免不必要的 merge 操作
$ git checkout master
$ git pull --rebase upstream master

8. 如果沒有發生衝突的話這樣應該就完成了本地的更新,再把更新後的 branch push 出去就行了
$ git push origin master

這時再回去看看自己的專案頁面,應該會發現已經同步到最新的狀態了!
PS. 修改 remote 名稱
$ git remote rename {oldName} {newName}


Supplement
Git doc - 6.2 GitHub - 參與一個專案
如果你想要參與一個你沒有推送權限的專案,你可以「fork」一份。這代表說 GitHub 會複製一份這個專案的副本給你,並且你對這副本有全部的權限。這副本會存在於你的帳號下,你可以對它進行推送...


2021年5月5日 星期三

[ Python 常見問題 ] Is it possible to automatically break into the debugger when a exception is thrown?

 Source From Here

Question
Is there a way to automatically start break into the debugger (import pdb; pdb.set_trace()) whenever a exception is thrown to inspect the local stack?

HowTo
I found what I was looking for in an answer to What is the simplest way of using Python pdb to in...use of an unhandled exception?:
- test.py
  1. #!/usr/bin/env python3  
  2. import pdb  
  3. import sys  
  4. import functools  
  5.   
  6.   
  7. def debug_on(*exceptions):  
  8.     if not exceptions:  
  9.         exceptions = (Exception, )  
  10.     def decorator(f):  
  11.         @functools.wraps(f)  
  12.         def wrapper(*args, **kwargs):  
  13.             try:  
  14.                 return f(*args, **kwargs)  
  15.             except exceptions:  
  16.                 pdb.post_mortem(sys.exc_info()[2])  
  17.         return wrapper  
  18.     return decorator  
  19.   
  20. @debug_on()  
  21. def buggy_function():  
  22.     print("About to cause exception!")  
  23.     raise Exception("Something wrong!")  
  24.     print("Unreachable...")  
  25.   
  26. if __name__ == '__main__':  
  27.     buggy_function()  
Let's test it:
# ./test.py
About to cause exception!
> /tmp/test.py(23)buggy_function()
-> raise Exception("Something wrong!")

(Pdb) list
  1. 18         return decorator  
  2. 19  
  3. 20     @debug_on()  
  4. 21     def buggy_function():  
  5. 22         print("About to cause exception!")  
  6. 23  ->     raise Exception("Something wrong!")  
  7. 24         print("Unreachable...")  
  8. 25  
  9. 26     if __name__ == '__main__':  
  10. 27         buggy_function()  
  11. [EOF]  
(Pdb) exit()


2021年5月2日 星期日

[Git 文章收集] 提升程式碼品質:使用 Pre-Commit (Git Hooks)

 Source From Here

Preface
在團隊協作開發中,程式碼品質是一個非常需要重視的問題,除了比較進階的重構以外,有沒有什麼自動檢查的方法可以提升程式碼品質呢?答案是有的,就是今天要討論的 pre-commit。

在程式碼 commit 進 branch 之前,如果能在 local 端阻擋一些低級失誤,就可以大大提升開發品質,在專案中使用 pre-commit 有以下好處:
* 自動化檢查程式碼排版規範快速又有效率(如 python PEP8)
* 低級的問題不會進到 code review
---- 多一點時間檢查程式邏輯,而不是基本錯誤(如排版)。
---- 人工檢查程式碼的時間很寶貴,減少人工即是增進效率。
* 低級的問題不會進到 CI/CD pipeline
---- pipeline 應該多一點綠勾勾,而不是滿滿 debug 的痕跡。

這次示範的 pre-commit 設定是基於 python 的 pre-commit framework:pre-commit。市面上的 pre-commit framework 其實不少,例如 js 的 Husky,或是自己撰寫 git hook 的 script 也是可以的。那就廢話不多說,開始介紹吧!

一、Pre-commit 基本使用
以 framework 設置 pre-commit 非常簡單,官方文件 淺顯易懂,只需要 pre-commit 的基本程式,再加上一個 yaml 設定檔即可完成!

1. Pre-commit 安裝
如下安裝工具:
# pip3 install pre-commit
# pre-commit install
pre-commit installed at .git/hooks/pre-commit

接著還需要:
* 撰寫配置文件 .pre-commit-config.yaml
* 產生一個新 commit (需要的套件安裝會和第一次 commit 時的檢查一起完成。)


2. Pre-commit 設定檔與使用流程
需要設定的只有 .pre-commit-config.yaml 這個 config 檔,將它置於專案下(也就是和 .git 同一層)即可,他的基礎格式如下:
  1. repos:  
  2. -   repo: https://github.com/pre-commit/pre-commit-hooks  
  3.     rev: v1.2.3  
  4.     hooks:  
  5.     -   ...  
一個實際使用的 .pre-commit-config.yaml 範例:
  1. repos:  
  2. -   repo: https://github.com/pre-commit/pre-commit-hooks  
  3.     rev: v2.2.3  
  4.     hooks:  
  5.     -   id: flake8  
  6.     -   id: detect-aws-credentials  
  7.     -   id: detect-private-key  
  8.     -   id: check-added-large-files  
  9.     -   id: check-merge-conflict  
  10.     -   id: check-json  
  11.     -   id: check-yaml  
  12. -   repo: https://github.com/ambv/black  
  13.     rev: stable  
  14.     hooks:  
  15.     - id: black  
  16.       language_version: python3.6  
  17. -   repo: https://github.com/Lucas-C/pre-commit-hooks-bandit  
  18.     rev: v1.0.4  
  19.     hooks:  
  20.     -   id: python-bandit-vulnerability-check  
  21.         args: [-l, --recursive, -x, tests]  
  22.         files: .py$  
  23. -   repo: https://github.com/asottile/reorder_python_imports  
  24.     rev: v1.6.1  
  25.     hooks:  
  26.     -   id: reorder-python-imports  
使用這個設定檔 commit 時的檢查訊息:


修改後再次 commit:


二、常用的 Git Hooks
Pre-commit 設置的本質上就是加上一個個 Git hooks,而市面上有很多 hooks 可以挑選,以目的來區分大抵上有幾種類別:

* 程式碼規範檢查
---- black:自動格式化 (formatter)
---- flake8:檢查程式碼不符合規範的地方 (checker, style & linting)。

* 安全檢查:
---- detect-aws-credentials
---- detect-private-key

* 格式檢查:
---- check-json
---- check-yaml

* 其他檢查:
---- check-added-large-files
---- check-merge-conflict

而每一個 hook 也可以針對各個 hook 需要的設定檔再做設定,例如 flake8 的客製化:
.flake8
  1. [flake8]  
  2. ignore = E203, E266, E501, W503, F403, F401  
  3. max-line-length = 79  
  4. max-complexity = 18  
  5. select = B,C,E,F,W,T4,B9  
  6. exclude =  
  7.     .git,  
  8.     __pycache__,  
  9.     build  
另一個常見的用法是用 hook 啟動 unit test,不過要當 test case 質量和數量到達一定程度時比較建議使用,不然測試常常壞掉,然後不得已常常下 --no-verify 參數繞過檢查,反而得不償失。

三、Pre-commit 的注意事項

1. Pre-commit 是一個用於自律的輕量本地端開發工具。
Pre-commit 不是強制性的,不是用來強硬擋下不符規範的 commit;其主要目的是更早的發現問題,有助於維持開發流程的健康狀態。
方法論設計理念上,pre-commit hooks 的作用域僅限 local,其他人不應該有權限限制你 local 的 git 指令,如強硬限制你必須使用的 hooks。急於 commit 可以使用 git commit --no-verify 繞過檢查,雖然這種作法有點不健康。

如果要強硬擋下的話應該使用 server 端的 pre-receive hook。Server 端的檢查雖然有強制性,但也有速度較慢、可能無法重現等缺點。

2. 在開發團隊中維護 hooks,在大量 repositories 中設定 pre-commit 的方法?
Git 2.9 之後有 "global" hooks 的功能,但還是不大適合用於所有 repositories 的設定。 由於存放 Git hooks 的 .git/hooks 目錄不會隨著專案複製,比較適合的做法還是另外開一個 repository 分享存放設定檔,讓團隊中的成員可以自行拷貝或連結進行設定,也比較方便追蹤 hooks 的更動。

Supplement
Getting Started with Python Pre-commit Hooks
Define rules to reject commits that don’t meet your requirements



2021年5月1日 星期六

[ 常見問題 ] 解決”/bin/rm: Argument list too long”刪除文件錯誤

 HowTo

執行 rm 後出現 ”/bin/rm: Argument list too long” 報錯,應該是目錄中的文件太多、處理時間太長導致的。
# ls | xargs -n 10 rm -rf

通過 xargs 命令,將文件 10 個分成一組,然後分批刪除,這樣就不會出現錯誤了。

備註信息:我們執行上面命令的時候,一定要謹慎,要在當前需要刪除文件的目錄下執行,不要在其他目錄中操作,要不你會刪除其他目錄文件的。

2021年4月29日 星期四

[ Python 文章收集 ] The Curious Case of Python's Context Manager

 Source From Here

Preface
Discovering the quirks of Python's context manager

Python’s context managers are great for resource management and stopping the propagation of leaked abstractions. You’ve probably used it while opening a file or a database connection. Usually it starts with a with statement like this:
  1. with open("file.txt""wt") as f:  
  2.     f.write("contents go here")  
In the above case, file.txt gets automatically closed when the execution flow goes out of the scope. This is equivalent to writing:
  1. try:  
  2.     f = open("file.txt""wt")  
  3.     text = f.write("contents go here")  
  4. finally:  
  5.     f.close()  
Writing Custom Context Managers
To write a custom context manager, you need to create a class that includes the __enter__ and __exit__ methods. Let’s recreate a custom context manager that will execute the same workflow as above:
  1. class CustomFileOpen:  
  2.     """Custom context manager for opening files."""  
  3.   
  4.     def __init__(self, filename, mode):  
  5.         self.filename = filename  
  6.         self.mode = mode  
  7.   
  8.     def __enter__(self):  
  9.         self.f = open(self.filename, self.mode)  
  10.         return self.f  
  11.   
  12.     def __exit__(self, *args):  
  13.         self.f.close()  
You can use the above class just like a regular context manager:
  1. with CustomFileOpen("file.txt""wt") as f:  
  2.     f.write("contents go here")  
From Generators to Context Managers
Creating context managers by writing a class with __enter__ and __exit__ methods, is not difficult. However, you can achieve better brevity by defining them using contextlib.contextmanager decorator. This decorator converts a generator function into a context manager. The blueprint for creating context manager decorators goes something like this:
  1. @contextmanager  
  2. def some_generator(<arguments>):  
  3.     <setup>  
  4.     try:  
  5.         yield <value>  
  6.     finally:  
  7.         <cleanup>  
When you use the context manager with the with statement:
  1. with some_generator(<arguments>) as <variable>:  
  2.     <body>  
It roughly translates to:
  1. <setup>  
  2. try:  
  3.     <variable> = <value>  
  4.     <body>  
  5. finally:  
  6.     <cleanup>  
The setup code goes before the try..finally block. Notice the point where the generator yields. This is where the code block nested in the with statement gets executed. After the completion of the code block, the generator is then resumed. If an unhandled exception occurs in the block, it’s re-raised inside the generator at the point where the yield occurred and then the finally block is executed. If no unhandled exception occurs, the code gracefully proceeds to the finally block where you run your cleanup code.

Let’s implement the same CustomFileOpen context manager with contextmanager decorator:
  1. from contextlib import contextmanager  
  2.   
  3. @contextmanager  
  4. def CustomFileOpen(filename, method):  
  5.     """Custom context manager for opening a file."""  
  6.   
  7.     f = open(filename, method)  
  8.     try:  
  9.         yield f  
  10.   
  11.     finally:  
  12.         f.close()  
Writing Context Managers as Decorators
You can use context managers as decorators also. To do so, while defining the class, you have to inherit from contextlib.ContextDecorator class. Let’s make a RunTime decorator that will be applied on a file-opening function. The decorator will:
* Print a user provided description of the function
* Print the time it takes to run the function

  1. from contextlib import ContextDecorator  
  2. from time import time  
  3.   
  4. class RunTime(ContextDecorator):  
  5.     """Timing decorator."""  
  6.   
  7.     def __init__(self, description):  
  8.         self.description = description  
  9.   
  10.     def __enter__(self):  
  11.         print(self.description)  
  12.         self.start_time = time()  
  13.   
  14.     def __exit__(self, *args):  
  15.         self.end_time = time()  
  16.         run_time = self.end_time - self.start_time  
  17.         print(f"The function took {run_time} seconds to run.")  
You can use the decorator like this:
  1. @RunTime("This function opens a file")  
  2. def custom_file_write(filename, mode, content):  
  3.     with open(filename, mode) as f:  
  4.         f.write(content)  
Using the function like this should return:
>>> print(custom_file_write("file.txt", "wt", "jello"))
This function opens a file
The function took 0.0005390644073486328 seconds to run.
None

You can also create the same decorator via contextlib.contextmanager decorator.
  1. from contextlib import contextmanager  
  2.   
  3. @contextmanager  
  4. def runtime(description):  
  5.   
  6.     print(description)  
  7.     start_time = time()  
  8.     try:  
  9.         yield  
  10.     finally:  
  11.         end_time = time()  
  12.         run_time = end_time - start_time  
  13.         print(f"The function took {run_time} seconds to run.")  
Nesting Contexts
You can nest multiple context managers to manage resources simultaneously. Consider the following dummy manager:
  1. from contextlib import contextmanager  
  2.   
  3.   
  4. @contextmanager  
  5. def get_state(name):  
  6.     print("entering:", name)  
  7.     yield name  
  8.     print("exiting :", name)  
  9.   
  10.   
  11. # multiple get_state can be nested like this  
  12. with get_state("A") as A, get_state("B") as B, get_state("C") as C:  
  13.     print("inside with statement:", A, B, C)  
Output:
entering: A
entering: B
entering: C
inside with statement: A B C
exiting : C
exiting : B
exiting : A

Notice the order they’re closed. Context managers are treated as a stack, and should be exited in reverse order in which they’re entered. If an exception occurs, this order matters, as any context manager could suppress the exception, at which point the remaining managers will not even get notified of this. The __exit__ method is also permitted to raise a different exception, and other context managers then should be able to handle that new exception.

Combining Multiple Context Managers
You can combine multiple context managers too. Let’s consider these two managers:
  1. from contextlib import contextmanager  
  2.   
  3.   
  4. @contextmanager  
  5. def a(name):  
  6.     print("entering a:", name)  
  7.     yield name  
  8.     print("exiting a:", name)  
  9.   
  10.   
  11. @contextmanager  
  12. def b(name):  
  13.     print("entering b:", name)  
  14.     yield name  
  15.     print("exiting b:", name)  
Now combine these two using the decorator syntax. The following function takes the above define managers a and b and returns a combined context manager ab.
  1. @contextmanager  
  2. def ab(a, b):  
  3.     with a("A") as A, b("B") as B:  
  4.         yield (A, B)  
This can be used as:
  1. with ab(a, b) as AB:  
  2.     print("Inside the composite context manager:", AB)  
Output:
entering a: A
entering b: B
Inside the composite context manager: ('A', 'B')
exiting b: B
exiting a: A

If you have variable numbers of context managers and you want to combine them gracefully, contextlib.ExitStack is here to help. Let’s rewrite context manager ab using ExitStack. This function takes the individual context managers and their arguments as tuples and returns the combined manager:
  1. from contextlib import contextmanager, ExitStack  
  2.   
  3.   
  4. @contextmanager  
  5. def ab(cms, args):  
  6.     with ExitStack() as stack:  
  7.         yield [stack.enter_context(cm(arg)) for cm, arg in zip(cms, args)]  
Then you can use it this way:
  1. with ab((a, b), ("A""B")) as AB:  
  2.     print("Inside the composite context manager:", AB)  
Output:
entering a: A
entering b: B
Inside the composite context manager: ['A', 'B']
exiting b: B
exiting a: A

ExitStack can be also used in cases where you want to manage multiple resources gracefully. For example, suppose, you need to create a list from the contents of multiple files in a directory. Let’s see, how you can do so while avoiding accidental memory leakage with robust resource management:
  1. from contextlib import ExitStack  
  2. from pathlib import Path  
  3.   
  4. # ExitStack ensures all files are properly closed after o/p  
  5. with ExitStack() as stack:  
  6.     streams = (  
  7.         stack.enter_context(open(fname, "r")) for fname in Path("src").rglob("*.py")  
  8.     )  
  9.     contents = [f.read() for f in streams]  
Using Context Managers to Create SQLAlchemy Session
If you are familiar with SQLALchemy, Python’s SQL toolkit and Object Relational Mapper, then you probably know the usage of Session to run a query. A Session basically turns any query into a transaction and makes it atomic. Context managers can help you write a transaction session in a very elegant way. A basic querying workflow in SQLAlchemy may look like this:
  1. from sqlalchemy import create_engine  
  2. from sqlalchemy.orm import sessionmaker  
  3. from contextlib import contextmanager  
  4.   
  5. # an Engine, which the Session will use for connection resources  
  6. some_engine = create_engine("sqlite://")  
  7.   
  8. # create a configured "Session" class  
  9. Session = sessionmaker(bind=some_engine)  
  10.   
  11.   
  12. @contextmanager  
  13. def session_scope():  
  14.     """Provide a transactional scope around a series of operations."""  
  15.     session = Session()  
  16.     try:  
  17.         yield session  
  18.         session.commit()  
  19.     except:  
  20.         session.rollback()  
  21.         raise  
  22.     finally:  
  23.         session.close()  
The excerpt above creates an in memory SQLite connection and a session_scope function with context manager. The session_scope function takes care of committing and rolling back in case of exception automatically. The session_scope function can be used to run queries in the following way:
  1. with session_scope() as session:  
  2.     myobject = MyObject("foo""bar")  
  3.     session.add(myobject)  


[Git 文章收集] GitLab -更新 Fork 出來的專案

  Source From  Here Question 在 GitLab 的介面上,可以透過 Fork 的功能,把別人的專案建立一個 fork 到自己的帳號底下,例如原始專案的網址是  http://gitlab/userA/project.git,  Fork 出來的專案網址...