2020年8月11日 星期二

[ Python 文章收集 ] Jupyter Magic - Timing(%%time %time %timeit)

 Source From Here

Preface
對於計時有兩個十分有用的魔法命令:%%time 和 %timeit如果你有些代碼運行地十分緩慢,而你想確定是否問題出在這裡,這兩個命令將會非常方便。

1.%%time 將會給出 cell 的代碼運行一次所花費的時間。
  1. %%time  
  2. import time  
  3. for _ in range(1000):  
  4.     time.sleep(0.01)# sleep for 0.01 seconds  
Output:
CPU times: user 196 ms, sys: 21.4 ms, total: 217 ms
Wall time: 11.6 s

注: window 下好像只能顯示 “Wall time”, Ubuntu16.4 可以正常顯示,其他系統未進行測試

2.%time 將會給出當前行的代碼運行一次所花費的時間。
  1. import numpy  
  2. %time numpy.random.normal(size=1000)  
Output:
Wall time: 1e+03 µs

3.%timeit 使用Python的timeit模塊,它將會執行一個語句 100,000次 (默認情況下),然後給出運行最快3次的平均值。
  1. import numpy  
  2. %timeit numpy.random.normal(size=100)  
Output:
12.8 µs ± 1.25 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Supplement
Jupyter - Built-in magic commands

2020年8月7日 星期五

[ 文章收集 ] Testing with GoMock: A Tutorial

 Source From Here

Preface
This is a quick tutorial on how to test code using the GoMock mocking library and the standard library testing package testingGoMock is a mock framework for Go. It enjoys a somewhat official status as part of the github.com/golang organization, integrates well with the built-in testing package, and provides a flexible expectation API.

The code snippets referenced in this post are available on GitHub: github.com/sgreben/testing-with-gomock

Installation
First, we need to install the gomock package github.com/golang/mock/gomock as well as the mockgen code generation tool github.com/golang/mock/mockgen. Technically, we could do without the code generation tool, but then we’d have to write our mocks by hand, which is tedious and error-prone.

Both packages can be installed using go get:
# go get github.com/golang/mock/gomock
# go get github.com/golang/mock/mockgen

We can verify that the mockgen binary was installed successfully by running:
# $GOPATH/bin/mockgen

This should output usage information and a flag list. Now that we’re all set up, we’re ready to test some code!

Basic Usage
Usage of GoMock follows four basic steps:
1. Use mockgen to generate a mock for the interface you wish to mock.
2. In your test, create an instance of gomock.Controller and pass it to your mock object’s constructor to obtain a mock object.
3. Call EXPECT() on your mocks to set up their expectations and return values
4. Call Finish() on the mock controller to assert the mock’s expectations

Let’s look at a small example to demonstrate the above workflow. To keep things simple, we’ll be looking at just two files — an interface Doer in the file doer/doer.go that we wish to mock and a struct User in user/user.go that uses the Doer interface. The interface that we wish to mock is just a couple of lines — it has a single method DoSomething that does something with an int and a string and returns an error:
doer/doer.go
  1. package doer  
  2.   
  3. type Doer interface {  
  4.     DoSomething(int, string) error  
  5. }  
Here’s the code that we want to test while mocking out the Doer interface:
  1. package user  
  2.   
  3. import "doer"  
  4.   
  5. type User struct {  
  6.     Doer doer.Doer  
  7. }  
  8.   
  9. func (u *User) Use() error {  
  10.     return u.Doer.DoSomething(123"Hello GoMock")  
  11. }  
Before all, let's clone the the project:
# git clone https://github.com/sgreben/testing-with-gomock.git
# cd $GOPATH/src/testing-with-gomock // enter root of project
# tree .

Our current project layout looks as follows:

We start by creating a directory mocks that will contain our mock implementations and then running mockgen on the doer package:
# rm -f mocks/* // Remove any exist file(s)
# # mockgen -destination=mocks/mock_doer.go -package=mocks testing-with-gomock/doer Doer
# cat mocks/mock_doer.go

Here, we have to create the directory mocks ourselves because GoMock won’t do it for us and will quit with an error instead. Here’s what the arguments given to mockgen mean:
1. -destination=mocks/mock_doer.go: put the generated mocks in the file path mocks/mock_doer.go.
2. -package=mocks: put the generated mocks in the package mocks
3. testing-with-gomock/doer: generate mocks for this package (search from $GOPATH)
4. Doer: generate mocks for this interface. This argument is required — we need to specify the interfaces to generate mocks for explicitly. We can, however specify multiple interfaces here as a comma-separated list (e.g. Doer1,Doer2).

If $GOPATH/bin was not in our $PATH, we’d have to call mockgen via $GOPATH/bin/mockgen. In the following we’ll assume that we have $GOPATH/bin in our $PATH.

As a result, our invocation of mockgen places a file mocks/mock_doer.go in our project. This is how such a generated mock implementation looks:
mocks/mock_doer.go
  1. // Code generated by MockGen. DO NOT EDIT.  
  2. // Source: testing-with-gomock/doer (interfaces: Doer)  
  3.   
  4. // Package mocks is a generated GoMock package.  
  5. package mocks  
  6.   
  7. import (  
  8.     reflect "reflect"  
  9.   
  10.     gomock "github.com/golang/mock/gomock"  
  11. )  
  12.   
  13. // MockDoer is a mock of Doer interface.  
  14. type MockDoer struct {  
  15.     ctrl     *gomock.Controller  
  16.     recorder *MockDoerMockRecorder  
  17. }  
  18.   
  19. // MockDoerMockRecorder is the mock recorder for MockDoer.  
  20. type MockDoerMockRecorder struct {  
  21.     mock *MockDoer  
  22. }  
  23.   
  24. // NewMockDoer creates a new mock instance.  
  25. func NewMockDoer(ctrl *gomock.Controller) *MockDoer {  
  26.     mock := &MockDoer{ctrl: ctrl}  
  27.     mock.recorder = &MockDoerMockRecorder{mock}  
  28.     return mock  
  29. }  
  30.   
  31. // EXPECT returns an object that allows the caller to indicate expected use.  
  32. func (m *MockDoer) EXPECT() *MockDoerMockRecorder {  
  33.     return m.recorder  
  34. }  
  35.   
  36. // DoSomething mocks base method.  
  37. func (m *MockDoer) DoSomething(arg0 int, arg1 string) error {  
  38.     m.ctrl.T.Helper()  
  39.     ret := m.ctrl.Call(m, "DoSomething", arg0, arg1)  
  40.     ret0, _ := ret[0].(error)  
  41.     return ret0  
  42. }  
  43.   
  44. // DoSomething indicates an expected call of DoSomething.  
  45. func (mr *MockDoerMockRecorder) DoSomething(arg0, arg1 interface{}) *gomock.Call {  
  46.     mr.mock.ctrl.T.Helper()  
  47.     return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DoSomething", reflect.TypeOf((*MockDoer)(nil).DoSomething), arg0, arg1)  
  48. }  
Note that the generated EXPECT() method is defined on the same object as the mock methods (in this case, DoSomething) — avoiding name clashes here is likely a reason for the non-standard all-uppercase name.

Next, we define a mock controller inside our test. mock controller is responsible for tracking and asserting the expectations of its associated mock objects.

We can obtain a mock controller by passing a value t of type *testing.T to its constructor, and then use it to construct a mock of the Doer interface. We also defer its Finish
  1. method — more on this later.  
  2. mockCtrl := gomock.NewController(t)  
  3. defer mockCtrl.Finish()  
  4.   
  5. mockDoer := mocks.NewMockDoer(mockCtrl)  
Suppose we want to assert that mockerDoer‘s Do method will be called once, with 123 and "Hello GoMock" as arguments, and will return nil. To do this, we can now call EXPECT() on the mockDoer to set up its expectations in our test. The call to EXPECT() returns an object (called a mock recorder) providing methods of the same names as the real object.

Calling one of the methods on the mock recorder specifies an expected call with the given arguments. You can then chain other properties onto the call, such as:
* the return value (via .Return(...))
* the number of times this call is expected to occur (via .Times(number), or via .MaxTimes(number) and .MinTimes(number))

In our case, the call looks like this:
  1. // Expect Do to be called once with 123 and "Hello GoMock" as parameters, and return nil from the mocked call.  
  2. mockDoer.EXPECT().DoSomething(123"Hello GoMock").Return(nil).Times(1)  
That’s it – we’ve now specified our first mock call! Here’s the complete example:
- user/user_test.go
  1. package user_test  
  2.   
  3. import (  
  4.     "testing"  
  5.   
  6.     "github.com/golang/mock/gomock"  
  7.     "testing-with-gomock/mocks"  
  8.     "testing-with-gomock/user"  
  9. )  
  10.   
  11. func TestUse(t *testing.T) {  
  12.     mockCtrl := gomock.NewController(t)  
  13.     defer mockCtrl.Finish()  
  14.   
  15.     mockDoer := mocks.NewMockDoer(mockCtrl)  
  16.     testUser := &user.User{Doer: mockDoer}  
  17.   
  18.     // Expect Do to be called once with 123 and "Hello GoMock" as parameters, and return nil from the mocked call.  
  19.     mockDoer.EXPECT().DoSomething(123"Hello GoMock").Return(nil).Times(1)  
  20.   
  21.     testUser.Use()  
  22. }  
Probably, it’s not obvious where in the code the mock’s expectations are asserted. This happens in the deferred Finish()It’s idiomatic to defer this call to Finish at the point of declaration of the mock controller — this way we don’t forget to assert the mock expectations later.

Finally, we’re ready to run our tests:
# go test -v testing-with-gomock/user
=== RUN TestUse
--- PASS: TestUse (0.00s)
PASS
ok testing-with-gomock/user 0.001s

If you need to construct more than one mock, you can reuse the mock controller — its Finish method will then assert the expectations of all mocks associated with the controller.

We might also want to assert that the value returned by the Use method is indeed the one returned to it by DoSomethingWe can write another test, creating a dummy error and then specifying it as a return value for mockDoer.DoSomething:
user/user_test.go
  1. func TestUseReturnsErrorFromDo(t *testing.T) {  
  2.     mockCtrl := gomock.NewController(t)  
  3.     defer mockCtrl.Finish()  
  4.   
  5.     dummyError := errors.New("dummy error")  
  6.     mockDoer := mocks.NewMockDoer(mockCtrl)  
  7.     testUser := &user.User{Doer:mockDoer}  
  8.   
  9.     // Expect Do to be called once with 123 and "Hello GoMock" as parameters, and return dummyError from the mocked call.  
  10.     mockDoer.EXPECT().DoSomething(123"Hello GoMock").Return(dummyError).Times(1)  
  11.   
  12.     err := testUser.Use()  
  13.   
  14.     if err != dummyError {  
  15.         t.Fail()  
  16.     }  
  17. }  
Using GoMock with go:generate
Running mockgen for each package and interface individually is cumbersome when there is a large number of interfaces/packages to mock. To alleviate this problem, the mockgen command may be placed in a special go:generate comment. In our example, we can add a go:generate comment just below the package statement of our doer.go:
doer/doer.go
  1. package doer  
  2.   
  3. //go:generate mockgen -destination=../mocks/mock_doer.go -package=mocks testing-with-gomock/doer Doer  
  4.   
  5. type Doer interface {  
  6.     DoSomething(int, string) error  
  7. }  
Note that at the point where mockgen is called, the current working directory is doer — hence we need to specify ../mocks/ as the directory to write our mocks to, not just mocks/.

We can now comfortably generate all mocks specified by such a comment by running:
# go generate ./...

from the project’s root directory. Note that there is no space between // and go:generate in the comment. This is required for go generate to pick up the comment as an instruction to process. A reasonable policy on where to put the go:generate comment and which interfaces to include is the following:
1. One go:generate comment per file containing interfaces to be mocked
2. Include all interfaces to generate mocks for in the call to mockgen
3. Put the mocks in a package mocks and write the mocks for a file X.go into mocks/mock_X.go.

This way, the mockgen call is close to the actual interfaces, while avoiding the overhead of separate calls and destination files for each interface.

Using argument matchers
Sometimes, you don’t care about the specific arguments a mock is called with. With GoMock, a parameter can be expected to have a fixed value (by specifying the value in the expected call) or it can be expected to match a predicate, called a Matcher. Matchers are used to represent ranges of expected arguments to a mocked method. The following matchers are pre-defined in GoMock:
* gomock.Any(): matches any value (of any type)
* gomock.Eq(x): uses reflection to match values that are DeepEqual to x
* gomock.Nil(): matches nil
* gomock.Not(m): (where m is a Matcher) matches values not matched by the matcher m
* gomock.Not(x): (where x is not a Matcher) matches values not DeepEqual to x

For example, if we don’t care about the value of the first argument to Do, we could write:
  1. mockDoer.EXPECT().DoSomething(gomock.Any(), "Hello GoMock")  
GoMock automatically converts arguments that are not of type Matcher to Eq matchers, so the above call is equivalent to:
  1. mockDoer.EXPECT().DoSomething(gomock.Any(), gomock.Eq("Hello GoMock"))  
You can define your own matchers by implementing the gomock.Matcher interface. For example, a matcher checking an argument’s type could be implemented as follows:
match/oftype.go
  1. package match  
  2.   
  3. import (  
  4.     "reflect"  
  5.     "github.com/golang/mock/gomock"  
  6. )  
  7.   
  8. type ofType struct{ t string }  
  9.   
  10. func OfType(t string) gomock.Matcher {  
  11.     return &ofType{t}  
  12. }  
  13.   
  14. func (o *ofType) Matches(x interface{}) bool {  
  15.     return reflect.TypeOf(x).String() == o.t  
  16. }  
  17.   
  18. func (o *ofType) String() string {  
  19.     return "is of type " + o.t  
  20. }  
We can then use our custom matcher like this:
// Expect Do to be called once with 123 and any string as parameters, and return nil from the mocked call.
  1. mockDoer.EXPECT().  
  2.     DoSomething(123, match.OfType("string")).  
  3.     Return(nil).  
  4.     Times(1)  
We’ve split the above call across multiple lines for readability. For more complex mock calls this is a handy way of making the mock specification more readable. Note that in Go we have to put the dot at the end of each line in a sequence of chained calls. Otherwise, the parser will consider the line ended and we’ll get a syntax error.

Asserting call order
The order of calls to an object is often important. GoMock provides a way to assert that one call must happen after another call, the .After method. For example,
  1. callFirst := mockDoer.EXPECT().DoSomething(1"first this")  
  2. callA := mockDoer.EXPECT().DoSomething(2"then this").After(callFirst)  
  3. callB := mockDoer.EXPECT().DoSomething(2"or this").After(callFirst)  
specifies that callFirst must occur before either callA or callB.

GoMock also provides a convenience function gomock.InOrder to specify that the calls must be performed in the exact order given. This is less flexible than using .After directly, but can make your tests more readable for longer sequences of calls:
  1. gomock.InOrder(  
  2.     mockDoer.EXPECT().DoSomething(1"first this"),  
  3.     mockDoer.EXPECT().DoSomething(2"then this"),  
  4.     mockDoer.EXPECT().DoSomething(3"then this"),  
  5.     mockDoer.EXPECT().DoSomething(4"finally this"),  
  6. )  
Under the hoodInOrder uses .After to chain the calls in sequence.

Specifying mock actions
Mock objects differ from real implementations in that they don’t implement any of their behavior — all they do is provide canned responses at the appropriate moment and record their calls. However, sometimes you need your mocks to do more than that. Here, GoMock‘s Do actions come in handy. Any call may be decorated with an action by calling .Do on the call with a function to be executed whenever the call is matched:
  1. mockDoer.EXPECT().  
  2.     DoSomething(gomock.Any(), gomock.Any()).  
  3.     Return(nil).  
  4.     Do(func(x int, y string) {  
  5.         fmt.Println("Called with x =",x,"and y =", y)  
  6.     })  
Complex assertions about the call arguments can be written inside Do actions. For example, if the first (int) argument of DoSomething should be less than or equal to the length of the second (string) argument, we can write:
  1. mockDoer.EXPECT().  
  2.     DoSomething(gomock.Any(), gomock.Any()).  
  3.     Return(nil).  
  4.     Do(func(x int, y string) {  
  5.         if x > len(y) {  
  6.             t.Fail()  
  7.         }  
  8.     })  
The same functionality could not be implemented using custom matchers, since we are relating the concrete values, whereas matchers only have access to one argument at a time.

Summary
In this post, we’ve seen how to generate mocks using mockgen and how to batch mock generation using go:generate comments and the go generate tool. We’ve covered the expectation API, including argument matchers, call frequency, call order and Do-actions.

Supplement
Youtube - How to mock in your Go tests - Golang Tutorial
Youtube - Unit Tests and Test Doubles like Mocks, Stubs & Fakes

[ Python 文章收集 ] Jupyter Magic - Timing(%%time %time %timeit)

  Source From  Here Preface 對於計時有兩個十分有用的魔法命令: %%time   和  %timeit .  如果你有些代碼運行地十分緩慢,而你想確定是否問題出在這裡 ,這兩個命令將會非常方便。 1.%%time 將會給出 cell 的代碼運行一次所花...