[ ML 文章收集 ] Use Keras Deep Learning Models with Scikit-Learn in Python

 Preface

(article source) Keras is one of the most popular deep learning libraries in Python for research and development because of its simplicity and ease of use. The scikit-learn library is the most popular library for general machine learning in Python.

In this post you will discover how you can use deep learning models from Keras with the scikit-learn library in Python. This will allow you to leverage the power of the scikit-learn library for tasks like model evaluation and model hyper-parameter optimization.

Let's import the necessary packages in advance for later sample code usage:

view plaincopy to clipboardprint?

1. import pandas as pd  
2. import numpy as np  
3. from keras.models import Sequential  
4. from keras.layers import Dense  
5. from keras.wrappers.scikit_learn import KerasClassifier  
6. from sklearn.model_selection import StratifiedKFold  
7. from sklearn.model_selection import cross_val_score  
8. from sklearn.model_selection import GridSearchCV  

Overview
Keras is a popular library for deep learning in Python, but the focus of the library is deep learning. In fact it strives for minimalism, focusing on only what you need to quickly and simply define and build deep learning models.

The scikit-learn library in Python is built upon the SciPy stack for efficient numerical computation. It is a fully featured library for general machine learning and provides many utilities that are useful in the development of deep learning models. Not least:

* Evaluation of models using resampling methods like k-fold cross validation.
* Efficient search and evaluation of model hyper-parameters.

The Keras library provides a convenient wrapper fo...ion estimators in scikit-learn. In the next sections, we will work through examples of using the KerasClassifier wrapper for a classification neural network created in Keras and used in the scikit-learn library.

Data Set
The test problem is the Pima Indians onset of diabetes classification dataset. This is a small dataset with all numerical attributes that is easy to work with. Download the dataset and place it in your currently working directly with the name diabetes.csv (update: download from here).

view plaincopy to clipboardprint?

1. feature_columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']  
2. target_columns = ['Outcome']  
3. pima_df = pd.read_csv('../../datas/kaggle_pima-indians-diabetes-database/diabetes.csv')  
4. pima_df.head()  

Evaluate Deep Learning Models with Cross Validation
The KerasClassifier and KerasRegressor classes in Keras take an argument build_fn which is the name of the function to call to get your model. You must define a function called whatever you like that defines your model, compiles it and returns it.

In the example, below we define a function create_model() that create a simple multi-layer neural network for the problem.

We pass this function name to the KerasClassifier class by the build_fn argument. We also pass in additional arguments of nb_epoch=150 and batch_size=10. These are automatically bundled up and passed on to the fit() function which is called internally by the KerasClassifier class.

In this example, we use the scikit-learn StratifiedKFold to perform 10-fold stratified cross-validation. This is a resampling technique that can provide a robust estimate of the performance of a machine learning model on unseen data.

We use the scikit-learn function cross_val_score() to evaluate our model using the cross-validation scheme and print the results.

view plaincopy to clipboardprint?

1. # Function to create model, required for KerasClassifier  
2. def create_model():  
3.     # create model  
4.     model = Sequential()  
5.     model.add(Dense(12, input_dim=8, activation='relu'))  
6.     model.add(Dense(8, activation='relu'))  
7.     model.add(Dense(1, activation='sigmoid'))  
8.     # Compile model  
9.     model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])  
10.     return model  
11.   
12. # fix random seed for reproducibility  
13. seed = 7  
14. numpy.random.seed(seed)  
15.   
16. # split into input (X) and output (Y) variables  
17. X = pima_df[feature_columns]  
18. Y = pima_df[target_columns]  
19. # create model  
20. model = KerasClassifier(build_fn=create_model, epochs=150, batch_size=10, verbose=0)  
21. # evaluate using 10-fold cross validation  
22. kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)  
23. results = cross_val_score(model, X, Y, cv=kfold)  
24. print(results.mean())  

Output:

0.7319207131862641

Running the example displays the skill of the model for each epoch. A total of 10 models are created and evaluated and the final average accuracy is displayed (73.2%).

Grid Search Deep Learning Model Parameters
The previous example showed how easy it is to wrap your deep learning model from Keras and use it in functions from the scikit-learn library.

In this example, we go a step further. The function that we specify to the build_fn argument when creating the KerasClassifier wrapper can take arguments. We can use these arguments to further customize the construction of the model. In addition, we know we can provide arguments to the fit() function.

In this example, we use a grid search to evaluate different configurations for our neural network model and report on the combination that provides the best-estimated performance.

The create_model() function is defined to take two arguments optimizer and init, both of which must have default values. This will allow us to evaluate the effect of using different optimization algorithms and weight initialization schemes for our network.

After creating our model, we define arrays of values for the parameter we wish to search, specifically:

* Optimizers for searching different weight values.
* Initializers for preparing the network weights using different schemes.
* Epochs for training the model for a different number of exposures to the training dataset.
* Batches for varying the number of samples before a weight update.

The options are specified into a dictionary and passed to the configuration of the GridSearchCV scikit-learn class. This class will evaluate a version of our neural network model for each combination of parameters (2 x 3 x 3 x 3 for the combinations of optimizers, initializations, epochs and batches). Each combination is then evaluated using the default of 3-fold stratified cross validation.

That is a lot of models and a lot of computation. This is not a scheme that you want to use lightly because of the time it will take. It may be useful for you to design small experiments with a smaller subset of your data that will complete in a reasonable time. This is reasonable in this case because of the small network and the small dataset (less than 1000 instances and 9 attributes).

Finally, the performance and combination of configurations for the best model are displayed, followed by the performance of all combinations of parameters:

view plaincopy to clipboardprint?

1. # Function to create model, required for KerasClassifier  
2. def create_model(optimizer='rmsprop', init='glorot_uniform'):  
3.     # create model  
4.     model = Sequential()  
5.     model.add(Dense(12, input_dim=8, kernel_initializer=init, activation='relu'))  
6.     model.add(Dense(8, kernel_initializer=init, activation='relu'))  
7.     model.add(Dense(1, kernel_initializer=init, activation='sigmoid'))  
8.     # Compile model  
9.     model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])  
10.     return model  
11.   
12.   
13. # create model  
14. model = KerasClassifier(build_fn=create_model, verbose=0)  
15. # grid search epochs, batch size and optimizer  
16. optimizers = ['rmsprop', 'adam']  
17. init = ['glorot_uniform', 'normal', 'uniform']  
18. epochs = [50, 100, 150]  
19. batches = [5, 10, 20]  
20. param_grid = dict(  
21.     optimizer=optimizers,  
22.     epochs=epochs,  
23.     batch_size=batches,   
24.     init=init  
25. )  
26.   
27. grid = GridSearchCV(estimator=model, param_grid=param_grid)  
28. grid_result = grid.fit(X, Y)  
29.   
30. # summarize results  
31. print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))  
32.   
33. means = grid_result.cv_results_['mean_test_score']  
34. stds = grid_result.cv_results_['std_test_score']  
35. params = grid_result.cv_results_['params']  
36.   
37. for mean, stdev, param in zip(means, stds, params):  
38.     print("%f (%f) with: %r" % (mean, stdev, param))  

Output:

Best: 0.764383 using {'batch_size': 5, 'epochs': 100, 'init': 'normal', 'optimizer': 'adam'}

This might take about 5 minutes to complete on your workstation executed on the CPU (rather than GPU). running the example shows the results below.

We can see that the grid search discovered that using a normal initialization scheme, adam optimizer, 100 epochs and a batch size of 5 achieved the best cross-validation score of approximately 76% on this problem.

For a fuller example of tuning hyperparameters with Keras, see the tutorial:
* How to Grid Search Hyperparameters for Deep Learning Models in Python With Keras