[ ML 文章收集 ] Keras - Plot training, validation and test set accuracy

 Source From Here

Question
As title.

HowTo
First step, let's prepare some fake training/test data:

view plaincopy to clipboardprint?

1. from sklearn.datasets import make_classification  
2. from sklearn.linear_model import LogisticRegression  
3. from sklearn.model_selection import train_test_split  
4. from sklearn.metrics import roc_curve  
5. from sklearn.metrics import roc_auc_score  
6. from matplotlib import pyplot  
7.   
8. # Generate 2 class dataset  
9. X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)  
10. # Split into train/test sets  
11. trainX, testX, trainy, testy = train_test_split(X, y, test_size=0.2, random_state=2)  

Check dataset shape:

view plaincopy to clipboardprint?

1. print(f"trainX.shape={trainX.shape}")  

Output:

trainX.shape=(800, 20)

Next is to generate a NN model:

view plaincopy to clipboardprint?

1. import keras  
2. import tensorflow as tf  
3. from keras.models import Sequential  
4. from keras.layers import Dense  
5. from keras.wrappers.scikit_learn import KerasClassifier  
6. from keras import regularizers  
7. from keras.optimizers import SGD  
8. from keras.layers import Dropout  
9. from keras.models import model_from_json  
10. from keras import backend as K  
11. from keras.callbacks import Callback  
12.   
13. def get_nn(input_shape):  
14.     # Initialising the ANN  
15.     classifier = Sequential()  
16.   
17.     # Adding the dropout layer  
18.     classifier.add(Dropout(0.01, input_shape=(input_shape,)))  
19.   
20.     # Adding the input layer and the first hidden layer  
21.     classifier.add(Dense(input_shape, kernel_initializer='uniform', activation='sigmoid'))  
22.   
23.     # Adding the input layer and the first hidden layer  
24.     classifier.add(Dense(input_shape*2, kernel_initializer='uniform', activation='sigmoid'))  
25.   
26.     # Adding the second hidden layer  
27.     classifier.add(  
28.         Dense(  
29.             units=int(input_shape),  
30.             kernel_initializer = 'uniform',  
31.             activation = 'sigmoid',  
32.             activity_regularizer=regularizers.l1(10e-5)  
33.         )  
34.     )  
35.   
36.     # Adding the third hidden layer  
37.     classifier.add(  
38.         Dense(  
39.             units=int(input_shape/2),  
40.             kernel_initializer = 'uniform',  
41.             activation = 'sigmoid',  
42.             activity_regularizer=regularizers.l1(10e-5)  
43.         )  
44.     )  
45.       
46.     # Adding the output layer  
47.     classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))  
48.   
49.     # Compiling the ANN  
50.     # https://keras.io/api/optimizers/  
51.     # opt = SGD(lr=0.01, momentum=0.9)  
52.     m_preccision = keras.metrics.Precision()  
53.     m_recall = keras.metrics.Recall()  
54.     # classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = [m_preccision, m_recall])  
55.     classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy', tf.keras.metrics.Precision()])  
56.       
57.     return classifier  
58.       
59. nn_model = get_nn(trainX.shape[1])  
60. nn_model.summary()  

Output:

view plaincopy to clipboardprint?

1. Model: "sequential_5"  
2. _________________________________________________________________  
3. Layer (type)                 Output Shape              Param #     
4. =================================================================  
5. dropout_5 (Dropout)          (None, 20)                0           
6. _________________________________________________________________  
7. dense_23 (Dense)             (None, 20)                420         
8. _________________________________________________________________  
9. dense_24 (Dense)             (None, 40)                840         
10. _________________________________________________________________  
11. dense_25 (Dense)             (None, 20)                820         
12. _________________________________________________________________  
13. dense_26 (Dense)             (None, 10)                210         
14. _________________________________________________________________  
15. dense_27 (Dense)             (None, 1)                 11          
16. =================================================================  
17. Total params: 2,301  
18. Trainable params: 2,301  
19. Non-trainable params: 0  

It is time for training and collect evaluation result:

view plaincopy to clipboardprint?

1. from matplotlib import pyplot as plt  
2.   
3. history = nn_model.fit(trainX, trainy, validation_split=0.1, epochs=50, batch_size=4)  
4. print(f"history.history.key()={history.history.keys()}")  

Output:

...
Epoch 50/50
180/180 [==============================] - 0s 3ms/step - loss: 0.1062 - accuracy: 0.9764 - precision_5: 0.9780 - val_loss: 0.4934 - val_accuracy: 0.8625 - val_precision_5: 0.8611
history.history.key()=dict_keys(['loss', 'accuracy', 'precision_5', 'val_loss', 'val_accuracy', 'val_precision_5'])

Now we can check out accuracy trending chart:

view plaincopy to clipboardprint?

1. plt.rcParams['figure.figsize'] = [10, 5]  
2. plt.plot(history.history['accuracy'])  
3. plt.plot(history.history['val_accuracy'])  
4. plt.title('model accuracy')  
5. plt.ylabel('accuracy')  
6. plt.xlabel('epoch')  
7. plt.legend(['train', 'val'], loc='upper left')  
8. plt.show()  

And then the loss trending chart:

view plaincopy to clipboardprint?

1. plt.plot(history.history['loss'])  
2. plt.plot(history.history['val_loss'])  
3. plt.title('model loss')  
4. plt.ylabel('loss')  
5. plt.xlabel('epoch')  
6. plt.legend(['train_loss', 'val_loss'], loc='upper left')  
7. plt.show()  

For the complete example, refer to this notebook.

Supplement
* How to Use ROC Curves and Precision-Recall Curves for Classification in Python
* How to make inline plots in Jupyter Notebook larger?