[ Python 常見問題 ] How to tell if tensorflow is using gpu acceleration from inside python shell?

 Source From Here

Question
I have installed tensorflow in my ubuntu 16.04 using the second answer here with ubuntu's builtin apt cuda installation.

Now my question is how can I test if tensorflow is really using gpu? I have a gtx 960m gpu. When I import tensorflow this is the output:

I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally

Is this output enough to check if tensorflow is using gpu ?

HowTo
To find out which device is used, you can enable log device placement like this:

>>> import tensorflow as tf
>>> sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(log_device_placement=True))
...
2020-10-14 12:06:45.480226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-10-14 12:06:45.480280: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-14 12:06:45.481261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-14 12:06:45.481291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-10-14 12:06:45.481301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-10-14 12:06:45.481455: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-14 12:06:45.481841: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-14 12:06:45.482189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 293 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
...

[ Python 常見問題 ] Can Keras with Tensorflow backend be forced to use CPU or GPU at will?

 Source From Here

Question
I have Keras installed with the Tensorflow backend and CUDA. I'd like to sometimes on demand force Keras to use CPU. Can this be done without say installing a separate CPU-only Tensorflow in a virtual environment? If so how? If the backend were Theano, the flags could be set, but I have not heard of Tensorflow flags accessible via Keras.

HowTo
If you want to force Keras to use CPU:

Way1

view plaincopy to clipboardprint?

1. import os  
2. os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"   # see issue #152  
3. os.environ["CUDA_VISIBLE_DEVICES"] = ""  

before Keras / Tensorflow is imported.

Way 2
Run your script as

$ CUDA_VISIBLE_DEVICES="" ./your_keras_code.py

See also
https://github.com/keras-team/keras/issues/152
https://github.com/fchollet/keras/issues/4613

[ 常見問題 ] Substring: How to Split a String

 Source From Here

Preface
In the example below we are looking at how to take the first x number of characters from the start of a string. If we know a character we want to separate on, like a space, we can use strings.Split() instead. But for this we’re looking to get the first 6 characters as a new string.

HowTo
To do this we first convert it into a rune, allowing for better support in different languages and allowing us to use it like an array. Then we pick the first characters using [0:6] and converting it back to a string.

Split Based on Position:

view plaincopy to clipboardprint?

1. package main  
2.   
3. import (  
4.     "fmt"  
5. )  
6.   
7. func main() {  
8.   
9.     myString := "Hello! This is a golangcode.com test ;)"  
10.   
11.     // Step 1: Convert it to a rune  
12.     a := []rune(myString)  
13.   
14.     // Step 2: Grab the num of chars you need  
15.     myShortString := string(a[0:6])  
16.   
17.     fmt.Println(myShortString)  
18. }  

Output:

Hello!

Split Based on Character:
The alternative way, using the strings package would be:

view plaincopy to clipboardprint?

1. package main  
2.   
3. import (  
4.     "fmt"  
5.     "strings"  
6. )  
7.   
8. func main() {  
9.   
10.     myString := "Hello! This is a golangcode.com test ;)"  
11.   
12.     strParts := strings.Split(myString, "!")  
13.   
14.     fmt.Println(strParts[0])  
15. }  

Output:

Hello

Supplement
* The Go Blog - Strings, bytes, runes and characters in Go

The Go language defines the word rune as an alias for the type int32, so programs can be clear when an integer value represents a code point. Moreover, what you might think of as a character constant is called a rune constant in Go...

[ 文章收集 ] How To Install Go 1.15 on Fedora 32/31 & CentOS/RHEL 8/7

 Source From Here

Preface
Go is an open source programming language developed by a team at Google. It provides easy to build simple, reliable, and efficient software’s. This language is designed for writing servers, that’s why it is using widely these days. Go has released latest version 1.15 in Aug 2020.

This tutorial will help you to install Go 1.15 on your Fedora 32/31/30/29/28/27 & CentOS/RHEL 8/7/6 systems. You can also use tutorials to install Go lang on Ubuntu and Debian systems.

Prerequisites
Log in to your Red Hat or its derivative system using ssh and upgrade to apply the latest security updates there. Execute the following command on the terminal.

# yum update

Step 1 – Install Go 1.15
Now download the Go language binary archive file using the following link. To find and download latest version available or 32 bit version go to official download page.

# wget https://dl.google.com/go/go1.15.2.linux-amd64.tar.gz

Now extract the downloaded archive and install it to the desired location on your system. For this tutorial, I am installing it under /usr/local directory. You can also put this under the home directory (for shared hosting) or other location.

# tar -xzf go1.15.2.linux-amd64.tar.gz
# mv go /usr/local

Step 2 – Setup Go Environment
Now you need to set up Go language environment variables for your project. Commonly you need to set 3 environment variables as GOROOT, GOPATH and PATH.

* GOROOT is the location where Go package is installed on your system

# export GOROOT=/usr/local/go

* GOPATH is the location of your work directory. For example my project directory is ~/Projects/Proj1 .

# export GOPATH=$HOME/Projects/Proj1

* Now set the PATH variable to access go binary system wide

# export PATH=$GOPATH/bin:$GOROOT/bin:$PATH

All the above environment will be set for your current session only. To make it permanent add above commands in ~/.bash_profile file.

Step 3 – Verify Installation
At this step, you have successfully installed and configured go language on your system. First, use the following command to check the Go version.

# go version
go version go1.15.2 linux/amd64

Now also verify all configured environment variables using following command.

# go env

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
...

[ Python 常見問題 ] Explicitly select items from a list or tuple

 Source From Here

Question
I have the following Python list (can also be a tuple):

view plaincopy to clipboardprint?

1. myList = ['foo', 'bar', 'baz', 'quux']  

How do I explicitly pick out items whose indices have no specific patterns? For example, I want to select [0,2,3]. Or from a very big list of 1000 items, I want to select [87, 342, 217, 998, 500]. Is there some Python syntax that does that? Something that looks like:

>>> myBigList[87, 342, 217, 998, 500]

HowTo
We can leverage list comprehension feature to accomplish it:

>>> my_big_list = list(range(1000))
>>> my_selected_list = [my_big_list[i] for i in [87, 342, 217, 998, 500]]
>>> my_selected_list
[87, 342, 217, 998, 500]

I compared the answers with python 3.8:

[ ML 文章收集 ] The Hyperparameter Cheat Sheet

 Source From Here

Preface
Before we dive in, let’s start with a quick definition. In machine learning, a hyperparameter (sometimes called a tuning or training parameter) is defined as any parameter whose value is set/chosen at the onset of the learning process. Whereas other parameter values are computed during training.

In this blog, we will discuss some of the important hyperparameters involved in the following machine learning classifiers: K-Nearest Neighbors, Decision Trees and Random Forests, AdaBoost and Gradient Boost, and Support Vector Machines. Specifically, I will focus on the hyperparameters that tend to have the greatest effect on the bias-variance tradeoff. Please note that the below lists are by no means meant to be exhaustive, and I encourage everyone to research each parameter on their own.

This blog assumes a basic understanding of each classifier, so we will skip a theory overview and mostly dive right into the tuning process utilizing Scikit Learn.

K-Nearest Neighbors (KNN)
In the KNN classifier (documentation), a data point is labeled based on its proximity to its neighbors. But how many neighbors should be considered in the classification?

N_neighbors (K)
Simply put, K is the number of neighbors that defines an unlabeled datapoint’s classification boundary.

At K=3, the black datapoint is labeled as red since red has blue outnumbered 2:1

K takes in a range of integers (default = 5), finds the K-nearest neighbors, calculates the distance from each unlabeled point to those K-neighbors. How distance is calculated is defined by the metrics parameter explained below.
Notes.

Important Note: K tends to be odd to avoid ties (i.e., if K = 4, this could result in a 2 Yes and 2 No, which would confuse the classifier).

Bias-Variance Tradeoff: in general, the smaller the K, the tighter the fit (of the model). Therefore, by decreasing K, you are decreasing bias and increasing variance, which leads to a more complex model.

Other Parameters for Consideration

* Leaf_size

Leaf_size determines how many observations are captured in each leaf of either the BallTree of KDTree algorithms, which ultimately make the classification. The default equals 30. You can tune leaf_size by passing in a range of integers, like n_neighbors, to find the optimal leaf size. It is important to note that leaf_size can have a serious effect on run time and memory usage. Because of this, you tend not to run it on leaf_sizes smaller than 30 (smaller leafs equates to more leafs).

* Weights

Weights is the function that weights the data when making a prediction. “Uniform” is an equal weighted function, while “distance” weights the points by the inverse of their distance (i.e., location matters!). Utilizing the “distance” function will result in closer data points having a more significant influence on the classification.

* Metric

Metric can be set to various distance metrics (see here) like Manhattan, Euclidean, Minkowski, or weighted Minkowski (default is “minkowski” with a p=2, which is the Euclidean distance). Which metric you choose is heavily dependent on what question you are trying to answer.

Note: we will skip the algorithm hyperparameter because it is preferred to set this parameter to “auto” so that the computer tells you which tree algorithm is best. I would recommend reading sklearn’s documentation for more information on the differences between the BallTree and KDTree algorithms.

Sample K optimizing code block (assumes F1 is best scoring metric


Decision Trees and Random Forests
When building a Decision Tree (documentation) and/or Random Forest (documentation), there are many important hyperparameters to be considered. Some of these are found in both classifiers while some are just specific to forests. Let’s take a look at the hyperparameters that are most likely to have the largest effect on bias and variance.

* N_estimators

N_estimators (only used in Random Forests) is the number of decision trees used in making the forest (default = 100). Generally speaking, the more uncorrelated trees in our forest, the closer their individual errors get to averaging out. However, more does not mean better since this can have an exponential effect on computation costs. After a certain point, there exists statistical evidence of diminishing returns. Bias-Variance Tradeoff: in theory, the more trees, the more overfit the model (low bias). However, when coupled with bagging, we need not worry.

* Max_depth

Max_depth is an integer that sets the maximum depth of the tree. The default is None, which means the nodes are expanded until all the leaves are pure (i.e., all the data belongs to a single class) or until all leaves contain less than the min_samples_split, which we will define next. Bias-Variance Tradeoff: increasing the max_depth leads to overfitting (low bias)

* Min_samples_split

Min_samples_split is the minimum number of samples required to split an internal node. Bias-Variance Tradeoff: the higher the minimum, the more “clustered” the decision will be, which could lead to underfitting (high bias).

* Min_samples_leaf

Min_samples_leaf defines the minimum number of samples needed at each leaf. The default input here is 1. Bias-Variance Tradeoff: similar to min_samples_split, if you do not allow the model to split (say because your min_samples_lear parameter is set too high) your model could be over generalizing the training data (high bias).

Other Parameters for Consideration
* Criterion

Criterion measures the quality of the split and receives either “gini”, for Gini impurity (default), or “entropy”, for information gain. Gini impurity is the probability of incorrectly classifying a randomly chosen datapoint if it were labeled according to the class distribution of the dataset. Entropy is a measure of chaos in your data set. If a split in the dataset results in lower entropy, then you have gained information (i.e., your data has become more decision useful) and the split is worthy of the additional computational costs.

GridSearchCV Optimization for Random Forests


AdaBoost and Gradient Boosting
By utilizing weak learners (aka “stumps”), boosting algos like AdaBoost (documentation) and Gradient Boosting (documentation) focus on what the model misclassifies. By overweighting these misclassified data points, the model focuses on what it got wrong in order to learn how to get them right.

Similar to Decision Trees and Random Forests, we will focus on the bias-variance tradeoff usual suspects.
* N_estimators

N_estimators is the maximum number of estimators at which boosting is terminated. If a perfect fit is reached, the algo is stopped. The default here is 50. Bias-Variance Tradeoff: the higher the number of estimators in your model the lower the bias.

Other Important Parameters
* Learning_rate

Learning_rate is the rate at which we are adjusting the weights of our model with respect to the loss gradient. In layman’s terms: the lower the learning_rate, the slower we travel along the slope of the loss function. Important note: there is a trade-off between learning_rate and n_estimators as a tiny learning_rate and a large n_estimators will not necessarily improve results relative to the large computational costs.

* Base_estimator/Loss

Base_estimator (AdaBoost) / Loss (Gradient Boosting) is the base estimator from which the boosted ensemble is built. For AdaBoost the default value is None, which equates to a Decision Tree Classifier with max depth of 1 (a stump). For Gradient Boosting the default value is deviance, which equates to Logistic Regression. If “exponential” is passed, the AdaBoost algorithm is used.

GridSearchCV Optimization for AdaBoost:


Support Vector Machines (SVM)
This classifier, which is formally defined by a separating hyperplane (let’s take a minute to appreciate how awesome the word hyperplane is), has many tuning parameters to consider, but we will only focus on three: C, Kernel, and Gamma.


* C

C is the regularization parameter. As the documentation notes, the strength of regularization is inversely proportional to C. Basically, this parameter tells the model how much you want to avoid being wrong. You can think of the inverse of C as your total error budget (summed across all training points), with a lower C value allowing for more error than a higher value of C. Bias-Variance Tradeoff: as previously mentioned, a lower C value allows for more error, which translates to higher bias.

* Gamma

Gamma determines how far the scope of influence of a single training points reaches. A low gamma value allows for points far away from the hyperplane to be considered in its calculation, whereas a high gamma value prioritizes proximity. Bias-Variance Tradeoff: think of gamma as inversely related to K in KNN, the higher the gamma, the tighter the fit (low bias).

A Very Important Hyperparameter

* Kernel

Kernel specifies which kernel should be used. Some of the acceptable strings are “linear”, “poly”, and “rbf”. Linear uses linear algebra to solve for the hyperplane, while poly uses a polynomial to solve for the hyperplane in a higher dimension ([color=brown]see Kernel Trick[/color]). RBF, or the radial basis function kernel, uses the distance between the input and some fixed point (either the origin or some of fixed point c) to make a classification assumption. More information on the Radial Basis Function can be found here.

GridSearchCV Optimization for SVM:


Final Word
We have seen multiple was to train a model using sklearn, specifically GridSearchCV. However, it is very, very important to keep in mind the bias-variance tradeoff, as well as the tradeoff between computational costs and scoring metrics. Ideally, we want a model with low bias and low variance to limit overall error, but is it really worth the extra run-time, memory, etc. for only slight improvements? I will let you answer that.