2017年5月30日 星期二

[Linux 文章收集] 10 Practical Linux Cut Command Examples to Select File Columns

Source From Here
Preface
Linux command cut is used for text processing. You can use this command to extract portion of text from a file by selecting columns. This tutorial provides few practical examples of cut command that you can use in your day to day command line activities. For most of the example, we’ll be using the following test file.
# cat test.txt
cat command for file oriented operations.
cp command for copy files or directories.
ls command to list out files and directories with its attributes.

10 Practical Linux Cut Command Examples

1. Select Column of Characters
To extract only a desired column from a file use -c option. The following example displays 2nd character from each line of a file test.txt
# cut -c2 test.txt
a
p
s

As seen above, the characters a, p, s are the second character from each line of the test.txt file.

2. Select Column of Characters using Range
Range of characters can also be extracted from a file by specifying start and end position delimited with -. The following example extracts first 3 characters of each line from a file called test.txt
# cut -c1-3 test.txt
cat
cp
ls

3. Select Column of Characters using either Start or End Position
Either start position or end position can be passed to cut command with -c option. The following specifies only the start position before the ‘-‘. This example extracts from 3rd character to end of each line from test.txt file.
# cut -c3- test.txt
t command for file oriented operations.
command for copy files or directories.
command to list out files and directories with its attributes.

The following specifies only the end position after the ‘-‘. This example extracts 8 characters from the beginning of each line from test.txt file.
# cut -c-8 test.txt
cat comm
cp comma
ls comma

The entire line would get printed when you don’t specify a number before or after the ‘-‘ as shown below.
# cut -c- test.txt
cat command for file oriented operations.
cp command for copy files or directories.
ls command to list out files and directories with its attributes.

4. Select a Specific Field from a File
Instead of selecting x number of characters, if you like to extract a whole field, you can combine option -f and -d. The option -f specifies which field you want to extract, and the option -d specifies what is the field delimiter that is used in the input file. The following example displays only first field of each lines from /etc/passwd file using the field delimiter : (colon). In this case, the 1st field is the username:
# cut -d':' -f1 /etc/passwd
root
daemon
bin
sys
sync
games
bala

5. Select Multiple Fields from a File
You can also extract more than one fields from a file or stdout. Below example displays username and home directory of users who has the login shell as “/bin/bash”.
# grep "/bin/bash" /etc/passwd | cut -d':' -f1,6
root:/root
bala:/home/bala

To display the range of fields specify start field and end field as shown below. In this example, we are selecting field 1 through 4, 6 and 7:
# grep "/bin/bash" /etc/passwd | cut -d':' -f1-4,6,7
root:x:0:0:/root:/bin/bash
bala:x:1000:1000:/home/bala:/bin/bash

6. Select Fields Only When a Line Contains the Delimiter
In our /etc/passwd example, if you pass a different delimiter other than : (colon), cut will just display the whole line. In the following example, we’ve specified the delimiter as | (pipe), and cut command simply displays the whole line, even when it doesn’t find any line that has | (pipe) as delimiter.
# grep "/bin/bash" /etc/passwd | cut -d'|' -f1
root:x:0:0:root:/root:/bin/bash
bala:x:1000:1000:bala,,,:/home/bala:/bin/bash

But, it is possible to filter and display only the lines that contains the specified delimiter using -s option. The following example doesn’t display any output, as the cut command didn’t find any lines that has | (pipe) as delimiter in the /etc/passwd file.
# grep "/bin/bash" /etc/passwd | cut -d'|' -s -f1

7. Select All Fields Except the Specified Fields
In order to complement the selection field list use option –complement. The following example displays all the fields from /etc/passwd file except field 7:
# grep "/bin/bash" /etc/passwd | cut -d':' --complement -s -f7
root:x:0:0:root:/root
bala:x:1000:1000:bala,,,:/home/bala

8. Change Output Delimiter for Display
By default the output delimiter is same as input delimiter that we specify in the cut -d option. To change the output delimiter use the option –output-delimiter as shown below. In this example, the input delimiter is : (colon), but the output delimiter is # (hash).
# grep "/bin/bash" /etc/passwd | cut -d':' -s -f1,6,7 --output-delimiter='#'
root#/root#/bin/bash
bala#/home/bala#/bin/bash

9. Change Output Delimiter to Newline
In this example, each and every field of the cut command output is displayed in a separate line. We still used –output-delimiter, but the value is $’\n’ which indicates that we should add a newline as the output delimiter.
# grep bala /etc/passwd | cut -d':' -f1,6,7 --output-delimiter=$'\n'
bala
/home/bala
/bin/bash

10. Combine Cut with Other Unix Command Output
The power of cut command can be realized when you combine it with the stdout of some other Unix command. Once you master the basic usage of cut command that we’ve explained above, you can wisely use cut command to solve lot of your text manipulation requirements.

The following example indicates how you can extract only useful information from the ps command output. We also showed how we’ve filtered the output of ps command using grep and sed before the final output was given to cut command. Here, we’ve used cut option -d and -f which we’ve explained in the above examples.
# ps axu | grep python | sed 's/\s\+/ /g' | cut -d' ' -f2,11-
2231 /usr/bin/python /usr/lib/unity-lens-video/unity-lens-video
2311 /usr/bin/python /usr/lib/unity-scope-video-remote/unity-scope-video-remote
2414 /usr/bin/python /usr/lib/ubuntuone-client/ubuntuone-syncdaemon
2463 /usr/bin/python /usr/lib/system-service/system-service-d
3274 grep --color=auto python


2017年5月29日 星期一

[Linux 常見問題] How does one output bold text in Bash?

Source From Here
Question
I'm writing a Bash script that prints some text to the screen:
  1. echo "Some Text"  
Can I format the text? I would like to make it bold.

How-To
The most compatible way of doing this is using tput to discover the right sequences to send to the terminal:
  1. bold=$(tput bold)  
  2. normal=$(tput sgr0)  
then you can use the variables $bold and $normal to format things:
  1. echo "this is ${bold}bold${normal} but this isn't"  
gives
this is bold but this isn't


2017年5月27日 星期六

[Linux 常見問題] How to assign name for a screen?

Source From Here
Question
I'm using the screen multiplexer tool on the command shell and open a lot of screens. I then forget which process ID associates with which task. I would like to set a name for a screen but can't find an option in the manpage.

Currently, listing the screens looks like this:
  1. There are screens on:  
  2.     5422.pts-1.aws1 (Detached)  
  3.     5448.pts-1.aws1 (Detached)  
  4.     5027.pts-1.aws1 (Detached)  
  5. 3 Sockets in /var/run/screen/S-sb.  
And I would like to see something like this:
  1. There are screens on:  
  2.     5422.logCleanWorker (Detached)  
  3.     5448.overNightLongTask(Detached)  
  4.     5027.databaseOverNightLongTask (Detached)  
  5. 3 Sockets in /var/run/screen/S-sb.  

How can I do this?

How-To
To create a new screen with the name foo, use
// -S sessionname: When creating a new session, this option can be used to specify a meaningful name for the session.
# screen -S foo

Then to reattach it, run
// -r sessionowner/[pid.tty.host]: resumes a detached screen session.
# screen -r foo # or use -x, as in
# screen -x foo # for "Multi display mode" (see the man page)


2017年5月26日 星期五

[Toolkit] Tensorflow 常見問題

Version of TensorFlow installed in my system?

Source From Here 
Question 
As title. 

How-To 
Pip installation 
Run: 
# python -c 'import tensorflow as tf; print(tf.__version__)' // for Python 2
# python3 -c 'import tensorflow as tf; print(tf.__version__)' // for Python 3

Virtualenv installation 
pip list | grep tensorflow will also show the version of Tensorflow installed.



I want to Update TensorFlow

Source From Here
Question
I had a TensorFlow V0.10 but I watne to update this version .

How-To
To upgrade any python package, use pip install <pkg_name> --upgrade. So in your case it would be:
# pip install tensorflow --upgrade // Just updated to 1.1.0


2017年5月23日 星期二

[Linux 常見問題] Linux yum command skip updating packages

Source From Here
Question
How can I skip any packages updating while using yum update command in CentOS / RedHat / RHEL / Fedora Linux server system?

How-To
yum command under RedHat / Centos has the -x or –exclude option. It will exclude a specific package by name or shell glob (* or ?) from updates on all repositories.

yum exclude package syntax
Use the following syntax ( you must login as the root user to use following commands):
# yum -x {package-name} update
# yum --exclude={package-name} update

For example, exclude httpd or kernel package from update, enter:
# yum -x httpd* update
# yum --exclude=kernel* update

Below will list available update related to docker and do updating without docker packages:
# yum check-update | grep docker // Check available update related to docker
docker.x86_64 2:1.12.6-16.el7.centos extras
docker-client.x86_64 2:1.12.6-16.el7.centos extras
docker-common.x86_64 2:1.12.6-16.el7.centos extras


# yum -y -x docker* update // Doing update without docker package(s)


2017年5月22日 星期一

[ Python 常見問題 ] How can I create basic timestamps or dates? (Python 3.4)

Source From Here 
Question 
As title. 

How-To 
Ultimately you want to review the datetime documentation and become familiar with the formatting variables, but here are some examples to get you started: 
  1. import datetime  
  2.   
  3. print('Timestamp: {:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now()))  
  4. print('Timestamp: {:%Y-%b-%d %H:%M:%S}'.format(datetime.datetime.now()))  
  5. print('Date now: %s' % datetime.datetime.now())  
  6. print('Date today: %s' % datetime.date.today())  
  7.   
  8. today = datetime.date.today()  
  9. print("Today's date is {:%b, %d %Y}".format(today))  
  10.   
  11. schedule = '{:%b, %d %Y}'.format(today) + ' - 6 PM to 10 PM Pacific'  
  12. schedule2 = '{:%B, %d %Y}'.format(today) + ' - 1 PM to 6 PM Central'  
  13. print('Maintenance: %s' % schedule)  
  14. print('Maintenance: %s' % schedule2)  
Output: 
Timestamp: 2017-05-22 19:36:31
Timestamp: 2017-May-22 19:36:31
Date now: 2017-05-22 19:36:31.683160
Date today: 2017-05-22
Today's date is May, 22 2017
Maintenance: May, 22 2017 - 6 PM to 10 PM Pacific
Maintenance: May, 22 2017 - 1 PM to 6 PM Central

Reference - strftime() and strptime() Behavior

2017年5月21日 星期日

[ Python 常見問題 ] Sorting a dict with tuples as values

Source From Here 
Question 
I have a structure that looks like this: 
  1. {'key_info':(rank,raw_data1,raw_data2),'key_info2':....}  
basically i need back a list of the keys in a sorted order, that is sorted based on the specific field in the tuple

How-To 
With help of API:sorted, you can do it this way: 
>>> test_dict = {}
>>> test_dict['John'] = (18, 'Taiwan')
>>> test_dict['Ken'] = (22, 'American')
>>> test_dict['Peter'] = (16, 'Japan')
>>> test_dict['Mary'] = (30, 'China')
>>> test_dict
{'Ken': (22, 'American'), 'John': (18, 'Taiwan'), 'Peter': (16, 'Japan'), 'Mary': (30, 'China')}
>>> sorted(test_dict.keys(), key=lambda k: test_dict[k][0], reverse=True) # Sorting with age in descending order
['Mary', 'Ken', 'John', 'Peter']
>>> sorted(test_dict.keys(), key=lambda k: test_dict[k][1]) # Sorting with country name in ascending order
['Ken', 'Mary', 'Peter', 'John']
>>> test_dict['Winson'] = (29, 'Canada')
>>> test_dict['Jacy'] = (24, 'China')
>>> sorted(test_dict.keys(), key=lambda k: (test_dict[k][1], test_dict[k][0])) # Sorting with country, then age in ascending order
['Ken', 'Winson', 'Jacy', 'Mary', 'Peter', 'John'] # Jacy=(24, 'China'); Mary=(30, 'China')


Supplement 
Python Doc - Sorting HOW TO 
Sort a list by multiple attributes? 

2017年5月19日 星期五

[ Algorithm ] 96. Unique Binary Search Trees

Source From Here
Question
Given n, how many structurally unique BST's (binary search trees) that store values 1...n? For example, Given n = 3, there are a total of 5 unique BST's.


How-To
First note that dp[k] represents the number of BST trees built from 1....k;

Then assume we have the number of the first 4 trees: dp[1] = 1 ,dp[2] =2 ,dp[3] = 5, dp[4] =14 , how do we get dp[5] based on these four numbers is the core problem here.

The essential process is: to build a tree, we need to pick a root node, then we need to know how many possible left sub trees and right sub trees can be held under that node, finally multiply them.

To build a tree contains {1,2,3,4,5}. First we pick 1 as root, for the left sub tree, there are none; for the right sub tree, we need count how many possible trees are there constructed from {2,3,4,5}, apparently it's the same number as {1,2,3,4}. So the total number of trees under "1" picked as root is dp[0] * dp[4] = 14. (assume dp[0] =1). Similarly, root 2 has dp[1]*dp[3] = 5 trees. root 3 has dp[2]*dp[2] = 4, root 4 has dp[3]*dp[1]= 5 and root 5 has dp[0]*dp[4] = 14. Finally sum the up and it's done.

Now, we may have a better understanding of the dp[k], which essentially represents the number of BST trees with k consecutive nodes. It is used as database when we need to know how many left sub trees are possible for k nodes when picking (k+1) as root.
  1. public int numTrees(int n) {  
  2.     int [] dp = new int[n+1];  
  3.     dp[0]= 1;  
  4.     dp[1] = 1;  
  5.     for(int level = 2; level <=n; level++)  
  6.         for(int root = 1; root<=level; root++)  
  7.             dp[level] += dp[level-root]*dp[root-1];  
  8.     return dp[n];  
  9. }  


[ Py DS ] Ch3 - Data Manipulation with Pandas (Part5)

Source From  Here   Pivot Tables   We have seen how the  GroupBy  abstraction lets us explore relationships within a dataset. A pivot ta...