2015年6月29日 星期一

[Linux 文章收集] Introducing Linux Network Namespaces

Source From Here 
Preface 
what are network namespaces? Generally speaking, an installation of Linux shares a single set of network interfaces and routing table entries. You can modify the routing table entries using policy routing (here’s an introduction I wrote and here’s a write-up on a potential use case for policy routing), but that doesn’t fundamentally change the fact that the set of network interfaces and routing tables/entries are shared across the entire OS. Network namespaces change that fundamental assumption. With network namespaces, you can have different and separate instances of network interfaces and routing tables that operate independent of each other

This concept is probably best illustrated through some examples. Along the way, I’ll introduce a few new ideas as well. First, though, I need to provide some assumptions. 
Throughout these examples, I’m using Ubuntu Server 12.04.3 LTS. Please note that support for network namespaces varies between Linux distributions; Ubuntu supports them but Red Hat doesn’t. If you’re thinking about using network namespaces, be sure your Linux distribution includes support.

Further, I’ll assume that you’re either running as root, or that you will prepend sudo to the commands listed here as necessary.

Creating and Listing Network Namespaces 
Creating/Deleting a network namespace is actually quite easy. Just use this command: 
ip netns { add | delete } <NETNSNAME>

For example, let’s say you wanted to create a namespace called “blue”. You’d use this command: 
# ip netns add blue
# ip netns
blue
...

You should see your network namespace listed there, ready for you to use. 

Assigning Interfaces to Network Namespaces 
Creating the network namespace is only the beginning; the next part is to assign interfaces to the namespaces, and then configure those interfaces for network connectivity. One thing that threw me off early in my exploration of network namespaces was that you couldn’t assign physical interfaces to a namespace. How in the world were you supposed to use them, then? 

It turns out you can only assign virtual Ethernet (veth) interfaces to a network namespace. Virtual Ethernet interfaces are an interesting construct; they always come in pairs, and they are connected like a tube—whatever comes in one veth interface will come out the other peer veth interface. As a result, you can use veth interfaces to connect a network namespace to the outside world via the “default” or “global” namespace where physical interfaces exist. 

Let’s see how that’s done. First, you’d create the veth pair: 
# ip link add veth0 type veth peer name veth1

Naturally, you could substitute other names for veth0 and veth1, if you wanted. You can verify that the veth pair was created using this command: 
# ip link list
...
67: veth1:  mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 8a:fc:3d:e2:04:69 brd ff:ff:ff:ff:ff:ff
68: veth0:  mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 4e:fb:40:f6:25:ef brd ff:ff:ff:ff:ff:ff

You should see a pair of veth interfaces (using the names you assigned in the command above) listed there. Right now, they both belong to the “default” or “global” namespace, along with the physical interfaces. 

Let’s say that you want to connect the global namespace to the blue namespace. To do that, you’ll need to move one of the veth interfaces to the bluenamespace using this command: 
# ip link set veth1 netns blue
# ip link list | grep veth1 // veth1 disappear

If you then run the ip link list command again, you’ll see that the veth1 interface has disappeared from the list. It’s now in the blue namespace, so to see it you’d need to run this command: 
# ip netns exec blue ip link list
1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
67: veth1:  mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 8a:fc:3d:e2:04:69 brd ff:ff:ff:ff:ff:ff

Whoa! That’s a bit of a complicated command. Let’s break it down: 
1. The first part, ip netns exec, is how you execute commands in a different network namespace.
2. Next is the specific namespace in which the command should be run (in this case, the blue namespace).
3. Finally, you have the actual command to be executed in the remote namespace. In this case, you want to see the interfaces in the blue namespace, so you run ip link list.

Configuring Interfaces in Network Namespaces 
Now that veth1 has been moved to the blue namespace, we need to actually configure that interface. Once again, we’ll use the ip netns exec command, this time to configure the veth1 interface in the blue namespace: 
# ip netns exec blue ifconfig veth1 10.1.1.1/24 up
# ip netns exec blue ifconfig veth1
veth1: flags=4099 mtu 1500
inet 10.1.1.1 netmask 255.255.255.0 broadcast 10.1.1.255
ether 8a:fc:3d:e2:04:69 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

In this case, you’re using ifconfig to assign an IP address to the veth1 interface and bring that interface up. (Note: you could use the ip addr, ip route, and ip linkcommands to accomplish the same thing.

Once the veth1 interface is up, you can verify that the network configuration of the blue namespace is completely separate by just using a few different commands. For example, let’s assume that your “global” namespace has physical interfaces in the 172.16.1.0/24 range, and your veth1 interface is in a separate namespace and assigned something from the 10.1.1.0/24 range. You could verify how network namespaces keep the network configuration separate using these commands: 
* ip addr list in the global namespace will not show any 10.1.1.0/24-related interfaces or addresses.
* ip netns exec blue ip addr list will show only the 10.1.1.0/24-related interfaces and addresses, and will not show any interfaces or addresses from the global namespace.
* Similarly, ip route list in each namespace will show different routing table entries, including different default gateways.

Connecting Network Namespaces to the Physical Network 
This part of it threw me for a while. I can’t really explain why, but it did. Once I’d figured it out, it was obvious. To connect a network namespace to the physical network, just use a bridge. In my case, I used an Open vSwitch (OVS) bridge, but a standard Linux bridge would work as well. Place one or more physical interfaces as well as one of the veth interfaces in the bridge, and—bam!—there you go. Naturally, if you had different namespaces, you’d probably want/need to connect them to different physical networks or different VLANs on the physical network.

2015年6月28日 星期日

[ 常見問題 ] Get docker container name by PID

Source From Here 
Question 
I have list of PID's and i need to get their docker container name going the other direction is easy ... Get PID of docker container by image name: 
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
417af6d7dbfa 52366bfd2985:latest "sh /home/root/scrip 19 minutes ago Up 19 minutes 4444/tcp, 5999/tcp insane_ardinghelli

# docker inspect --format '{{.State.Pid}}' insane_ardinghelli
64998

Any idea how to get the name by PID ? 

How-To 
// -q, --quiet=false Only display numeric IDs
# docker ps -q
417af6d7dbfa
# docker ps -q | xargs docker inspect --format '{{.Name}}'
/insane_ardinghelli

Supplement 
Docker Doc - Using the command line 
"xargs" All-IN-One Tutorial Guide

[Linux 文章收集] "xargs" All-IN-One Tutorial Guide

Source From Here
Preface
xargs is a just like "awk" ,"find" & "grep" commands processes the standard input on all unix flavoured operating sysems. Basically "xargs" is used to remove or do some operation on long list of file names which were produced by "find" & "grep" commands.

Usually many UNIX operating system doesn't accept such a long list of argument.UNIX xargs command divide that list into sub-list with acceptable length and made it work. For example I'd like to find out all *.sh file located in 100s of sub-directories and move them to another directory called ~/back.scripts. How do I use command line args with xargs to achieve the same?

as per man page "xargs" is used to execute a command, passing constructed argument list(s). The arguments are typically a long list of filenames (generated byls or find etc) that are passed to xargs via a pipe.

Some features:
* xargs can execute the command supplying some initial arguments directly, and reading the remaining arguments from standard input (or piped input).
* xargs passes arguments to command in several bundles, this allows command to process more arguments than it could normally handle at once.
* Arguments in the standard input must be separated by unquoted blank characters, or unescaped blank characters or newline characters.
* Characters can be quoted by enclosing them in "double-quotes" (non-double-quote and non-newline chars only).
* Characters can be quoted by enclosing them in 'apostrophes' (non-apostrophe and non-newline chars only).
* Any unquoted character can be escaped by preceding it with a backslash.

Exit Status
This command returns the following exit values:


Examples
Find all the .conf files under /etc/ and pass to the ls command, -print0 is required if any filenames contain whitespace.:
# find /etc/ -name "*.conf" -print0 | xargs -0 ls -l

Find all files in the work folder, pass to grep and search for 'profit':
# find ./work -print | xargs grep "profit"


{} as the argument list marker
{} is the default argument list marker. You need to use {} this with various command which take more than two arguments at a time. For example mv command need to know the file name. The following will find all .sh files in or below the current directory and move them to ~/.old.files directory:
# find . -name "*.sh" -print0 | xargs -0 -I {} mv {} ~/back.scripts

You can rename {} to something else. In the following example {} is renamed as file. This is more readable as compare to previous example:
# find . -name "*.sh" -print0 | xargs -0 -I file mv file ~/back.scripts

Where,
-0 If there are blank spaces or characters (including newlines) many commands will not work. This option take cares of file names with blank space.
-I Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character.

10 Popular "XARGS" Command Examples:
1) With& Without "xargs" observation:
In this example of xargs command we will see how output changes with use of xargs command in unix or Linux. Here is the output of find command withoutxargs first and than with xargs, you can clearly see that multiline output is converted into single line:
# find . -name "*sh*"
./.bash_history
./.bash_profile
./.bash_profile.cf-before-edit
./.cshrc
...

# find . -name "*bash*" | xargs
./.bash_history ./.bash_profile ./.bash_profile.cf-before-edit ...

2) Xargs with grep:
When you use "xargs" in conjusction with find and grep , the grep will look for the specific word in each file in the from the standard input.
# find . -name "*.sh" | xargs grep "ksh"

In the above example first find all .sh files from current directory or below and than on each .sh file look for word "ksh".

3) Covert muti-line output into single line
Best example of xargs is converting output of one command into one line. For example you can run any command and then combine xargs to convert output into single line. here is an example xargs in unix which does that.
# ls -1 *.sh
linux_sysinfo.sh
aix_sysinfo.sh
audit_script.sh
chperm_messages.sh

# ls -1 *.sh | xargs
linux_sysinfo.sh aix_sysinfo.sh audit_script.sh chperm_messages.sh

4) To Delete temporary files using xargs & find:
Another common example of xargs command in unix is removing temporary files from system.
# find /tmp -name "*.tmp" | xargs rm

This will remove all .tmp file from /tmp or below directory. xargs in unix is very fast as compared to deleting single file at a time which can also be done by usingfind command alone.

5) xargs -0 to handle space in file name
Above example of xargs command in unix will not work as expected if any of file name contains space or new line on it. To avoid this problem we use find -print0to produce null separated file name and xargs -0 to handle null separated items. Here is an example of xargs command in unix which can handle file name with spaces and newline:
# find /tmp -name "*.tmp" -print0 | xargs -0 rm

6) Counting number of lines/words/characters in each file using xargs & find:
"find" in conjuction with "xargs" and "wc" we can count number of lines/words/characters in each file under a particular directory.
# ls -1 *.sh | xargs wc -l 
112 linux_sysinfo.sh
85 aix_sysinfo.sh
35 audit_script.sh
18 chperm_messages.sh
250 total

Note: you can use '-c' & '-w' with wc to obtain number of characters and words respectively.

7) xargs and cut command in Unix:
Though most of xargs examples in unix will be along with find and grep command but xargs is not just limited to this two it can also be used with any command which generated long list of input for example we can use xargs with cut command in unix. In below example of unix xargs we will xargs example with cutcommand. for using cut command let's first create a test file with some data e.g.
# cat fruits.txt
Orange,Greenorange
Mango,Redmango
Banana,Pinkbanana


// Now we will display name of actual fruit from first column using xargs command in one line:
# cut -d, -f1 fruits.txt | sort | xargs
Banana Mango Orange

8) To insert file names into the middle of command lines, type:
This command sequence renames all files in the current directory by adding .old to the end of each name. The -I flag tells the xargs command to insert each line of the ls directory listing where {} (braces) appear. If the current directory contains the files chap1, chap2, and chap3, this constructs the following commands:
// -t, --verbose: Print the command line on the standard error output before executing it.
# ls | xargs -t -I {} mv {} {}.old
# mv chap1 chap1.old
# mv chap2 chap2.old
# mv chap3 chap3.old

9) To run a command on files that you select individually, type:
This command sequence allows you to select files to add to the lib.a library. The -p flag tells the xargs command to display each ar command it constructs and to ask if you want to run it. Type y to run the command. Press the any other key if you do not want to run the command.
// -p, --interactive: Prompt the user about whether to run each command line and read a line from the terminal.
// -n max-args, --max-args=max-args: Use at most max-args arguments per command line.

# ls | xargs -p -n 1 ar r lib.a
ar r lib.a chap1 ?...
ar r lib.a chap2 ?...
ar r lib.a chap3 ?...

10) To construct a command that contains a specific number of arguments and to insert those arguments into the middle of a command line, type:
// If the current directory contains files chap1 through chap10, the output constructed will be the following
# ls | xargs -n6 | xargs -I {} echo {} - some files in the directory
chap1 chap2 chap3 chap4 chap5 chap6 - some files in the directory
chap7 chap8 chap9 chap10 - some file in the directory


Supplement
10 Xargs Command Examples in Linux / UNIX
This tutorials explains the usage of xargs command using few simple examples...


2015年6月26日 星期五

[ 常見問題 ] Docker - Stop / remove all Docker containers

Source From Here
One liner to stop / remove all of Docker containers:
// -a, --all=false Show all containers (default shows just running)
// -q, --quiet=false Only display numeric IDs

# echo $(docker ps -a -q)
6ca1c890e3f2 607d3ad2e4c8 67430c23a16d cc86b1ba697c 9ff17926b736 f12b657aa412 ef217701a1b7 33f15b0c22f2 1fb091625c6d
# docker stop $(docker ps -a -q) // Stop all containers
# docker rm $(docker ps -a -q) // Remove all containers


2015年6月25日 星期四

[ Python 常見問題 ] Python speed testing - Time Difference - milliseconds

Source From Here 
Question 
What is the proper way to compare 2 times in Python in order to speed test a section of code? I tried reading the API docs. I'm not sure I understand thetimedelta thing. So far I have this code: 
  1. from datetime import datetime  
  2.   
  3. tstart = datetime.now()  
  4. print t1  
  5.   
  6. # code to speed test  
  7.   
  8. tend = datetime.now()  
  9. print t2  
  10. # what am I missing?  
  11. # I'd like to print the time diff here  
How-To 
datetime.timedelta is just the difference between two datetimes ... so it's like a period of time, in days / seconds / microseconds: 
>>> import datetime
>>> a = datetime.datetime.now()
>>> b = datetime.datetime.now()
>>> c = b - a

>>> c
datetime.timedelta(0, 4, 316543)
>>> c.days
0
>>> c.seconds
4
>>> c.microseconds
316543

Be aware that c.microseconds only returns the microseconds portion of the timedelta! For timing purposes always use c.total_seconds(). You can do all sorts of maths with datetime.timedelta, eg: 
>>> c / 10
datetime.timedelta(0, 0, 431654)

It might be more useful to look at CPU time instead of wallclock time though ... that's operating system dependant though ... under Unix-like systems, check out the 'time' command.

2015年6月24日 星期三

[ Python 常見問題 ] Python list function argument names

Source From Here 
Question 
Is there a way to get the argument names a function takes? For example: 
  1. def foo(bar, buz):  
  2.     pass  
I can use magical_way this way: 
  1. magical_way(foo) == ["bar""buz"]  
How-To 
Use the inspect method from Python's standard library (the cleanest, most solid way to perform introspection). 

inspect.getargspec(f) returns the names and default values of f's arguments: 
Get the names and default values of a Python function’s arguments. A tuple of four things is returned: (args, varargs, keywords, defaults)args is a list of the argument names (it may contain nested lists). varargs and keywords are the names of the * and ** arguments or None. defaults is a tuple of default argument values or None if there are no default arguments; if this tuple has n elements, they correspond to the last n elements listed in args.

A testing example: 
>>> import inspect
>>> def f(a, b='b', c=3, *args, **kwards):
... pass
...
>>> inspect.getargspec(f)
ArgSpec(args=['a', 'b', 'c'], varargs='args', keywords='kwards', defaults=('b', 3))
>>> inspect.getargspec(f)[0] # If you are only interested in the defined argument names
['a', 'b', 'c']

If you only want the names and don't care about special forms *a, **k, 
  1. import inspect  
  2.   
  3. ef magical_way(f):  
  4.    return inspect.getargspec(f)[0]  

Supplement 
Getting method parameter names in python 5 answers 
Getting list of parameter names inside python function 4 answers 
Python *args and **kwargs?

[ Py DS ] Ch3 - Data Manipulation with Pandas (Part5)

Source From  Here   Pivot Tables   We have seen how the  GroupBy  abstraction lets us explore relationships within a dataset. A pivot ta...