[Linux 常見問題] Quickly create a large file on a Linux system

 Source From Here

Question
How can I quickly create a large file on a Linux (Red Hat Linux) system?

dd will do the job, but reading from /dev/zero and writing to the drive can take a long time when you need a file several hundreds of GBs in size for testing... If you need to do that repeatedly, the time really adds up. I don't care about the contents of the file, I just want it to be created quickly. How can this be done?

Using a sparse file won't work for this. I need the file to be allocated disk space.

How-To
dd from the other answers is a good solution, but it is slow for this purpose. In Linux (and other POSIX systems), we have fallocate, which uses the desired space without having to actually writing to it, works with most modern disk based file systems, very fast. For example:

# fallocate -l 10G gentoo_root.img

[Linux 常見問題] grep + add time out after some time if not find the relevant match

Source From Here
Question
I use the following command syntax to search params in my script

# grep -qsRw -m1 "any_param" /dir/..../

Some times the search take a very long time. The question is how to add time out to grep command. For example after 20 seconds grep will break out

How-To
There is a Linux command timeout that can do this for you. For example:

# time grep -R -m 1 -e "pattern to search" Github/

real 0m3.627s
user 0m1.824s
sys 0m1.798s

# echo $?
1 // Return code from command grep

# timeout 2s grep -R -m 1 -e "pattern to search" Github/ // Force the grep to timeout after 2 second
# echo $?
124 // Return code from command timeout

[Linux 常見問題] Linux and Unix Test Disk I/O Performance With dd Command

Source From Here
Question
How can I use dd command on a Linux to test I/O performance of my hard disk drive? How do I check the performance of a hard drive including the read and write speed on a Linux operating systems? You can use the following commands on a Linux or Unix-like systems for simple I/O performance test:

* dd command : It is used to monitor the writing performance of a disk device on a Linux and Unix-like system.
* hdparm command : It is used to get/set hard disk parameters including test the reading and caching performance of a disk device on a Linux based system.

In this tutorial you will learn how to use the dd command to test disk I/O performance.

Use dd command to monitor the reading and writing performance of a disk device

1. Open a shell prompt. Or login to a remote server via ssh.
2. Use the dd command to measure server throughput (write speed) dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
3. Use the dd command to measure server latency dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 oflag=dsync

The dd command is useful to find out simple sequential I/O performance.

Understanding dd command options
In this example, I’m using RAID-10 (Adaptec 5405Z with SAS SSD) array running on a Ubuntu Linux 14.04 LTS server. The basic syntax is as follows to find out server throughput:

view plaincopy to clipboardprint?

1. # Syntax  
2. # dd if=/dev/input.file  of=/path/to/output.file  bs=block-size  count=number-of-blocks  oflag=dsync  
3.   
4. ## GNU dd syntax ##  
5. ##########################################################  
6. ##***[Adjust bs and count as per your needs and setup]**##  
7. ##########################################################  
8. dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync  
9. dd if=/dev/zero of=/tmp/test2.img bs=64M count=1 oflag=dsync  
10. dd if=/dev/zero of=/tmp/test3.img bs=1M count=256 conv=fdatasync  
11. dd if=/dev/zero of=/tmp/test4.img bs=8k count=10k  
12. dd if=/dev/zero of=/tmp/test4.img bs=512 count=1000 oflag=dsync  
13.   
14. ## OR alternate syntax for GNU/dd ##  
15. dd if=/dev/zero of=/tmp/testALT.img bs=1G count=1 conv=fdatasync  

Sample outputs:

# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 2.38434 s, 450 MB/s

Please note that one gigabyte was written for the test1.img and 450 MB/s was server throughput for this test. Where:

* if=/dev/zero (if=/dev/input.file) : The name of the input file you want dd the read from.
* of=/tmp/test1.img (of=/path/to/output.file) : The name of the output file you want dd write the input.file to.
* bs=1G (bs=block-size) : Set the size of the block you want dd to use. 1 gigabyte was written for the test. Please note that Linux will need 1GB of free space in RAM. If your test system does not have sufficient RAM available, use a smaller parameter for bs (such as 128MB or 64MB and so on).
* count=1 (count=number-of-blocks): The number of blocks you want dd to read.
* oflag=dsync (oflag=dsync) : Use synchronized I/O for data. Do not skip this option. This option get rid of caching and gives you good and accurate results
* conv=fdatasyn: Again, this tells dd to require a complete “sync” once, right before it exits. This option is equivalent to oflag=dsync.

Finding server latency time
In this example, 512 bytes were written one thousand times to get RAID10 server latency time:

# dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 oflag=dsync

Sample outputs:

1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.60362 s, 848 kB/s

Please note that server throughput and latency time depends upon server/application load too. So I recommend that you run these tests on a newly rebooted server as well as peak time to get better idea about your workload. You can now compare these numbers with all your devices.

But why the server throughput and latency time are so low?
Low values does not mean you are using slow hardware. The value can be low because of the HARDWARE RAID10 controller’s cache.

Use hdparm command to see buffered and cached disk read speed
I suggest you run the following commands 2 or 3 times Perform timings of device reads for benchmark and comparison purposes:

view plaincopy to clipboardprint?

1. ### Buffered disk read test for /dev/sda ##  
2. hdparm -t /dev/sda1  
3. ## OR ##  
4. hdparm -t /dev/sda  

To perform timings of cache reads for benchmark and comparison purposes again run the following command 2-3 times (note the -T option):

view plaincopy to clipboardprint?

1. ## Cache read benchmark for /dev/sda ###  
2. hdparm -T /dev/sda1  
3. ## OR ##  
4. hdparm -T /dev/sda  

OR combine both tests:

view plaincopy to clipboardprint?

1. hdparm -Tt /dev/sda  

Sample outputs:


Again note that due to filesystems caching on file operations, you will always see high read rates.

Use dd command on Linux to test read speed
To get accurate read test data, first discard caches before testing by running the following commands:

view plaincopy to clipboardprint?

1. flush  
2. echo 3 | sudo tee /proc/sys/vm/drop_caches  
3. time dd if=/path/to/bigfile of=/dev/null bs=8k  

One execution result:

# time dd if=/tmp/test1.img of=/dev/null bs=8k
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB) copied, 0.315225 s, 3.4 GB/s

real 0m0.319s
user 0m0.050s
sys 0m0.269s

Linux Laptop example
Run the following command:

view plaincopy to clipboardprint?

1. ### Debian Laptop Throughput With Cache ##  
2. dd if=/dev/zero of=/tmp/laptop.bin bs=1G count=1 oflag=direct  
3.   
4. ### Deactivate the cache ###  
5. hdparm -W0 /dev/sda  
6.   
7. ### Debian Laptop Throughput Without Cache ##  
8. dd if=/dev/zero of=/tmp/laptop.bin bs=1G count=1 oflag=direct  

[Linux 文章收集] Bash - How to trim string

Source From Here
How-To
You can use a bash parameter expansion, sed, cut and tr to trim a string.

Bash parameter expansion
Let's use bash parameter expansion to remove all whitespace characters from the variable foo:
- test1.sh

view plaincopy to clipboardprint?

1. #!/bin/sh  
2. foo="Hello    world."  
3. echo "${foo//[[:space:]]/}"  

Execution result:

# ./test1.sh
Helloworld.

Using sed command to trim the front/back space
To remove only space characters before and after string use sed:
- test2.sh

view plaincopy to clipboardprint?

1. #!/bin/sh  
2. foo="      Hello    world.    "  
3. echo "${foo}" | sed -e 's/^[[:space:]]*//'  

Execution result:

# ./test2.sh
Hello world.

Using xargs to remove space
Easy to use and remember way how to remove whitespaces before and after word:
- test3.sh

view plaincopy to clipboardprint?

1. #!/bin/sh  
2. foo="      Hello    world.    "  
3. foo_remove_space=`echo ${foo} | xargs`  
4. echo "foo_remove_space='${foo_remove_space}'"  

Execution result:

# ./test3.sh
foo_remove_space='Hello world.'

Supplement
* Split string by delimiter and get N-th element
* Linux 系統 xargs 指令範例與教學

[Linux 文章收集] Check Top Processes sorted by RAM or CPU Usage in Linux

Source From Here 
Preface 
The following command will show the list of top processes ordered by RAM and CPU use in descendant form (remove the pipeline and head if you want to see the full list): 

# ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

Sample Output: 
 

Brief explanation of above options used in above command. 

The -o (or –format) option of ps allows you to specify the output format. A favorite of mine is to show the processes’ PIDs (pid), PPIDs (pid), the name of the executable file associated with the process (cmd), and the RAM and CPU utilization (%mem and %cpu, respectively). 

Additionally, I use --sort to sort by either %mem or %cpu. By default, the output will be sorted in ascendant form, but personally I prefer to reverse that order by adding a minus sign in front of the sort criteria. To add other fields to the output, or change the sort criteria, refer to the OUTPUT FORMAT CONTROL section in the man page of ps command. 

Supplement 
* Linux 用 ps 與 top 指令找出最耗費 CPU 與記憶體資源的程式 

這裡介紹如何在 Linux 中使用 ps 與 top 指令列出系統上最吃 CPU 與記憶體的程式。作為 Linux 系統的管理者，時常都需要查看系統的負載狀況，如果系統中出現不正常的程式，吃掉太多的 CPU 或記憶體資源，就會影響系統的效能，太嚴重的話甚至會造成當機等狀況。

[Linux 常見問題] How does one output bold text in Bash?

Source From Here
Question
I'm writing a Bash script that prints some text to the screen:

view plaincopy to clipboardprint?

1. echo "Some Text"  

Can I format the text? I would like to make it bold.

How-To
The most compatible way of doing this is using tput to discover the right sequences to send to the terminal:

view plaincopy to clipboardprint?

1. bold=$(tput bold)  
2. normal=$(tput sgr0)  

then you can use the variables $bold and $normal to format things:

view plaincopy to clipboardprint?

1. echo "this is ${bold}bold${normal} but this isn't"  

gives

this is bold but this isn't