[Linux 文章收集] Linux find the memory used by a program / process using pmap command

Source From Here
Introduction
You can find the memory used by a program (process) by looking into /proc directory or using standard command such as ps or top. However, you must calculate all memory usage by hand i.e. add Shared Memory + mapped file + total virtual memory size of the process + Resident Set Size + non-swapped physical memory used by process. So how do you find the memory used by a process or program under Linux? Use a tool called pmap. It reports the memory map of a process or processes.

pmap examples
To display process mappings, type

# ps aux | grep java | grep -v grep
root 32337 51.6 8.1 4845484 806488 pts/0 Sl 11:25 86:10 java -cp ../../GPhishing.jar:../../libs/* phishnet.srv.CPTUpdateDaemon
// pmap
# pmap 32337
32337: java -cp ../../GPhishing.jar:../../libs/* phishnet.srv.CPTUpdateDaemon
0000000000400000 4K r-x-- java
0000000000600000 4K r---- java
0000000000601000 4K rw--- java
000000000243d000 132K rw--- [ anon ]
000000075dc00000 27136K rw--- [ anon ]
000000075f680000 11776K ----- [ anon ]
...
00007fff5db0e000 132K rw--- [ stack ]
00007fff5dbfe000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 4845484K

The -x (Or --extended: Show the extended format.)option can be used to provide information about the memory allocation and mapping types per mapping. The amount of resident, non-shared anonymous, and locked memory is shown for each mapping:

# pmap -x 32337
32337: java -cp ../../GPhishing.jar:../../libs/* phishnet.srv.CPTUpdateDaemon
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 4 4 0 r-x-- java
0000000000600000 4 4 4 r---- java
0000000000601000 4 4 4 rw--- java
000000000243d000 132 28 28 rw--- [ anon ]
...
---------------- ------- ------- -------
total kB 4851260 809964 792780

[Linux 文章收集] Use BASH nullglob To Verify *.c Files Exists or Not In a Directory

Source From Here
Introduction
Can you explain me usage of nullglob variable under BASH? How do I check for any *.c files in any directory?

BASH shell has the following two special variables to control pathname expansion. Bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern.

- nullglob

If set, bash allows patterns which match no files to expand to a null string, rather than themselves. This is useful to check for any *.mp3 or *.cpp files in directory.

- dotglob

If set, bash includes filenames beginning with a . in the results of pathname expansion.

How do I set and unset nullglob variable?
Use shopt command to toggle the values of variables. The -s option enable nullglob effects and the -u option disable nullglob option.

# shopt -s nullglob // enable 
# shopt -u nullglob // disable

Here is sample shell script to see if *.mp3 exists or not in a directory:
- listFE.sh

view plaincopy to clipboardprint?

1. #!/bin/sh  
2. old=$(pwd)  
3. [ $# -lt 2 ] && echo -e "\t[Info] Give arg1=Folder path; arg2=File extension!\n" && exit 1  
4. [ -d $1 ] && cd $1 || exit 2  
5.   
6. shopt -s nullglob  
7. found=0  
8. for i in *.$2; do  
9.     echo "File $i found" # or take other action  
10.     found=1  
11. done  
12. shopt -u nullglob  
13. [ $found -eq 0 ] && echo "Directory is empty"  
14. cd $old  

One usage example:

// List all file with extension .mp3 under folder 'empty'
# ./listFE.sh empty mp3
File test.mp3 found

Without nullglob i will expand to *. only if there are no files in given directory. You can also use GNU find command to find out if directory is empty or not i.e. check for any *.c files in a directory called ~/project/editor:

// -maxdepth 0: Do not scan for sub directories.
// -empty : File is empty and is either a regular file or a directory.
// -exec echo {} directory is empty. \; : Display message if directory is empty.
# find ~/project/editor -maxdepth 0 -empty -exec echo {} directory is empty. \;

[ 英語發燒句 ] Part-7

32. "這還用說" 

A. Boy oh boy! The prices in this city are so high!
B. You are telling me! I've spent half of my vacation money already!

Note. 類似說法 No kidding / You;ave got that right / Tell me about it. 

33. "老樣子" 

A. Hi, John. How's work going?
B. Same old, same old. But in this economy, I can't complain.

Note. 類似說法 More of the same / Nothing new / Nothing special. 

34. "(某人) 就是這樣" 

A. I can't believe how many jokes Gary told at the party last night.
B. That's Gary for you. He always tries to make people laugh.

35. "無所謂, 管它的" 

A. A lot of celebrities eat at this restaurant.
B. I couldn't care less. I just want the food to taste good.

Note. 類似說法 It makes no difference ot me. 

36. "別客氣, 請自便" 

A. Can I borrow your bicycle this weekend?
B. Be my guest. I'll be out of town.

37. "丟臉; 沒面子" 

A. Tom said he could beat anyone at tennis.
B. Well, if he plays me, he's going to end up with eggs on his face.

Note. 類似說法 red in the face / looking like a fool 

[Linux 常見問題] How do I pump traffic using tcpreplay at 100 MBps, 500 MBps and 1Gbps speeds?

Source From Here
Question
I used the -R and -K (Preloads packets into RAM before sending.) option but it doesnt seem to be working as I captured the pumped traffic using tcpdump and the number of packets that I see there dont seem to match the number of packets that I expect in the time frame.

How-To
First of all make sure you are using the latest version, available here. You will want to use the -K and --mbps (or -M, Replay packets at a given Mbps.) options, for example:

# tcpreplay -i eth7 -K --mbps 1000 smallFlows.pcap 
File Cache is enabled
Actual: 14261 packets (9216531 bytes) sent in 0.073761 seconds.
Rated: 124951275.0 Bps, 999.61 Mbps, 193340.65 pps
Flows: 1209 flows, 16390.77 fps, 14243 flow packets, 18 non-flow
Statistics for network device: eth7
Attempted packets: 14261
Successful packets: 14261
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0

When you attempt to move to higher speeds (e.g. 10GigE) you may need to generate a bigger block of data by using the --loop (Loop through the capture file X times.) option. Also with Tcpreplay version 4.0 there are the more advanced--netmap (Write packets directly to netmap enabled network adapter.) and --unique-ip (Modify IP addresses each loop iteration to generate unique flows. This option must appear in combination with the following options: loop.) options which on a properly set up system, will achieve near wire rate and very high flows/sec. More information available at Tcpreplay How To. Here is an example:

# tcpreplay -i eth7 -K --mbps 9500 --loop 100 --netmap --unique-ip smallFlows.pcap 
Switching network driver for eth7 to netmap bypass mode... done!
File Cache is enabled
Actual: 1426100 packets (921653100 bytes) sent in 0.776133 seconds.
Rated: 1187493767.1 Bps, 9499.95 Mbps, 1837442.80 pps
Flows: 120900 flows, 155772.27 fps, 1424300 flow packets, 1800 non-flow
Statistics for network device: eth7
Attempted packets: 1426100
Successful packets: 1426100
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
Switching network driver for eth7 to normal mode... done!

[ 英語發燒句 ] Part-6

26. "一分錢一分貨" 

A. Wow! I can't believe how expensive the meals on this menu are.
B. They're certainly not cheap, but you pay for quality.

27. "放馬過來" 

A. Hey! Feel like losing a tennis match this afternoon?
B. You think you can beat me on the court? Bring it on!

28. "雞婆" 

A. You should fold your T-shirts so that they stack neatly in your drawer.
B. Quit being such a backseat driver! Just let me fold the laundry the way I want to.

29. "時間抓的很緊" 

A. We should make it to the theater on time if we don't cut it too close.
B. In that case, we had better get going soon!

30. "有印象; 想起來" 

A. Do you know a man named Steven Fitz?
B. His name doesn't ring a bell.

31. "讓人煩心; 惹毛某人" 

A. Are you going home already?
B. Yeah, the loud music in here is getting on my nerves.

[ 英語發燒句 ] Part-5

23. "少裝傻" 

A. I have no idea how your car got damaged.
B. Don't play dumb. I know it was you.

Note. dumb 表 "啞的", 在口語中亦指 "愚笨的", play dumb 即表 "裝聾作啞", 裝傻之意. 

24. "不要慌" 

A. My dog run away. I'm so sad. I don't know what I can do!
B. Hold yourself together. Let's search the neighborhood for him.

25. "我樂翻了" 

A. How are things going with the girl you are dating?
B. Great! I'm on cloud nine.

[ 英語發燒句 ] Part-4

19. "我們雙方各退一步吧" 

A. There is no way I'm going to agree to the terms of this contract.
B. Let's meet each other halfway then.

20. "我有預感" 

A. Why do you look so worried.
B. I think something bad is about to happen. I can feel it in my bone.

21. "別掃興" 

A. I don't want to go to the beach today. Just go without me.
B. Oh, come on! Don't be such a spoilsport.

Note. Don't spoil the fun / Don't be such a party pooper / Don't be such a wet blanket. 

22. "他們老是不對盤" 

A. It is true that Jack and Mike don't get along very well?
B. Yeah. For some reason, they are always butting heads with each other.

[ 英語發燒句 ] Part-3

13. "多此一舉" 

A. Let's come up with a system for scheduling meetings with clients.
B. There's no reason to reinvent the wheel. Let's just buy some scheduling software for the computer.

14. "達成共識" 

A. Is there going to be a sales meeting this afternoon.
B. Yeah. The boss wants us to be all on the same page before we go to the trade show.

15. "加油" 

A. There's only minute left in the game.
B. Here we go, Titans. Here we go!

16. "小題大作" 

A. Oh. no! We don't have enough eggs for breakfast tomorrow morning!
B. Don't make a mountain out of a molehill. I'll pick some up in the morning.

17. "沉不住氣" 

A. I'm so unhappy with the service at this restaurant. I'm going to speak with the manager.
B. Go ahead and do it. Try not to lose your cool, though.

18. "有樣學樣" 

A. After my son saw the other children throwing rocks into the lake, he started to do the same.
B. Well, monkey see, monkey do.

[ 英語發燒句 ] Part-2

7. "說曹操, 曹操到" 

A. Hi Jane! Hi Mike! What are you guys doing?
B. Well, speak of the devil, Tom! We were just talking about you.

8. "吹噓" 

A. Is Kevin really that good at basketball?
B. He says he is, but I think he's just blowing smoke.

Note. 可用動詞 brag 或 boast 來表示. 

9. 胡說八道 

A. Do you want to listen to the politician's speech tonight?
B. I never listen to politicians. So many of them are just full of hot air.

Note. full of beans, full of bull, full of it. 

10. 挫挫銳氣; 消消氣焰 

A. Tim is always bragging about how great of a a tennis player he is.
B. He should play with my brother Alex some time. I promise you that Alex will take him down a notch.

11. "多管閒事" 

A. How do you like the new office manager?
B. She's driving me crasy. She insists on having a finger in every pie.

12. "住嘴" 

A. The children are arguing about who gets to play with the toy.
B. Well, tell them to put a lid on it. Otherwise, I'm going to take the toy away.

Note. be quiet, shut up, shut your trap, pipe down, zip it, hush up. 

[ InAction Note ] Ch5. Message Storage - The KahaDB message store

Preface: 
The JMS specification supports two types of message delivery: persistent and non-persistent. A message delivered with the persistent delivery property must be logged to stable storage. For nonpersistent messages, a JMS provider must make best efforts to deliver the message, but it won’t be logged to stable storage. 

ActiveMQ supports both of these types of message delivery and can also be configured to support message recovery, an in-between state where messages are cached in memory. ActiveMQ supports a pluggable strategy for message storage and provides storage options for in-memory, file-based, and relational databases. 

Persistent messages are used if you want messages to always be available to a message consumer after they’ve been delivered to the broker, even if that consumer isn’t running when the message was sent. Once a message has been consumed and acknowledged by a message consumer, it’s typically deleted from the broker’s message store. 

Nonpersistent messages are typically used for sending notifications or real-time data. You should use nonpersistent messages when performance is critical and guaranteed delivery of the message isn’t required. 

How are messages stored by ActiveMQ? 
It’s important to gain some basic knowledge of the storage mechanisms for messages in an ActiveMQ message store. This will aid in configuration and provide an awareness of what takes place in the ActiveMQ broker during the delivery of persistent messages. Messages sent to queues and topics are stored differently, because there are some storage optimizations that can be made with topics that don’t make sense with queues, as we’ll explain. 

Storage for queues is straightforward—messages are basically stored in first in, first out order (FIFO). See figure 5.1 for a depiction of this. One message is dispatched to a single consumer at a time. Only when that message has been consumed and acknowledged can it be deleted from the broker’s message store. 
 

For durable subscribers to a topic, each consumer gets a copy of the message. In order to save storage space, only one copy of a message is stored by the broker. A durable subscriber object in the store maintains a pointer to its next stored message and dispatches a copy of it to its consumer as shown in figure 5.2. The message store is implemented in this manner because each durable subscriber could be consuming messages at different rates or they may not all be running at the same time. Also, because every message can potentially have many consumers, a message can’t be deleted from the store until it’s been successfully delivered to every interested durable subscriber. 
 

The KahaDB message store: 
The recommended message store for general-purpose messages since ActiveMQ version 5.3 is KahaDB. This is a file-based message store that combines a transactional journal, for reliable message storage and recovery, with good performance and scalability. 

The KahaDB store is a file-based, transactional store that’s been tuned and designed for the fast storage of messages. The aim of the KahaDB store is to be easy to use and as fast as possible. Its use of a file-based message database means there’s no prerequisite for a third-party database. This message store enables ActiveMQ to be downloaded and running in literally minutes. In addition, the structure of the KahaDB store has been streamlined especially for the requirements of a message broker. 

The KahaDB message store uses a transactional log for its indexes and only uses one index file for all its destinations. It’s been used in production environments with 10,000 active connections, each connection having a separate queue. The configurability of the KahaDB store means that it can be tuned for most usage scenarios, from high throughput applications (for example, trading platforms), to storing large amounts of messages (for example, GPS tracking). 

To enable the KahaDB store for ActiveMQ, you need to configure the  element in the activemq.xml configuration file. Here’s a minimal configuration for the KahaDB message store: 

If you want to embed an ActiveMQ broker inside an application, the message store can also be configured programmatically. Here’s an example of a programmatic configuration for KahaDB: 

 

Although the example seems small, it’s enough to create an ActiveMQ broker using the KahaDB message store and listen for ActiveMQ clients connecting over TCP. For more information about embedding ActiveMQ, see chapter 8. 

The KahaDB message store internals 
The KahaDB message store is the fastest of all the provided message store implementations. Its speed is the result of the combination of a fast transactional journal comprised of data log files, the highly optimized indexing of message IDs, and in-memory message caching. Figure 5.3 provides a high-level diagram of the KahaDB message store. 
 

The diagram provides a view of the three distinct parts of the KahaDB message store including the following: 
- The data logs 

The data logs act as a message journal, which consists of a rolling log of messages and commands (such as transactional boundaries and message deletions) stored in data files of a certain length. When the maximum length of the currently used data file has been reached, a new data file is created. All the messages in a data file are reference counted, so that once every message in that data file is no longer required, the data file can be removed or archived. In the data logs, messages are only appended to the end of the current data file, so storage is fast.

- The cache 

The cache holds messages temporarily if there are active consumer(s) for the messages. If there are active consumers, messages are dispatched at the same time they’re scheduled to be stored. If messages are acknowledged in time, they don’t need to be written to disk.

- The BTree indexes 

The BTree indexes hold references to the messages in the data logs that are indexed by their message ID. The indexes maintain the FIFO data structure for queues and the durable subscriber pointers to their topic messages. The redo log is used only if the ActiveMQ broker hasn’t shut down cleanly, and are used to insure the integrity of the BTree index is maintained.

The KahaDB message store directory structure: 
When you start an ActiveMQ broker configured to use a KahaDB store, a directory will automatically be created in which the persistent messages are stored. This directory structure is shown in figure 5.4. 
 

Inside of the KahaDB directory, the following directory and file structures can be found: 
- db log files 

KahaDB stores messages into data log files named db-.log of a predefined size. When a data log is full, a new one will be created, and the log number incremented. When there are no more references to any of the messages in the data log file, it’ll be deleted or archived.

- archive directory 

This exists only if archiving is enabled. The archive is used to store data logs that are no longer needed by KahaDB, making it possible to replay messages from the archived data logs at a later point. If archiving isn’t enabled (the default), data logs that are no longer in use are deleted from the file system.

- db.data 

This file contains the persistent BTree indexes to the messages held in the message data logs.

- db.redo 

This is the redo file, used for recovering the BTree indexes if the KahaDB message store starts after a hard stop.

Configuring the KahaDB message store: 
The KahaDB message store can be configured in the activemq.xml file. Its configuration options control the different tuning parameters (table 5.1): 
 
Table 5.1 Configuration options available for the KahaDB message store 

ActiveMQ provides a pluggable API for message stores, and there are three additional implementations to KahaDB that are shipped with ActiveMQ: 

* The AMQ message store—A file-based message store designed for performance
* The JDBC message store—A message store based on JDBC
* The Memory message store—A memory-based message store

We’ll look at the use cases and configuration for these additional message stores in the next three sections.

[ 英語發燒句 ] Part-1

1. "到此為此" 

A. Do you think we should keep seeing each other?
B. We both know this relationship isn't going to work out, so let's leave it at that.

2. "放馬後炮" 

A. David says that if you had made these changes to your presentation, you would have made the sale.
B. Hit's just a Monday morning quarterback. He couldn't have made a better presentation with the information I had at that time.

3. "原來如此" 

A. I wonder why they put up a wall of plants and flowers at the art gallery.
B. They are promoting the city's flower festival.
A. Now it's clear as day.

Note. I see / I got it / No wonder / Now I get it. / That makes sense. 

4. "他有毛病嗎?" 

A. The client wants to use our product for six months before he pays for it.
B. What planet is he from? Tell him that we don't do business that way.

5. "背黑鍋" 

A. I don't think I can finish the project on time.
B. Well, you'll have to tell that to the boss. I'm going to take the heat for your failure.

Note.: 除了用 take the heat, 類似用法還有: be a scapegoat / be left holding the bag 

6. "格格不入" 

A. Do you want to go to dance club tomorrow night?
B. Not really. I don't know how to dance, so I'd just feel out of place.

Note.: not fit in / a fish out of water