[文章收集] Snort : Firing up OpenAppID

Source From Here
Preface
The purpose of this post is to provide guidance to Snort users who would like to try out Snort 2.9.7.0 and the OpenAppID features that it comes with. It is not intended to guide the reader through setting up Snort from scratch, there are plenty of docs on how to set up Snort at http://www.snort.org/docs.

Snort
Before we can download the latest Snort source code and compile it, we have a new prerequisite to fulfill before we can compile Snort. You'll need to install Luajit, which is used to define application detectors with the new OpenAppID.

Step #1, Prep the system.
You’ll need to download the following files from snort.org download page:

* daq-xxx.tar.gz
* snort-2.9.7.0_xxx.tar.gz
* snort-openappid-detectors.xxx.tgz
* snortrules-snapshot-xxx.tar.gz

Step #2, Install Snort requirements
On my Ubuntu system this was as easy as the following command.

# sudo apt-get install openssl libssl-dev build-essential g++ flex bison zlib1g-dev autoconf libtool libpcap-dev libpcre3-dev libdumbnet-dev build-essential

If you Linux is CentOS distribution, you can refer to CentOS LuaJIT Installation instructions

Libdnet-1.11 and LuaJIT were compiled and used so we were running the same versions as those who were developing OpenAppID. Besides that, we still have a few packages to compile and install:

# mkdir ~/build
// Put daq-xxx.tar.gz, snort-2.9.7.0_xxx.tar.gz, snort-openappid-detectors.xxx.tgz and snortrules-snapshot-xxx.tar.gz into ~/build
# cd ~/build
# wget http://prdownloads.sourceforge.net/libdnet/libdnet-1.11.tar.gz
# tar xzvf libdnet-1.11.tar.gz
# cd libdnet-1.11/
# ./configure
# make
# sudo make install
# cd ..
# wget http://luajit.org/download/LuaJIT-2.0.2.tar.gz
# tar xzvf LuaJIT-2.0.2.tar.gz
# cd LuaJIT-2.0.2/
# make
# sudo make install
# cd ..
# tar -xzvf daq-2.0.2.tar.gz
# cd daq-2.0.2/
# ./configure
# make
# sudo make install
# sudo ldconfig
# cd .. 

Step #3 Building Snort with OpenAppID support.
Now we’re all set to download and install the RC (Release candidate) version of Snort. At the time this was written that was 2.9.7.0 rc (snort-2.9.7.0_rc.tar.gz). Grab the latest source from snort.org’s Download Snort page. Then we can extract and build.

# wget https://www.snort.org/downloads/snortdev/snort-2.9.7.0_rc.tar.gz
# tar -xvf snort-2.9.7.0_rc.tar.gz
# cd snort-2.9.7.0_rc
# ./configure --enable-sourcefire --enable-open-appid
# make
# sudo make install
# which snort
/usr/local/bin/snort # Here is the installed path
# /usr/local/bin/snort --version

,,_ -*> Snort! <*-
o" )~ Version 2.9.7.0_rc GRE (Build 147)

Note the new flag on the configure command. We are adding that switch to enable OpenAppID.

Step #4 Configure Snort without OpenAppID enabled
Before we jump into the new capabilities of OpenAppID, it’s best to make sure the foundations are functioning. Snort needs some configuration files and directories to be made. (Using sudo if necessary)

# mkdir /etc/snort # For configuration
# mkdir /var/log/snort # For log data
# mkdir /usr/local/lib/snort_dynamicrules # For dynamic rules
# mkdir /etc/snort/rules # For normal text rules
# touch /etc/snort/white_list.rules # For white lists
# touch /etc/snort/black_list.rules # For black lists

A set of configuration files are included in the snort tarball. These can be copied into your /etc/snort/ directory.

# cd ~/build/snort-2.9.7.0_rc/etc/
# cp attribute_table.dtd file_magic.conf snort.conf unicode.map classification.config gen-msg.map reference.config threshold.conf /etc/snort/

The last thing to do is to add some of the VRT supplied rules to enable snort to detect ‘bad’ stuff. We’ve untarred unto a temp directory called ‘crules’, the name has no meaning.

# cd ~/build
# mkdir crules
# mv snortrules-snapshot-xxxx.tar.gz crules // Change xxxx to your version number
# cd crules/
# tar -zxvf snortrules-snapshot-xxxx.tar.gz
# cp -r preproc_rules /etc/snort
# sudo cp -r rules /etc/snort/
# sudo cp -r so_rules /etc/snort/

The next step is to configure the /etc/snort/snort.conf file. They key changes made to get Snort working were:

# vi /etc/snort/snort.conf

view plaincopy to clipboardprint?

1. ...  
2. RULE_PATH /etc/snort/rules  
3. SO_RULE_PATH /etc/snort/rules  
4. PREPROC_RULE_PATH /etc/snort/rules  
5. WHITE_LIST_PATH /etc/snort  
6. BLACK_LIST_PATH /etc/snort  
7. ...  

To test Snort’s config the -T option can be used. If Snort has any problems with configuration, it will let you know.

# snort -c /etc/snort/snort.conf -T
...
Snort successfully validated the configuration!
Snort exiting

If you encounter problem look like:

Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_appid_preproc.so... ERROR: Failed to load /usr/local/lib/snort_dynamicpreprocessor//libsf_appid_preproc.so: libluajit-5.1.so.2: cannot open shared object file: No such file or directory
Fatal Error, Quitting..

You can try below approach:

# ldd /usr/local/lib/snort_dynamicpreprocessor/libsf_appid_preproc.so # Check problem
linux-vdso.so.1 => (0x00007fff731ff000)
libluajit-5.1.so.2 => not found
libdnet.1 => /usr/lib64/libdnet.1 (0x00007f6b773f0000)
...
# find /usr/ -name libluajit-5.1.so.2 # Check where is the required .so
/usr/local/lib/libluajit-5.1.so.2
# ln -s /usr/local/lib/libluajit-5.1.so.2 /lib64/libluajit-5.1.so.2 # Let's create a soft link for it
# snort -c /etc/snort/snort.conf -T # Try again

Application Detector Package
Step #5 Enabling OpenAppID in Snort.
The next step is to download the Application Detector Package. This entirely new content contains what is at the heart of the OpenAppID feature. You can download it from the Download Snort page on snort.org (currently: snort-openappid.tar.gz).

# cd ~/build
# tar -zxvf ./snort-openappid-detectors.xxxx.tgz # Remember to change xxxx to your downloaded version
# mkdir /usr/local/lib/openappid
# mv odp/ /usr/local/lib/openappid/

Once that is unpacked you will see an odp directory. Here you will see the appMapping.data file, which includes metadata about the application detectors. A simple way to explore what application detectors are included in this first release is to examine the second column of this file. If you’d like to see the list, try the following command.

# find /usr/local/lib/openappid/ -name appMapping.data
/usr/local/lib/openappid/odp/appMapping.data
# cat /usr/local/lib/openappid/odp/appMapping.data | cut -f2 | grep Facebook
Facebook Apps
Facebook
Facebook Chat
Facebook Comment
Facebook Read Email
...

You will also see the libs subdirectory, which contains the Lua libraries needed to run application detection. The port and lua subdirectories contain the applications detectors themselves. The port detectors are simple YAML files to patch port and application. The lua directory contains the bulk of the detectors. These lua scripts perform far more powerful detection.

Configure snort.conf
There are several changes to make in our snort.conf file. We must add the preprocessor command for OpenAppID and configure the output command.
Additionally, I suggest commenting out the following line.

view plaincopy to clipboardprint?

1. # path to dynamic rules libraries   
2. #dynamicdetection directory /usr/local/lib/snort_dynamicrules  <---comment class="keyword" nbsp="" span="" style="background-color: inherit; border: none; color: #006699; font-weight: bold; margin: 0px; padding: 0px;">this line  

At present time there there isn't a shared object rule release specifically for 2.9.7.0, so it is easier to skip this for now.

Configure OpenAppID Preprocessor
Our next step is to add the configuration for the OpenAppID preprocessor to the snort.conf file. Find the lines for the reputation preprocessor. This should be followed by commented text for “Step 6.” Just after the reputation preprocessor and before Step 6 we will add another preprocessor setting.

view plaincopy to clipboardprint?

1. ...  
2. preprocessor reputation: \  
3.    memcap 500, \  
4.    priority whitelist, \  
5.    nested_ip inner, \  
6.    whitelist $WHITE_LIST_PATH/white_list.rules, \  
7.    blacklist $BLACK_LIST_PATH/black_list.rules  
8.   
9. preprocessor appid : app_stats_filename appstats-unified.log, app_stats_period 60, app_detector_dir /usr/local/lib/openappid  

This will turn on the OpenAppID preprocessor. The first step simply names the configuration file to which application statistics will be logged. The second gives the time period used to sample this data. Finally, you must point out the directory which contains the odp directory we extracted from the Open App ID Detector package.

Configure Output
The final step in configuring our snort.conf file will be to change the output command. Everyone should be using the Unified2 output module already, but there is a new command to add new the new event field for the application in use. Look into Step 6 find the lines explaining the unified2 output type. In that section add the following line.

view plaincopy to clipboardprint?

1. ###################################################  
2. # Step #6: Configure output plugins  
3. # For more information, see Snort Manual, Configuring Snort - Output Modules  
4. ###################################################  
5. ...  
6. output unified2: filename snort.log, limit 128, appid_event_types  
7. ...  

Testing Snort
Now let’s fire up Snort.

# /usr/local/bin/snort -c /etc/snort/snort.conf -i eth?

Where eth? is which ever interface you will be monitoring with (e.g. eth0). To start Snort, the following command was used.

# snort -c /etc/snort/snort.conf --daq afpacket -i eth0 -k none

If there aren't any errors you will see the final line output by Snort indicating a successful startup, as “Commencing packet processing.” If you examine the output above that you will see a new section for AppID Configuration. This will include the configuration given to the preprocessor for reference.

To test the new deployment, fire up a web browser to test out application detection. For my test I used Chrome to visit cnn.com.
If you are still examining the console output of the Snort process you will see something to the effect of “Opening /var/log/snort/appstats-u2.log.xxxxxx for output.” Ending the Snort process will also write what data has not yet been output.

Examining Application Statistics
When you downloaded the Snort package, it included a tools subdirectory. In here you will find a couple of programs for changing the unified2 binary format to readable text. In this case we will use the u2openappid program to print the application statistics that Snort is collecting for us.

# u2openappid /var/log/snort/appstats-u2.log.xxxxxxx
statTime="1393807860",appName="chrome",txBytes="6043",rxBytes="111267"
statTime="1393807860",appName="dns",txBytes="8708",rxBytes="38103"
statTime="1393807860",appName="http",txBytes="200399",rxBytes="1444070"
statTime="1393807860",appName="cnn.com",txBytes="198478",rxBytes="1557970"
statTime="1393807860",appName="doubleclick",txBytes="5543",rxBytes="2598"
statTime="1393807860",appName="truste",txBytes="1829",rxBytes="12208"
statTime="1393807860",appName="washington_time",txBytes="2210",rxBytes="1401"
statTime="1393807860",appName="turner_broadcas",txBytes="1785",rxBytes="1316"
statTime="1393807860",appName="moat",txBytes="5707",rxBytes="3453"
...

From this you can see a number of entries. In this case the above were generated from my visit to cnn.com. You can see that they share the same statTime. OpenAppID identifies the client application (Chrome), DNS request, protocol (http), and web app (cnn.com). The rest of the entries are other web apps embedded within CNN’s web page. To further illustrate the data we can get form OpenAppID, here is the output from when I instead use Firefox to visit Netflix.

statTime="1393809840",appName="google",txBytes="3500",rxBytes="5171"
statTime="1393809840",appName="firefox",txBytes="11695",rxBytes="20818"
statTime="1393809840",appName="http",txBytes="11695",rxBytes="20818"
statTime="1393809840",appName="netflix",txBytes="1072",rxBytes="1262"
statTime="1393809840",appName="https",txBytes="18330",rxBytes="33984"
statTime="1393809840",appName="verisign",txBytes="1450",rxBytes="2668"

Supplement
* Youtube - OpenAppID Install
* CentOS LuaJIT Installation instructions

LuaJIT is a Just-In-Time Compiler (JIT) for the Lua programming language.