Preface
Files used in this exercise:
In this exercise you will begin to get acquainted with the Hadoop tools. You will manipulate files in HDFS, the Hadoop Distributed File System.
Exercise
Before starting the exercises, run the course setup script in a terminal window:
Hadoop
Hadoop is already installed, configured, and running on your virtual machine. Most of your interaction with the system will be through a command-line wrapper called hadoop. If you run this program with no arguments, it prints a help message. To try this, run the below command in a terminal window:
The hadoop command is subdivided into several subsystems. For example, there is a subsystem for working with files in HDFS and another for launching and managing MapReduce processing jobs.
Step1: Exploring HDFS
The subsystem associated with HDFS in the Hadoop wrapper program is called FsShell. This subsystem can be invoked with command hadoop fs.
1. In the terminal window, enter
You see a help messge describing all the commands associated with the FsShell subsystem.
2. Enter:
This shows you the contents of the root directory in HDFS. There will be multiple entries, one of which is /user. Individual users have a "home" directory under this directory, named after their username.
Step2: Uploading Files
Besides browsing the existing filesystem, another important thing you can do with FsShell is to upload new data into HDFS.
1. Change directories to the local filesystem directory containing the sample data we will be using in the course.
If you perform a regular Linux ls command in this directory, you will see a few files, including two named shakespeare.tar.gz and shakespeare-stream.tar.gz. Both of those contain the complete works of Shakespeare in text format, but with different formats and organizations. For now, we will work with shakespeare.tar.gz.
2. Unzip shakespeare.tar.gz by running
This creates a directory named shakespeare/ containing several files on your local filesystem.
3. Insert this directory into HDFS:
This copies the local shakespeare directory and its contents into a remote HDFS directory named /user/training/shakespeare.
4. List the contents of your HDFS home directory now:
You should see an entry for the shakespeare directory. If you don't pass a directory name to the -ls command, it assumes you mean your home directory, i.e./user/training. Any relative path will based on your home directory too!
5. We also have Web server log file, which we will put into HDFS for use in the future exercise:
The file is currently compressed using GZip. Rather than extract the file to the local disk and then upload it, we will extract and upload in one step. Now, extrack and upload the file in one step. The -c option to gunzip uncompresses to standard output, and the dash (-) in the below command takes whatever is being sent to its standard input and places that data in HDFS:
6. Run the hadoop fs -ls command to verify that the log file is in your HDFS home directory
7. The access log file is quite large - around 500 MB. Create a small version of this file, consisting only of its first 5000 lines, and store the smaller version in HDFS. You can use the smaller version for testing in subsequent exercises.
Step3: Viewing and Manipulating Files
Now let's view some of the data you just copied into HDFS.
1. Enter
This lists the contents of the /user/training/shakespeare HDFS directory.
2. The glossary file included in the compressed file you began with is not strictly a work of Shakespere, let's remove it:
3. Enter:
This prints the last 50 lines of Henry IV, Part 1 to your terminal. This command is handy for viewing the output of MapReduce programs. Very often, an individual output file of a MapReduce program is very large, making it inconvenient to view the entire file in the terminal.
4. To download a file to work with on the local filesystem use the fs -get command. This command takes two arguments: an HDFS path and a local path. It copies the HDFS contents into the local filesystem:
Other Commands
Useful arguments for users of a hadoop cluster from hadoop command:
Commands useful for administrators of a hadoop cluster can refer here.
Supplement
* Apache Hadoop 2.5.1 - Command Menu
2014年12月29日 星期一
2014年12月28日 星期日
[ RubyAlg ] MIT Linear Algebra, Spring 2005 - Lec10
Source From Here
4 Fundamental Sub-Space: C(A), N(A), R(A)=C(A^T), N(A^T)
這邊將介紹 4 個基礎的 Vector Sub-Space. 除了已經過的 Column Space 與 Null Space, 另外兩個如下:
R(A) = C(A^T): Row sub-space
N(A^T) = The left nullspace of A
Let A be an mxn Matrix:
For C(A):
For C(A^T):
For N(A):
For N(A^T):
首先來看 N(A^T), 也就是 Null Space of A^T:
因為 EA=R, 接著來看一些運算:
而 Matrix E 放在 Matrix A 的左邊說明對 row 進行操作, 舉 EA=R 的 row1 為例:
由上面的結果可以知道 C(A^T) 的 rank=2 (跟 C(A) 的 rank 一樣). 接著考慮 EA=0 可知知道 [-1, 0, 1] 是 N(A^T) 的解 (A 的 -1*row0 加上 A 的 row2 得到 [0,0,0,0])
如果硬解的話, 可以參考下面運算過程:
Connection between Row Space and Column Space
Consider A as a matrix mxn:
1. dim(C(A)) = dim(C(A^T)) = r
2. dim(C(A)) = n - r
3. dim(C(A^T)) = m - r
4 Fundamental Sub-Space: C(A), N(A), R(A)=C(A^T), N(A^T)
這邊將介紹 4 個基礎的 Vector Sub-Space. 除了已經過的 Column Space 與 Null Space, 另外兩個如下:
R(A) = C(A^T): Row sub-space
N(A^T) = The left nullspace of A
Let A be an mxn Matrix:
For C(A):
For C(A^T):
For N(A):
For N(A^T):
首先來看 N(A^T), 也就是 Null Space of A^T:
因為 EA=R, 接著來看一些運算:
而 Matrix E 放在 Matrix A 的左邊說明對 row 進行操作, 舉 EA=R 的 row1 為例:
由上面的結果可以知道 C(A^T) 的 rank=2 (跟 C(A) 的 rank 一樣). 接著考慮 EA=0 可知知道 [-1, 0, 1] 是 N(A^T) 的解 (A 的 -1*row0 加上 A 的 row2 得到 [0,0,0,0])
如果硬解的話, 可以參考下面運算過程:
Connection between Row Space and Column Space
Consider A as a matrix mxn:
1. dim(C(A)) = dim(C(A^T)) = r
2. dim(C(A)) = n - r
3. dim(C(A^T)) = m - r
[CCDH] Exercise15 - Creating an Inverted Index (P54)
Preface
Files and Directories Used in this Exercise
In this exercise, you will write a MapReduce job that produces an inverted index.
For this lab you will use an alternative input, provided in the file invertedIndexInput.tgz. When decompressed, this archive contains a directory of files; each is a Shakespeare play formatted as follows:
Each line contains:
This format can be read directly using the KeyValueTextInputFormat class provided in the Hadoop API. This input format presents each line as one record to your Mapper, with the part before the tab character as the key, and the part after the tab as the value.
Given a body of text in this form, your indexer should produce an index of all the words in the text. For each word, the index should have a list of all the locations where the words appears. For example, for the word "honeysuckle" your output should look like this:
The index should contain such an entry for every word in the text.
Lab Experiment
Prepare The Input Data
1. Extract the invertedIndexInput directory and upload to HDFS:
Define The MapReduce Solution
Remember that for this program you use a special input format to suit the form of your data, so your driver class will do for it:
2. Implement driver class:
Note that the exercise requires you to retrieve the file name - since that is the name of the play. The Context object can be used to retrieve the name of the file.
2. Implement the Mapper class
The Reducer will output inverted index information for key as word and value as exist location list:
3. Implement the Reducer class
4. Build project and run MapReduce job
5. Check result
Files and Directories Used in this Exercise
In this exercise, you will write a MapReduce job that produces an inverted index.
For this lab you will use an alternative input, provided in the file invertedIndexInput.tgz. When decompressed, this archive contains a directory of files; each is a Shakespeare play formatted as follows:
Each line contains:
This format can be read directly using the KeyValueTextInputFormat class provided in the Hadoop API. This input format presents each line as one record to your Mapper, with the part before the tab character as the key, and the part after the tab as the value.
Given a body of text in this form, your indexer should produce an index of all the words in the text. For each word, the index should have a list of all the locations where the words appears. For example, for the word "honeysuckle" your output should look like this:
The index should contain such an entry for every word in the text.
Lab Experiment
Prepare The Input Data
1. Extract the invertedIndexInput directory and upload to HDFS:
Define The MapReduce Solution
Remember that for this program you use a special input format to suit the form of your data, so your driver class will do for it:
2. Implement driver class:
- package solution;
- import org.apache.hadoop.fs.Path;
- import org.apache.hadoop.io.Text;
- import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
- import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
- import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;
- import org.apache.hadoop.mapreduce.Job;
- import org.apache.hadoop.conf.Configured;
- import org.apache.hadoop.conf.Configuration;
- import org.apache.hadoop.util.Tool;
- import org.apache.hadoop.util.ToolRunner;
- public class InvertedIndex extends Configured implements Tool {
- public int run(String[] args) throws Exception {
- if (args.length != 2) {
- System.out.printf("Usage: InvertedIndex );
- return -1;
- }
- Job job = new Job(getConf());
- job.setJarByClass(InvertedIndex.class);
- job.setJobName("Inverted Index");
- /*
- * We are using a KeyValueText file as the input file.
- * Therefore, we must call setInputFormatClass.
- * There is no need to call setOutputFormatClass, because the
- * application uses a text file for output.
- */
- job.setInputFormatClass(KeyValueTextInputFormat.class); // Here setup our customized input format
- FileInputFormat.setInputPaths(job, new Path(args[0]));
- FileOutputFormat.setOutputPath(job, new Path(args[1]));
- job.setMapperClass(IndexMapper.class);
- job.setReducerClass(IndexReducer.class);
- job.setOutputKeyClass(Text.class);
- job.setOutputValueClass(Text.class);
- boolean success = job.waitForCompletion(true);
- return success ? 0 : 1;
- }
- public static void main(String[] args) throws Exception {
- int exitCode = ToolRunner.run(new Configuration(), new InvertedIndex(), args);
- System.exit(exitCode);
- }
- }
2. Implement the Mapper class
- package solution;
- import java.io.IOException;
- import org.apache.hadoop.fs.Path;
- import org.apache.hadoop.io.Text;
- import org.apache.hadoop.mapreduce.lib.input.FileSplit;
- import org.apache.hadoop.mapreduce.Mapper;
- public class IndexMapper extends Mapper
{ - @Override
- public void map(Text key, Text value, Context context) throws IOException,
- InterruptedException {
- /*
- * Get the FileSplit for the input file, which provides access
- * to the file's path. You need the file's path because it
- * contains the name of the play.
- */
- FileSplit fileSplit = (FileSplit) context.getInputSplit();
- Path path = fileSplit.getPath();
- /*
- * Call the getName method on the Path object to retrieve the
- * file's name, which is the name of the play. Then append
- * "@" and the line number to the play's name. The resulting
- * string is the location of the words on that line.
- */
- String wordPlace = path.getName() + "@" + key.toString();
- Text location = new Text(wordPlace);
- /*
- * Convert the line to lower case.
- */
- String lc_line = value.toString().toLowerCase();
- /*
- * Split the line into words. For each word on the line,
- * emit an output record that has the word as the key and
- * the location of the word as the value.
- */
- for (String word : lc_line.split("\\W+")) {
- if (word.length() > 0) {
- context.write(new Text(word), location);
- }
- }
- }
- }
3. Implement the Reducer class
- package solution;
- import java.io.IOException;
- import org.apache.hadoop.io.Text;
- import org.apache.hadoop.mapreduce.Reducer;
- /**
- * On input, the reducer receives a word as the key and a set
- * of locations in the form "play name@line number" for the values.
- * The reducer builds a readable string in the valueList variable that
- * contains an index of all the locations of the word.
- */
- public class IndexReducer extends Reducer
{ - private static final String SEP = ",";
- @Override
- public void reduce(Text key, Iterable
values, Context context) - throws IOException, InterruptedException {
- StringBuilder valueList = new StringBuilder();
- boolean firstValue = true;
- /*
- * For each "play name@line number" in the input value set:
- */
- for (Text value : values) {
- /*
- * If this is not the word's first location, add a comma to the
- * end of valueList.
- */
- if (!firstValue) {
- valueList.append(SEP);
- } else {
- firstValue = false;
- }
- /*
- * Convert the location to a String and append it to valueList.
- */
- valueList.append(value.toString());
- }
- /*
- * Emit the index entry.
- */
- context.write(key, new Text(valueList.toString()));
- }
- }
5. Check result
訂閱：
文章 (Atom)
[Git 常見問題] error: The following untracked working tree files would be overwritten by merge
Source From Here 方案1： // x -----删除忽略文件已经对 git 来说不识别的文件 // d -----删除未被添加到 git 的路径中的文件 // f -----强制运行 # git clean -d -fx 方案2： 今天在服务器上 gi...
-
CNN 卷積神經網路簡介 STEP1. 卷積神經網路介紹 CNN 卷積神經網路可以分成兩大部分: * 影像的特徵提取 : 透過 Convolution 與 Max Pooling 提取影像特徵. * Fully connected Feedforward n... -
Understanding the core indexing classes : As you saw in our Indexer class ( A simple application ), you need the following classes to...
-
前言 : 為什麼程序管理這麼重要呢？這是因為： * 首先，本章一開始就談到的，我們在操作系統時的各項工作其實都是經過某個 PID 來達成的 (包括你的 bash 環境)， 因此，能不能進行某項工作，就與該程序的權限有關了。 * 再來，如果您的 Linux 系統是個...
