[Toolkit] Keras - MNIST 手寫數字辨識使用 CNN

CNN 卷積神經網路簡介 

STEP1. 卷積神經網路介紹 
CNN 卷積神經網路可以分成兩大部分: 

* 影像的特徵提取: 透過 Convolution 與 Max Pooling 提取影像特徵.
* Fully connected Feedforward network: Flatten layers, hidden layers and output layers

STEP2. 卷積運算 (Convolution) 
卷積運算的原理是將一個影像透過卷積運算的 Filter weight(s) 產生多個影像, 在上面第一層的 Convolution 為例: 
1. 先以隨機方式產生 16 個 3x3 的 filter weight(S) 

2. 要轉換的影像由左而右, 由上而下透過 filter weight 產生新影像的值: 

3. 使用 16 個 filter weight 產生 16 個影像 

STEP3. Max-Pooling 運算說明 
Max-Pool 運算可以將影像縮減取樣 (downsampling), 如下圖: 原本影像是 4x4, 經過 Max-Pool 運算後, 影像大小為 2x2: 

downsampling 有以下好處: 

* 減少需要處理的資料點: 減少後續運算所需時間.
* 讓影像位置的差異變小: 例如手寫數字 7, 位置上下左右可能不同, 但是位置不同可能影響辨識. 減少影像大小讓數字的位置差異變小.
* 參數的數量和計算量下降: 這在一定程度上也控制了 Overfitting 的狀況.

進行資料前處理 (Preprocess) 
CNN (Convolution Neural Network) 與 MLP 進行資料的前處理方式有所不同, 說明如下: 

* MLP : image reshape (60000, 784): MLP 因為直接送進神經元處理, 所以 60,000 筆轉換為一筆成 28x28 = 784 個神經元輸入.
* CNN : image reshape (60000, 28, 28, 1): CNN 因為必須先進行卷積與池化 (Max-Pool) 運算, 所以必須保留影像的維度. 因此 60,000 筆轉換成一筆成 28 (長) x 28(寬) x 1(高) 的影像單位.

STEP1. 資料讀取與轉換 

view plaincopy to clipboardprint?

1. #!/usr/bin/env python3  
2. from keras.datasets import mnist  
3. from keras.utils import np_utils  
4. import numpy as np  
5. np.random.seed(10)  
6.   
7. # Read MNIST data  
8. (X_Train, y_Train), (X_Test, y_Test) = mnist.load_data()  
9.   
10. # Translation of data  
11. X_Train40 = X_Train.reshape(X_Train.shape[0], 28, 28, 1).astype('float32')  
12. X_Test40 = X_Test.reshape(X_Test.shape[0], 28, 28, 1).astype('float32')  

STEP2. 將 Features 進行標準化與 Label 的 Onehot encoding 

view plaincopy to clipboardprint?

1. # Standardize feature data  
2. X_Train40_norm = X_Train40 / 255  
3. X_Test40_norm = X_Test40 /255  
4.   
5. # Label Onehot-encoding  
6. y_TrainOneHot = np_utils.to_categorical(y_Train)  
7. y_TestOneHot = np_utils.to_categorical(y_Test)  

建立模型 
接著會依照下面流程圖建立模型: 

STEP1. 建立卷積層與池化層 

view plaincopy to clipboardprint?

1. from keras.models import Sequential  
2. from keras.layers import Dense,Dropout,Flatten,Conv2D,MaxPooling2D  
3.   
4. model = Sequential()  
5. # Create CN layer 1  
6. model.add(Conv2D(filters=16,  
7.                  kernel_size=(5,5),  
8.                  padding='same',  
9.                  input_shape=(28,28,1),  
10.                  activation='relu'))  
11. # Create Max-Pool 1  
12. model.add(MaxPooling2D(pool_size=(2,2)))  
13.   
14. # Create CN layer 2  
15. model.add(Conv2D(filters=36,  
16.                  kernel_size=(5,5),  
17.                  padding='same',  
18.                  input_shape=(28,28,1),  
19.                  activation='relu'))  
20.   
21. # Create Max-Pool 2  
22. model.add(MaxPooling2D(pool_size=(2,2)))  
23.   
24. # Add Dropout layer  
25. model.add(Dropout(0.25))  

STEP2. 建立神經網路 
- 建立平坦層 
下面程式碼建立平坦層, 將之前步驟已經建立的池化層2, 共有 36 個 7x7 維度的影像轉換成 1 維向量, 長度是 36x7x7 = 1764, 也就是對應到 1764 個神經元: 

view plaincopy to clipboardprint?

1. model.add(Flatten())  

- 建立 Hidden layer 

view plaincopy to clipboardprint?

1. model.add(Dense(128, activation='relu'))  
2. model.add(Dropout(0.5))  

- 建立輸出層 
最後建立輸出層, 共有 10 個神經元, 對應到 0~9 共 10 個數字. 並使用 softmax 激活函數 進行轉換 (softmax 函數可以將神經元的輸出轉換成每一個數字的機率): 

view plaincopy to clipboardprint?

1. model.add(Dense(10, activation='softmax'))  

STEP3. 查看模型的摘要 

view plaincopy to clipboardprint?

1. model.summary()  
2. print("")  

進行訓練 
接著我們使用 Back Propagation 進行訓練. 

STEP1. 定義訓練並進行訓練 

view plaincopy to clipboardprint?

1. # 定義訓練方式  
2. model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])  
3.   
4. # 開始訓練  
5. train_history = model.fit(x=X_Train4D_norm,  
6.                           y=y_TrainOneHot, validation_split=0.2,  
7.                           epochs=10, batch_size=300, verbose=2)  

在 compile 方法中: 

* loss: 設定 Loss Function, 這邊選定 Cross Entropy 作為 Loss Function.
* optimizer: 設定訓練時的優化方法, 在深度學習使用 adam (Adam: A Method for Stochastic Optimization) 可以更快收斂, 並提高準確率.
* metrics: 設定評估模型的方式是 accuracy 準確率.

訓練過程的輸出如下: 

22s - loss: 0.4872 - acc: 0.8478 - val_loss: 0.0968 - val_acc: 0.9722
Epoch 2/10
22s - loss: 0.1407 - acc: 0.9591 - val_loss: 0.0631 - val_acc: 0.9808
Epoch 3/10
22s - loss: 0.1029 - acc: 0.9689 - val_loss: 0.0516 - val_acc: 0.9838
...
Epoch 8/10
22s - loss: 0.0513 - acc: 0.9841 - val_loss: 0.0345 - val_acc: 0.9898
Epoch 9/10
22s - loss: 0.0454 - acc: 0.9866 - val_loss: 0.0342 - val_acc: 0.9902
Epoch 10/10
22s - loss: 0.0428 - acc: 0.9874 - val_loss: 0.0330 - val_acc: 0.9903

STEP2. 畫出 accuracy 執行結果 
之前的訓練步驟產生的 accuracy 與 loss 都會記錄在 train_history 變數. 底下將常用的函數定義在 utils.py: 
- utils.py 

view plaincopy to clipboardprint?

1. import os  
2.   
3. def isDisplayAvl():  
4.     return 'DISPLAY' in os.environ.keys()  
5.   
6. import matplotlib.pyplot as plt  
7. def plot_image(image):  
8.     fig = plt.gcf()  
9.     fig.set_size_inches(2,2)  
10.     plt.imshow(image, cmap='binary')  
11.     plt.show()  
12.   
13. def plot_images_labels_predict(images, labels, prediction, idx, num=10):  
14.     fig = plt.gcf()  
15.     fig.set_size_inches(12, 14)  
16.     if num > 25: num = 25  
17.     for i in range(0, num):  
18.         ax=plt.subplot(5,5, 1+i)  
19.         ax.imshow(images[idx], cmap='binary')  
20.         title = "l=" + str(labels[idx])  
21.         if len(prediction) > 0:  
22.             title = "l={},p={}".format(str(labels[idx]), str(prediction[idx]))  
23.         else:  
24.             title = "l={}".format(str(labels[idx]))  
25.         ax.set_title(title, fontsize=10)  
26.         ax.set_xticks([]); ax.set_yticks([])  
27.         idx+=1  
28.     plt.show()  
29.   
30. def show_train_history(train_history, train, validation):  
31.     plt.plot(train_history.history[train])  
32.     plt.plot(train_history.history[validation])  
33.     plt.title('Train History')  
34.     plt.ylabel(train)  
35.     plt.xlabel('Epoch')  
36.     plt.legend(['train', 'validation'], loc='upper left')  
37.     plt.show()  

接著便可以使用函數 show_train_history 顯示 accuracy 在 train 與 evaluation 的差異與 loss 在 train 與 evaluation 的差異如下: 

view plaincopy to clipboardprint?

1. from utils import *  
2. if isDisplayAvl():  
3.     show_train_history(train_history, 'acc', 'val_acc')  
4.     show_train_history(train_history, 'loss', 'val_loss')  

執行結果如下: 
- Training accuracy vs Evaluation accuracy 

- Training loss vs Evaluation loss 

評估模型準確率與進行預測 
我們已經完成訓練, 接下來要使用 test 測試資料集來評估準確率. 

STEP1. 評估模型準確率 

view plaincopy to clipboardprint?

1. scores = model.evaluate(X_Test4D_norm, y_TestOneHot)  
2. print()  
3. print("\t[Info] Accuracy of testing data = {:2.1f}%".format(scores[1]*100.0))  

STEP2. 預測結果 

view plaincopy to clipboardprint?

1. print("\t[Info] Making prediction of X_Test4D_norm")  
2. prediction = model.predict_classes(X_Test4D_norm)  # Making prediction and save result to prediction  
3. print()  
4. print("\t[Info] Show 10 prediction result (From 240):")  
5. print("%s\n" % (prediction[240:250]))  

STEP3. 顯示前 10 筆預測結果 

view plaincopy to clipboardprint?

1. if isDisplayAvl():  
2.     plot_images_labels_predict(X_Test, y_Test, prediction, idx=240)  

STEP4. 顯示 Confusion Matrix 

view plaincopy to clipboardprint?

1. import pandas as pd  
2. print("\t[Info] Display Confusion Matrix:")  
3. print("%s\n" % pd.crosstab(y_Test, prediction, rownames=['label'], colnames=['predict']))  

完整代碼連結如下: 

* ch8_1.py: 主程式
* utils.py: 相關使用函式

Supplement 
* ML Lecture 10: Convolutional Neural Network 
* TensorFlow : Tutorials 02 - Convolutional Neural Network 
* Save and Load Your Keras Deep Learning Models

[ Gradle IA ] Ch2 - Next-generation builds with Gradle (Part1)

Preface 
This Chapter covers 

■ Understanding how Gradle compares to other build tools
■ Describing Gradle’s compelling feature set
■ Installing Gradle
■ Writing and executing a simple Gradle script
■ Running Gradle on the command line

For years, builds had the simple requirements of compiling and packaging software. But the landscape of modern software development has changed, and so have the needs for build automation. Today, projects involve large and diverse software stacks, incorporate multiple programming languages, and apply a broad spectrum of testing strategies. With the rise of agile practices, builds have to support early integration of code as well as frequent and easy delivery to test and production environments. 

Established build tools continuously fall short in meeting these goals in a simple but customizable fashion. How many times have your eyes glazed over while looking at XML to figure out how a build works? And why can’t it be easier to add custom logic to your build? All too often, when adding on to a build script, you can’t shake the feeling of implementing a workaround or hack. I feel your pain. There has to be a better way of doing these things in an expressive and maintainable way. There is—it’s called Gradle. 

Gradle is the next evolutionary step in JVM-based build tools. It draws on lessons learned from established tools like Ant and Maven and takes their best ideas to the next level. Following a build-by-convention approach, Gradle allows for declaratively modeling your problem domain using a powerful and expressive domain-specific language (DSL) implemented in Groovy instead of XML. Because Gradle is a JVM native, it allows you to write custom logic in the language you’re most comfortable with, be it Java or Groovy. 

In this chapter, we’ll compare existing JVM-language build tools with the features Gradle has to offer. Later, you’ll learn how Gradle can help you automate your software delivery process in the context of a continuous delivery deployment pipeline. To get a first taste of what it’s like to use Gradle, you’ll install the runtime, write a simple build script, and run it on the command line. Join me on an exciting journey as we explore the world of Gradle. 

Why Gradle? Why now? 
If you’ve ever dealt with build systems, frustration may be one of the feelings that comes up when thinking about the challenges you’ve faced. Shouldn’t the build tool naturally help you accomplish the goal of automating your project? Instead, you had to compromise on maintainability, usability, flexibility, extendibility, or performance. 

Let’s say you want to copy a file to a specific location under the condition that you’re building the release version of your project. To identify the version, you check a string in the metadata describing your project. If it matches a specific numbering scheme (for example, 1.0-RELEASE), you copy the file from point A to point B. From an outside perspective, this may sound like a trivial task. If you have to rely on XML, the build language of many traditional tools, expressing this simple logic becomes a nightmare. The build tool’s response is to add scripting functionality through nonstandard extension mechanisms. You end up mixing scripting code with XML or invoking external scripts from your build logic. It’s easy to imagine that you’ll need to add more and more custom code over time. As a result, you inevitably introduce accidental complexity, and maintainability goes out the window. Wouldn’t it make sense to use an expressive language to define your build logic in the first place? 

Here’s another example. Maven follows the paradigm of convention over configuration by introducing a standardized project layout and build lifecycle for Java projects. That’s a great approach if you want to ensure a unified application structure for a greenfield project—a project that lacks any constraints imposed by prior work. However, you may be the lucky one who needs to work on one of the many legacy projects that follow different conventions. One of the conventions Maven is very strict about is that one project needs to produce one artifact, such as a JAR file. But how do you create two different JAR files from one source tree without having to change your project structure? Just for this purpose, you’d have to create two separate projects. Again, even though you can make this happen with a workaround, you can’t shake off the feeling that your build process will need to adapt to the tool, not the tool to your build process. 

These are only some of the issues you may have encountered with existing solutions. Often you’ve had to sacrifice nonfunctional requirements to model your enterprise’s automation domain. But enough with the negativity—let’s see how Gradle fits into the build tool landscape. 

Evolution of Java build tools 
Let’s look at how build tools have evolved over the years. As I discussed in chapter 1, two tools have dominated building Java projects: Ant and Maven. Over the course of years, both tools significantly improved and extended their feature set. But even though both are highly popular and have become industry standards, they have one weak point: build logic has to be described in XML. XML is great for describing hierarchical data, but falls short on expressing program flow and conditional logic. As a build script grows in complexity, maintaining the build code becomes a nightmare. 

Ant’s first official version was released in 2000. Each element of work (a target in Ant’s lingo) can be combined and reused. Multiple targets can be chained to combine single units of work into full workflows. For example, you might have one target for compiling Java source code and another one for creating a JAR file that packages the class files. Building a JAR file only makes sense if you first compiled the source code. In Ant, you make the JAR target depend on the compile target. Ant doesn’t give any guidance on how to structure your project. Though it allows for maximum flexibility, Ant makes each build script unique and hard to understand. External libraries required by your project were usually checked into version control, because there was no sophisticated mechanism to automatically pull them from a central location. Early versions of Ant required a lot of discipline to avoid repetitive code. Its extension mechanism was simply too weak. As a result, the bad coding practice of copying and pasting code was the only viable option. To unify project layouts, enterprises needed to impose standards. 

Maven 1, released in July 2004, tried to ease that process. It provided a standardized project and directory structure, as well as dependency management. Unfortunately, custom logic is hard to implement. If you want to break out of Maven’s conventions, writing a plugin, called a Mojo, is usually the only solution. The name Mojo might imply a straightforward, easy, and sexy way to extend Maven; in reality, writing a plugin in Maven is cumbersome and overly complex. 

These days a lot of people are looking for alternatives to established build tools. We see a shift from using XML to a more expressive and readable language to define builds. A build tool that carries on this idea is Gant, a DSL on top of Ant written in Groovy. Using Gant, users can now combine Groovy’s language features with their existing knowledge of Ant without having to write XML. Even though it wasn’t part of the core Maven project, a similar approach was proposed by the project Maven Polyglot that allows you to write your build definition logic, which is the project object model (POM) file, in Groovy, Ruby, Scala, or Clojure. 

We’re on the cusp of a new era of application development: polyglot programming. Many applications today incorporate multiple programming languages, each of which is best suited to implement a specific problem domain. It’s not uncommon to face projects that use client-side languages like JavaScript that communicate with a mixed, multilingual backend like Java, Groovy, and Scala, which in turn calls off to a C++ legacy application. It’s all about the right tool for the job. Despite the benefits of combining multiple programming languages, your build tool needs to fluently support this infrastructure as well. JavaScript needs to be merged, minified, and zipped, and your server-side and legacy code needs to be compiled, packaged, and deployed. 

Gradle fits right into that generation of build tools and satisfies many requirements of modern build tools (figure 2.1). It provides an expressive DSL, a convention over configuration approach, and powerful dependency management. It makes the right move to abandon XML and introduce the dynamic language Groovy to define your build logic. Sounds compelling, doesn’t it? Keep reading to learn about Gradle’s feature set and how to get your boss on board. 

Why you should choose Gradle 
If you’re a developer, automating your project is part of your day-to-day business. Don’t you want to treat your build code like any other piece of software that can be extended, tested, and maintained? Let’s put software engineering back into the build. Gradle build scripts are declarative, readable, and clearly express their intention. Writing code in Groovy instead of XML, sprinkled with Gradle’s build-by-convention philosophy, significantly cuts down the size of a build script and is far more readable (see figure 2.2). 

It’s impressive to see how much less code you need to write in Gradle to achieve the same goal. With Gradle you don’t have to make compromises. Where other build tools like Maven propose project layouts that are “my way or the highway,” Gradle’s DSL allows for flexibility by adapting to nonconventional project structures. 
Gradle’s motto 

“Make the impossible possible, make the possible easy, and make the easy elegant” (adapted quote from Moshé Feldenkrais).

Never change a running system, you say? Your team already spent a lot of time on establishing your project’s build code infrastructure. Gradle doesn’t force you to fully migrate all of your existing build logic. Good integration with other tools like Ant and Maven is at the top of Gradle’s priority list. We’ll take a deeper look at Gradle’s integration features and potential migration strategies in Chapter 9. 

The market seems to be taking notice of Gradle. In spring 2010, Gradle was awarded the Springy award for the most innovative open source project (http://www.springsource.org/node/2871). ThoughtWorks, a highly regarded software development consultancy, periodically publishes a report on emerging technologies, languages, and tools—their so-called technology radar. The goal of the technology radar is to help decision makers in the software industry understand trends and their effect on the market. In their latest edition of the report from May 2013 (http://thoughtworks.fileburst.com/assets/technology-radar-may-2013.pdf), Gradle was rated with the status Adopt, indicating a technology that should be adopted by the industry. 

Gradle found adopters early on, even before a 1.0 version was released. Popular open source projects like Groovy and Hibernate completely switched to Gradle as the backbone for their builds. Every Android project ships with Gradle as the default build system. Gradle also had an impact on the commercial market. Companies like Orbitz, EADS, and Software AG embraced Gradle as well, to name just a few. VMware, the company behind Spring and Grails, made significant investments in choosing Gradle. Many of their software products, such as the Spring framework and Grails, are literally built on the trust that Gradle can deliver. 

Gradle’s compelling feature set 
Let’s take a closer look at what sets Gradle apart from its competitors: its compelling feature set (see figure 2.3). To summarize, Gradle is an enterprise-ready build system, powered by a declarative and expressive Groovy DSL. It combines flexibility and effortless extendibility with the idea of convention over configuration and support for traditional dependency management. Backed by a professional services company (Gradleware) and strong community involvement, Gradle is becoming the numberone choice build solution for many open source projects and enterprises. 

Expressive build language and deep API 
The key to unlocking Gradle’s power features within your build script lies in discovering and applying its domain model, as shown in figure 2.4. 

As you can see in the figure, a build script directly maps to an instance of type Project in Gradle’s API. In turn, the dependencies configuration block in the build script invokes the method dependencies() of the project instance. Like most APIs in the Java world, it’s available as HTML Javadoc documentation on Gradle’s website at https://docs.gradle.org/current/javadoc/. Who would have known? You’re actually dealing with code. Without knowing it, you generate an object representation of your build logic in memory. In Chapter 4, we’ll explore many of Gradle’s API classes and how they’re represented in your build script. 

Each element in a Gradle script has a one-to-one representation with a Java class; however, some of the elements have been sugarcoated with a sprinkle of Groovy syntax. Having a Groovy-fied version of a class in many cases makes the code more compact than its Java counterpart and allows for using new language features like closures. 

Gradle can’t know all the requirements specific to your enterprise build. By exposing hooks into lifecycle phases, Gradle allows for monitoring and configuring your build script’s execution behavior. Let’s assume you have the very unique requirement of sending out an email to the development team whenever a unit test failure occurs. The way you want to send an email (for example, via SMTP or a third-party email service provider) and the list of recipients are very specific to your build. Other builds using Gradle may not be interested in this feature at all. By writing a custom test listener that’s notified after the test execution lifecycle event, you can easily incorporate this feature for your build. 

Gradle establishes a vocabulary for its model by exposing a DSL implemented in Groovy. When dealing with a complex problem domain, in this case the task of building software, being able to use a common language to express your logic can be a powerful tool. Let’s look at some examples. Most common to builds is the notation of a unit of work that you want to get executed. Gradle describes this unit of work as a task. Part of Gradle’s standard DSL is the ability to define tasks very specific to compiling and packaging Java source code. It’s a language for building Java projects with its own vocabulary that doesn’t need to be relevant to other contexts. 

Another example is the way you can express dependencies to external libraries, a very common problem solved by build tools. Out-of-the-box Gradle provides you with two configuration blocks for your build script that allow you to define the dependencies and repositories that you want to retrieve them from. If the standard DSL elements don’t fit your needs, you can even introduce your own vocabulary through Gradle’s extension mechanism. 

This may sound a little nebulous at first, but once you’re past the initial hurdle of learning the build language, creating maintainable and declarative builds comes easy. A good place to start is the Gradle Build Language Reference Guide at https://gradle.org/docs. Gradle’s DSL can be extended. You may want to change the behavior of an existing task or add your own idioms for describing your business domain. Gradle offers you plenty of options to do so. 

Gradle is Groovy 
Prominent build tools like Ant and Maven define their build logic through XML. As we all know, XML is easy to read and write, but can become a maintenance nightmare if used in large quantities. XML isn’t very expressive. It makes it hard to define complex custom logic. Gradle takes a different approach. Under the hood, Gradle’s DSL is written with Groovy providing syntactic sugar on top of Java. The result is a readable and expressive build language. All your scripts are written in Groovy as well. Being able to use a programming language to express your build needs is a major plus. You don’t have to be a Groovy expert to get started. Because Groovy is written on top of Java, you can migrate gradually by trying out its language features. You could even write your custom logic in plain Java—Gradle couldn’t care less. Battlescarred Groovy veterans will assure you that using Groovy instead of Java will boost your productivity by orders of magnitude. A great reference guide is the book Groovy in Action, Second Edition by Dirk Konig et al. (Manning, 2009) For a primer on Groovy. 

Flexible conventions 
One of Gradle’s big ideas is to give you guidelines and sensible defaults for your projects. Every Java project in Gradle knows exactly where source and test class file are supposed to live, and how to compile your code, run unit tests, generate Javadoc reports, and create a distribution of your code. All of these tasks are fully integrated into the build lifecycle. If you stick to the convention, there’s only minimal configuration effort on your part. In fact, your build script is a one-liner. Seriously! Do you want to learn more about building a Java project with Gradle? Well, you can—we’ll cover it in Chapter 3. Figure 2.5 illustrates how Gradle introduces conventions and lifecycle tasks for Java projects. 

Default tasks are provided that make sense in the context of a Java project. For example, you can compile your Java production source code, run tests, and assemble a JAR file. Every Java project starts with a standard directory layout. It defines where to find production source code, resource files, and test code. Convention properties are used to change the defaults. 

The same concept applies to other project archetypes like Scala, Groovy, web projects, and many more. Gradle calls this concept build by convention. The build script developer doesn’t need to know how this is working under the hood. Instead, you can concentrate on what needs to be configured. Gradle’s conventions are similar to the ones provided by Maven, but they don’t leave you feeling boxed in. Maven is very opinionated; it proposes that a project only contains one Java source directory and only produces one single JAR file. This is not necessarily reality for many enterprise projects. Gradle allows you to easily break out of the conventions. On the opposite side of the spectrum, Ant never gave you a lot of guidance on how to structure your build script, allowing for a maximum level of flexibility. Gradle takes the middle ground by offering conventions combined with the ability to easily change them. Szczepan Faber, one of Gradle’s core engineers, put it this way on his blog: “Gradle is an opinionated framework on top of an unopinionated toolkit.” (Monkey Island, “opinionated or not,” June 2, 2012, http://monkeyisland.pl/2012/06/02/opinionated-or-not/.) 

Robust and powerful dependency management 
Software projects are usually not self-contained. All too often, your application code uses a third-party library providing existing functionality to solve a specific problem. Why would you want to reinvent the wheel by implementing a persistence framework if Hibernate already exists? Within an organization, you may be the consumer of a component or module implemented by a different team. External dependencies are accessible through repositories, and the type of repository is highly dependent on what your company prefers. Options range from a plain file system to a full-fledged enterprise repository. External dependencies may have a reference to other libraries or resources. We call these transitive dependencies. 

Gradle provides an infrastructure to manage the complexity of resolving, retrieving, and storing dependencies. Once they’re downloaded and put in your local cache, they’re made available to your project. A key requirement of enterprise builds is reproducibility. Recall the story of Tom and Joe from chapter 1. Do you remember the last time your coworker said, “But it works on my box”? Builds have to produce the same result on different machines, independent of the contents of your local cache. Dependency managers like Ivy and Maven in their current implementation cannot fully guarantee reproducibility. Why is that? Whenever a dependency is downloaded and stored in the local cache, it doesn’t take into account the artifact’s origin. In situations where the repository is changed for a project, the cached dependency is considered resolved, even though the artifact’s content may be slightly different. At worst, this will cause a failing build that’s extremely hard to debug. Another common complaint specific to Ivy is the fact that dependency snapshot versions, artifacts currently under development with the naming convention –SNAPSHOT, aren’t updated correctly in the local cache, even though it changed on the repository and is marked as changing. There are many more scenarios where current solutions fall short. Gradle provides its own configurable, reliable, and efficient dependency management solution. We’ll have a closer look at its features in Chapter 5. 

Large enterprise projects usually consist of multiple modules to separate functionality. In the Gradle world, each of the submodules is considered a project that can define dependencies to external libraries or other modules. Additionally, each subproject can be run individually. Gradle figures out for you which of the subproject dependencies need to be rebuilt, without having to store a subproject’s artifact in the local cache. 

Scalable builds 
For some companies, a large project with hundreds of modules is reality. Building and testing minor code changes can consume a lot of time. You may know from personal experience that deleting old classes and resources by running a cleanup task is a natural reflex. All too often, you get burned by your build tool not picking up the changes and their dependencies. What you need is a tool that’s smart enough to only rebuild the parts of your software that actually changed. Gradle supports incremental builds by specifying task inputs and outputs. It reliably figures out for you which tasks need to be skipped, built, or partially rebuilt. The same concept translates to multimodule projects, called partial builds. Because your build clearly defines the dependencies between submodules, Gradle takes care of rebuilding only the necessary parts. No more running clean by default! 

Automated unit, integration, and functional tests are part of the build process. It makes sense to separate short-running types of tests from the ones that require setting up resources or external dependencies to be run. Gradle supports parallel test execution. This feature is fully configurable and ensures that you’re actually taking advantage of your processor’s cores. The buck doesn’t stop here. Gradle is going to support distributing test execution to multiple machines in a future version. I’m sorry to tell you, but the days of reading your Twitter feed between long builds are gone. 

Developers run builds many times during development. That means starting a new Gradle process each time, loading all its internal dependencies, and running the build logic. You’ll notice that it usually takes a couple of seconds before your script actually starts to execute. To improve the startup performance, Gradle can be run in daemon mode. In practice, the Gradle command forks a daemon process, which not only executes your build, but also keeps running in the background. Subsequent build invocations will piggyback on the existing daemon process to avoid the startup costs. As a result, you’ll notice a far snappier initial build execution. 

Effortless extendibility 
Most enterprise builds are not alike, nor do they solve the same problems. Once you’re past the initial phase of setting up your basic scripts, you’ll want to implement custom logic. Gradle is not opinionated about the way you implement that code. Instead, it gives you various choices to pick from, depending on your specific use case. The easiest way to implement custom logic is by writing a task. Tasks can be defined directly in your build script without special ceremony. If you feel like complexity takes over, you may want to explore the option of a custom task that allows for writing your logic within a class definition, making structuring your code easy and maintainable. If you want to share reusable code among builds and projects, plugins are your best friend. Representing Gradle’s most powerful extension mechanism, plugins give you full access to Gradle’s API and can be written, tested, and distributed like any other piece of software. Writing a plugin is surprisingly easy and doesn’t require a lot of additional descriptors. 

Integration with other build tools 
Wouldn’t it be a huge timesaver to be able to integrate with existing build tools? Gradle plays well with its predecessors Ant, Maven, and Ivy, as shown in figure 2.6. 

If you’re coming from Ant, Gradle doesn’t force you to fully migrate your build infrastructure. Instead, it allows you to import existing build logic and reuse standard Ant tasks. Gradle builds are 100% compatible with Maven and Ivy repositories. You can retrieve dependencies and publish your own artifacts. Gradle provides a converter for existing Maven builds that can translate the build logic into a Gradle build script. 

Existing Ant scripts can be imported into your Gradle build seamlessly and used as you’d use any other external Gradle script. Ant targets directly map to Gradle tasks at runtime. Gradle ships with the Ant libraries and exposes a helper class to your scripts called AntBuilder, which fully blends into Gradle’s DSL. It still looks and feels like Ant’s XML, but without the pointy brackets. Ant users will feel right at home, because they don’t have to transition to Gradle syntax right away. Migrating from Ant to Gradle is also a no-brainer. You can take baby steps by reusing your existing Ant logic while using Gradle’s benefits at the same time. 

Gradle aims to reach a similar depth of integration with Maven. At the time of writing, this hasn’t been realized yet. In the long run, Maven POMs and plugins will be treated as Gradle natives. Maven and Ivy repositories have become an important part of today’s build infrastructure. Imagine a world without Maven Central to help access specific versions of your favorite project dependencies. Retrieving dependencies from a repository is only one part of the story; publishing to them is just as important. With a little configuration, Gradle can upload your project’s artifact for companywide or public consumption. 

Community-driven and company-backed 
Gradle is free to use and ships with the Apache License 2.0. After its first release in April 2008, a vibrant community quickly started to form around it. Over the past five years, open source developers have made major contributions to Gradle’s core code base. Being hosted on GitHub turned out to be very beneficial to Gradle. Code changes can be submitted as pull requests and undergo a close review process by the core committers before making it into the code base. If you’re coming from other build tools like Maven, you may be used to a wide range of reusable plugins. Apart from the standard plugins shipped with the runtime, the Gradle community releases new functionality almost daily. Throughout the book, you’ll use many of the standard plugins shipped with Gradle. 

Gradleware is the technical service and support company behind Gradle. Not only does it provide professional advice for Gradle itself, it aims for a wide range of enterprise automation consulting. The company is backed by high-caliber engineers very experienced in the domain. Recently, Gradleware started to air free webinars to spark interest for newcomers and deepen knowledge for experienced Gradle users. 

Icing on the cake: additional features 
Don’t you hate having to install a new runtime for different projects? Gradle Wrapper to the rescue! It allows for downloading and installing a fresh copy of the Gradle runtime from a specified repository on any machine you run the build on. This process is automatically triggered on the first execution of the build. The Wrapper is especially useful for sharing your builds with a distributed team or running them on a CI platform. 

Gradle is also equipped with a rich command-line interface. Using command-line options, you can control everything from specifying the log level, to excluding tests, to displaying help messages. This is nothing special; other tools provide that, too. Some of the features stand out, though. Gradle allows for running commands in an abbreviated, camel-cased form. In practice, a command named runMyAwesomeTask would be callable with the abbreviation rMAT. Handy, isn’t it? Even though this book presents most of its examples by running commands in a shell, bear in mind that Gradle provides an out-of-the-box graphical user interface. 

The bigger picture: continuous delivery 
Being able to build your source code is only one aspect of the software delivery process. More importantly, you want to release your product to a production environment to deliver business value. Along the way, you want to run tests, build the distribution, analyze the code for quality-control purposes, potentially provision a target environment, and deploy to it. 

There are many benefits to automating the whole process. First and foremost, delivering software manually is slow, error-prone, and nerve-wracking. I’m sure every one of us hates the long nights due to a deployment gone wrong. With the rise of agile methodologies, development teams are able to deliver software faster. Release cycles of two or three weeks have become the norm. Some organizations like Etsy and Flickr even ship code to production several times a day! Optimally, you want to be able to release software by selecting the target environment simply by pressing a button. Practices like automated testing, CI, and deployment feed into the general concept of continuous delivery. 

In this book, we’ll look at how Gradle can help get your project from build to deployment. It’ll enable you to automate many of the tasks required to implement continuous delivery, be they compiling your source code, deploying a deliverable, or calling external tools that help you with implementing the process. For a deep dive on continuous delivery and all of its aspects, I recommend Continuous Delivery: Reliable Software Releases th...by Jez Humble and David Farley (Addison Wesley, 2010). 

Automating your project from build to deployment 
Continuous delivery introduces the concept of a deployment pipeline, also referred to as the build pipeline. A deployment pipeline represents the technical implementation of the process for getting software from version control into your production environment. The process consists of multiple stages, as shown in figure 2.7. 

■ Commit stage: 

Reports on the technical health level of your project. The main stakeholder of this phase is the development team as it provides feedback about broken code and finds “code smells.” The job of this stage is to compile the code, run tests, perform code analysis, and prepare the distribution.

■ Automated acceptance test stage: 

Asserts that functional and nonfunctional requirements are met by running automated tests.

■ Manual test stage: 

Verifies that the system is actually usable in a test environment. Usually, this stage involves QA personnel to verify requirements on the level of user stories or use cases.

■ Release stage: 

Either delivers the software to the end user as a packaged distribution or deploys it to the production environment.

Let’s see what stages of the deployment pipeline can benefit from project automation. It’s obvious that the manual test stage can be excluded from further discussion, because it only involves manual tasks. This book mainly focuses on using Gradle in the commit and automated acceptance test stages. The concrete tasks we’re going to look at are: 

■ Compiling the code
■ Running unit and integration tests
■ Performing static code analysis and generating test coverage
■ Creating the distribution
■ Provisioning the target environment
■ Deploying the deliverable
■ Performing smoke and automated functional tests

Figure 2.8 shows the order of tasks within each of the stages. While there are no hard rules that prevent you from skipping specific tasks, it’s recommended that you follow the order. For example, you could decide to compile your code, create the distribution, and deploy it to your target environment without running any tests or static code analysis. However, doing so increases the risk of undetected code defects and poor code quality. 

Topics like infrastructure provisioning, automated deployment, and smoke testing can also be applied to the release stage. In practice, applying these techniques to a production environment is more complex than in a controlled test environment. In a production environment, you may have to deal with clustered and distributed server infrastructures, zero-downtime release rollouts, and automated rollbacks to the previous release. 

Covering these advanced topics would go beyond the scope of this book. However, there are some great examples of deployment management tools in the wild that you may want to check out, such as Asgard, a web-based cloud management and deployment tool built and used by Netflix (https://github.com/Netflix/asgard). But enough pure theory—let’s get your feet wet by installing Gradle on your machine and building your first project. In chapter 3, we’ll go even further by exploring how to implement and run a complex Java project using Gradle. 

Supplement 
* Ch2 - Next-generation builds with Gradle (Part2) 
* Gradle Moto