Prediction is another common use for neural networks. A predictive neural network will attempt to predict future values based on present and past values. Such neural networks are called temporal neural networks because they operate over time. This chapter will introduce temporal neural networks and the support classes that Encog provides for them.
In this chapter, you will see two applications of Encog temporal neural networks. First, we will look at how to use Encog to predict sunspots. Sunspots are reasonably predictable and the neural network should be able to learn future patterns by analyzing past data. Next, we will examine a simple case of applying a neural network to making stock market predictions.
Before we look at either example we must see how a temporal neural network actually works. A temporal neural network is usually either a feedforward or simple recurrent network. Structured properly, the feedforward neural networks shown so far could be structured as a temporal neural network by assigning certain input and output neurons.
How a Predictive Neural Network Works
A predictive neural network uses its inputs to accept information about current data and uses its outputs to predict future data. It uses two “windows,” a future window and a past window. Both windows must have a window size, which is the amount of data that is either predicted or is needed to predict. To see the two windows in action, consider the following data.
Consider a temporal neural network with a past window size of five and a future window size of two. This neural network would have five input neurons and two output neurons. We would break the above data among these windows to produce training data. The following data shows one such element of training data.
Of course the data above needs to be normalized in some way before it can be fed to the neural network. The above illustration simply shows how the input and output neurons are mapped to the actual data. To get additional data, both windows are simply slid forward. The next element of training data would be as follows.
You would continue sliding the past and future windows forward as you generate more training data. Encog contains specialized classes to prepare data in this format. Simply specify the size of the past, or input, window and the future, or output, window. These specialized classes will be discussed in the next section.
Using the Encog Temporal Dataset
The Encog temporal dataset is contained in the following package org.encog.ml.data.temporal. There are a few classes that make up the Encog temporal dataset. These classes are as follows (encog 3.3):
The TemporalDataDescription class describes one unit of data that is either used for prediction or output. The TemporalError class is an exception that is thrown if there is an error while processing the temporal data. The TemporalMLDataSet class operates just like any Encog dataset and allows the temporal data to be used for training. The TemporalPoint class represents one point of temporal data.
To begin using a TemporalMLDataSet we must instantiate it as follows:
The RAW type specifies that the data points should be passed on to the neural network unmodified. The PERCENT CHANGE specifies that each point should be passed on as a percentage change. The DELTA CHANGE specifies that each point should be passed on as the actual change between the two values. If you are normalizing the data yourself, you would use the RAW type. Otherwise, it is very likely you would use the PERCENT CHANGE type.
Next, provide the raw data to train the temporal network from. To do this, create TemporalPoint objects and add them to the temporal dataset. Each TemporalPoint object can contain multiple values, i.e. have the same number of values in each temporal data point as in the TemporalDataDescription objects. The following code shows how to define a temporal data point.
Application to Sunspots
In this section we will see how to use Encog to predict sunspots, which are fairly periodic and predictable. A neural network can learn this pattern and predict the number of sunspots with reasonable accuracy. The output from the sunspot prediction program is shown below. Of course, the neural network first begins training and will train until the error rate falls below six percent (PredictSunspot):
Once the network has been trained, it tries to predict the number of sunspots between 1960 and 1978. It does this with at least some degree of accuracy. The number displayed is normalized and simply provides an idea of the relative number of sunspots. A larger number indicates more sunspot activity; a lower number indicates less sunspot activity.
There are two prediction numbers given: the regular prediction and the closed-loop prediction. Both prediction types use a past window of 30 and a future window of 1. The regular prediction simply uses the last 30 values from real data. The closed loop starts this way and, as it proceeds, its own predictions become the input as the window slides forward. This usually results in a less accurate prediction because any mistakes the neural network makes are compounded.
We will now examine how this program was implemented. This program can be found at PredictSunspot. As you can see, the program has sunspot data hardcoded near the top of the file. This data was taken from a C-based neural network example program. You can find the original application at the following URL: http://www.neural-networks-at-your-fingertips.com/bpn.html
The older, C-based neural network example was modified to make use of Encog. You will notice that the Encog version is much shorter than the Cbased version. This is because much of what the example did was already implemented in Encog. Further, the Encog version trains the network faster because it makes use of resilient propagation, whereas the C-based example makes use of backpropagation.
This example goes through a two-step process for using the data. First, the raw data is normalized. Then, this normalized data is loaded into a TemporalMLDataSet object for temporal training. The normalizeSunspots method is called to normalize the sunspots. This method is shown below.
Now that the sunspot data has been normalized, it should be converted to temporal data. This is done by calling the generateTraining method, shown below.
The data is now ready for training. This dataset is trained using resilient propagation. This process is the same as those used many times earlier in this book. Once training is complete, we will attempt to predict sunspots using the application. This is done with the predict method, which is shown here.
This will display a list of all of the sunspot predictions made by Encog. In the next section we will see how Encog can automatically pull current market information and attempt to predict stock market directions.
Using the Encog Market Dataset
Encog also includes a dataset specifically designed for stock market data. This dataset is capable of downloading data from external sources. Currently, the only external source included in Encog is Yahoo Finance. The Encog market dataset is built on top of the temporal dataset and most classes in the Encog market dataset descend directly from corresponding classes in the temporal data set. The following classes make up the Encog Market Dataset package:
The MarketDataDescription class represents one piece of market data that is part of either the past or future window. It descends from the TemporalDataDescription class. It consists primarily of a TickerSymbol object and a MarketDataType enumeration. The ticker symbol specifies the security to include and the MarketDataType specifies what type of data from this security to use. The available data types are listed below.
These are the market data types criteria currently supported by Encog. They are all represented inside of the MarketDataType enumeration.
The MarketMLDataSet class is descended from the TemporalMLDataSet. This is the main class when creating market-based training data for Encog. This class is an Encog dataset and can be trained. If any errors occur, the MarketError exception will be thrown.
The MarketPoint class descends from the TemporalPoint. You will usually not deal with this object directly, as Encog usually downloads market data from Yahoo Finance. The following code shows the general format for using the MarketMLDataSet class. First, create a loader. Currently, the YahooFinanceLoader is the only public loader available for Encog.
Application to the Stock Market
We will now look at an example of applying Encog to stock market prediction. This program attempts to predict the direction of a single stock based on past performance. This is a very simple stock market example and is not meant to offer any sort of investment advice. First, let’s explain how to run this example. There are four distinct modes in which this example can be run, depending on the command line argument that was passed. These arguments are summarized below.
To begin the example you should run the main class, which is named MarketPredict. The following sections will show how this example generates data, trains and then evaluates the resulting neural network. This application is located at here. Each of these modes to use this program will now be covered.
Generating Training Data
The first step is to generate the training data. The example is going to download about eight years worth of financial information to train with. It takes some time to download and process this information. The data is downloaded and written to an Encog EG file. The class MarketBuildTraining provides this functionality. All work performed by this class is in the static method named generate. This method is shown below.
Training the Neural Network
Training the neural network is very simple. The network and training data are already created and stored in an EG file. All that the training class needs to do is load both of these resources from the EG file and begin training. The MarketTrain class does this. The static method train performs all of the training. This method is shown here.
At this point, the neural network is trained. To further train the neural network, run the training again or move on to evaluating the neural network. If you train the same neural network again using resilient propagation, the error rate will initially spike. This is because the resilient propagation algorithm must reestablish proper delta values for training.
One challenge with neural networks is determining the optimal architecture for the hidden layers. Should there be one hidden layer or two? How many neurons should be in each of the hidden layers? There are no easy answers to these questions. Generally, it is best to start with a neural network with one hidden layer and double the number of hidden neurons as input neurons. There are some reports that suggest that the second hidden layer has no advantages, although this is often debated. Other reports suggest a second hidden layer can sometimes lead to faster convergence. For more information, see the hidden layer page on the Heaton Research wiki. http://www.heatonresearch.com/wiki/Hidden_Layers
One utility provided by Encog is the incremental pruning class. This class allows you to use a brute force technique to determine an optimal hidden layer configuration. Calling the market example with the prune argument will perform an incremental prune. This will try a number of different hidden layer configurations to attempt to find the best one. This command begins by loading a training set to memory.
The user may also specify the number and sizes of the hidden layers to try. Each call to addHiddenLayer specifies the lower and upper bound to try. The first call to addHiddenLayer specifies the range for the first hidden layer. Here we specify to try hidden layer one sizes from 5 to 50. Because the lower point is not zero, we are required to have a first hidden layer.
Evaluating the Neural Network
We are now ready to evaluate the neural network using the trained neural network from the last section and gauge its performance on actual current stock market data. The MarketEvaluate class contains all of the evaluation code. There are two important methods used during the evaluation process. The first is the determineDirection method which tell the direction it will move the next day.
The resulting data is returned to the calling method. Now that we have covered the support methods, it is time to learn how the actual training occurs. The static method evaluate performs the actual evaluation. This method is shown below.
The following code snippet shows the output of this application when launched once. Because it uses data preceding the current date, the results will be different when run. These results occur because the program is attempting to predict percent movement on Apple Computer’s stock price.
Here, the program had an accuracy of 60%, which is very good for this simple neural network. Accuracy rates generally range from 30-40% when this program was run at different intervals. This is a very simple stock market predictor and should not be used for any actual investing. It shows how to structure a neural network to predict market direction.