5.1 Questions about Processes

How to I set a stopping criterion for training?
The stopping criterion is part of the statistic process that has the data on which the criterion is based. Thus, if you want to stop training based on sum-squared-error, the SE_Stat (see section 12.8.1 Summed-Error Statistics) is where to go to set the criterion. The criterion is always set in the statistic that is located in the loop_stats of the process you wish to stop. So, if you wan to stop training, look in the TrainProcess (see section 12.4.2 Iterating over Epochs: TrainProcess) for the statistic.
How do I run my network in the background for training?
Often networks take a while to learn, and it is nice to be able to run them in the background for a day or two. To do this, get the network all setup to train, save it, and then use the following type of command:
unix > pdp++ -nogui -f startup.css <myproj> <extra args> &
where `pdp++' refers to the PDP++ executable you are using (bp++, cs++, etc), and -nogui tells it to not open up the graphical user interface, and -f startup.css is a script file that will automate the process of loading the project and running it. For more information on different startup arguments for PDP++, see section 9.3 Startup Arguments to PDP++. Also, check out the comments at the top of `startup.css' (located in the `css/include' directory) to see what kinds of assumptions it makes about your project. Also see section 9.6 Project Scripts for more useful scripts that you can use. See section 9.4 Signals to Control a PDP++ Process for information on how to control the process once it is running in the background.
How do I automatically save my networks after training?
This can be accomplished by creating a SaveNetsProc in the loop_procs group of an appropriate Batch or Train process. This process names the network with the name field of the network object plus the current epoch and, if present, batch counters. You can set the mod value of this process to save the networks at different intervals. An example project using this is `demo/bp/gridsearch_xor.proj.gz'. There is a CSS script version of this process called `save_networks.css' in the `css/include' directory, as well as a modified version that saves networks that get below some criterion (which is presumably higher than the stopping criterion), called `save_good_nets.css'. These scripts are meant to be attached to ScriptProcess objects created in the loop_procs group of an appropriate Batch or Train process. They basically just issue a save command on the network, but they also give the network a name based on the current epoch and/or batch counters. See these scripts for more detailed documentation.
How do I change one of the processes in my schedule process hierarchy?
All objects can be changed to a different type using the Change Type menu option (ChangeMyType function in CSS). In the project view, just select the object you want to change, and hit the Change Type button, and it will change the type, and reset all the relevant pointers so your process hierarchy should remain intact.
How do I remove a higher-level process object without losing the lower-level ones?
If you remove a process object by using the .processes/ Remove option, it will automatically remove all the processes below the one removed. However, if you do Object/Close on the object itself, it will just remove itself and close-up the gap between the remaining processes (in terms of stat aggregation, etc). The Rmv Obj(s) button on the project view also does this kind of remove. Finally, you can use the Structure/RemoveSuperProc and RemoveSubProc menu options on a process object itself to remove its parent or sub process.
Why can't I change the environment or network pointer in my process?
The environment and network pointers are automatically inherited from the parent process in the hierarchy (this is true for stats and other process objects hanging off of a given process as well). This means that you can only change these at the top-level process in a given hierarchy. On the other hand, it means that when you want to change these pointers, you only have to change them in one place and they are automatically propagated to all processes (see section 12.1 The Basic Features of all Processes).
Where do I create a given statistic process?
Each statistic process "knows" where to create itself in the standard schedule process hierarchy. This default location will be the one shown when you use the NewStat menu option in the .processes menu on the project. Use this to create your statistic. In general, most statistics should be created in a relatively low-level process, such as the TrialProcess, since most stats deal with data that changes event-by-event. Remember that you create the stat at the lowest level so that it can compute the raw information, and these low-level values can be aggregated up the processing hierarchy if higher-level information is desired, see section 12.8 Different Types of Statistics.
Why can't I change the name of my statistic?
First, the name of the statistic object is not the same as the label that is associated with that object in the log (though they are related), so changing one will not change the other. The Stat object names are automatically set to reflect the aggregation and layer settings of the stat, see section 12.8 Different Types of Statistics. This is true as long as the name contains the type name of the stat (e.g., SE_Stat), so if you want to give a stat a custom name, make sure it doesn't include the type name. To change the label associated with the stat as it appears in the log, you need to edit the StatVal object(s) that contains the value of the statistic. This object contains a name field, which is what determines the label used by the log. However, as it appears on most stats (e.g., SE_Stat), the name field is hidden, so you have to use the CSS script to set it. Typing:
css> .processes[1].loop_stats[0].se.Edit();
will pull up an editor for the epoch level SE_Stat StatVal (which is the member se of an SE_Stat) in the `demo/bp/xor.proj.gz' demo, and allow you to edit the name. Alternatively you can use PDP++Root/Object/Browe and click your way through the process hierarchy until you get to the variable you want to change, and then hit Select to edit it. You will have to do a GetHeaders on the log to see this new name. Other stats like ScriptStat have a list of StatVals, which, when edited, allow you to edit the names directly. An easier alternative is to simply change the display_name of the ViewSpec in the relevant log. However, this will not survive a call to GetHeaders, nor will it show up in the log file saved to disk.
How do I get the epoch counter into my testing process?
The epoch counter, which is on the TrainProcess is not directly available to a testing process, which typically is created at the epoch level. Thus, you need to add a EpochCounterStat to the final stats of the process where you want to see the epoch counter. This gets the counter off of the network itself. To get other counters from the training process hierarchy (i.e., the batch counter), use the ProcCounterStat. For more info, see section 12.8.6 Miscellaneous Other Stat Types.
How do I record the activations of the units in my network?
This is accomplished by creating a MonitorStat, which gathers information from the network and makes it available for logging. The easiest way to do this is through the NetView (see section 10.6 Network Viewer), where you can select the objects to be monitored, the variable to monitor, and it will create an appropriate MonitorStat for you. Note that the monitor stat is created in the schedule process that corresponds to the time-grain at which you want the information recorded. It is the log of this process which will show the values you are recording. For more info, see section 12.8.2 Monitoring Network State Variables.
How do I determine which event is closest to the output my network actually produced?
The ClosestEventStat does exactly this, see section 12.8.3 Finding The Event Closest to the Current Output Pattern.
How do I create a cross-validation setup?
Cross-validation is accomplished by periodically testing during training. Thus, you simply need to create a testing process hierarchy, (at the Epoch level), and link your testing epoch process into the loop_procs of the training process. You should set the modulo factor (in the mod field of the process) of your testing epoch process to reflect the frequency with which you want to perform testing. See `demo/bp_misc/bp_xval.proj.gz' for a demonstration of a cross-validation training setup.
How do I set the stopping criteria for training based on testing error (e.g., for cross-validation)?
Add an aggregator stat of the testing epoch stat in the final_stats of your training epoch process (i.e., make an SE_Stat there, set its time_agg.from to point to your test epoch SE_Stat, use the LAST operator). You will want to create aggregates of this stat (which is itself an aggregator) using LAST, and set the stopping criterion in the aggregator of this stat in the loop_stats of the train process. The project `demo/bp_misc/bp_dig_xval.proj.gz' contains an example of this.
How do I graph both training and testing error (e.g., for cross-validation)?
If you follow the steps for the previous item, a graph log at the level of the training epoch process will show both the training and testing error. Again, see the demo project.
How do I make a hybrid network of two different types of
algorithms? Since all schedule process types assume that a given network has all of the appropriate types of objects (i.e., BpUnit's, BpCon's, etc for backprop), you have to create two separate networks, and then link them together with a BridgeProcess, see section 12.6.3 Linking Networks Together with a Bridge.