AdaBoost Multiclass Classifier
AdaBoost (short for “Adaptive Boosting”) is a popular boosting classification algorithm.
The AdaBoost algorithm performs well on a variety of data sets except some noisy data ([Friedman98], [Zhu2005]).
The library supports two methods for the algorithms:
- SAMME, or Stagewise Additive Modeling using a Multi-class Exponential loss function [Zhu2005]
- SAMME.R, which is a modification of SAMME method for Real-valued returned probabilities from weak learner
Details
Given
feature vectors
of size
, a vector of class labels
where
in case of binary classification and
, where
is a number of classes,
describes the class
the feature vector
belongs to,
and
is a weak learner algorithm, the problem is to build an AdaBoost classifier.
Training Stage
SAMME method
The following scheme shows the major steps of the SAMME algorithm:
- Initialize weights
for
- For
:
- Train the weak learner
using weights
.
- Choose a confidence value
, where
- Update
, where
is a normalization factor.
- Output the final hypothesis:
SAMME algorithm in case of binary classification is equal to the AdaBoost algorithm from [Friedman98].
SAMME.R method
The following scheme shows the major steps of the SAMME.R algorithm:
- Initialize weights
for
- For
:
- Train the weak learner
using weights
.
- Receive the weighed class probability estimates from weak learner:
- For
, set
:
- For
, update
:
whereis a normalization factor,
,
- Output the final hypothesis:
Prediction Stage
SAMME method
Given the AdaBoost classifier and
feature vectors
,
the problem is to calculate the final class
:
SAMME.R method
Given the AdaBoost classifier and
feature vectors
,
the problem is to calculate the final class
:
where
is as defined above in Training Stage.
Batch Processing
AdaBoost classifier follows the general workflow described in Classification Usage Model.
Training
For a description of the input and output, refer to Classification Usage Model.
At the training stage, an AdaBoost classifier has the following parameters:
Parameter | Default Value | Description |
---|---|---|
algorithmFPType | float | The floating-point type that the algorithm uses for intermediate computations. Can be float or double . |
method | defaultDense | Available methods for computation of the AdaBoost algorithm:
|
weakLearnerTraining | Pointer to an object of the classification stump training class | Pointer to the training algorithm of the weak learner. By default, a classification stump weak learner is used. |
weakLearnerPrediction | Pointer to an object of the classification stump prediction class | Pointer to the prediction algorithm of the weak learner. By default, a classification stump weak learner is used. |
accuracyThreshold | AdaBoost training accuracy. | |
maxIterations | The maximal number of iterations for the algorithm. | |
learningRate | Multiplier for each classifier to shrink its contribution. | |
nClasses | The number of classes. | |
resultsToCompute | The 64-bit integer flag that specifies which extra characteristics of AdaBoost to compute.
Current version of the library only provides the following option: computeWeakLearnersErrors |
Output
In addition to classifier output, AdaBoost calculates the result described below.
Pass the
Result ID
as a parameter to the methods that access the result of your algorithm.
For more details, see Algorithms.Result ID | Result |
---|---|
weakLearnersErrors | A numeric table computeWeakLearnersErrors option is on.By default, this result is an object of the HomogenNumericTable class,
but you can define the result as an object of any class derived from NumericTable . |
Prediction
For a description of the input and output, refer to Classification Usage Model.
At the prediction stage, an AdaBoost classifier has the following parameters:
Parameter | Default Value | Description |
---|---|---|
algorithmFPType | float | The floating-point type that the algorithm uses for intermediate computations. Can be float or double . |
method | defaultDense | Performance-oriented computation method, the only method supported by the AdaBoost classifier at the prediction stage. |
weakLearnerPrediction | Pointer to an object of the classification stump prediction class | Pointer to the prediction algorithm of the weak learner. By default, a classification stump weak learner is used. |
nClasses | The number of classes. |
Examples
C++ (CPU)
Batch Processing:
Java*
There is no support for Java on GPU.
Batch Processing: