Linear discriminant analysis (LDA), normal discriminant analysis (NDA), canonical variates analysis (CVA), or discriminant function analysis is a generalization of Fisher's linear discriminant, a method used in statistics and other fields, to find a linear combination of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.
LDA is closely related to analysis of variance (ANOVA) and regression analysis, which also attempt to express one dependent variable as a linear combination of other features or measurements. However, ANOVA uses categorical independent variables and a continuous dependent variable, whereas discriminant analysis has continuous independent variables and a categorical dependent variable (i.e. the class label).Analyzing Quantitative Data: An Introduction for Social Researchers, Debra Wetcher-Hendricks, p.288 Logistic regression and probit regression are more similar to LDA than ANOVA is, as they also explain a categorical variable by the values of continuous independent variables. These other methods are preferable in applications where it is not reasonable to assume that the independent variables have a normal distribution, which is a fundamental assumption of the LDA method.
LDA is also closely related to principal component analysis (PCA) and factor analysis in that they both look for linear combinations of variables which best explain the data. LDA explicitly attempts to model the difference between the classes of data. PCA, in contrast, does not take into account any difference in class, and factor analysis builds the feature combinations based on similarities rather than differences. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.
LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.Abdi, H. (2007) "Discriminant correspondence analysis." In: N.J. Salkind (Ed.): Encyclopedia of Measurement and Statistic. Thousand Oaks (CA): Sage. pp. 270–275.
Discriminant analysis is used when groups are known a priori (unlike in cluster analysis). Each case must have a score on one or more quantitative predictor measures, and a score on a group measure.Büyüköztürk, Ş. & Çokluk-Bökeoğlu, Ö. (2008). Discriminant function analysis: Concept and application. Egitim Arastirmalari - Eurasian Journal of Educational Research, 33, 73-92. In simple terms, discriminant function analysis is classification - the act of distributing things into groups, classes or categories of the same type.
History The original dichotomous discriminant analysis was developed by Sir Ronald Fisher in 1936.Cohen et al. Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences 3rd ed. (2003). Taylor & Francis Group. It is different from an ANOVA or MANOVA, which is used to predict one (ANOVA) or multiple (MANOVA) continuous dependent variables by one or more independent categorical variables. Discriminant function analysis is useful in determining whether a set of variables is effective in predicting category membership.
LDA for two classes
Consider a set of observations (also called features, attributes, variables or measurements) for each sample of an object or event with known class . This set of samples is called the training set in a supervised learning context. The classification problem is then to find a good predictor for the class of any sample of the same distribution (not necessarily from the training set) given only an observation .
LDA approaches the problem by assuming that the conditional probability density functions and are both the normal distribution with mean and covariance parameters and , respectively. Under this assumption, the Bayes-optimal solution is to predict points as being from the second class if the log of the likelihood ratios is bigger than some threshold T, so that:
Without any further assumptions, the resulting classifier is referred to as quadratic discriminant analysis (QDA).
LDA instead makes the additional simplifying homoscedasticity assumption (i.e. that the class covariances are identical, so ) and that the covariances have full rank. In this case, several terms cancel:
because both sides are scalar and transpose to each other ( is Hermitian)
and the above decision criterion becomes a threshold on the dot product
for some threshold constant c, where
This means that the criterion of an input being in a class is purely a function of this linear combination of the known observations.
It is often useful to see this conclusion in geometrical terms: the criterion of an input being in a class is purely a function of projection of multidimensional-space point onto vector (thus, we only consider its direction). In other words, the observation belongs to if corresponding is located on a certain side of a hyperplane perpendicular to . The location of the plane is defined by the threshold .
Assumptions The assumptions of discriminant analysis are the same as those for MANOVA. The analysis is quite sensitive to outliers and the size of the smallest group must be larger than the number of predictor variables.
Multivariate normality: Independent variables are normal for each level of the grouping variable. Homogeneity of variance/covariance (homoscedasticity): Variances among group variables are the same across levels of predictors. Can be tested with Box's M statistic. It has been suggested, however, that linear discriminant analysis be used when covariances are equal, and that quadratic discriminant analysis may be used when covariances are not equal. Independence: Participants are assumed to be randomly sampled, and a participant's score on one variable is assumed to be independent of scores on that variable for all other participants.
It has been suggested that discriminant analysis is relatively robust to slight violations of these assumptions,Peter A. Lachenbruch. (1975). Discriminant analysis. NY: Hafner and it has also been shown that discriminant analysis may still be reliable when using dichotomous variables (where multivariate normality is often violated).Klecka, William R. (1980). Discriminant analysis. Quantitative Applications in the Social Sciences Series, No. 19. Thousand Oaks, CA: Sage Publications.
Discriminant functions Discriminant analysis works by creating one or more linear combinations of predictors, creating a new latent variable for each function. These functions are called discriminant functions. The number of functions possible is either where = number of groups, or (the number of predictors), whichever is smaller. The first function created maximizes the differences between groups on that function. The second function maximizes differences on that function, but also must not be correlated with the previous function. This continues with subsequent functions with the requirement that the new function not be correlated with any of the previous functions.
Given group , with sets of sample space, there is a discriminant rule such that if , then . Discriminant analysis then, finds "good" regions of to minimize classification error, therefore leading to a high percent correct classified in the classification table.Hardle, W., Simar, L. (2007). Applied Multivariate Statistical Analysis. Springer Berlin Heidelberg. pp. 289–303.
Each function is given a discriminant score to determine how well it predicts group placement. Structure Correlation Coefficients: The correlation between each predictor and the discriminant score of each function. This is a zero-order correlation (i.e., not corrected for the other predictors).Garson, G. D. (2008). Discriminant function analysis. https://web.archive.org/web/20080312065328/http://www2.chass.ncsu.edu/garson/pA765/discrim.htm. Standardized Coefficients: Each predictor's weight in the linear combination that is the discriminant function. Like in a regression equation, these coefficients are partial (i.e., corrected for the other predictors). Indicates the unique contribution of each predictor in predicting group assignment. Functions at Group Centroids: Mean discriminant scores for each grouping variable are given for each function. The farther apart the means are, the less error there will be in classification.
Discrimination rules
Maximum likelihood: Assigns to the group that maximizes population (group) density.Hardle, W., Simar, L. (2007). Applied Multivariate Statistical Analysis. Springer Berlin Heidelberg. pp. 289-303. Bayes Discriminant Rule: Assigns to the group that maximizes , where πi represents the prior probability of that classification, and represents the population density. Fisher's linear discriminant rule: Maximizes the ratio between SSbetween and SSwithin, and finds a linear combination of the predictors to predict group.
Eigenvalues An eigenvalue in discriminant analysis is the characteristic root of each function. It is an indication of how well that function differentiates the groups, where the larger the eigenvalue, the better the function differentiates. This however, should be interpreted with caution, as eigenvalues have no upper limit. The eigenvalue can be viewed as a ratio of SSbetween and SSwithin as in ANOVA when the dependent variable is the discriminant function, and the groups are the levels of the IV. This means that the largest eigenvalue is associated with the first function, the second largest with the second, etc..
Effect size Some suggest the use of eigenvalues as effect size measures, however, this is generally not supported. Instead, the canonical correlation is the preferred measure of effect size. It is similar to the eigenvalue, but is the square root of the ratio of SSbetween and SStotal. It is the correlation between groups and the function. Another popular measure of effect size is the percent of variance for each function. This is calculated by: where is the eigenvalue for the function and is the sum of all eigenvalues. This tells us how strong the prediction is for that particular function compared to the others. Percent correctly classified can also be analyzed as an effect size. The kappa value can describe this while correcting for chance agreement.
Canonical discriminant analysis for k classes Canonical discriminant analysis (CDA) finds axes (k − 1 canonical coordinates, k being the number of classes) that best separate the categories. These linear functions are uncorrelated and define, in effect, an optimal k − 1 space through the n-dimensional cloud of data that best separates (the projections in that space of) the k groups. See "Multiclass LDA" for details below.
Because LDA uses canonical variates, it was initially often referred as the "method of canonical variates" or canonical variates analysis (CVA).
Fisher's linear discriminant The terms Fisher's linear discriminant and LDA are often used interchangeably, although Fisher's original article actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as normally distributed classes or equal class covariances.
Suppose two classes of observations have means and covariances . Then the linear combination of features will have means and variances for . Fisher defined the separation between these two distributions to be the ratio of the variance between the classes to the variance within the classes:
This measure is, in some sense, a measure of the signal-to-noise ratio for the class labelling. It can be shown that the maximum separation occurs when