Motivations

There are several interrelated reasons for granulating variables in this fashion:

• Based on prior domain knowledge, there is no expectation that minute variations in temperature (e.g., the difference between 80–80.7 °F (26.7–27.1 °C)) could have an influence on behaviors driving the number of health-club applications. For this reason, any "regularity" which our learning algorithms might detect at this level of resolution would have to be spurious, as an artifact of overfitting. By coarsening the temperature variable into intervals the difference between which we do anticipate (based on prior domain knowledge) might influence number of health-club applications, we eliminate the possibility of detecting these spurious patterns. Thus, in this case, reducing resolution is a method of controlling overfitting.
• By reducing the number of intervals in the temperature variable (i.e., increasing its grain size), we increase the amount of sample data indexed by each interval designation. Thus, by coarsening the variable, we increase sample sizes and achieve better statistical estimation. In this sense, increasing granularity provides an antidote to the so-called curse of dimensionality, which relates to the exponential decrease in statistical power with increase in number of dimensions or variable cardinality.
• Independent of prior domain knowledge, it is often the case that meaningful regularities (i.e., which can be detected by a given learning methodology, representational language, etc.) may exist at one level of resolution and not at another.

Benefits of value granulation: Implications here exist at the resolution of ${\displaystyle \{X_{i},Y_{j}\}}$ that do not exist at the higher resolution of ${\displaystyle \{x_{i},y_{j}\};}$ in particular, ${\displaystyle \forall x_{i},y_{j}:x_{i}\not \to y_{j},}$ while at the same time, ${\displaystyle \forall X_{i}\exists Y_{j}:X_{i}\leftrightarrow Y_{j}.}$

For example, a simple learner or pattern recognition system may seek to extract regularities satisfying a conditional probability threshold such as ${\displaystyle p(Y=y_{j}|X=x_{i})\geq \alpha .}$ In the special case where ${\displaystyle \alpha =1,}$ this recognition system is essentially detecting logical implication of the form ${\displaystyle X=x_{i}\rightarrow Y=y_{j}}$ or, in words, "if ${\displaystyle X=x_{i},}$ then ${\displaystyle Y=y_{j}}$". The system's ability to recognize such implications (or, in general, conditional probabilities exceeding threshold) is partially contingent on the resolution with which the system analyzes the variables.

As an example of this last point, consider the feature space shown to the right. The variables may each be regarded at two different resolutions. Variable ${\displaystyle X}$ may be regarded at a high (quaternary) resolution wherein it takes on the four values ${\displaystyle \{x_{1},x_{2},x_{3},x_{4}\}}$ or at a lower (binary) resolution wherein it takes on the two values ${\displaystyle \{X_{1},X_{2}\}.}$ Similarly, variable ${\displaystyle Y}$ may be regarded at a high (quaternary) resolution or at a lower (binary) resolution, where it takes on the values ${\displaystyle \{y_{1},y_{2},y_{3},y_{4}\}}$ or ${\displaystyle \{Y_{1},Y_{2}\},}$ respectively. At the high resolution, there are no detectable implications of the form ${\displaystyle X=x_{i}\rightarrow Y=y_{j},}$ since every ${\displaystyle x_{i}}$ is associated with more than one ${\displaystyle y_{j},}$ and thus, for all ${\displaystyle x_{i},}$ ${\displaystyle p(Y=y_{j}|X=x_{i})<1.}$ However, at the low (binary) variable resolution, two bilateral implications become detectable: ${\displaystyle X=X_{1}\leftrightarrow Y=Y_{1}}$ and ${\displaystyle X=X_{2}\leftrightarrow Y=Y_{2}}$, since every ${\displaystyle X_{1}}$ occurs iff ${\displaystyle Y_{1}}$ and ${\displaystyle X_{2}}$ occurs iff ${\displaystyle Y_{2}.}$ Thus, a pattern recognition system scanning for implications of this kind would find them at the binary variable resolution, but would fail to find them at the higher quaternary variable resolution.

Issues and methods

It is not feasible to exhaustively test all possible discretization resolutions on all variables in order to see which combination of resolutions yields interesting or significant results. Instead, the feature space must be preprocessed (often by an entropy analysis of some kind) so that some guidance can be given as to how the discretization process should proceed. Moreover, one cannot generally achieve good results by naively analyzing and discretizing each variable independently, since this may obliterate the very interactions that we had hoped to discover.

A sample of papers that address the problem of variable discretization in general, and multiple-variable discretization in particular, is as follows: Chiu, Wong & Cheung (1991), Bay (2001), Liu et al. (2002), Wang & Liu (1998), Zighed, Rabaséda & Rakotomalala (1998), Catlett (1991), Dougherty, Kohavi & Sahami (1995), Monti & Cooper (1999), Fayyad & Irani (1993), Chiu, Cheung & Wong (1990), Nguyen & Nguyen (1998), Grzymala-Busse & Stefanowski (2001), Ting (1994), Ludl & Widmer (2000), Pfahringer (1995), An & Cercone (1999), Chiu & Cheung (1989), Chmielewski & Grzymala-Busse (1996), Lee & Shin (1994), Liu & Wellman (2002), Liu & Wellman (2004).

Variable transformation

A number of classical methods, such as principal component analysis, multidimensional scaling, factor analysis, and structural equation modeling, and their relatives, fall under the genus of "variable transformation." Also in this category are more modern areas of study such as dimensionality reduction, projection pursuit, and independent component analysis. The common goal of these methods in general is to find a representation of the data in terms of new variables, which are a linear or nonlinear transformation of the original variables, and in which important statistical relationships emerge. The resulting variable sets are almost always smaller than the original variable set, and hence these methods can be loosely said to impose a granulation on the feature space. These dimensionality reduction methods are all reviewed in the standard texts, such as Duda, Hart & Stork (2001), Witten & Frank (2005), and Hastie, Tibshirani & Friedman (2001).

Variable aggregation

A different class of variable granulation methods derive more from data clustering methodologies than from the linear systems theory informing the above methods. It was noted fairly early that one may consider "clustering" related variables in just the same way that one considers clustering related data. In data clustering, one identifies a group of similar entities (using a "measure of similarity" suitable to the domain — Martino, Giuliani & Rizzi (2018)), and then in some sense replaces those entities with a prototype of some kind. The prototype may be the simple average of the data in the identified cluster, or some other representative measure. But the key idea is that in subsequent operations, we may be able to use the single prototype for the data cluster (along with perhaps a statistical model describing how exemplars are derived from the prototype) to stand in for the much larger set of exemplars. These prototypes are generally such as to capture most of the information of interest concerning the entities.

A Watanabe-Kraskov variable agglomeration tree. Variables are agglomerated (or "unitized") from the bottom-up, with each merge-node representing a (constructed) variable having entropy equal to the joint entropy of the agglomerating variables. Thus, the agglomeration of two m-ary variables ${\displaystyle X_{1},X_{2}}$ having individual entropies ${\displaystyle H(X_{1}),H(X_{2})}$ yields a single m²-ary variable ${\displaystyle X_{1,2}}$ with entropy ${\displaystyle H(X_{1,2})=H(X_{1},X_{2}).}$ When ${\displaystyle X_{1},X_{2}}$ are highly dependent (i.e., redundant) and have large mutual information ${\displaystyle I(X_{1};X_{2}),}$ then ${\displaystyle H(X_{1,2})\ll H(X_{1})+H(X_{2})}$ because ${\displaystyle H(X_{1},X_{2})=H(X_{1})+H(X_{2})-I(X_{1};X_{2}),}$ and this would be considered a parsimonious unitization or aggregation.

Similarly, it is reasonable to ask whether a large set of variables might be aggregated into a smaller set of prototype variables that capture the most salient relationships between the variables. Although variable clustering methods based on linear correlation have been proposed (Duda, Hart & Stork 2001;Rencher 2002), more powerful methods of variable clustering are based on the mutual information between variables. Watanabe has shown (Watanabe 1960;Watanabe 1969) that for any set of variables one can construct a polytomic (i.e., n-ary) tree representing a series of variable agglomerations in which the ultimate "total" correlation among the complete variable set is the sum of the "partial" correlations exhibited by each agglomerating subset (see figure). Watanabe suggests that an observer might seek to thus partition a system in such a way as to minimize the interdependence between the parts "... as if they were looking for a natural division or a hidden crack."

One practical approach to building such a tree is to successively choose for agglomeration the two variables (either atomic variables or previously agglomerated variables) which have the highest pairwise mutual information (Kraskov et al. 2003). The product of each agglomeration is a new (constructed) variable that reflects the local joint distribution of the two agglomerating variables, and thus possesses an entropy equal to their joint entropy. (From a procedural standpoint, this agglomeration step involves replacing two columns in the attribute-value table—representing the two agglomerating variables—with a single column that has a unique value for every unique combination of values in the replaced columns (Kraskov et al. 2003). No information is lost by such an operation; however, if one is exploring the data for inter-variable relationships, it would generally not be desirable to merge redundant variables in this way, since in such a context it is likely to be precisely the redundancy or dependency between variables that is of interest; and once redundant variables are merged, their relationship to one another can no longer be studied.

Equivalence class granulation

We illustrate with an example. Consider the attribute-value system below:

Sample Information System

Object	${\displaystyle P_{1}}$	${\displaystyle P_{2}}$	${\displaystyle P_{3}}$	${\displaystyle P_{4}}$	${\displaystyle P_{5}}$
${\displaystyle O_{1}}$	1	2	0	1	1
${\displaystyle O_{2}}$	1	2	0	1	1
${\displaystyle O_{3}}$	2	0	0	1	0
${\displaystyle O_{4}}$	0	0	1	2	1
${\displaystyle O_{5}}$	2	1	0	2	1
${\displaystyle O_{6}}$	0	0	1	2	2
${\displaystyle O_{7}}$	2	0	0	1	0
${\displaystyle O_{8}}$	0	1	2	2	1
${\displaystyle O_{9}}$	2	1	0	2	2
${\displaystyle O_{10}}$	2	0	0	1	0

When the full set of attributes ${\displaystyle P=\{P_{1},P_{2},P_{3},P_{4},P_{5}\}}$ is considered, we see that we have the following seven equivalence classes or primitive (simple) concepts:

${\displaystyle {\begin{cases}\{O_{1},O_{2}\}\\\{O_{3},O_{7},O_{10}\}\\\{O_{4}\}\\\{O_{5}\}\\\{O_{6}\}\\\{O_{8}\}\\\{O_{9}\}\end{cases}}}$

Thus, the two objects within the first equivalence class, ${\displaystyle \{O_{1},O_{2}\},}$ cannot be distinguished from one another based on the available attributes, and the three objects within the second equivalence class, ${\displaystyle \{O_{3},O_{7},O_{10}\},}$ cannot be distinguished from one another based on the available attributes. The remaining five objects are each discernible from all other objects. Now, let us imagine a projection of the attribute value system onto attribute ${\displaystyle P_{1}}$ alone, which would represent, for example, the view from an observer which is only capable of detecting this single attribute. Then we obtain the following much coarser equivalence class structure.

${\displaystyle {\begin{cases}\{O_{1},O_{2}\}\\\{O_{3},O_{5},O_{7},O_{9},O_{10}\}\\\{O_{4},O_{6},O_{8}\}\end{cases}}}$

This is in a certain regard the same structure as before, but at a lower degree of resolution (larger grain size). Just as in the case of value granulation (discretization/quantization), it is possible that relationships (dependencies) may emerge at one level of granularity that are not present at another. As an example of this, we can consider the effect of concept granulation on the measure known as attribute dependency (a simpler relative of the mutual information).

To establish this notion of dependency (see also rough sets), let ${\displaystyle [x]_{Q}=\{Q_{1},Q_{2},Q_{3},\dots ,Q_{N}\}}$ represent a particular concept granulation, where each ${\displaystyle Q_{i}}$ is an equivalence class from the concept structure induced by attribute set Q. For example, if the attribute set Q consists of attribute ${\displaystyle P_{1}}$ alone, as above, then the concept structure ${\displaystyle [x]_{Q}}$ will be composed of

${\displaystyle {\begin{aligned}Q_{1}&=\{O_{1},O_{2}\},\\Q_{2}&=\{O_{3},O_{5},O_{7},O_{9},O_{10}\},\\Q_{3}&=\{O_{4},O_{6},O_{8}\}.\end{aligned}}}$

The dependency of attribute set Q on another attribute set P, ${\displaystyle \gamma _{P}(Q),}$ is given by

${\displaystyle \gamma _{P}(Q)={\frac {\left|\sum _{i=1}^{N}{\underline {P}}Q_{i}\right|}{\left|\mathbb {U} \right|}}\leq 1}$

That is, for each equivalence class ${\displaystyle Q_{i}}$ in ${\displaystyle [x]_{Q},}$ we add up the size of its "lower approximation" (see rough sets) by the attributes in P, i.e., ${\displaystyle {\underline {P}}Q_{i}.}$ More simply, this approximation is the number of objects which on attribute set P can be positively identified as belonging to target set ${\displaystyle Q_{i}.}$ Added across all equivalence classes in ${\displaystyle [x]_{Q},}$ the numerator above represents the total number of objects which—based on attribute set P—can be positively categorized according to the classification induced by attributes Q. The dependency ratio therefore expresses the proportion (within the entire universe) of such classifiable objects, in a sense capturing the "synchronization" of the two concept structures ${\displaystyle [x]_{Q}}$ and ${\displaystyle [x]_{P}.}$ The dependency ${\displaystyle \gamma _{P}(Q)}$ "can be interpreted as a proportion of such objects in the information system for which it suffices to know the values of attributes in P to determine the values of attributes in Q" (Ziarko & Shan 1995).

Having gotten definitions now out of the way, we can make the simple observation that the choice of concept granularity (i.e., choice of attributes) will influence the detected dependencies among attributes. Consider again the attribute value table from above:

Sample Information System

Object	${\displaystyle P_{1}}$	${\displaystyle P_{2}}$	${\displaystyle P_{3}}$	${\displaystyle P_{4}}$	${\displaystyle P_{5}}$
${\displaystyle O_{1}}$	1	2	0	1	1
${\displaystyle O_{2}}$	1	2	0	1	1
${\displaystyle O_{3}}$	2	0	0	1	0
${\displaystyle O_{4}}$	0	0	1	2	1
${\displaystyle O_{5}}$	2	1	0	2	1
${\displaystyle O_{6}}$	0	0	1	2	2
${\displaystyle O_{7}}$	2	0	0	1	0
${\displaystyle O_{8}}$	0	1	2	2	1
${\displaystyle O_{9}}$	2	1	0	2	2
${\displaystyle O_{10}}$	2	0	0	1	0

Consider the dependency of attribute set ${\displaystyle Q=\{P_{4},P_{5}\}}$ on attribute set ${\displaystyle P=\{P_{2},P_{3}\}.}$ That is, we wish to know what proportion of objects can be correctly classified into classes of ${\displaystyle [x]_{Q}}$ based on knowledge of ${\displaystyle [x]_{P}.}$ The equivalence classes of ${\displaystyle [x]_{Q}}$ and of ${\displaystyle [x]_{P}}$ are shown below.

${\displaystyle [x]_{Q}}$	${\displaystyle [x]_{P}}$
${\displaystyle {\begin{cases}\{O_{1},O_{2}\}\\\{O_{3},O_{7},O_{10}\}\\\{O_{4},O_{5},O_{8}\}\\\{O_{6},O_{9}\}\end{cases}}}$	${\displaystyle {\begin{cases}\{O_{1},O_{2}\}\\\{O_{3},O_{7},O_{10}\}\\\{O_{4},O_{6}\}\\\{O_{5},O_{9}\}\\\{O_{8}\}\end{cases}}}$

The objects that can be definitively categorized according to concept structure ${\displaystyle [x]_{Q}}$ based on ${\displaystyle [x]_{P}}$ are those in the set ${\displaystyle \{O_{1},O_{2},O_{3},O_{7},O_{8},O_{10}\},}$ and since there are six of these, the dependency of Q on P, ${\displaystyle \gamma _{P}(Q)=6/10.}$ This might be considered an interesting dependency in its own right, but perhaps in a particular data mining application only stronger dependencies are desired.

We might then consider the dependency of the smaller attribute set ${\displaystyle Q=\{P_{4}\}}$ on the attribute set ${\displaystyle P=\{P_{2},P_{3}\}.}$ The move from ${\displaystyle Q=\{P_{4},P_{5}\}}$ to ${\displaystyle Q=\{P_{4}\}}$ induces a coarsening of the class structure ${\displaystyle [x]_{Q},}$ as will be seen shortly. We wish again to know what proportion of objects can be correctly classified into the (now larger) classes of ${\displaystyle [x]_{Q}}$ based on knowledge of ${\displaystyle [x]_{P}.}$ The equivalence classes of the new ${\displaystyle [x]_{Q}}$ and of ${\displaystyle [x]_{P}}$ are shown below.

${\displaystyle [x]_{Q}}$	${\displaystyle [x]_{P}}$
${\displaystyle {\begin{cases}\{O_{1},O_{2},O_{3},O_{7},O_{10}\}\\\{O_{4},O_{5},O_{6},O_{8},O_{9}\}\end{cases}}}$	${\displaystyle {\begin{cases}\{O_{1},O_{2}\}\\\{O_{3},O_{7},O_{10}\}\\\{O_{4},O_{6}\}\\\{O_{5},O_{9}\}\\\{O_{8}\}\end{cases}}}$

Clearly, ${\displaystyle [x]_{Q}}$ has a coarser granularity than it did earlier. The objects that can now be definitively categorized according to the concept structure ${\displaystyle [x]_{Q}}$ based on ${\displaystyle [x]_{P}}$ constitute the complete universe ${\displaystyle \{O_{1},O_{2},\ldots ,O_{10}\}}$, and thus the dependency of Q on P, ${\displaystyle \gamma _{P}(Q)=1.}$ That is, knowledge of membership according to category set ${\displaystyle [x]_{P}}$ is adequate to determine category membership in ${\displaystyle [x]_{Q}}$ with complete certainty; In this case we might say that ${\displaystyle P\rightarrow Q.}$ Thus, by coarsening the concept structure, we were able to find a stronger (deterministic) dependency. However, we also note that the classes induced in ${\displaystyle [x]_{Q}}$ from the reduction in resolution necessary to obtain this deterministic dependency are now themselves large and few in number; as a result, the dependency we found, while strong, may be less valuable to us than the weaker dependency found earlier under the higher resolution view of ${\displaystyle [x]_{Q}.}$

In general it is not possible to test all sets of attributes to see which induced concept structures yield the strongest dependencies, and this search must be therefore be guided with some intelligence. Papers which discuss this issue, and others relating to intelligent use of granulation, are those by Y.Y. Yao and Lotfi Zadeh listed in the #References below.

Component granulation

Another perspective on concept granulation may be obtained from work on parametric models of categories. In mixture model learning, for example, a set of data is explained as a mixture of distinct Gaussian (or other) distributions. Thus, a large amount of data is "replaced" by a small number of distributions. The choice of the number of these distributions, and their size, can again be viewed as a problem of concept granulation. In general, a better fit to the data is obtained by a larger number of distributions or parameters, but in order to extract meaningful patterns, it is necessary to constrain the number of distributions, thus deliberately coarsening the concept resolution. Finding the "right" concept resolution is a tricky problem for which many methods have been proposed (e.g., AIC, BIC, MDL, etc.), and these are frequently considered under the rubric of "model regularization".