zbMATH — the first resource for mathematics

Feature-based cluster validation for high-dimensional data. (English) Zbl 1157.68445
Gammerman, A. (ed.), Artificial intelligence and applications. Machine learning. As part of the 26th IASTED international multi-conference on applied informatics. Calgary: International Association of Science and Technology for Development (IASTED); Anaheim, CA: Acta Press (ISBN 978-0-88986-710-9/CD-ROM). 232-239 (2008).
Summary: Cluster validation is commonly used to determine the optimal number of clusters in a data set. Despite the success of distance-based validity indexes, their efficacy decreases rapidly when dealing with high-dimensional data. The present paper introduces a feature-based cluster validation criterion which can cope with said situation. In contrast to distance-based methods, our criterion evaluates similarity in terms of shared relevant features between data.
The idea is based on the identification of the \`\` core” features which are correlated within the description of each of the discovered clusters. The individual quality of each cluster is then evaluated through the frequency of the core features with respect to that of the non-core features within the cluster, while the between-cluster isolation is measured by means of the overlap coefficient between clusters, considering only the core features within the clusters. The overall clustering quality is measured by a weighted combination of the within and between cluster correlation coefficients, which enables choosing an appropriate number of clusters according to the purpose of clustering. Furthermore, our validation can prune out unreliable clusters which have no correlated features and thus no specific description of their content. Extensive experiments on the Reuters-21578 collection are conducted to show the effectiveness of our validation criterion.
For the entire collection see [Zbl 1154.68012].
68T10 Pattern recognition, speech recognition
68P05 Data structures
PDF BibTeX Cite