Distributed Silhouette Algorithm: Evaluating Clustering on Big Data

Marco Gaido
3/24/2023
cs.DCcs.AIcs.DS

No confident code implementation found for this paper.

Abstract

In the big data era, the key feature that each algorithm needs to have is the possibility of efficiently running in parallel in a distributed environment. The popular Silhouette metric to evaluate the quality of a clustering, unfortunately, does not have this property and has a quadratic computational complexity with respect to the size of the input dataset. For this reason, its execution has been hindered in big data scenarios, where clustering had to be evaluated otherwise. To fill this gap, in this paper we introduce the first algorithm that computes the Silhouette metric with linear complexity and can easily execute in parallel in a distributed environment. Its implementation is freely available in the Apache Spark ML library.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Lumbermark: Resistant Clustering by Chopping Up Mutual Reachability Minimum Spanning Trees

83% match1 repo2026

Parallelization of the K-Means Algorithm with Applications to Big Data Clustering

83% match2024

Accuracy Evaluation of Overlapping and Multi-resolution Clustering Algorithms on Large Datasets

83% match2 repos222019

Superior Parallel Big Data Clustering through Competitive Stochastic Sample Size Optimization in Big-means

82% match2024

Data Aggregation for Hierarchical Clustering

81% match1 repo8312023

Settling Time vs. Accuracy Tradeoffs for Clustering Big Data

81% match2024

Cite this paper

@article{gaido2023distributed,
  title  = {Distributed Silhouette Algorithm: Evaluating Clustering on Big Data},
  author = {Marco Gaido},
  year   = {2023},
  eprint = {2303.14102},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.14102v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.14102.svg)](https://codeofpaper.com/paper/2303.14102)

Discussion