Estimation of Spectral Clustering Hyper Parameters
share
Robust automation of analysis procedures capable of handling diverse data sets is critical for high data throughput experiments at the Linac Coherent Light Source (LCLS). One challenge encountered in this process is determining the number of clusters required for the execution of conventional clustering algorithms. It is demonstrated here that bi-cross validation of the inverted and regularized Laplacian used in the spectral clustering algorithm, yields a robust minimum at the predicted number of clusters and kernel hyper parameters. These results indicate that the process of estimating the number of clusters should not be divorced from the process of estimating other hyper parameters. Applying this method to LCLS x-ray scattering data demonstrates the ability to identify clusters of dropped shots without manually setting boundaries on detector fluence and provides a path towards identifying rare events.
READ FULL TEXT