Hopkins statistic

The Hopkins statistic (introduced by Brian Hopkins and John Gordon Skellam) is a way of measuring the cluster tendency of a data set.[1] It belongs to the family of sparse sampling tests. It acts as a statistical hypothesis test where the null hypothesis is that the data is generated by a Poisson point process and are thus uniformly randomly distributed.[2] If individuals are aggregated, then its value approaches 0, and if they are randomly distributed, the value tends to 0.5.[3]

Preliminaries

A typical formulation of the Hopkins statistic follows.[2]

Let X {\displaystyle X} be the set of n {\displaystyle n} data points.
Generate a random sample X ∼ {\displaystyle {\overset {\sim }{X}}} of m ≪ n {\displaystyle m\ll n} data points sampled without replacement from X {\displaystyle X} .
Generate a set Y {\displaystyle Y} of m {\displaystyle m} uniformly randomly distributed data points.
Define two distance measures,
u i , {\displaystyle u_{i},} the minimum distance (given some suitable metric) of y i ∈ Y {\displaystyle y_{i}\in Y} to its nearest neighbour in X {\displaystyle X} , and
w i , {\displaystyle w_{i},} the minimum distance of x ∼ i ∈ X ∼ ⊆ X {\displaystyle {\overset {\sim }{x}}_{i}\in {\overset {\sim }{X}}\subseteq X} to its nearest neighbour x j ∈ X , x i ∼ ≠ x j . {\displaystyle x_{j}\in X,\,{\overset {\sim }{x_{i}}}\neq x_{j}.}

Definition

With the above notation, if the data is d {\displaystyle d} dimensional, then the Hopkins statistic is defined as:[4]

H = ∑ i = 1 m u i d ∑ i = 1 m u i d + ∑ i = 1 m w i d {\displaystyle H={\frac {\sum _{i=1}^{m}{u_{i}^{d}}}{\sum _{i=1}^{m}{u_{i}^{d}}+\sum _{i=1}^{m}{w_{i}^{d}}}}\,}

Under the null hypotheses, this statistic has a Beta(m,m) distribution.

Notes and references

  1. ^ Hopkins, Brian; Skellam, John Gordon (1954). "A new method for determining the type of distribution of plant individuals". Annals of Botany. 18 (2). Annals Botany Co: 213–227. doi:10.1093/oxfordjournals.aob.a083391.
  2. ^ a b Banerjee, A. (2004). "Validating clusters using the Hopkins statistic". 2004 IEEE International Conference on Fuzzy Systems (IEEE Cat. No.04CH37542). Vol. 1. pp. 149–153. doi:10.1109/FUZZY.2004.1375706. ISBN 0-7803-8353-2. S2CID 36701919.
  3. ^ Aggarwal, Charu C. (2015). Data Mining. Cham: Springer International Publishing. p. 158. doi:10.1007/978-3-319-14142-8. ISBN 978-3-319-14141-1. S2CID 13595565.
  4. ^ Cross, G.R.; Jain, A.K. (1982). "Measurement of clustering tendency". Theory and Application of Digital Control: 315-320. doi:10.1016/B978-0-08-027618-2.50054-1.

External links

  • http://www.sthda.com/english/wiki/assessing-clustering-tendency-a-vital-issue-unsupervised-machine-learning