Count-distinct problem

Instance: A stream of elements ${\displaystyle x_{1},x_{2},\ldots ,x_{s}}$ with repetitions, and an integer ${\displaystyle m}$. Let ${\displaystyle n}$ be the number of distinct elements, namely ${\displaystyle n=|\left\{{x_{1},x_{2},\ldots ,x_{s}}\right\}|}$, and let these elements be ${\displaystyle \left\{{e_{1},e_{2},\ldots ,e_{n}}\right\}}$.

Objective: Find an estimate ${\displaystyle {\widehat {n}}}$ of ${\displaystyle n}$ using only ${\displaystyle m}$ storage units, where ${\displaystyle m\ll n}$.

An example of an instance for the cardinality estimation problem is the stream: ${\displaystyle a,b,a,c,d,b,d}$. For this instance, ${\displaystyle n=|\left\{{a,b,c,d}\right\}|=4}$.

The naive solution to the problem is as follows:

 Initialize a counter, c, to zero, ${\displaystyle c\leftarrow 0}$.
 Initialize an efficient dictionary data structure, D, such as hash table or search tree in which insertion and membership can be performed quickly.  
 For each element ${\displaystyle x_{i}}$, a membership query is issued. 
     If ${\displaystyle x_{i}}$ is not a member of D (${\displaystyle x_{i}\notin D}$)
         Add ${\displaystyle x_{i}}$ to D
         Increase c by one, ${\displaystyle c\leftarrow c+1}$
     Otherwise (${\displaystyle x_{i}\in D}$) do nothing.
 Output ${\displaystyle n=c}$.

As long as the number of distinct elements is not too big, D fits in main memory and an exact answer can be retrieved. However, this approach does not scale for bounded storage, or if the computation performed for each element ${\displaystyle x_{i}}$ should be minimized. In such a case, several streaming algorithms have been proposed that use a fixed number of storage units.

To handle the bounded storage constraint, streaming algorithms use a randomization to produce a non-exact estimation of the distinct number of elements, ${\displaystyle n}$. State-of-the-art estimators hash every element ${\displaystyle e_{j}}$ into a low-dimensional data sketch using a hash function, ${\displaystyle h(e_{j})}$. The different techniques can be classified according to the data sketches they store.

In its weighted version, each element is associated with a weight and the goal is to estimate the total sum of weights. Formally,

Instance: A stream of weighted elements ${\displaystyle x_{1},x_{2},\ldots ,x_{s}}$ with repetitions, and an integer ${\displaystyle m}$. Let ${\displaystyle n}$ be the number of distinct elements, namely ${\displaystyle n=|\left\{{x_{1},x_{2},\ldots ,x_{s}}\right\}|}$, and let these elements be ${\displaystyle \left\{{e_{1},e_{2},\ldots ,e_{n}}\right\}}$. Finally, let ${\displaystyle w_{j}}$ be the weight of ${\displaystyle e_{j}}$.

Objective: Find an estimate ${\displaystyle {\widehat {w}}}$ of ${\displaystyle w=\sum _{j=1}^{n}w_{j}}$ using only ${\displaystyle m}$ storage units, where ${\displaystyle m\ll n}$.

An example of an instance for the weighted problem is: ${\displaystyle a(3),b(4),a(3),c(2),d(3),b(4),d(3)}$. For this instance, ${\displaystyle e_{1}=a,e_{2}=b,e_{3}=c,e_{4}=d}$, the weights are ${\displaystyle w_{1}=3,w_{2}=4,w_{3}=2,w_{4}=3}$ and ${\displaystyle \sum {w_{j}}=12}$.

As an application example, ${\displaystyle x_{1},x_{2},\ldots ,x_{s}}$ could be IP packets received by a server. Each packet belongs to one of ${\displaystyle n}$ IP flows ${\displaystyle e_{1},e_{2},\ldots ,e_{n}}$. The weight ${\displaystyle w_{j}}$ can be the load imposed by flow ${\displaystyle e_{j}}$ on the server. Thus, ${\displaystyle \sum _{j=1}^{n}{w_{j}}}$ represents the total load imposed on the server by all the flows to which packets ${\displaystyle x_{1},x_{2},\ldots ,x_{s}}$ belong.

Any extreme order statistics estimator (min/max sketches) for the unweighted problem can be generalized to an estimator for the weighted problem .[11] For example, the weighted estimator proposed by Cohen et al.[5] can be obtained when the continuous max sketches estimator is extended to solve the weighted problem. In particular, the HyperLogLog algorithm [6] can be extended to solve the weighted problem. The extended HyperLogLog algorithm offers the best performance, in terms of statistical accuracy and memory usage, among all the other known algorithms for the weighted problem.

• Count–min sketch
• Streaming algorithm
• Maximum likelihood
• Minimum-variance unbiased estimator