Cumulative Gain

DCG is a refinement of simpler measure, Cumulative Gain (CG).^[2] Cumulative Gain is the sum of the graded relevance values of all results in a search result list. CG does not take into account the rank (position) of a result in the result list. The CG at a particular rank position ${\displaystyle p}$ is defined as:

${\displaystyle \mathrm {CG_{p}} =\sum _{i=1}^{p}rel_{i}}$

Where ${\displaystyle rel_{i}}$ is the graded relevance of the result at position ${\displaystyle i}$.

The value computed with the CG function is unaffected by changes in the ordering of search results. That is, moving a highly relevant document ${\displaystyle d_{i}}$ above a higher ranked, less relevant, document ${\displaystyle d_{j}}$ does not change the computed value for CG (assuming ${\displaystyle i,j\leq p}$). Based on the two assumptions made above about the usefulness of search results, (N)DCG is usually preferred over CG. Cumulative Gain is sometimes called Graded Precision.

Discounted Cumulative Gain

The premise of DCG is that highly relevant documents appearing lower in a search result list should be penalized as the graded relevance value is reduced logarithmically proportional to the position of the result.

The usual formula of DCG accumulated at a particular rank position ${\displaystyle p}$ is defined as:^[1]

${\displaystyle \mathrm {DCG_{p}} =\sum _{i=1}^{p}{\frac {rel_{i}}{\log _{2}(i+1)}}=rel_{1}+\sum _{i=2}^{p}{\frac {rel_{i}}{\log _{2}(i+1)}}}$

Previously there was no theoretically sound justification for using a logarithmic reduction factor^[3] other than the fact that it produces a smooth reduction. But Wang et al. (2013)^[2] gave theoretical guarantee for using the logarithmic reduction factor in Normalized DCG (NDCG). The authors show that for every pair of substantially different ranking functions, the NDCG can decide which one is better in a consistent manner.

An alternative formulation of DCG^[4] places stronger emphasis on retrieving relevant documents:

${\displaystyle \mathrm {DCG_{p}} =\sum _{i=1}^{p}{\frac {2^{rel_{i}}-1}{\log _{2}(i+1)}}}$

The latter formula is commonly used in industry including major web search companies^[5] and data science competition platforms such as Kaggle.^[6]

These two formulations of DCG are the same when the relevance values of documents are binary;^[3]: 320 ${\displaystyle rel_{i}\in \{0,1\}}$.

Note that Croft et al. (2010) and Burges et al. (2005) present the second DCG with a log of base e, while both versions of DCG above use a log of base 2. When computing NDCG with the first formulation of DCG, the base of the log does not matter, but the base of the log does affect the value of NDCG for the second formulation. Clearly, the base of the log affects the value of DCG in both formulations.

Convex and smooth approximations to DCG have also been developed, for use as an objective function in gradient based learning methods^[7].

Normalized DCG

Search result lists vary in length depending on the query. Comparing a search engine's performance from one query to the next cannot be consistently achieved using DCG alone, so the cumulative gain at each position for a chosen value of ${\displaystyle p}$ should be normalized across queries. This is done by sorting all relevant documents in the corpus by their relative relevance, producing the maximum possible DCG through position ${\displaystyle p}$, also called Ideal DCG (IDCG) through that position. For a query, the normalized discounted cumulative gain, or nDCG, is computed as:

${\displaystyle \mathrm {nDCG_{p}} ={\frac {DCG_{p}}{IDCG_{p}}}}$,

where IDCG is ideal discounted cumulative gain,

${\displaystyle \mathrm {IDCG_{p}} =\sum _{i=1}^{|REL_{p}|}{\frac {rel_{i}}{\log _{2}(i+1)}}}$

and ${\displaystyle REL_{p}}$ represents the list of relevant documents (ordered by their relevance) in the corpus up to position p.

The nDCG values for all queries can be averaged to obtain a measure of the average performance of a search engine's ranking algorithm. Note that in a perfect ranking algorithm, the ${\displaystyle DCG_{p}}$ will be the same as the ${\displaystyle IDCG_{p}}$ producing an nDCG of 1.0. All nDCG calculations are then relative values on the interval 0.0 to 1.0 and so are cross-query comparable.

The main difficulty encountered in using nDCG is the unavailability of an ideal ordering of results when only partial relevance feedback is available.