Skip to main content

📝 Normalized Discounted Cumulative Gain (NDCG)

Description​

< What is it? >​

  • Meaning: Normalized Discounted Cumulative Gain (NDCG) measures how well a ranked list places relevant items near the top. It considers both how relevant each item is and where it appears.

  • Example: suppose three movies have these relevance labels for a user:

    MovieRelevance
    A3 — highly relevant
    B1 — somewhat relevant
    C0 — irrelevant
    RankingOrderQuality
    IdealA → B → CMost relevant movie first
    WorseC → B → AMost relevant movie last

    NDCG gives the ideal ranking 1.0 and the worse ranking a lower score.

Key points​

< Gain, discount, and normalization >​

  • Gain: reward relevant items. This page uses the common gain formula 2rel−12^{rel}-1, where relrel is a nonnegative relevance label.

  • Discounted: reduce the reward for items appearing farther down the list. For the top KK results:

    DCG@K=∑i=1K2reli−1log⁡2(i+1)DCG@K=\sum_{i=1}^{K}\frac{2^{rel_i}-1}{\log_2(i+1)}

    Here, ii is the position, starting at 1, and relirel_i is the label of the item at that position. Sum only available positions when fewer than KK items are returned.

  • Normalized: divide by the best possible score for the evaluated items' relevance labels:

    NDCG@K=DCG@KIDCG@KNDCG@K=\frac{DCG@K}{IDCG@K}

    IDCG is the DCG of the ideal ordering: sort the evaluation set by relevance before taking the top KK. TensorFlow Ranking documents this normalization and configurable gain and discount functions.

  • Example: using the movie labels above:

    RankingDCG@3NDCG@3
    A → B → C7.6311.000
    C → B → A4.1310.541

< Using NDCG in recommendations >​

  • Evaluate each request: NDCG@10 evaluates the ordering of the first 10 recommendations. Calculate it separately for each recommendation request, then average across requests.
  • Separate labels from predictions: relevance labels come from evaluation data, such as ratings, human judgments, or defined engagement grades. Predicted scores determine the order; they are not the relevance labels used to calculate NDCG.
  • Keep the evaluation setup consistent: use the same cutoff, gain formula, candidate-set policy, and tie-breaking rule when comparing models. Evaluating only retrieved candidates measures their ordering; evaluating against the broader catalog can also reflect missed relevant items.
  • Handle zero ideal gain: if all relevance labels are zero, the ratio is undefined. The implementation below assigns 0.0; document this convention because evaluation tools may handle these requests differently.

Comparison​

< NDCG vs. Mean Reciprocal Rank (MRR) >​

  • What each metric rewards:

    MetricWhat it rewards
    MRRPlacing the first relevant item near the top
    NDCGPlacing multiple relevant items near the top, accounting for relevance grades

    NDCG can use graded or binary relevance. MRR requires a yes/no relevance decision and ignores relevant items after the first match.

Implementation​

< Calculate NDCG with Python >​

  • Input: item relevance is A: 3, B: 1, C: 0. Each score dictionary contains model scores for these same items. This example uses exponential gain and preserves dictionary order to break score ties.

    from math import log2

    relevance = {"A": 3, "B": 1, "C": 0}

    def dcg(labels, k):
        return sum(
            (2 ** rel - 1) / log2(position + 1)
            for position, rel in enumerate(labels[:k], start=1)
        )

    def evaluate(scores, k):
        order = sorted(scores, key=scores.get, reverse=True)
        labels = [relevance[item] for item in order]
        actual = dcg(labels, k)
        ideal = dcg(sorted(relevance.values(), reverse=True), k)
        ndcg = actual / ideal if ideal > 0 else 0.0
        return order, actual, ndcg

    for name, scores in [
        ("Ideal", {"A": 0.9, "B": 0.5, "C": 0.1}),
        ("Worse", {"A": 0.1, "B": 0.5, "C": 0.9}),
    ]:
        order, actual, ndcg = evaluate(scores, k=3)
        print(f"{name}: {order}, DCG@3={actual:.3f}, NDCG@3={ndcg:.3f}")

  • Verified output:

    Ideal: ['A', 'B', 'C'], DCG@3=7.631, NDCG@3=1.000
    Worse: ['C', 'B', 'A'], DCG@3=4.131, NDCG@3=0.541

Reference​