📝 Random Forest

Description

< What is it? >

• A random forest combines many decision trees to make a classification or regression prediction. Each tree learns from a randomized version of the training problem, and the forest combines their outputs. This is an ensemble model.
• Why combine trees? A single deep tree can change substantially when its training data changes. Averaging diverse trees reduces prediction variance. Google's Random Forest lesson explains this approach.
• Example: to predict whether a user will click a movie, every tree receives the same candidate features at inference. The forest combines the trees' predictions into a click-probability estimate.

Key points

< How training works >

• Sample training rows: for each tree, draw a bootstrap sample—sample rows with replacement. Some rows appear multiple times and others are left out. This sampling-and-aggregation approach is called bagging.

• Randomize candidate features: at each split, consider a random subset of features, then choose a useful split from those candidates. The tree still learns its split rules from the data and labels; its predictions are not random guesses.

• Train independent trees: each tree learns from its own sample without depending on earlier trees' predictions. Trees can therefore be trained in parallel. Row and feature randomness help make their errors less correlated. See Google's explanation.

  Training rows and labels
       ├── Bootstrap sample A → Tree A
       ├── Bootstrap sample B → Tree B
       └── Bootstrap sample C → Tree C

< How inference combines predictions >

• Classification: in scikit-learn, each tree estimates class probabilities from its reached leaf. The forest averages these probabilities and predicts the class with the largest average. This can differ from taking a majority vote of hard class labels. See the RandomForestClassifier API.

• Regression: average the numeric predictions from the trees. For example, tree predictions of 4.0, 3.5, and 4.5 produce a forest prediction of 4.0. See the ensemble guide.

• Input and output: each input row contains the same features, in the same order, as training. Classification returns a class label or class-probability vector; regression returns a number.

  One candidate's features
       ├── Tree A → prediction A ─┐
       ├── Tree B → prediction B ─┼→ Average → Final prediction
       └── Tree C → prediction C ─┘

< Important parameters >

• Control size and diversity: the following parameters apply to scikit-learn's forest estimators.

  Parameter	Meaning
  n_estimators	Number of trees; more trees increase computation and memory
  max_depth	Maximum depth of each tree
  min_samples_leaf	Minimum training samples in a leaf; larger values smooth predictions
  max_features	Number or fraction of candidate features considered at each split
  bootstrap	Whether each tree uses sampling with replacement
  n_jobs	Parallel workers
  random_state	Seed for reproducible sampling and split randomness

  Choose settings using validation data. Adding trees does not replace controlling tree complexity. Parameter reference

Comparison

< Random Forest vs. XGBoost >

Both Random Forest and XGBoost are ensemble methods based on decision trees, but they use fundamentally different strategies to combine those trees.

• The Core Difference

  	Random Forest	XGBoost
  Ensemble type	Bagging (parallel)	Boosting (sequential)
  How trees are built	Independently, on random subsets	Sequentially, each fixing prior errors
  Tree depth	Deep (low bias, high variance)	Shallow (high bias, low variance)
  Training speed	Fast (parallelizable)	Slower (sequential by nature)
  Overfitting risk	Lower (averaging reduces variance)	Higher (needs regularization)
  Hyperparameters	Fewer, easier to tune	Many, needs careful tuning
  Interpretability	Moderate	Moderate (with feature importance)

• Random Forest: Bagging

  Idea: Build many independent decision trees, each on a random bootstrap sample of the data and a random subset of features. Then average (regression) or vote (classification).

  Data → Bootstrap sample 1 → Tree 1 ─┐
       → Bootstrap sample 2 → Tree 2 ─┼→ Average/Vote → Prediction
       → Bootstrap sample 3 → Tree 3 ─┘

  Key traits:

  • Trees are independent — can be trained in parallel.
  • Reduces variance (averaging many noisy trees).
  • Individual trees are deep and overfit; the ensemble fixes that.
  • Uses bagging (bootstrap aggregating) + feature randomness.

• XGBoost: Boosting (Gradient Boosting)

  Idea: Build trees sequentially. Each new tree is trained to predict the residual errors of the previous ensemble. Combine them with a weighted sum.

  Tree 1 → predicts y
  Tree 2 → predicts (y - Tree1)
  Tree 3 → predicts (y - Tree1 - Tree2)
  ...
  Final = Tree1 + Tree2 + Tree3 + ...

  Key traits:

  • Trees are dependent — each one learns from the previous.
  • Reduces bias (each tree corrects prior mistakes).
  • Individual trees are shallow (weak learners).
  • Uses gradient descent on the loss function to guide tree building.
  • Adds regularization (L1/L2) to prevent overfitting — this is XGBoost's key innovation.

• A Useful Analogy

  	Analogy
  Random Forest	Ask 100 independent experts and average their opinions
  XGBoost	Train one expert, then train a second to fix the first's mistakes, then a third to fix the second's, etc.

• The Family Tree

  Decision Trees
  ├── Bagging
  │   └── Random Forest
  └── Boosting
      ├── AdaBoost (1995)
      └── Gradient Boosting (1999)
          ├── XGBoost (2014)
          ├── LightGBM (2017)
          └── CatBoost (2017)

  XGBoost is not a descendant of Random Forest. They're siblings under the broader umbrella of tree ensembles. XGBoost descends from gradient boosting, which is a different lineage.

• Key Similarities

  • Both are tree-based ensembles.
  • Both handle non-linear relationships well.
  • Both work on tabular data (often beat neural nets).
  • Both provide feature importance.
  • Both are off-the-shelf — minimal preprocessing needed.
  • Both are interpretable-ish (compared to deep learning).

• Key Differences in Practice

  Aspect	Random Forest	XGBoost
  Default performance	Good baseline	Often SOTA on tabular
  Tuning effort	Low	High
  Training time	Fast	Slower
  Prediction time	Fast (parallel trees)	Slower (sequential sum)
  Handles missing values	Needs imputation	Built-in
  Regularization	Implicit (averaging)	Explicit (L1/L2)
  Overfitting	Hard to overfit	Easy to overfit without tuning

• When to Use Which

  Use Random Forest when...	Use XGBoost when...
  You want a quick, solid baseline	You want maximum accuracy
  You have limited time for tuning	You can afford hyperparameter search
  You want parallel training	You have sequential time budget
  You want robustness to overfitting	You have lots of data
  You want simplicity	You want to win Kaggle competitions

• Modern Context

  Both are part of the broader gradient-boosted decision tree (GBDT) ecosystem:

  Model	Notes
  XGBoost	The original high-performance GBDT; still widely used
  LightGBM	Faster, lower memory (Microsoft)
  CatBoost	Best for categorical features (Yandex)
  Random Forest	Still a strong, simple baseline

  For tabular data, XGBoost / LightGBM / CatBoost usually win. Random Forest is a great fallback when you need something quick and reliable.

• TL;DR

  Random Forest = bagging. Many independent deep trees, averaged. Reduces variance. Parallel, simple, robust.

  XGBoost = boosting. Many sequential shallow trees, each fixing prior errors. Reduces bias. Sequential, powerful, needs tuning.

  They're siblings, not ancestors — both are tree ensembles, but they use opposite strategies. Random Forest is the reliable baseline; XGBoost is the competition-winning workhorse.

Implementation

< Train a movie-click classifier >

• Concrete data: each row represents a user–movie impression with features [genre_match, watch_hours_weekly]. The label is 1 for clicked and 0 for not clicked. This small synthetic dataset illustrates the API; it does not establish performance on new users.

  python -m pip install numpy scikit-learn

• Code: train a forest, then predict two new candidate rows. Inference requires features but no click labels.

  import numpy as np
  from sklearn.ensemble import RandomForestClassifier
  
  # Columns: [genre_match (0/1), watch_hours_weekly]
  X_train = np.array([
      [0, 2], [0, 4], [0, 6], [0, 8],
      [1, 2], [1, 4], [1, 6], [1, 8],
  ])
  clicked = np.array([0, 0, 0, 0, 1, 1, 1, 1])
  
  model = RandomForestClassifier(
      n_estimators=100,
      max_depth=3,
      max_features="sqrt",
      bootstrap=True,
      random_state=42,
      n_jobs=1,
  )
  model.fit(X_train, clicked)
  
  # Two candidate movies for a user who watches 5 hours per week.
  movie_ids = np.array(["movie_A", "movie_B"])
  X_new = np.array([[0, 5], [1, 5]])
  labels = model.predict(X_new)
  probabilities = model.predict_proba(X_new)
  click_column = np.flatnonzero(model.classes_ == 1)[0]
  click_scores = probabilities[:, click_column]
  
  print("Input:", X_new.tolist())
  print("Classes:", model.classes_.tolist())
  print("Predicted labels:", labels.tolist())
  print("Probability shape:", probabilities.shape)
  print("Click probabilities:", click_scores.round(3).tolist())
  order = np.argsort(-click_scores, kind="stable")
  print("Ranked movies:", movie_ids[order].tolist())

• Output interpretation: input shape is (2, 2) and class-probability shape is (2, 2). Probability columns follow model.classes_; selecting class 1 gives one click score per movie. For illustration, if the click scores are [0.1, 0.9], the output is:

  Input: [[0, 5], [1, 5]]
  Classes: [0, 1]
  Predicted labels: [0, 1]
  Probability shape: (2, 2)
  Click probabilities: [0.1, 0.9]
  Ranked movies: ['movie_B', 'movie_A']

  This is illustrative output, not a captured run of the training code. Run the example to obtain its actual learned probabilities. Sorting classifier probabilities is a pointwise ranking approach; this forest was trained on clicks, not a pairwise or listwise ranking loss.

< See how averaging produces a score >

• Runnable calculation: suppose three trees return the following click probabilities for the same two movies. These are supplied example values, not outputs from the trained forest above.

  import numpy as np
  
  # Rows: trees. Columns: movie_A, movie_B.
  tree_click_probabilities = np.array([
      [0.1, 0.8],
      [0.2, 0.9],
      [0.3, 0.7],
  ])
  forest_scores = tree_click_probabilities.mean(axis=0)
  print("Forest click probabilities:", forest_scores.round(3).tolist())

  Verified output:

  Forest click probabilities: [0.2, 0.8]

Troubleshoot

< Common problems >

• Training performance is high but validation performance is poor: reduce depth or increase min_samples_leaf, and check for leakage. For time-based recommendation data, evaluate on later requests and compute historical features only from past information.
• The forest is too large: limit tree count or depth and measure memory and inference latency alongside quality.
• Feature importance is misleading: impurity-based importance can favor features with many possible split values. Consider permutation importance on held-out data; correlated features still complicate interpretation. See the scikit-learn ensemble guide.
• Probabilities are used for decisions: check calibration on held-out data. A predicted click probability should be validated before interpreting it as a reliable frequency estimate.

Related ideas

• Decision Tree explains the split rules used by individual trees.
• Extreme Gradient Boosting (XGBoost) explains boosting and learning to rank.
• Recommendation System explains candidate retrieval and ranking.
• Confidence Calibration discusses probability reliability.
• Model Lifecycle and Deployment covers training, evaluation, and serving.

Reference

• Google: Random Forest
• Google: Gradient boosting
• scikit-learn: RandomForestClassifier
• scikit-learn: Ensemble methods
• XGBoost: Random Forests
• XGBoost: Learning to rank