Skip to main content

πŸ“ Reinforcement Learning (RL)

Description​

< What is it? >​

  • Reinforcement Learning (RL) is a learning paradigm in which an agent learns to choose actions from feedback about their consequences. Its objective is to maximize expected cumulative reward, rather than only the reward from the next action. Applications include robot control, games, resource allocation, and language-model post-training.
  • Example: a robot must reach the end of a corridor. It receives βˆ’1 for each move that does not reach the goal and +10 for reaching it. Moving toward the goal may initially receive a negative reward, but it enables a larger reward later. The agent learns which actions produce the best overall outcome.
  • Learning signal: the environment supplies rewards and new observations. A training example does not usually specify the correct action for every situation. Google’s Introduction to Machine Learning introduces this reward-driven learning process.

Definition​

< Core terminology >​

  • The components: the corridor example gives each term a concrete meaning.

    TermMeaningCorridor example
    AgentThe decision-making learnerThe robot’s controller
    EnvironmentThe system the agent interacts withThe corridor and its movement rules
    State, sts_tInformation sufficient to describe the current situation for predicting transitions and rewardsCurrent position: 0, 1, 2, or 3
    Observation, oto_tInformation available to the agent; it may reveal only part of the stateThe position sensor reading
    Action, ata_tA choice available to the agentMove left or right
    Reward, rt+1r_{t+1}Numerical feedback after an actionβˆ’1 for a non-goal move; +10 on reaching position 3
    Policy, Ο€\piA rule or probability distribution for choosing actionsChoose right at positions 0, 1, and 2
    EpisodeOne run from a starting condition to terminationStart at 0 and finish at the goal
    Return, GtG_tAccumulated future reward, optionally discountedReward from the remaining journey
    Value functionExpected return under a specified policyVΟ€(s)V^\pi(s) values a state; QΟ€(s,a)Q^\pi(s,a) values an action in that state

  • Markov Decision Process (MDP): a common mathematical formulation specifies states, actions, transition probabilities, rewards, and a discount factor. The Markov assumption means that the current state and action provide the information needed to predict the next-state and reward distributions. When observations omit relevant information, the policy may need history or memory. See Key Concepts in RL.

Key points​

< The interaction and learning loop >​

  • Training cycle: the agent collects experience and uses it to improve its policy or value estimates. Updates may happen after each interaction or after collecting a batch.

    The agent's policy selects actions that affect the environment. Observations feed the policy; observations, actions, and rewards feed the learning algorithm, which improves the policy during training.

    The policy is the agent’s decision rule. The learning algorithm improves it directly or through learned action values; the dashed arrow represents updates during training.

  • Experience record: a transition commonly contains (state, action, reward, next_state, terminated). For example, (2, "right", 10, 3, True) records reaching the goal. This is observed experience, not a supplied label saying that β€œright” is the correct action.

  • Training versus inference: training changes the learned parameters or table entries. At inference, the deployed policy chooses an action from an observation; receiving a reward does not automatically update the model unless the application explicitly runs a learning procedure.

< Reward, return, and delayed credit >​

  • Discounted return: for an episode ending at time TT, a common objective uses:

    Gt=βˆ‘k=0Tβˆ’tβˆ’1Ξ³krt+k+1G_t = \sum_{k=0}^{T-t-1} \gamma^k r_{t+k+1}

    The discount factor Ξ³\gamma determines how strongly future rewards contribute. With Ξ³=0\gamma=0, only the immediate reward counts. Values closer to 1 give later rewards more weight; finite episodic tasks can also use Ξ³=1\gamma=1.

  • Worked example: the route 0 β†’ 1 β†’ 2 β†’ 3 produces rewards βˆ’1, βˆ’1, +10. With Ξ³=0.9\gamma=0.9:

    G0=βˆ’1+0.9(βˆ’1)+0.92(10)=6.2G_0 = -1 + 0.9(-1) + 0.9^2(10) = 6.2

    Its undiscounted total reward is 8. The discounted return is smaller because the goal reward arrives after two earlier actions.

  • Credit assignment: the final reward must influence the earlier choices that made it possible. A value-based method propagates estimated future returns backward through visited states; a policy-gradient method adjusts action probabilities using return or advantage estimates.

< Exploration and exploitation >​

  • Exploration tries actions to discover their consequences. Exploitation chooses actions that currently appear best. Always exploiting early estimates can prevent the agent from finding a better strategy.
  • Epsilon-greedy example: with Ο΅=0.2\epsilon=0.2, choose a random action on 20% of decisions and a highest-value action on the remaining 80%. Random exploration can also select the currently best action. Exploration is often reduced as training progresses.

< How Q-learning updates an action value >​

  • Update rule: tabular Q-learning stores a number for each state–action pair and updates it toward an observed reward plus an estimated future value:

    Q(s,a)←Q(s,a)+Ξ±[r+Ξ³max⁑aβ€²Q(sβ€²,aβ€²)βˆ’Q(s,a)]Q(s,a) \leftarrow Q(s,a) + \alpha \left[r + \gamma\max_{a'}Q(s',a') - Q(s,a)\right]

    Here, sβ€²s' is the next state, aβ€²a' ranges over its available actions, and Ξ±\alpha is the learning rate. For a terminal transition, the future-value term is zero. See the original Q-learning paper.

  • One numerical update: initially, suppose Q(2, right) = 0. Moving right reaches the goal with reward 10. With Ξ±=0.2\alpha=0.2, the updated value is 0 + 0.2 Γ— (10 βˆ’ 0) = 2. Repeated updates bring it toward 10; earlier states then learn that moving right leads to a valuable future state.

  • Neural-network version: a Deep Q Network (DQN) replaces the table with a neural network that estimates action values. A loss measures disagreement with a reward-based target, and backpropagation updates the network. Tabular Q-learning itself does not require a neural network.

< Reinforcement learning in LLM post-training >​

  • Mapping to language generation: the prompt and generated prefix form the policy’s context; a token is an action, and an answer or tool-use trajectory can receive a reward. Training adjusts the model to increase expected reward on the task distribution.
  • Reward sources: Reinforcement Learning from Human Feedback (RLHF) can use a reward model learned from human preferences. Reinforcement Learning with Verifiable Rewards (RLVR) uses checkable outcomes, such as whether generated code passes tests. A reward need not be a human rating on every generated answer.
  • Optimization and parameter updates: Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) are policy-optimization methods. Low-Rank Adaptation (LoRA) determines which parameters are trained and can be combined with an RL objective. These names describe different parts of the training setup. The TRL GRPO documentation provides reward-function and training examples.
  • Application boundary: an Agentic RAG workflow does not become reinforcement learning merely because it calls tools repeatedly. RL requires a reward-driven learning update. Editing instructions is prompt engineering; it does not update model parameters, as explained in Google’s Machine Learning Crash Course.

Comparison​

< Reinforcement learning vs. supervised learning >​

  • Choosing the learning setup: the distinction concerns the objective and feedback, not whether the implementation uses a neural network.

    AspectSupervised learningReinforcement learning
    Training feedbackDesired output for an inputRewards associated with actions and outcomes
    Typical goalPredict labels or target values accuratelyChoose actions with high expected return
    ExamplePredict whether a logged user–item pair was clickedChoose recommendations to improve a long-term session objective
    ConsequencesStandard training treats examples as a supplied datasetActions can change future states and available experience
    Main challengeGeneralization to unseen examplesGeneralization, exploration, and delayed credit assignment
    When usefulA prediction task has suitable input–target examplesDecisions affect later outcomes and reward feedback is available

  • Recommendation example: training a click predictor from click/not-click labels is supervised learning. Using predicted click probability to sort items does not by itself make the system RL. A contextual bandit can learn from immediate recommendation feedback; full sequential RL additionally models consequences for future decisions.

< Main families of reinforcement learning methods >​

  • What is learned: these categories overlap; an actor–critic method also performs policy optimization.

    FamilyWhat it learns or usesExamplesStrengthLimitation
    Value-basedAction values used to select actionsTabular Q-learning, DQNDirect action comparison; experience can be reusedTables do not scale; standard DQN needs discrete actions
    Policy-gradientA parameterized action-selection policyREINFORCEDirectly optimizes policy behaviorGradient estimates can have high variance
    Actor–criticAn actor chooses actions; a critic estimates valuePPO with a value critic, Soft Actor-Critic (SAC)Value estimates help guide policy updatesMore interacting components to train and tune
    Model-basedKnown or learned environment dynamics for planning or trainingWorld-model methodsCan use predicted experience to reduce real interactionsModel errors can mislead planning or policy learning

    See Kinds of RL Algorithms.

  • Two separate distinctions: model-free means no explicit environment-dynamics model is used for planning or learning; the agent can still contain a neural network. On-policy methods use experience from the policy being optimized, while off-policy methods can learn from experience collected by another behavior policy. Offline RL learns from a fixed dataset without collecting new interactions during training; it is not simply another name for off-policy RL.

Implementation​

< Learn a corridor policy with tabular Q-learning >​

  • Input and task: the environment has positions [0, 1, 2, 3], starts at 0, and ends at 3. Actions are left and right; moving left at 0 stays at 0. Reaching 3 gives +10; every other move gives βˆ’1. No labeled correct route is supplied.

  • Runnable Python: this example uses only the standard library. q[state][action] is the learned table. Training uses exploration; inference selects the action with the highest learned value.

    import random

    rng = random.Random(7)
    actions = ("left", "right")
    goal = 3
    q = [[0.0, 0.0] for _ in range(goal + 1)]
    alpha, gamma, epsilon = 0.2, 0.9, 0.2


    def step(state, action):
        next_state = max(0, min(goal, state + (-1 if action == 0 else 1)))
        terminated = next_state == goal
        reward = 10.0 if terminated else -1.0
        return next_state, reward, terminated


    # Training: learn from consequences, without correct-action labels.
    for episode in range(1000):
        state = 0
        for _ in range(30):
            if rng.random() < epsilon:
                action = rng.randrange(2)  # Explore.
            else:
                action = max(range(2), key=lambda a: q[state][a])

            next_state, reward, terminated = step(state, action)
            future = 0.0 if terminated else max(q[next_state])
            target = reward + gamma * future
            q[state][action] += alpha * (target - q[state][action])
            state = next_state
            if terminated:
                break
        # The 30-step collection limit is not a terminal state of this task.

    print("state | Q(left) | Q(right) | chosen action")
    for state in range(goal):
        action = max(range(2), key=lambda a: q[state][a])
        print(f"{state} | {q[state][0]:.2f} | {q[state][1]:.2f} | {actions[action]}")

    # Inference: use the learned table without changing it.
    state, total_reward = 0, 0.0
    path = [state]
    for _ in range(30):
        action = max(range(2), key=lambda a: q[state][a])
        state, reward, terminated = step(state, action)
        path.append(state)
        total_reward += reward
        if terminated:
            break
    print("Path:", " -> ".join(map(str, path)))
    print(f"Total reward: {total_reward:.1f}")

  • Verified output: running the code produces:

    state | Q(left) | Q(right) | chosen action
    0 | 4.58 | 6.20 | right
    1 | 4.58 | 8.00 | right
    2 | 6.20 | 10.00 | right
    Path: 0 -> 1 -> 2 -> 3
    Total reward: 8.0

  • Reading the result: for inference input state = 0, the learned values are approximately [4.58, 6.20], so the output action is right. These values estimate discounted return; they are not probabilities. Q(0, right) = 6.20 matches the earlier discounted-return calculation, while the printed total reward is the undiscounted sum 8.0.

  • Evaluation scope: this deterministic toy verifies the learning mechanism. A realistic evaluation should freeze the policy and measure return, success rate, and task costs across separate evaluation episodes, varied scenarios, and multiple training seeds. One successful route does not establish generalization.

< Libraries for larger tasks >​

  • Environment and training tools: choose the library according to the task rather than treating an environment API as a learning algorithm.

    ToolRoleTypical use
    GymnasiumDefines environments and reset() / step() interfacesSimulated control tasks and custom environments
    Stable-Baselines3Provides PyTorch implementations of RL algorithmsTraining and evaluating DQN, PPO, or SAC agents
    Hugging Face TRLProvides language-model training workflowsReward-based LLM post-training with GRPO

Troubleshoot​

< Common training problems >​

  • Diagnose behavior as well as reward: monitor actual task outcomes alongside the scalar objective.

    SymptomPossible causeWhat to check
    The agent never reaches the goalRare rewards or insufficient explorationStart with an easier task and inspect sampled trajectories
    Reward improves but behavior is undesirableThe reward rewards a shortcutInspect high-reward trajectories and revise the objective
    Training is unstableLarge updates or poorly scaled inputs and rewardsCheck normalization, learning rate, and results across seeds
    Training looks good but evaluation failsOverfitting to training scenariosUse separate evaluation scenarios and compare with a simple baseline
    Values are wrong near episode boundariesTermination and collection cutoffs are confusedUse zero future value at true termination; retain bootstrapping at an external time limit

  • Termination versus truncation: Gymnasium returns both terminated and truncated. Either can end data collection for an episode, but an external time limit does not necessarily remove the state’s future value. See Handling Time Limits.

Q & A​

< Does reinforcement learning require a neural network? >​

  • No: tabular Q-learning stores values in a table. Deep reinforcement learning uses neural networks to represent policies, value functions, or environment models when tables are impractical.

< Who defines the reward, and is it a training label? >​

  • The task designer defines how reward is measured: it can come from game rules, simulator outcomes, user feedback, a learned reward model, or an automated verifier. It evaluates an outcome rather than normally identifying the correct action. Algorithms can construct training targets from it, such as Q-learning’s reward + discounted future value.

Video Tutorial​

Reference​