Reinforcement Learning: Short Notes (b)

Reinforcement Learning (RL) is a machine learning paradigm where an agent learns to make sequential decisions by interacting with an environment. At each time step, the agent chooses an action and receives a reward (possibly delayed). The goal is to learn a behavior that maximizes the expected long-term cumulative reward.

Key elements of the RL loop are modeled using a Markov decision process (MDP): states, actions, transition dynamics, and rewards. RL often uses policy to decide actions, and value function to evaluate how good outcomes are. A central set of relationships are the Bellman equations that express value as immediate reward plus discounted future value.

Objective (return and discounting)

The agent typically maximizes the discounted sum of rewards (the return):

$$G_t=\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}$$

where $\gamma \in [0,1)$ is the discount factor.

Core learning ingredients (what an RL “system” needs)

1) Agent–environment interaction

RL is fundamentally about repeated interaction:

1. Observe current state
2. Choose an action
3. Receive reward
4. Move to the next state, then repeat

2) Policy and value

• A policy defines behavior.
• A value function estimates long-term usefulness of states or state-action pairs.

Common value targets:

• $V^\pi(s)$: expected return starting in state $s$ and following policy $\pi$.
• $Q^\pi(s,a)$: expected return starting in state $s$, taking action $a$, then following policy $\pi$.

3) Exploration vs exploitation

RL must balance:

• Exploitation: choose actions known to yield high reward
• Exploration: try uncertain actions to improve future decisions

This is formalized with exploration strategies such as epsilon-greedy or stochastic policies (e.g., softmax), and sometimes upper confidence bound (UCB)-style methods.

Pro Tip
Start with small toy environments (gridworld/cartpole-like) to validate that your learning signal improves before scaling to larger problems.

Major families of reinforcement learning methods

A) Value-based methods

Value-based RL learns value functions (especially action-value function $Q(s,a)$) and chooses actions that maximize value.

• [Q-learning]{def="Off-policy control method using Bellman optimality updates for Q values"} updates toward an optimal action-value using a greedy target.
• [SARSA]{def="On-policy control method updating Q using the action actually taken next"} updates using the next action from the current policy.

Value-based methods are typically represented as:

• TD learning: $$\text{TD error}=\big(r+\gamma \,V(s')\big)-V(s)$$

and use it to reduce prediction error.

B) Policy-based methods

Policy-based RL directly learns the policy parameters $\theta$ by gradient ascent on expected return. A classic approach is:

• REINFORCE (Monte Carlo policy gradients)

Policy-gradient intuition:

• Increase probability of actions that led to higher-than-expected return
• Decrease probability of actions that led to worse outcomes

Key concepts:

• advantage function $A(s,a)$ helps reduce variance.
• baseline (e.g., $V(s)$) is commonly used.

C) Actor–critic methods

Actor–critic combines both:

• Actor: policy (chooses actions)
• Critic: value function (evaluates actions)

The critic provides a learning signal (often via TD error or advantage), and the actor updates accordingly.

Warning
With deep RL, training can be unstable; common mitigations include target networks, experience replay, and reward normalization (depending on the algorithm).