Understand it in plain language

Reinforcement learning studies how an agent chooses actions in an environment to improve expected cumulative reward. State or observation describes the situation, a policy chooses an action, and reward provides feedback. Delayed consequences and exploration make this different from simply matching labelled examples. A reward is a design choice, not a guarantee that the behaviour is what people intended.

How the AI approach works

Reinforcement learning addresses sequential decisions. An agent observes a situation, chooses an action, receives a reward, and encounters a new situation. A policy maps observations or states to action choices. The objective is usually expected return: a sum of rewards, often discounted so distant rewards receive less weight. Maximizing only the next reward can miss a better long-term outcome.

The Markov decision process is a useful formal model with states, actions, transition behavior, rewards, and a discount factor. Its state must contain the information needed to predict the next-state distribution. If the agent only sees incomplete observations, memory or a belief about the hidden state may be necessary. The environment is not simply a labelled spreadsheet.

Value-based methods estimate how good states or state–action pairs are. Q-learning updates an action value toward reward plus the discounted best estimated value of the next state. Policy-gradient methods adjust the policy directly; actor–critic methods combine policy and value learning. Model-based methods learn or use a transition model to plan. These families make different tradeoffs in data efficiency, stability, and computational cost.

Exploration is how an agent obtains evidence about actions it has not tried enough; exploitation uses its current best choice. Safe exploration is difficult in real systems, so start in a suitable simulator or controlled environment. A badly designed reward can encourage shortcuts that earn points without achieving the intended purpose.

Break the idea into steps

1. Define actions, observations, rewards, and stopping conditions.
2. Distinguish trying an unfamiliar action from exploiting a known one.
3. Test for shortcuts that increase reward while missing the real goal.

Worked example: connect the method to the task

A robot in a small grid gets a reward for reaching a charging station and a penalty per step. Going toward an immediate small reward can be worse than taking several steps to a larger terminal reward. Q-values can propagate the future benefit backward through experienced transitions.

The experiment below trains a tabular Q-learning policy in a tiny deterministic line world. Its known states and safe reset make experimentation easy. Real robots add incomplete observations, uncertainty, costly mistakes, and much harder simulation-to-reality transfer.

Algorithm experiment: predict, change, explain

This Q-learning agent learns in a tiny line world. Set gamma to 0 and inspect the resulting greedy path: immediate rewards alone cannot propagate the distant goal benefit. The step limit prevents a stuck policy from looping forever.

import random
random.seed(7)

# States 0..4; 4 is terminal. Actions: -1 left, +1 right.
q = [[0.0, 0.0] for _ in range(5)]
actions = [-1, 1]
alpha, gamma, epsilon = 0.3, 0.9, 0.3
for episode in range(400):
    state = 0
    for step in range(50):
        action = (random.randrange(2) if random.random() < epsilon
                  else max(range(2), key=lambda a: q[state][a]))
        next_state = max(0, min(4, state + actions[action]))
        terminal = next_state == 4
        reward = 1.0 if terminal else -0.05
        target = reward + (0.0 if terminal else gamma * max(q[next_state]))
        q[state][action] += alpha * (target - q[state][action])
        state = next_state
        if terminal:
            break
state, path = 0, [0]
for _ in range(10):
    action = max(range(2), key=lambda a: q[state][a])
    state = max(0, min(4, state + actions[action]))
    path.append(state)
    if state == 4:
        break
print("greedy path:", path)
print("reached goal:", state == 4)

greedy path: [0, 1, 2, 3, 4]
reached goal: True

Change one thing. Run it again.

Run this small, self-contained experiment in standard Python. No package installation or paid API is required. Change the suggested variable and explain the effect on the method before applying it to real data.

Let the editor finish loading, load this example, then press Run.

The free editor uses OneCompiler, an external service. Code you run is sent there. Do not enter passwords, API keys, or private student data. If an example is sent before the editor is ready, wait and load it again.

Apply the idea: check your reasoning

Before you move on

• Explain the main idea without repeating the technical label.
• Identify the approach, its assumptions, and the evidence needed to trust the result.
• Complete the challenge, then compare your reasoning with the answer.

Sources & a deeper next step

The refresh uses primary papers and official documentation. These optional links are in English; you can understand this lesson without reading them first. Product versions and availability can change.

• UC Berkeley CS 188: AI foundations ↗
• Spinning Up: reinforcement-learning concepts ↗