From Pixels to Decisions: How AVs Make Choices on the Road

Knowing what's around you is only half of driving. The other half is deciding what to do next — and doing it safely.

Autonomous vehicles turn sensor data into decisions through a loop of perception → prediction → planning → control. Let's walk through that loop using a concrete example.

Step 1: Perception – "What's around me?"

First, the AV takes in data from cameras, lidar, radar, and other sensors, and answers:

• Where are the lanes?
• Where are other cars, cyclists, and pedestrians?
• Are there traffic lights, stop signs, or cones?
• Is there anything unusual, like a vehicle stopped in the lane?

The result is a structured map of the scene, not just raw pixels.

Step 2: Prediction – "What will they do next?"

Next, the prediction system asks:

• Where will other vehicles be in 1–5 seconds?
• Is that pedestrian about to cross?
• Is that cyclist likely to move into my lane?
• Is that car waiting to turn, or rolling forward?

To answer this, AVs use models trained on many examples of human behavior:

• Cars turning left at intersections
• Pedestrians waiting, then stepping into crosswalks
• Cyclists merging or signaling turns

The output is essentially many possible futures with probabilities attached:

• "The pedestrian is 80% likely to cross soon."
• "The oncoming car is 95% likely to continue straight."

Step 3: Planning – "What's the safest, smoothest move?"

Given all those possible futures, the planner chooses a path and a set of actions that:

• Avoid collisions
• Obey traffic rules
• Keep the ride comfortable
• Reach the destination efficiently

For example, at a four-way stop:

1. The planner sees other cars arriving at different times.
2. It considers scenarios — who goes first, who yields.
3. It chooses a conservative plan:
   • Come to a full stop
   • Wait your turn
   • Proceed cautiously, with room for others' mistakes

Planners often evaluate many candidate paths and speeds, then pick the one with the lowest "cost" in terms of risk, discomfort, and inefficiency.

Step 4: Control – "Turn the wheel, press the pedals"

Once a plan is chosen, the control system turns it into precise commands:

• How much to steer
• How hard to accelerate
• How quickly to brake

This happens dozens of times per second. If something changes — a pedestrian starts crossing unexpectedly, a car cuts in — the whole loop runs again, updating perception, predictions, and the plan.

A full loop example: Unprotected left turn

Unprotected left turns are tricky, even for humans. Here's how an AV might handle it:

1. Perception: Detect oncoming traffic, the intersection geometry, lane markings, and any pedestrians in the crosswalk.

2. Prediction: Estimate the speed and future positions of oncoming cars. Predict whether pedestrians will step off the curb.

3. Planning: If gaps in traffic are too small, plan to wait. When a safe gap appears, plan a smooth left-turn path that clears the intersection without rushing.

4. Control: Apply throttle and steering to follow the planned path. Be ready to brake if a new vehicle appears or a pedestrian behaves unexpectedly.

All of this is happening while continuously re-evaluating the situation.

Safety first, speed second

Importantly, planners are usually biased toward:

• Slower, more cautious maneuvers
• Extra buffer around vulnerable road users
• Yielding when in doubt

That's why AVs sometimes feel more conservative than human drivers. The system is tuned to "err on the side of safety," even if it means waiting a bit longer or leaving more space.

The big picture

At a high level, you can think of AV decision-making as:

See → Understand → Imagine possible futures → Pick the safest plan → Execute → Repeat.

It's not magic or "gut feeling." It's a continuous, data-driven loop designed to produce safe, predictable behavior on the road.