Understanding forecast uncertainty

COVER PHOTO: Abby Lewis

SOURCE: Cheong, L., Bleisch, S., Kealy, A., Tolhurst, K., Wilkening, T., & Duckham, M. (2016). Evaluating the impact of visualization of wildfire hazard upon decision-making under uncertainty. International Journal of Geographical Information Science, 30(7), 1377-1404. DOI: 10.1080/13658816.2015.1131829

Predicting the Future

A family in California agonizes over whether to pack all their things and move out of their house after seeing that there is a 30% chance their house will be burnt in a forest fire today. A mayor pours over hurricane forecasts to determine what guidelines they should offer to people in their city. State officials sort through coronavirus forecasts to determine the best course of action for their state. 

Forecasts allow us to make informed decisions about how to respond to future scenarios. However, forecasts can’t tell us exactly what will happen in the future. Forecasts often give the likelihood of a given event (e.g. 30% chance of fire) or a range of possible scenarios (150,000–200,000 deaths from COVID-19). We then use that probabilistic assessment to determine the best course of action. Understanding the range of possible outcomes—we call this the uncertainty of the forecast—gives us a better ability to make informed decisions. 

If that all sounds a bit abstract, let’s think of this in terms of weather forecasts. Temperature forecasts typically don’t give any indication of the range of possible temperatures you might expect at a given time; they don’t indicate any uncertainty. However, rain forecasts do. A forecast for a 30% chance of rain indicates that it most likely will not rain, but it is still much more likely to rain than if there were a 0% chance of rain (as a side note—most weather forecasts overestimate the chance of rain, a phenomenon coined the “wet bias”). How you respond to a 30% chance of rain will likely be different than if the forecast simply said “no, it will not rain.”

Uncertainty visualization

Forecast uncertainty can be pretty difficult to wrap your head around, and this makes it harder for people to make decisions informed by forecasts. To help with this problem, people have come up with a variety of different ways to visualize forecast uncertainty. 

Public Health Uncertainty

One example that we have seen a lot recently is coronavirus graphs like these (below) that show observed data leading up to today and a shaded area with likely outcomes in the future. These forecasts are composed of many separate predictions, each of which has slightly different assumptions about how contagious the virus is, how socially isolated people will be, how effective treatment can be, etc. to cover various potential outcomes. The shaded interval shows the range of outcomes that contains 95% of these predictions, and the lines inside those shaded areas show the average prediction.

Hurricane Forecast Uncertainty

Another familiar example of uncertainty visualization is NOAA’s hurricane forecasts. These forecasts show the potential path a hurricane may take as it hits land. Different points on the line indicate what time it may reach that location, and the width of the cone increases further into the future because we are less sure of where the hurricane may be at that point. These visualizations provide a lot of information in a very clean format, but they can be difficult to interpret. People often think that the hurricane itself gets larger over time, when in reality the uncertainty increases but the size of the hurricane stays the same.

Forest Fire Uncertainty

Yet another option for uncertainty visualization can be seen in the forest fire likelihood plot below. In this image, different colors represent different levels of fire risk. This map is relatively easy to understand, but it still misrepresents the forecast by showing clear boundaries between risk levels, when the reality is much more nuanced.

Using uncertainty visualizations for decision making

To better understand how people use uncertainty visualizations to make decisions, Dr. Lisa Cheong and colleagues from the University of Melbourne, Australia developed an interactive, incentive-based study. They created six different ways of visualizing forest fire risk and asked participants to use these visualizations to decide whether they should stay home or leave to avoid the fire. Participants won (and earned money) if they interpreted the map correctly; so they got money if they chose to stay home and their house was not impacted by the fire or if they chose to leave and their house did end up being impacted by the fire. Whether or not their house was impacted by the fire was determined randomly based on the probability specified in the map.

The visualizations that the researchers used were primarily map-based, like the fire risk image shown above. One map used distinct colors to demonstrate different risk levels (as above), one used boundaries but no colors, and one option just showed the percent likelihood in text.

As it turned out, participants did just about the same with both of the different map options, but they did better when the only information they got was the text-based percent likelihood. However, participants generally preferred the color-based representation over the plain-text representation. This is interesting because it indicates that people think they are getting more information from the map, but in reality, that information is getting in the way of their decision making.

However, Cheong and colleagues also did a second study. This time they only gave participants 5 seconds to look at the uncertainty visualization and make their decision. Under the increased time pressure, participants did significantly better with the color-based maps than either the text-based option or the boundary-only map option. It seems that when people have very little time to make a decision, color-based map options are easier to process than reading text.

Importantly, this research demonstrates that the way forecast uncertainty is represented can affect how we interpret the same information, and this can cause us to behave in different ways.

Understanding uncertainty

Forecasts are all around us: from the weather to the economy to global pandemics. As you go about your week, pay attention to how uncertainty is visualized in these different forecasts. Is there uncertainty at all? Where does it come from? How is it visualized? And how does that affect your interpretation of the forecast? The more accurately we understand forecast uncertainty, the more we can be certain we can be that we are making the best possible decisions given the information we have available.