How accurate are weather forecasts? 7.3M forecasts, 30 cities

2 November 20266 min read

What we measured

Whether a forecast is good depends on what it is compared with. We scored each forecast against a competitor that needs no model. For rain the competitor is "it will not rain". For temperature it is "tomorrow will be like today".

The trust horizon is the lead time at which a forecast stops beating that free competitor. Skill = 1 − forecast mean error / competitor mean error. Below zero, the forecast is worse than the competitor. The study covers two years and leads of 1 to 7 days.

Main results

  • In eight cities the rain forecast scored below the competitor at every measured lead: Ankara, Konya, Kayseri, Eskişehir, Denizli, Isparta, Kars and Van.
  • The longest rain horizon is in Rize: 5.74 days.
  • In two cities the temperature forecast never beat yesterday: Antalya and Mersin.
  • In Trabzon the temperature forecast is still ahead at day 7, the last lead in the archive.
  • Ankara's temperature horizon is 5.92 days and its rain horizon is zero. In one city, the answer changes with the question.

Why rain forecasts fall behind in dry cities

It rains in Ankara about one hour in twenty, so "it will not rain" is right roughly 95% of the time. To clear that bar, a forecast must catch enough real rain to pay for its false alarms. The result does not say the model is bad. It says how strong the free guess already is in that city.

All 30 cities

CityRain horizon (days)95% intervalTemperature horizon (days)95% interval
Ankara0.000.00-1.005.925.58-6.26
Denizli0.000.00-1.294.274.11-4.44
Eskişehir0.000.00-1.326.255.88-6.64
Isparta0.000.00-0.004.234.11-4.34
Kars0.000.00-0.004.353.74-4.91
Kayseri0.000.00-1.214.974.62-5.35
Konya0.000.00-1.376.135.88-6.31
Van0.000.00-1.414.153.88-4.32
Şanlıurfa1.180.00-2.475.995.55-6.21
Erzurum1.390.00-2.253.973.32-5.43
Diyarbakır1.720.00-2.845.024.51-5.52
Mersin1.750.00-3.710.000.00-2.33
Sivas1.810.00-2.915.555.15-5.97
Balıkesir2.031.30-2.705.505.08-5.94
Çanakkale2.221.66-2.895.685.08-7.00
Antalya2.280.00-4.180.000.00-0.00
Tokat2.631.52-3.405.775.34-6.31
Gaziantep2.641.44-3.716.175.80-6.51
Aydın2.711.84-3.424.164.04-4.28
Malatya2.711.80-4.094.804.56-5.09
İstanbul2.821.78-3.595.514.99-6.13
İzmir2.861.72-4.115.735.39-6.50
Samsun3.181.96-3.574.724.51-4.96
Adana3.361.45-4.064.974.69-5.40
Bursa3.432.89-3.995.985.62-6.55
Zonguldak4.403.90-4.904.844.69-5.01
Trabzon4.453.95-4.887.005.75-7.00
Ordu4.594.14-5.064.904.74-5.19
Muğla5.614.55-6.554.504.35-4.66
Rize5.745.11-6.714.174.06-4.28

7.00 is the archive's last lead; measurement stops there and the true horizon may be longer. 0.00 means the forecast scored below the competitor at every measured lead. Where the interval is not 0.00-0.00, "can never win" is not a claim we can make.

What this does not show

  • It does not rank weather apps. We did not test apps one by one.
  • It does not use rain gauges. The reference truth is ERA5 reanalysis, which is also a model.
  • It does not represent every part of a city. Each city is one grid point, and rain is local.
  • It does not measure overall model quality. The question is only whether it rains in a given hour.

How to read a forecast

  1. In a dry city, do not rely on the hourly rain icon alone. Read nearby hours and the probability together.
  2. Temperature forecasts stay useful longer in most cities. Do not expect the same trust for rain.
  3. Always score a forecasting system against a free competitor.

Sources

More writing

High AUC is not a decision: one churn system, five datasetsWe ran Vinctum Aegis on five public churn datasets. On four, AUC was 0.76-0.93 and no customer got an intervene decision. The cause sat in the decision's inputs.A churn prediction is not a decision: from risk score to actionA churn model tells you who is likely to leave, not whom to spend on. A worked example of why the same 78% risk means two different decisions for two customers.