How accurate are weather forecasts? 7.3M forecasts, 30 cities
What we measured
Whether a forecast is good depends on what it is compared with. We scored each forecast against a competitor that needs no model. For rain the competitor is "it will not rain". For temperature it is "tomorrow will be like today".
The trust horizon is the lead time at which a forecast stops beating that free competitor. Skill = 1 − forecast mean error / competitor mean error. Below zero, the forecast is worse than the competitor. The study covers two years and leads of 1 to 7 days.
Main results
- In eight cities the rain forecast scored below the competitor at every measured lead: Ankara, Konya, Kayseri, Eskişehir, Denizli, Isparta, Kars and Van.
- The longest rain horizon is in Rize: 5.74 days.
- In two cities the temperature forecast never beat yesterday: Antalya and Mersin.
- In Trabzon the temperature forecast is still ahead at day 7, the last lead in the archive.
- Ankara's temperature horizon is 5.92 days and its rain horizon is zero. In one city, the answer changes with the question.
Why rain forecasts fall behind in dry cities
It rains in Ankara about one hour in twenty, so "it will not rain" is right roughly 95% of the time. To clear that bar, a forecast must catch enough real rain to pay for its false alarms. The result does not say the model is bad. It says how strong the free guess already is in that city.
All 30 cities
| City | Rain horizon (days) | 95% interval | Temperature horizon (days) | 95% interval |
|---|---|---|---|---|
| Ankara | 0.00 | 0.00-1.00 | 5.92 | 5.58-6.26 |
| Denizli | 0.00 | 0.00-1.29 | 4.27 | 4.11-4.44 |
| Eskişehir | 0.00 | 0.00-1.32 | 6.25 | 5.88-6.64 |
| Isparta | 0.00 | 0.00-0.00 | 4.23 | 4.11-4.34 |
| Kars | 0.00 | 0.00-0.00 | 4.35 | 3.74-4.91 |
| Kayseri | 0.00 | 0.00-1.21 | 4.97 | 4.62-5.35 |
| Konya | 0.00 | 0.00-1.37 | 6.13 | 5.88-6.31 |
| Van | 0.00 | 0.00-1.41 | 4.15 | 3.88-4.32 |
| Şanlıurfa | 1.18 | 0.00-2.47 | 5.99 | 5.55-6.21 |
| Erzurum | 1.39 | 0.00-2.25 | 3.97 | 3.32-5.43 |
| Diyarbakır | 1.72 | 0.00-2.84 | 5.02 | 4.51-5.52 |
| Mersin | 1.75 | 0.00-3.71 | 0.00 | 0.00-2.33 |
| Sivas | 1.81 | 0.00-2.91 | 5.55 | 5.15-5.97 |
| Balıkesir | 2.03 | 1.30-2.70 | 5.50 | 5.08-5.94 |
| Çanakkale | 2.22 | 1.66-2.89 | 5.68 | 5.08-7.00 |
| Antalya | 2.28 | 0.00-4.18 | 0.00 | 0.00-0.00 |
| Tokat | 2.63 | 1.52-3.40 | 5.77 | 5.34-6.31 |
| Gaziantep | 2.64 | 1.44-3.71 | 6.17 | 5.80-6.51 |
| Aydın | 2.71 | 1.84-3.42 | 4.16 | 4.04-4.28 |
| Malatya | 2.71 | 1.80-4.09 | 4.80 | 4.56-5.09 |
| İstanbul | 2.82 | 1.78-3.59 | 5.51 | 4.99-6.13 |
| İzmir | 2.86 | 1.72-4.11 | 5.73 | 5.39-6.50 |
| Samsun | 3.18 | 1.96-3.57 | 4.72 | 4.51-4.96 |
| Adana | 3.36 | 1.45-4.06 | 4.97 | 4.69-5.40 |
| Bursa | 3.43 | 2.89-3.99 | 5.98 | 5.62-6.55 |
| Zonguldak | 4.40 | 3.90-4.90 | 4.84 | 4.69-5.01 |
| Trabzon | 4.45 | 3.95-4.88 | 7.00 | 5.75-7.00 |
| Ordu | 4.59 | 4.14-5.06 | 4.90 | 4.74-5.19 |
| Muğla | 5.61 | 4.55-6.55 | 4.50 | 4.35-4.66 |
| Rize | 5.74 | 5.11-6.71 | 4.17 | 4.06-4.28 |
7.00 is the archive's last lead; measurement stops there and the true horizon may be longer. 0.00 means the forecast scored below the competitor at every measured lead. Where the interval is not 0.00-0.00, "can never win" is not a claim we can make.
What this does not show
- It does not rank weather apps. We did not test apps one by one.
- It does not use rain gauges. The reference truth is ERA5 reanalysis, which is also a model.
- It does not represent every part of a city. Each city is one grid point, and rain is local.
- It does not measure overall model quality. The question is only whether it rains in a given hour.
How to read a forecast
- In a dry city, do not rely on the hourly rain icon alone. Read nearby hours and the probability together.
- Temperature forecasts stay useful longer in most cities. Do not expect the same trust for rain.
- Always score a forecasting system against a free competitor.