Verifying hurricane Irma’s projected path

By Jan Verkade (jan@forecastverification.com)

Between Wednesday, August 30th and Tuesday, September 12, the US National Hurricane Center issued 52 advisories pertaining to Tropical Storm and Hurricane Irma. The NHC advisories described her observed and projected location as well as her characteristics. They were posted on the web in a number of formats including text bulletins and cone plots.

The NHC advisories were issued four times daily at 3am, 9am, 3pm and 9pm (all times in UTC). The forecast cycle cycle for each advisory begins 3 hours prior to the issuance of the advisory products, i.e. at 12am, 6am, 12pm or 6pm. Here, these latter “reference times” are referred to as T0 (“T-Zero”). Notably, the advisories contained predicted positions at T0 plus 12h, 24h, 36h, 48h, 72h, 96h and 120h – here referred to as forecast lead times.

As the advisories contained both forecasted and observed locations of the storm’s center, this offered the possibility to verify the forecasts. Forecasts of the location Irma’s center only were verified; I made no attempt to verify other variables such as wind speed and direction.

The forecast advisories

Essentially, the forecast is deterministic in nature: multiple zero-width point locations form a forecast track. The NHC does, however, address uncertainty in the forecast,

NHC tropical cyclone forecast tracks can be in error. This forecast uncertainty is conveyed by the track forecast “cone”.

This cone is formed by smoothing over multiple circles with diameters that vary (and increase) with forecast lead time.  The NHC states how the circle diameters are determined:

The size of each circle is set so that two-thirds of historical official forecast errors over a 5-year sample fall within the circle.

The diameters of the circles are taken from here and listed in the table below.

lead time [h]Radius [US nm]Radius [km]
122954
244583
3663117
4878144
72107198
96159294
120211391

If we would interpret the cones as a probabilistic forecast, we would expect, on average, two out of three advisories to ‘hit’ the circles – and 1 out of 3 to miss it. Note, however, that the NHC does NOT explicitly say that the forecast should be thus interpreted! We will, however, test this 2/3 probability of a hit and 1/3 probability of a miss against the record of observations. In verification terms, we will test the reliability of that probabilistic forecast.

The data

The data was obtained by downloading all 52 advisory text bulletins and extracting the relevant passages. It was then combined into a table listing, for each forecast, both the forecast and observed location of Irma’s center. The table’s contents can be downloaded as a .csv file..

Note that the data contains locations with a resolution of 1/10th of a degree. The advisories address the fact that there is observational uncertainty: the location is only accurate within so many nautical miles. In the present analysis, this observational uncertainty is not taken into account.

AdvisoryReference TimeValid TimeLead time [h]Forecast latitude (WGS84)Forecast longitude (WGS84)Observed latitude (WGS84)Observed longitude (WGS84)
12017-08-30 12:00:002017-08-31 00:00:001216.7-31.916.3-31.8
12017-08-30 12:00:002017-08-31 12:00:002417.3-33.816.7-33.4
12017-08-30 12:00:002017-09-01 00:00:003617.9-35.717.5-35.1
12017-08-30 12:00:002017-09-01 12:00:004818.2-37.618.3-37.2
12017-08-30 12:00:002017-09-02 12:00:007218.7-41.918.9-42.5
12017-08-30 12:00:002017-09-03 12:00:009617.7-46.517.8-47.9
12017-08-30 12:00:002017-09-04 12:00:0012017-5116.8-52.6
22017-08-30 18:00:002017-08-31 06:00:001216.8-32.716.4-32.4
22017-08-30 18:00:002017-08-31 18:00:002417.3-34.517.1-34.3
22017-08-30 18:00:002017-09-01 06:00:003617.8-36.218-36
22017-08-30 18:00:002017-09-01 18:00:004818.2-38.218.8-38.5
22017-08-30 18:00:002017-09-02 18:00:007218.2-4318.7-44
22017-08-30 18:00:002017-09-03 18:00:009617.3-47.817.8-49.2
22017-08-30 18:00:002017-09-04 18:00:0012016.5-52.516.7-53.9
32017-08-31 00:00:002017-08-31 12:00:001216.8-33.616.7-33.4
32017-08-31 00:00:002017-09-01 00:00:002417.3-35.517.5-35.1
32017-08-31 00:00:002017-09-01 12:00:003617.7-37.318.3-37.2
32017-08-31 00:00:002017-09-02 00:00:004818-39.519.1-39.7
32017-08-31 00:00:002017-09-03 00:00:007217.6-44.318.4-45.6
32017-08-31 00:00:002017-09-04 00:00:009616.8-4917.3-50.4
32017-08-31 00:00:002017-09-05 00:00:0012016.2-53.516.7-55.1
42017-08-31 06:00:002017-08-31 18:00:001217-34.217.1-34.3
42017-08-31 06:00:002017-09-01 06:00:002417.6-3618-36
42017-08-31 06:00:002017-09-01 18:00:003618-37.918.8-38.5
42017-08-31 06:00:002017-09-02 06:00:004818.2-40.119.1-41.1
42017-08-31 06:00:002017-09-03 06:00:007217.5-4518.2-46.9
42017-08-31 06:00:002017-09-04 06:00:009616.7-49.517-51.7
42017-08-31 06:00:002017-09-05 06:00:0012016-53.516.6-56.4
52017-08-31 12:00:002017-09-01 00:00:001217.5-35.217.5-35.1
52017-08-31 12:00:002017-09-01 12:00:002418-37.118.3-37.2
52017-08-31 12:00:002017-09-02 00:00:003618.2-39.219.1-39.7
52017-08-31 12:00:002017-09-02 12:00:004818.1-41.618.9-42.5
52017-08-31 12:00:002017-09-03 12:00:007217-46.517.8-47.9
52017-08-31 12:00:002017-09-04 12:00:009616-5116.8-52.6
52017-08-31 12:00:002017-09-05 12:00:0012016-55.516.7-57.7
62017-08-31 18:00:002017-09-01 06:00:001217.8-36.218-36
62017-08-31 18:00:002017-09-01 18:00:002418.2-38.318.8-38.5
62017-08-31 18:00:002017-09-02 06:00:003618.3-40.719.1-41.1
62017-08-31 18:00:002017-09-02 18:00:004817.9-42.918.7-44
62017-08-31 18:00:002017-09-03 18:00:007216.8-47.517.8-49.2
62017-08-31 18:00:002017-09-04 18:00:009616-5216.7-53.9
62017-08-31 18:00:002017-09-05 18:00:0012016.5-56.516.9-59.2
72017-09-01 00:00:002017-09-01 12:00:001218.1-3718.3-37.2
72017-09-01 00:00:002017-09-02 00:00:002418.3-39.219.1-39.7
72017-09-01 00:00:002017-09-02 12:00:003618.2-41.518.9-42.5
72017-09-01 00:00:002017-09-03 00:00:004817.7-43.818.4-45.6
72017-09-01 00:00:002017-09-04 00:00:007216.7-48.217.3-50.4
72017-09-01 00:00:002017-09-05 00:00:009616.3-52.416.7-55.1
72017-09-01 00:00:002017-09-06 00:00:0012017-5717.2-60.4
82017-09-01 06:00:002017-09-01 18:00:001218.5-38.218.8-38.5
82017-09-01 06:00:002017-09-02 06:00:002418.4-40.519.1-41.1
82017-09-01 06:00:002017-09-02 18:00:003618.2-42.918.7-44
82017-09-01 06:00:002017-09-03 06:00:004817.6-45.218.2-46.9
82017-09-01 06:00:002017-09-04 06:00:007216.5-49.317-51.7
82017-09-01 06:00:002017-09-05 06:00:009616.5-53.516.6-56.4
82017-09-01 06:00:002017-09-06 06:00:0012017.5-5817.7-61.9
92017-09-01 12:00:002017-09-02 00:00:001218.7-39.519.1-39.7
92017-09-01 12:00:002017-09-02 12:00:002418.5-41.818.9-42.5
92017-09-01 12:00:002017-09-03 00:00:003618-44.218.4-45.6
92017-09-01 12:00:002017-09-03 12:00:004817.3-46.617.8-47.9
92017-09-01 12:00:002017-09-04 12:00:007216.2-5116.8-52.6
92017-09-01 12:00:002017-09-05 12:00:009616.7-55.216.7-57.7
92017-09-01 12:00:002017-09-06 12:00:0012018-59.518.1-63.3
102017-09-01 18:00:002017-09-02 06:00:001218.9-40.919.1-41.1
102017-09-01 18:00:002017-09-02 18:00:002418.5-43.418.7-44
102017-09-01 18:00:002017-09-03 06:00:003617.9-45.718.2-46.9
102017-09-01 18:00:002017-09-03 18:00:004817.2-47.917.8-49.2
102017-09-01 18:00:002017-09-04 18:00:007216.4-52.216.7-53.9
102017-09-01 18:00:002017-09-05 18:00:009617-56.516.9-59.2
102017-09-01 18:00:002017-09-06 18:00:0012019-6118.5-64.7
112017-09-02 00:00:002017-09-02 12:00:001219-42.318.9-42.5
112017-09-02 00:00:002017-09-03 00:00:002418.5-44.618.4-45.6
112017-09-02 00:00:002017-09-03 12:00:003617.9-46.917.8-47.9
112017-09-02 00:00:002017-09-04 00:00:004817.3-4917.3-50.4
112017-09-02 00:00:002017-09-05 00:00:007216.8-53.316.7-55.1
112017-09-02 00:00:002017-09-06 00:00:009618-57.517.2-60.4
112017-09-02 00:00:002017-09-07 00:00:0012020.5-62.519.1-66
122017-09-02 06:00:002017-09-02 18:00:001218.8-43.718.7-44
122017-09-02 06:00:002017-09-03 06:00:002418.3-46.118.2-46.9
122017-09-02 06:00:002017-09-03 18:00:003617.7-48.317.8-49.2
122017-09-02 06:00:002017-09-04 06:00:004817.1-50.417-51.7
122017-09-02 06:00:002017-09-05 06:00:007217-54.716.6-56.4
122017-09-02 06:00:002017-09-06 06:00:009618.4-59.317.7-61.9
122017-09-02 06:00:002017-09-07 06:00:0012021-64.519.7-67.7
132017-09-02 12:00:002017-09-03 00:00:001218.5-4518.4-45.6
132017-09-02 12:00:002017-09-03 12:00:002418-47.417.8-47.9
132017-09-02 12:00:002017-09-04 00:00:003617.4-49.517.3-50.4
132017-09-02 12:00:002017-09-04 12:00:004817.1-51.616.8-52.6
132017-09-02 12:00:002017-09-05 12:00:007217.4-5616.7-57.7
132017-09-02 12:00:002017-09-06 12:00:009619.2-60.618.1-63.3
132017-09-02 12:00:002017-09-07 12:00:0012022-6620.1-69
142017-09-02 18:00:002017-09-03 06:00:001218-46.518.2-46.9
142017-09-02 18:00:002017-09-03 18:00:002417.4-48.817.8-49.2
142017-09-02 18:00:002017-09-04 06:00:003617-50.917-51.7
142017-09-02 18:00:002017-09-04 18:00:004816.8-5316.7-53.9
142017-09-02 18:00:002017-09-05 18:00:007217.5-57.516.9-59.2
142017-09-02 18:00:002017-09-06 18:00:009619.5-62.518.5-64.7
142017-09-02 18:00:002017-09-07 18:00:0012022-6820.7-70.4
152017-09-03 00:00:002017-09-03 12:00:001217.8-4817.8-47.9
152017-09-03 00:00:002017-09-04 00:00:002417.2-50.117.3-50.4
152017-09-03 00:00:002017-09-04 12:00:003616.7-52.316.8-52.6
152017-09-03 00:00:002017-09-05 00:00:004816.5-54.516.7-55.1
152017-09-03 00:00:002017-09-06 00:00:007217.7-59.317.2-60.4
152017-09-03 00:00:002017-09-07 00:00:009619.9-64.519.1-66
152017-09-03 00:00:002017-09-08 00:00:0012022.5-69.521.1-71.8
162017-09-03 06:00:002017-09-03 18:00:001217.5-49.117.8-49.2
162017-09-03 06:00:002017-09-04 06:00:002416.8-51.317-51.7
162017-09-03 06:00:002017-09-04 18:00:003616.5-53.416.7-53.9
162017-09-03 06:00:002017-09-05 06:00:004816.5-55.716.6-56.4
162017-09-03 06:00:002017-09-06 06:00:007217.8-60.617.7-61.9
162017-09-03 06:00:002017-09-07 06:00:009620-6619.7-67.7
162017-09-03 06:00:002017-09-08 06:00:0012022.5-71.521.5-73.2
172017-09-03 12:00:002017-09-04 00:00:001217.1-50.117.3-50.4
172017-09-03 12:00:002017-09-04 12:00:002416.6-52.216.8-52.6
172017-09-03 12:00:002017-09-05 00:00:003616.4-54.316.7-55.1
172017-09-03 12:00:002017-09-05 12:00:004816.7-56.616.7-57.7
172017-09-03 12:00:002017-09-06 12:00:007218.2-61.818.1-63.3
172017-09-03 12:00:002017-09-07 12:00:009620.4-67.120.1-69
172017-09-03 12:00:002017-09-08 12:00:0012022.5-7221.8-74.7
182017-09-03 18:00:002017-09-04 06:00:001217.2-51.417-51.7
182017-09-03 18:00:002017-09-04 18:00:002416.8-53.516.7-53.9
182017-09-03 18:00:002017-09-05 06:00:003616.8-55.816.6-56.4
182017-09-03 18:00:002017-09-05 18:00:004817.3-58.216.9-59.2
182017-09-03 18:00:002017-09-06 18:00:007219.1-63.518.5-64.7
182017-09-03 18:00:002017-09-07 18:00:009621.2-68.820.7-70.4
182017-09-03 18:00:002017-09-08 18:00:0012023-73.522-76
192017-09-04 00:00:002017-09-04 12:00:001216.7-52.616.8-52.6
192017-09-04 00:00:002017-09-05 00:00:002416.5-54.816.7-55.1
192017-09-04 00:00:002017-09-05 12:00:003616.8-57.116.7-57.7
192017-09-04 00:00:002017-09-06 00:00:004817.4-59.617.2-60.4
192017-09-04 00:00:002017-09-07 00:00:007219.3-6519.1-66
192017-09-04 00:00:002017-09-08 00:00:009621.2-70.221.1-71.8
192017-09-04 00:00:002017-09-09 00:00:0012022.8-74.522.1-77.2
202017-09-04 06:00:002017-09-04 18:00:001216.7-53.916.7-53.9
202017-09-04 06:00:002017-09-05 06:00:002416.6-56.216.6-56.4
202017-09-04 06:00:002017-09-05 18:00:003617-58.616.9-59.2
202017-09-04 06:00:002017-09-06 06:00:004817.8-61.217.7-61.9
202017-09-04 06:00:002017-09-07 06:00:007219.6-66.819.7-67.7
202017-09-04 06:00:002017-09-08 06:00:009621-7221.5-73.2
202017-09-04 06:00:002017-09-09 06:00:0012022-76.522.3-78.3
212017-09-04 12:00:002017-09-05 00:00:001216.5-54.816.7-55.1
212017-09-04 12:00:002017-09-05 12:00:002416.7-57.116.7-57.7
212017-09-04 12:00:002017-09-06 00:00:003617.3-59.617.2-60.4
212017-09-04 12:00:002017-09-06 12:00:004818.2-62.418.1-63.3
212017-09-04 12:00:002017-09-07 12:00:007219.9-6820.1-69
212017-09-04 12:00:002017-09-08 12:00:009621.2-73.321.8-74.7
212017-09-04 12:00:002017-09-09 12:00:0012022-77.822.7-79.3
222017-09-04 18:00:002017-09-05 06:00:001216.6-56.216.6-56.4
222017-09-04 18:00:002017-09-05 18:00:002417-58.716.9-59.2
222017-09-04 18:00:002017-09-06 06:00:003617.8-61.317.7-61.9
222017-09-04 18:00:002017-09-06 18:00:004818.7-64.118.5-64.7
222017-09-04 18:00:002017-09-07 18:00:007220.4-69.720.7-70.4
222017-09-04 18:00:002017-09-08 18:00:009621.6-74.822-76
222017-09-04 18:00:002017-09-09 18:00:0012023-7923.1-80.2
232017-09-05 00:00:002017-09-05 12:00:001216.9-57.416.7-57.7
232017-09-05 00:00:002017-09-06 00:00:002417.5-6017.2-60.4
232017-09-05 00:00:002017-09-06 12:00:003618.3-62.718.1-63.3
232017-09-05 00:00:002017-09-07 00:00:004819.3-65.519.1-66
232017-09-05 00:00:002017-09-08 00:00:007221.2-7121.1-71.8
232017-09-05 00:00:002017-09-09 00:00:009622.5-7622.1-77.2
232017-09-05 00:00:002017-09-10 00:00:0012024-8023.4-80.9
242017-09-05 06:00:002017-09-05 18:00:001217-58.916.9-59.2
242017-09-05 06:00:002017-09-06 06:00:002417.7-61.517.7-61.9
242017-09-05 06:00:002017-09-06 18:00:003618.6-64.218.5-64.7
242017-09-05 06:00:002017-09-07 06:00:004819.6-6719.7-67.7
242017-09-05 06:00:002017-09-08 06:00:007221.2-72.521.5-73.2
242017-09-05 06:00:002017-09-09 06:00:009622.4-77.222.3-78.3
242017-09-05 06:00:002017-09-10 06:00:0012024-8123.7-81.3
252017-09-05 06:00:002017-09-05 18:00:001217-58.916.9-59.2
252017-09-05 06:00:002017-09-06 06:00:002417.7-61.517.7-61.9
252017-09-05 06:00:002017-09-06 18:00:003618.6-64.218.5-64.7
252017-09-05 06:00:002017-09-07 06:00:004819.6-6719.7-67.7
252017-09-05 06:00:002017-09-08 06:00:007221.2-72.521.5-73.2
252017-09-05 06:00:002017-09-09 06:00:009622.4-77.222.3-78.3
252017-09-05 06:00:002017-09-10 06:00:0012024-8123.7-81.3
262017-09-05 12:00:002017-09-06 00:00:001217.2-60.317.2-60.4
262017-09-05 12:00:002017-09-06 12:00:002418.1-6318.1-63.3
262017-09-05 12:00:002017-09-07 00:00:003619.1-65.919.1-66
262017-09-05 12:00:002017-09-07 12:00:004820.1-68.720.1-69
262017-09-05 12:00:002017-09-08 12:00:007221.4-7421.8-74.7
262017-09-05 12:00:002017-09-09 12:00:009622.7-78.322.7-79.3
262017-09-05 12:00:002017-09-10 12:00:0012024.4-81.224.5-81.5
272017-09-05 18:00:002017-09-06 06:00:001217.6-61.817.7-61.9
272017-09-05 18:00:002017-09-06 18:00:002418.5-64.618.5-64.7
272017-09-05 18:00:002017-09-07 06:00:003619.5-67.319.7-67.7
272017-09-05 18:00:002017-09-07 18:00:004820.4-70.120.7-70.4
272017-09-05 18:00:002017-09-08 18:00:007221.6-75.322-76
272017-09-05 18:00:002017-09-09 18:00:009622.7-79.323.1-80.2
272017-09-05 18:00:002017-09-10 18:00:0012024.4-81.525.6-81.7
282017-09-06 00:00:002017-09-06 12:00:001218.1-63.118.1-63.3
282017-09-06 00:00:002017-09-07 00:00:002419.1-65.919.1-66
282017-09-06 00:00:002017-09-07 12:00:003620.1-68.520.1-69
282017-09-06 00:00:002017-09-08 00:00:004821-71.221.1-71.8
282017-09-06 00:00:002017-09-09 00:00:007222-76.222.1-77.2
282017-09-06 00:00:002017-09-10 00:00:009623.2-79.523.4-80.9
282017-09-06 00:00:002017-09-11 00:00:0012025-81.526.8-81.7
292017-09-06 06:00:002017-09-06 18:00:001218.6-64.618.5-64.7
292017-09-06 06:00:002017-09-07 06:00:002419.6-67.319.7-67.7
292017-09-06 06:00:002017-09-07 18:00:003620.6-7020.7-70.4
292017-09-06 06:00:002017-09-08 06:00:004821.3-72.521.5-73.2
292017-09-06 06:00:002017-09-09 06:00:007222.5-7722.3-78.3
292017-09-06 06:00:002017-09-10 06:00:009624-8023.7-81.3
292017-09-06 06:00:002017-09-11 06:00:0012026.5-80.928.2-82.2
302017-09-06 12:00:002017-09-07 00:00:001219-66.219.1-66
302017-09-06 12:00:002017-09-07 12:00:002420.2-6920.1-69
302017-09-06 12:00:002017-09-08 00:00:003621.2-71.721.1-71.8
302017-09-06 12:00:002017-09-08 12:00:004821.9-74.221.8-74.7
302017-09-06 12:00:002017-09-09 12:00:007222.9-78.122.7-79.3
302017-09-06 12:00:002017-09-10 12:00:009625.2-8024.5-81.5
302017-09-06 12:00:002017-09-11 12:00:0012029-80.529.6-82.7
312017-09-06 18:00:002017-09-07 06:00:001219.6-67.619.7-67.7
312017-09-06 18:00:002017-09-07 18:00:002420.7-70.420.7-70.4
312017-09-06 18:00:002017-09-08 06:00:003621.5-73.121.5-73.2
312017-09-06 18:00:002017-09-08 18:00:004822.2-75.622-76
312017-09-06 18:00:002017-09-09 18:00:007223.2-7923.1-80.2
312017-09-06 18:00:002017-09-10 18:00:009626-80.225.6-81.7
312017-09-06 18:00:002017-09-11 18:00:0012031-8130.9-83.5
322017-09-07 00:00:002017-09-07 12:00:001220.3-68.820.1-69
322017-09-07 00:00:002017-09-08 00:00:002421.3-71.721.1-71.8
322017-09-07 00:00:002017-09-08 12:00:003622.1-74.121.8-74.7
322017-09-07 00:00:002017-09-09 00:00:004822.7-76.322.1-77.2
322017-09-07 00:00:002017-09-10 00:00:007224-79.323.4-80.9
322017-09-07 00:00:002017-09-11 00:00:009627.4-80.326.8-81.7
322017-09-07 00:00:002017-09-12 00:00:0012032-8131.9-84.4
332017-09-07 06:00:002017-09-07 18:00:001220.7-70.520.7-70.4
332017-09-07 06:00:002017-09-08 06:00:002421.7-73.121.5-73.2
332017-09-07 06:00:002017-09-08 18:00:003622.3-75.522-76
332017-09-07 06:00:002017-09-09 06:00:004822.8-77.422.3-78.3
332017-09-07 06:00:002017-09-10 06:00:007224.5-8023.7-81.3
332017-09-07 06:00:002017-09-11 06:00:009628.5-80.528.2-82.2
342017-09-07 12:00:002017-09-08 00:00:001221.1-71.721.1-71.8
342017-09-07 12:00:002017-09-08 12:00:002421.9-74.221.8-74.7
342017-09-07 12:00:002017-09-09 00:00:003622.5-76.522.1-77.2
342017-09-07 12:00:002017-09-09 12:00:004823.1-78.422.7-79.3
342017-09-07 12:00:002017-09-10 12:00:007225.5-8024.5-81.5
342017-09-07 12:00:002017-09-11 12:00:009630.1-80.629.6-82.7
352017-09-07 18:00:002017-09-08 06:00:001221.6-73.121.5-73.2
352017-09-07 18:00:002017-09-08 18:00:002422.2-75.522-76
352017-09-07 18:00:002017-09-09 06:00:003622.7-77.622.3-78.3
352017-09-07 18:00:002017-09-09 18:00:004823.4-79.323.1-80.2
352017-09-07 18:00:002017-09-10 18:00:007226.2-80.725.6-81.7
352017-09-07 18:00:002017-09-11 18:00:009631.5-8230.9-83.5
362017-09-08 00:00:002017-09-08 12:00:001221.8-74.421.8-74.7
362017-09-08 00:00:002017-09-09 00:00:002422.3-76.722.1-77.2
362017-09-08 00:00:002017-09-09 12:00:003622.9-78.622.7-79.3
362017-09-08 00:00:002017-09-10 00:00:004823.7-79.923.4-80.9
362017-09-08 00:00:002017-09-11 00:00:007227.2-81.126.8-81.7
362017-09-08 00:00:002017-09-12 00:00:009632-83.531.9-84.4
372017-09-08 06:00:002017-09-08 18:00:001222.1-75.722-76
372017-09-08 06:00:002017-09-09 06:00:002422.6-77.822.3-78.3
372017-09-08 06:00:002017-09-09 18:00:003623.3-79.423.1-80.2
372017-09-08 06:00:002017-09-10 06:00:004824.5-80.423.7-81.3
372017-09-08 06:00:002017-09-11 06:00:007228-81.528.2-82.2
382017-09-08 12:00:002017-09-09 00:00:001222.4-77.122.1-77.2
382017-09-08 12:00:002017-09-09 12:00:002422.8-7922.7-79.3
382017-09-08 12:00:002017-09-10 00:00:003623.7-80.323.4-80.9
382017-09-08 12:00:002017-09-10 12:00:004825-8124.5-81.5
382017-09-08 12:00:002017-09-11 12:00:007229.3-82.329.6-82.7
392017-09-08 18:00:002017-09-09 06:00:001222.4-78.222.3-78.3
392017-09-08 18:00:002017-09-09 18:00:002423-79.923.1-80.2
392017-09-08 18:00:002017-09-10 06:00:003624.1-8123.7-81.3
392017-09-08 18:00:002017-09-10 18:00:004825.7-81.525.6-81.7
392017-09-08 18:00:002017-09-11 18:00:007230.6-8330.9-83.5
402017-09-09 00:00:002017-09-09 12:00:001222.6-79.122.7-79.3
402017-09-09 00:00:002017-09-10 00:00:002423.3-80.623.4-80.9
402017-09-09 00:00:002017-09-10 12:00:003624.5-81.424.5-81.5
402017-09-09 00:00:002017-09-11 00:00:004826.5-81.926.8-81.7
402017-09-09 00:00:002017-09-12 00:00:007231.6-83.831.9-84.4
412017-09-09 06:00:002017-09-09 18:00:001222.9-8023.1-80.2
412017-09-09 06:00:002017-09-10 06:00:002423.8-81.123.7-81.3
412017-09-09 06:00:002017-09-10 18:00:003625.4-81.825.6-81.7
412017-09-09 06:00:002017-09-11 06:00:004827.7-82.428.2-82.2
422017-09-09 12:00:002017-09-10 00:00:001223.4-80.723.4-80.9
422017-09-09 12:00:002017-09-10 12:00:002424.5-81.624.5-81.5
422017-09-09 12:00:002017-09-11 00:00:003626.4-82.126.8-81.7
422017-09-09 12:00:002017-09-11 12:00:004829-82.829.6-82.7
432017-09-09 18:00:002017-09-10 06:00:001224.1-81.323.7-81.3
432017-09-09 18:00:002017-09-10 18:00:002425.7-8225.6-81.7
432017-09-09 18:00:002017-09-11 06:00:003628.3-82.628.2-82.2
432017-09-09 18:00:002017-09-11 18:00:004831.2-83.830.9-83.5
442017-09-10 00:00:002017-09-10 12:00:001224.7-81.724.5-81.5
442017-09-10 00:00:002017-09-11 00:00:002426.8-82.426.8-81.7
442017-09-10 00:00:002017-09-11 12:00:003629.5-83.429.6-82.7
442017-09-10 00:00:002017-09-12 00:00:004832.2-8531.9-84.4
452017-09-10 06:00:002017-09-10 18:00:001225.4-8225.6-81.7
452017-09-10 06:00:002017-09-11 06:00:002427.8-82.828.2-82.2
452017-09-10 06:00:002017-09-11 18:00:003630.5-84.130.9-83.5
462017-09-10 12:00:002017-09-11 00:00:001226.6-82.126.8-81.7
462017-09-10 12:00:002017-09-11 12:00:002429.2-8329.6-82.7
462017-09-10 12:00:002017-09-12 00:00:003631.9-84.631.9-84.4
472017-09-10 18:00:002017-09-11 06:00:001228-82.528.2-82.2
472017-09-10 18:00:002017-09-11 18:00:002430.7-83.930.9-83.5
482017-09-11 00:00:002017-09-11 12:00:001229.4-82.829.6-82.7
482017-09-11 00:00:002017-09-12 00:00:002431.9-84.731.9-84.4
492017-09-11 06:00:002017-09-11 18:00:001230.8-83.730.9-83.5
502017-09-11 12:00:002017-09-12 00:00:001232-84.531.9-84.4

The analysis

Two verification ‘products’ were developed: a table with the relative number of hits (hurricane center observations within the projected circles) and maps that showed the locations of the projected circles as well as the actual location at that time.

The graphs were produced as follows:

  • First, the latitudes and longitudes were converted from their original WGS84 projection to the Mercator projection. This is a projection that is more or less in metres.
  • Based on the NHC forecast circle radii, circles were drawn around the projected center locations.
  • Actual observations were then added to the map.

The tables were produced as follows:

  • The distance between the forecast and observed locations was determined by applying the haversine formula.
  • By comparing with the forecast circle radii, it was determined whether the observation fell within the projected circle.
  • For each lead time separately, the fraction of forecasts that turned out to be hits was calculated.

Results

Below video gives a fair idea of forecast quality. Here are some viewing guides.

  • First, 12-hour forecasts and their verifying observations are shown. Then, 24-hour forecasts, 36h, etc. The forecast lead time is shown in the plot title.
  • The plot time is shown in the plot’s subtitle. Plot time progresses.
  • At plot time, forecasts of future center positions exist. These are the circles slightly ahead of the animated path.
  • The actual location (at plot time) of Irma’s center is indicated by a small black dot.
  • The associated forecast at plot time – that was made X hours ago – is then highlighted.
  • If we interpret the circles to be probabilistic forecasts, about 2/3 of the small black dots should be within the highlighted forecast circles (a hit). This also means that about 1/3 should be outside (these are misses). Look at the video and try and guess if this is indeed the case.

Calculating the distances between forecast and observed locations yielded the following ‘errors’:

The analysis of hits and misses yields the following table. Note that the hit rate should be approx. 2/3 or .67 in order for the forecasts to be reliable.

Forecast lead time
[h]
Hit rate
[-]
Average 'error'
[km]
Average 'error'
[US nm]
120.962815
240.94826
360.916937
480.849450
720.913673
960.92187101
1200.88243131

Conclusions

The hit rate of the forecast cone circles varies with lead time and is in the order of 90%. Given recent forecast performance

The size of each circle is set so that two-thirds of historical official forecast errors over a 5-year sample fall within the circle.

this is higher than expected. For whatever reason, the Irma forecasts were sharper than the average of the forecasts made in the previous five years.

If we would have interpreted this 2/3 as a probabilistic forecast, we would have found the Irma forecasts to be underconfident! This is rather unusual. More often, probability forecasts turn out to be overconfident.

In order for the a priori probabilities to have matched the observed frequencies, the suggested probabilities associated with the circles (two-thirds, or approx 67%) would have had to have been higher (approx 85-95%). Alternatively, the circle diameters would have had to have been smaller – resulting in a more narrow cone.

Note that here, we applied the principle of forecast post-processing: we assessed past performance and deduced probabilities from that. In this case, our assumption (that past performance was a measure of present performance) did not hold. That is, not in the case of this particular set of 52 forecasts. Possibly, a more physical approach to uncertainty estimation would have been better. I am thinking of ensembles.

Tools used

For this exercise, a number of tools was used. Downloading of the cone plots and the advisory HTML files was done using a Linux bash script that called wget. The cone plots were combined in an animated GIF using FFMPEG. Processing and plotting was done in R using the ggmap, rgeos and the geosphere package:

  • D. Kahle and H. Wickham. ggmap: Spatial Visualization with ggplot2. The R Journal, 5(1), 144-161. URL http://journal.r-project.org/archive/2013-1/kahle-wickham.pdf
  • Robert J. Hijmans (2016). geosphere: Spherical Trigonometry. R package version 1.5-5. https://CRAN.R-project.org/package=geosphere
  • R Core Team (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
  • Roger Bivand and Colin Rundel (2017). rgeos: Interface to Geometry Engine – Open Source (GEOS). R package version 0.3-23. https://CRAN.R-project.org/package=rgeos

All tools are available as open source software. Open source rocks!

Leave a Reply