CE 397 Statistics in Water Resources

Exercise 10—Personal Exercise

Precipitation Patterns in Space and Time

By:  Fernando Salas, Clark Siler, Ahmad Tavakoly, and David Maidment

The University of Texas at Austin

April 2009

Contents

Characterizing Spatial Patterns in Time. 1

Introduction. 1

Goals of the Exercise: 2

Computer and Data Requirements. 2

NEXRAD Data. 2

Downloading and Ingesting NEXRAD Data into ArcMap. 5

Storm Tracking using ArcMap. 12

Precipitation Histograms. 22

Summary. 26

Tracking Precipitation in Space and Time. 27

Mapping Storm in GIS. 27

Spatial Analyst. 29

Cell Statistics. 30

Zonal Statistics. 32

Hyetograph. 33

Summary. 34

Introduction

In this exercise we will explore how to characterize spatial patterns in time specifically extreme storms. How does one track a storm in space and time? How does the intensity of a storm change with time? We would like to build on the geostatistics processes we’ve learned in class and incorporate a time dimension in the analysis. In order to learn how to deal with data in space and time, we will utilize NEXRAD (Next Generation Radar) data in NetCDF (Network Common Data Form) file format. NetCDF files are a subset of gridded datasets that incorporate a third dimension in addition to the standard space dimensions (i.e. lat/long coordinates). This third dimension could be anything from time to elevation or pressure level. We will focus on using time as the third dimension. NetCDF files are hosted by the Unidata program at the University Corporation for Atmospheric Research (UCAR) and they are the ones in charge with the standard development of and updates to netCDF files. For our purposes, we will use ArcMap, a GIS (Geographic Information System) program, to perform some data manipulation and analysis. ArcMap ingests netCDF files easily and will allow us to run analyses on large batches of data rather swiftly.

Goals of the Exercise:

In this exercise we will investigate how one can characterize storm events through space and time. First, we will look at the path of a storm and study how its movements change as it moves over land. For the purposes of this class we have constructed a very simple method by which one can track a storm through space and time. Next, we will introduce a method for studying the intensity of a storm and how it changes in space and time. Then, we will briefly introduce open-ended approach to obtaining histogram information for raster data. Finally, we will examine the precipitation generated by the storm, as viewed by a fixed location in space, and create a precipitation hyetograph.

Computer and Data Requirements

This exercise will require the use of ArcMap, with the Spatial Analyst extension enabled. The version of ArcMap used in the LRC has this capability, and instructions to enable such are provided herein.

The data for this exercise is accessible at:

http://www.ce.utexas.edu/prof/maidment/STATWR2009/Ex10/Ex10Data.zip

NEXRAD Data

In the last 5 to 10 years, there have been enormous steps taken in the field of climatology to publish climate data online for both public and private access. This has been made possible by a large amount of inter-agency communication and integration. Agencies such as the National Climatic Data Center (NCDC) and Unidata have teamed up to provide climate data both readily and sustainably. By use of the THREDDS (Thematic Realtime Environmental Distributed Data Services) server, agencies have been able to publish copious amounts of climate data online.

The Center for Research in Water Resources (CRWR) here at the University of Texas has compiled a list of some of these datasets which are available online. Instead of having to search for and register with a variety of different websites, CRWR provides direct links to catalogs of climate data for direct and easy access. The CRWR website can be found below:

http://data.crwr.utexas.edu/

Go to the webpage and click on Web Coverage Service (WCS).

Here we can see a list of different climate datasets. For this exercise, we are only concerned with NEXRAD (precipitation) data however know that there is a lot of historical climate data available online. NEXRAD is a network of high resolution Doppler weather radars located all over the United States and are operated by the National Weather Service (NWS), an agency under the National Oceanic and Atmospheric Administration (NOAA). NEXRAD detects precipitation and wind which makes it good for tracking storms.

Click on Next Generation Radar (NEXRAD).

This page provides some documentation for the dataset we have selected. Information such as the variables provided, spatial extent and frequency of the data are available. In summary, the NEXRAD catalog we will look at provides historical hourly 4 km grids covering the continental United States dating back to 2002. NEXRAD has actually been around since 1992 however only data since 2002 is available through the THREDDS server. The NWS houses data since 1992 however the process in which one can access the data is a bit more complicated. The NWS link from where one can access NEXRAD data can be seen below.

http://lwf.ncdc.noaa.gov/oa/radar/radardata.html

Next click on the link provided. This will take us to the NEXRAD data catalog provided by NCDC hosted on the THREDDS server. Here we can see multiple folders containing NEXRAD data. The folders labeled in years contain hourly NEXRAD data. The folder labeled 6hr/ contains accumulated 6 hour NEXRAD data whereas the folder labeled day/ contains accumulated daily NEXRAD data. Because we will be working with netCDF files which allow us to add a time dimension, we can either download data for each timestamp we desire separately or download data that has been aggregated into one file for all the timestamps we desire. Aggregated accumulated daily data can be obtained through the files labeled ST4_Agg_YYYY-TEST.nc (YYYY being the four digit year desired.)

As a demonstration, we will walk through the procedure for which one can download NEXRAD data using both methods mentioned above.

Downloading and Ingesting NEXRAD Data into ArcMap

As mentioned above, one can download NEXRAD data in one of two ways: download separate grids for each timestamp or download one aggregated file containing the grids for each timestamp.

To download data in separate files, first go to the NEXRAD catalog.

http://www.ncdc.noaa.gov/thredds/catalog/radar/StIV/catalog.html

First select a year of interest, say 2005 and click on the folder. Click on the folder labeled 1hr/.

A long list of files will appear. Let’s look at Hurricane Katrina as an example. Scroll to the file named ST4.2005082912.01h.

This file contains the data for August 29, 2005 at noon. Click on the file and select NetCDFSubset.

The following page is the location of the NEXRAD netCDF file we are interested in. Here check the Total Precipitation box. If we wanted to, we could select bounds for the data since the default file contains data over the entire continental United States. Under Spatial Subset, we would check Bounding Box and enter the lat/long coordinates of the area we desire however we will skip over this part for now. Since there is only one timestamp at this location, we do not have to mess with the Choose Time Subset section. Click Submit and save the file.

We now have a netCDF file containing NEXRAD precipitation data for August 29, 2005 at noon.

Now that we have downloaded some data, we can ingest the file into ArcMap for viewing. Open ArcMap and click Ok to open an empty map. Now simply go to the folder where the data is located and click and drag it into the layers panel in ArcMap.

We now can see the NEXRAD data in our ArcMap display. The white circle in the raster image is Hurricane Katrina as it made landfall on August 29, 2005. If we want we could add a map of the United States and change the symbology of the data to display it a bit nicer. See below. It is important to note that the units of the raster are in millimeters per hour.

The second method for downloading NEXRAD data is in aggregated format. Go back to the NEXRAD catalog and this time click on one of the ST4_Agg_YYYY-TEST.nc files. Let’s look at Katrina again. Click on ST4_Agg_2005-Test.nc and then just as before click on NetCDFSubset. Check the Total Precipitation box. Again, we could choose a bounding box for our data but we will refrain from doing so.

This time we will select a Time Subset. Click on Time Range and then enter the Starting and Ending timestamps as follows:

Click Submit and save the file. This will download accumulated daily NEXRAD precipitation data from August 29, 2005 to August 31, 2005 all in one netCDF file. We can ingest this into ArcMap in the same manner as described above. Once the data is loaded, ArcMap will only display the data from the first day, August 29, 2005. This time, the units for the displayed data are in millimeters per day. We can cycle through the other days by right-clicking on the Total Precipitation layer and selecting Properties.

Select the NetCDF tab. We can change the timestamp of the data displayed in ArcMap by clicking in the value cell like so. Select 8/31/2005 12:00:00 PM and click Ok.

We can see that within a couple of days, Hurricane Katrina changed in form and traveled across the United States rather quickly.

Daily Precipitation on August 29, 2005

Daily Precipitation on August 31, 2005

In summary, we can see that netCDF files have the potential to be very powerful. Although we haven’t actually run through any analysis, the ability to download data readily and ingest it into analytical software is a very powerful tool. Very quickly we were able to download data for Hurricane Katrina and see how its spatial patterns progressed over time. Now we will look at how one can manipulate gridded data in ArcMap for further study.

Storm Tracking using ArcMap

For this exercise, we will be looking at Tropical Storm Erin which made its way across the state of Texas in August of 2007. The storm made landfall along the gulf coast near Lamar, Texas around the 16th of August and left Texas on the 19th of August headed towards Oklahoma. The storm resulted in 16 fatalities and caused severe flooding throughout.

All the data and files we will need for this exercise can be found in a geodatabase named TexasNEXRAD_Erin. The geodatabase can be found in a zipped file on the class website.

http://www.ce.utexas.edu/prof/maidment/STATWR2009/Ex10/Ex10Data.zip

Download the zipped file from the website and extract all the files into an empty folder of your choosing.

To begin the storm tracking process, open ArcMap. Click Ok to open a new empty map. Note, we will first look at the final product of the storm tracking procedure and then describe the manner in which this product was obtained. In the TexasNEXRAD_Erin geodatabase, we will find a feature dataset named SpatialData containing a shapefile of the state of Texas. We can add this shapefile to our map by clicking on the Add Data icon on the ArcMap toolbar.

Navigate to the folder in which you saved the geodatabase and open it.

Open the SpatialData feature dataset and Add Texas to the map.

Next Add the entire Centroids feature dataset to the map. If a spatial reference dialog box appears, just Click Ok.

The Centroids feature dataset contains points which represent the “center” of Tropical Storm Erin as it progressed over the state of Texas. Center is in quotation marks here because when we began to go through the storm tracking process, WE designated the center of a tropical storm to be the area of highest precipitation. Climatologists don’t necessarily consider this the center of a tropical storm. Many times, the point of lowest pressure within a tropical storm is considered to be the center however for the purpose of this exercise, our definition will suffice.

Once that data is loaded into ArcMap, one can see exactly how Tropical Storm Erin traveled over Texas. The points are labeled as follows: C200708DDTT where DD is the two digit day in August and TT is the time of day in military time. We can compare this storm track with the one provided by the National Hurricane Center (NHC). How does our storm track estimate compare to that of the NHC?

The method in which this data was analyzed followed a simple criterion for defining the center of a tropical storm: area in which one hour accumulated precipitation was greater than or equal to 50 millimeters.

The first step in this process involved downloading netCDF files using the same method described above (see Downloading and Ingesting NEXRAD Data into ArcMap) for the days and hours of interest. The netCDF file for August 16 hour 20 can be seen below as an example. This file is available in the geodatabase and is called Nexrad1620.

In order to evaluate the whole storm, we downloaded hourly data for a period of five days: August 16 to August 20. Once we had our entire set of netCDF files stored on a local hard drive, we used the reclassify tool in ArcMap to select out the raster values which were greater than or equal to 50 millimeters. The procedure for this is as follows. First, click on the toolbox icon in the toolbar . Once the toolbox is displayed, click on Spatial Analyst Tools followed by Reclass. The Reclassify command allows us to essentially create a histogram of our gridded data. We can select bin ranges and count the number of grid cells which fall within each bin. The tool will reclassify each raster cell with a number depending on the bin in which it falls.

Note: We need to make sure that we have the Spatial Analyst extension toggled in ArcMap. Go to Tools on the top menu bar and then Click Extensions. Make sure there is a checkmark next to Spatial Analyst. Click Close.

Click on Reclassify. Select Nexrad1620 as the input raster and leave the Reclass Field as Value. Click on Classify and Select Equal Interval as the Classification Method. Change the number of Classes to 2. Next Change the Break Values to 50 and 1000. This will create bins from 0 to 50 and 50 to 1000. 1000 is just an upper bound and can be set to any extreme value on the high end as long as it is greater than the greatest value in the input raster. Click Ok.

Now highlight the first row and click Delete Entries. Choose a location for the Output Raster and finally check the Change Missing Values to No Data option. Select Ok.

A new raster will be added to our existing map. We should be able to see a small cluster of cells at the center of Tropical Storm Erin. The cells are labeled 2 as was stated in the Reclassify window. These cells represent the original NEXRAD precipitation cells that have a value greater than or equal to 50 millimeters per hour (our definition of the center of a storm).

We can now proceed to create centroids from the new raster we created but first we must transform the cluster of raster cells to a polygon. ArcMap can do this using the Raster to Polygon tool. This tool can be located in: ArcToolbox | Conversion Tools |From Raster |Raster to Polygon.

Select the reclassified raster as the Input Raster and then choose a location to save the polygon shapefile. Click Ok.

A polygon should have been added to our map and should sit right above the cluster of cells we created in the step before (as shown by the arrow, above).

Using this newly created polygon, we can finally find the centroid of this polygon, which we use as a surrogate for the storm’s center. In the toolbox, navigate to ArcToolbox | Data Management Tools | Features | Features to Point. Select the newly created polygon as the input feature and again choose a location for the point shapefile. Click OK.

We have now created a point by which we can consider the center of Tropical Storm Erin on August 16 at hour 20.

This is the end product of our analysis however this is only for one timestamp. In order to create the first map with multiple centroids on it, we built a model in ArcMap through Model Builder and performed the same process described above for each time stamp desired. A screenshot of the model can be seen below however it is not necessary to create a model to perform this task. Model builder only aids in automating some of the actions performed above.

The results of this limited exercise are shown below, again, including an inset picture showing the storm’s track, as reported by the National Hurricane Center. How well do you think they match?

Precipitation Histograms

In the previous section (Storm Tracking) of this exercise, certain NEXRAD data were used to track the Tropical Storm Erin as it passed overland through Texas. We return to these data and introduce additional methods for characterizing the statistical properties of the NEXRAD precipitation data. However, some groundwork is required first.

You will recall that the NEXRAD NetCDF data are rasters reporting hourly precipitation data for each cell in the designated range. Certain of ArcGIS’ basic ArcCatalog tools were used to prepare the NEXRAD data for further processing. These include the Reclassify (ArcCatalog | Spatial Analyst Tools | Reclass | Reclassify) tool and the Raster to Polygon (ArcCatalog | Conversion Tools | From Raster | Raster to Polygon) tool. These tools are described briefly below:

Reclassify – this tool takes a raster (NEXRAD in this case) and produces a new raster using an array of values from which new values are drawn. For example, bins of 10 units were used to “round” the values of the original NEXRAD raster cells to conform to the specified bins. The resulting rasters are found in the TexasNEXRAD_Erin geodatabase, following a BreaksR**** naming convention (where **** is a substitute for date and hour, similar to before).

Raster to Polygon – this tool takes rasters as input and creates polygon feature classes from the outline of the raster. For this case, rasters corresponding to greater than ten units of precipitation were used as input, resulting in rasters showing the geographic locations of precipitation. These polygons are found in the TexasNEXRAD_Erin geodatabase, in the TenPolygons feature dataset, following a FinalTenP**** naming convention (where **** is a substitute for date and hour, similar to before).

In order to produce histograms of NEXRAD precipitation data, load appropriate data into a new ArcMap document. This includes the following items from the TexasNEXRAD_Erin geodatabase:

·        Texas – from TexasNEXRAD_Erin | Spatial Data

·        Reclassified Rasters (BreaksR****) – from TexasNEXRAD_Erin

·        Precipitation Polygons (FinalTenP****) – from TexasNEXRAD_Erin | TenPolygons

The result may look like the following:

As we saw earlier, ArcMap’s Spatial Analyst is a powerful tool for analyzing raster data like that which we are working with. We’ve used the Spatial Analyst tools, but now we’ll use the Spatial Analyst toolbar.

Make the toolbar visible by selecting Tools | Customize and checking Spatial Analyst in the list in the Toolbars tab of the dialog.

The Spatial Analyst toolbar should now be visible.

We’ll now explore a useful feature of the Spatial Analyst toolbar: Zonal Histogram. Launch the dialog by choosing Spatial Analyst | Zonal Histogram from the toolbar.

In the resulting dialog, choose one of the FinalTenP**** polygons for the zone dataset, a BreaksR**** raster for the value raster, and a name and location for the output table.

Running this tool creates a tabular histogram of the number of cells within the zone dataset (the polygon) of each value.

The results for the 1601 datasets are shown below:

Please note that the value rasters (BreaksR****) have values of 1, 2, 3, etc. These correspond to the following:

Note further that the attributes of the histogram table may not appear automatically. To view this, choose the Source tab of ArcMap’s table of contents view (the window on the left-hand side), locate the table from the list of items in the map, and right-click on the table and choose Open.

Zonal histograms can be created following this pattern for each of the raster / polygon combinations. We’ve seen the power afforded by tabular histograms in various course exercises this semester. This exercise has shown a way to obtain a geospatially derived set of histogram data. These histograms may be useful for a variety of kinds of analyses. These histograms provide an additional means for analyzing and presenting statistical data. Armed with this information, you can conduct research-specific analyses into the distribution of rainfall over an area, say. The possibilities are great.

Summary

We now have a method by which one can track a severe storm in space and time. We were not able to download data for the storm track displayed by the NHC however we can see from the map above that the tracks follow the same patterns. Since we have the timestamps for each of the centroids we create, we could proceed and find the velocity of Tropical Storm Erin as it moved over the state of Texas. We will leave this adventure to you. For demonstration purposes, we only chose 11 times to display the storm track; however, with the tools described above we could have easily tracked the storm with a higher frequency.

In addition to tracking a storm as it moves through space (and time), we’ve introduced methods for obtaining histograms of reclassified geospatial data. These histograms may be useful for a gamut of analyses, including analyzing the distribution of various intensities of NEXRAD precipitation in specific defined areas.

Tracking Precipitation in Space and Time

Previous sections have introduced methods for tracking storms and analyzing distributions of storm effects (e.g. precipitation distributions). This section walks through the process of using the NEXRAD data for Tropical Storm Erin to determine the amount of rainfall experienced in a specific geographic location over a defined time period. We will measure and track precipitation in Bexar County—the county where San Antonio is located—using the NEXRAD data. This tracking will involve determining the amount (depth) of rainfall in a given time period, and can be used to create a rainfall hyetograph—the temporal distribution of precipitation.

Mapping Storm in GIS

To get started, launch a new instance of ArcMap and choose to work with a new empty map. As before, choose to Add Data  and navigate to the location for the TexasNEXRAD_Erin geodatabase.

In the SpatialData feature dataset, add the Counties and BexarCounty feature classes. These are geographic representations of Texas’ counties and the individual county of Bexar, respectively.

Rather than go through the full process of importing specific NEXRAD files, representing them as layers, and exporting them to rasters, choose to Add Data once again and add the Nexrad16** layer files from the Layer directory, which was included in the zipped exercise data. These files store the symbology and reference to the NEXRAD rasters from the TexasNEXRAD_Erin geodatabase. These rasters are the raster representations of the NetCDF NEXRAD data, and have already been processed for easier use in ArcMap. The data come from August 2007, and the naming scheme includes the source, the date, and the hour (e.g. Nexrad1617 corresponds to the NEXRAD data from the seventeenth hour of August 16, 2007).

The following is a representation of the data in ArcMap, with one hour of NEXRAD data shown. Bexar County is shown selected (cyan in picture).

Now that the appropriate data is added to ArcMap, let’s use its tools to gather meaningful information.

Spatial Analyst

As we saw earlier, ArcMap’s Spatial Analyst is a powerful tool for analyzing raster data like that which we are working with. We’ve used the Spatial Analyst tools and briefly introduced the Spatial Analyst toolbar (using the Histogram function); now we’ll explore additional capabilities of this tool. To use the Spatial Analyst toolbar, we must, again, be working with the proper extension enabled (you can check this by clicking on Tools | Extensions and making sure that Spatial Analyst is checked).

As a review, make the toolbar visible by selecting Tools | Customize and checking Spatial Analyst in the list in the Toolbars tab of the dialog.

The Spatial Analyst toolbar should now be visible.

Cell Statistics

Let’s first determine the amount of rainfall that occurred over Bexar County in the selected eight-hour period (associated with the NEXRAD rasters on the map). The Spatial Analyst function, Cell Statistics, is helpful here. On the toolbar, choose Spatial Analyst | Cell Statistics.

In the resulting dialog, select all the Nexrad16** rasters, click on Add ->, and specify Sum as the overlay statistic. The result will be a temporary raster that is the sum of precipitation values from all the NEXRAD rasters. (This process may take some time.) The output raster can be saved by changing the output location, if desired.

The result of the Cell Statistics operation is shown below. You can see that Bexar County had the brunt of the precipitation in this period (Harris County—where Houston lies—also had significant rain in this time period).

As fun as seeing the aggregated precipitation over the state of Texas is, we still want to know the precipitation depth in Bexar County. The above raster’s colors and associated values represent accumulated precipitation at each cell; we want the precipitation for the whole county. Spatial Analyst can help us with this, too.

Zonal Statistics

On the toolbar, choose Spatial Analyst | Zonal Statistics to get the following dialog.

Ensure that the county feature class, BexarCounty, is set as the zone dataset, Sum is the value raster (“Sum” is the name of the temporary raster created above, which we’re using to get the precipitation depth), Chart Statistic is turned off, and choose a location and name for the output table that is convenient for you.

When the zonal statistics process is complete, you may see results like the following:

The table is of interest, particularly the MEAN field, which reports an average depth of precipitation in Bexar County of 115 mm (~4.5 inches).

Please recall that this depth of rain is over an eight-hour period. Perhaps we are interested in the incremental depths of rainfall. These are easily obtained using a similar process as above, but instead of using the temporary raster for the whole period, we’ll execute Zonal Statistics for each individual NEXRAD hourly raster and track the results.

Launch the Zonal Statistics dialog again by choosing Spatial Analyst | Zonal Statistics on the toolbar. Repeat the process used above for the Sum raster, but choose each Nexrad16** raster in turn, being sure to specify an output location for each. You can also record the MEAN value from each to plot a hyetograph.

Hyetograph

After logging the resulting values, a hyetograph can be produced with any standard graphing software. The following are tabular and graphical representations of the hyetograph corresponding to the average hourly depth of rainfall in Bexar County, TX during the eight-hour period of analysis (Aug 16, 2007).

Summary

This section was a brief introduction to tracking NEXRAD precipitation in space and time. We have more fully explored the capabilities of the Spatial Analysis tools, and have a more solid footing in using rasters to obtain useful information. In addition to determining the overall rainfall depth in a specified location over a defined time period, we have used the Spatial Analysis tools to produce a rainfall hyetograph—the distribution of rainfall over time. These tools unlock doors of analyzing data as well as presenting data in meaningful ways.