Writing Valid Requests for NWIS

The USGS National Water Information System (NWIS) is capable of handling a wide range of requests. A few features in Hydrofunctions are set up to help you write a successful request.



In [1]:

    
# First, import hydrofunctions.
import hydrofunctions as hf

What can we specify?

The NWIS can handle data requests that specify:

Where: we need to specify which stations we are interested in.
Service: the NWIS provides daily averages ('dv') and 'instantaneous values' ('iv')
When: we can specify a range of dates, a period of time before now, or just get the most recent observation.
What: we can specify which parameter we want, or just get everything collected at the site.
the data service we want.

The only required element is a station:



In [2]:

    
minimum_request = hf.NWIS('01585200')









    



Requested data from https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200

Since we only specified the where, the NWIS will assume the following elements:

Service: if not specified, provide the daily average value ('dv')
When: if a start_date or period is not given, then provide the most recent reading.
What: if you don't ask for a specific parameter (parameterCd), you will get everything.

Let's see what our request came back with:



In [3]:

    
minimum_request









    Out[3]:





USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <0 * Minutes>  Discharge, cubic feet per second
Start: 2019-07-22 00:00:00+00:00
End:   2019-07-22 00:00:00+00:00

Here's what the data look like in table form:



In [4]:

    
minimum_request.df()









    Out[4]:







  
    
      
      USGS:01585200:00060:00003_qualifiers
      USGS:01585200:00060:00003
    
    
      datetimeUTC
      
      
    
  
  
    
      2019-07-22 00:00:00+00:00
      P
      1.39

Different ways to specify which site you want

You can specify a site four different ways:

as a number or list of site numbers
using stateCd and a two letter postal code to retrieve every site in the state
using countyCd and a FIPS code to retrieve every site in a county or list of counties
using bBox to retrieve everything inside of a bounding box of latitudes and longitudes.

You are required to set one of these parameters, but only one.

All of these parameters are demonstrated in Selecting Sites

Different ways to specify time

You can specify time in three different ways:

if you specify nothing, you'll get the most recent reading.
period will return up to 999 days of the most recent data: period='P11D'
start_date will return all of the data starting at this date: start_date='2014-12-31'

If you specify a start_date, you can also specify an end_date, which is given in the same format.

What happens when you make a bad request?

The power of the NWIS also makes it easy to make mistakes. So, we've added a series of helpful error messages to let you know when something went wrong, and why it went wrong.



In [6]:

    
# For example, let's mistpye one of our parameters that worked so well above:
notSoGoodNWIS = hf.NWIS('01585200', 'xx', period='P200D')









    



---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-cbd93523cd36> in <module>
      1 # For example, let's mistpye one of our parameters that worked so well above:
----> 2 notSoGoodNWIS = hf.NWIS('01585200', 'xx', period='P200D').get_data()

c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\station.py in __init__(self, site, service, start_date, end_date, stateCd, countyCd, bBox, parameterCd, period, file)
    129                                         bBox=bBox,
    130                                         parameterCd=parameterCd,
--> 131                                         period=period
    132                                         )
    133             try:

c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\hydrofunctions.py in get_nwis(site, service, start_date, end_date, stateCd, countyCd, bBox, parameterCd, period)
    217     """
    218 
--> 219     service = typing.check_NWIS_service(service)
    220 
    221     if (parameterCd == 'all'):

c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\typing.py in check_NWIS_service(input)
    114         raise TypeError("The NWIS service type accepts 'dv' for daily values, "
    115                         "or 'iv' for instantaneous values. Actual value: "
--> 116                         "{}".format(input))
    117 
    118 

TypeError: The NWIS service type accepts 'dv' for daily values, or 'iv' for instantaneous values. Actual value: xx

Okay, maybe I shouldn't have typed 'xx' for our service.

Some errors get caught by hydrofunctions, but some don't. Sometimes we end up asking NWIS for something that doesn't make sense, or something that it doesn't have, or maybe NWIS isn't available. In this case, hydrofunctions will receive an error message from NWIS and help you figure out what went wrong.



In [7]:

    
# Let's ask for the impossible: the start date is AFTER the end date:
badRequest = hf.get_nwis('01585200', 'dv', '2017-12-31', '2017-01-01')









    



Requested data from https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200&startDT=2017-12-31&endDT=2017-01-01






    



c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\hydrofunctions.py:627: SyntaxWarning: The NWIS returned a code of 400.
400 Bad Request - This often occurs if the URL arguments are inconsistent. For example, if you submit a request using a startDT and an endDT with the period argument. An accompanying error should describe why the request was bad.
Error message from NWIS: Bad Request

URL used in this request: https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200&startDT=2017-12-31&endDT=2017-01-01
  warnings.warn(msg, SyntaxWarning)

Getting help

I probably shouldn't have started with all of the things that go wrong! My point is that we've got ya.

Where can you go to learn how to do things the RIGHT way?

But we also have a few built-in helpers that you can use right here, right now:

help() and ? will list the docstring for whatever object you are curious about
dir() and .\<TAB> will tell you about available methods.



In [8]:

    
# Use the help() function to see all of the parameters for a function, their default values, 
# and a short explanation of what it all means. Or you can type ?hf.NWIS to access the same information.
help(hf.NWIS)









    



Help on class NWIS in module hydrofunctions.station:

class NWIS(Station)
 |  NWIS(site=None, service='dv', start_date=None, end_date=None, stateCd=None, countyCd=None, bBox=None, parameterCd='all', period=None, file=None)
 |  
 |  A class for working with data from the USGS NWIS service.
 |  
 |  description
 |  
 |  Args:
 |      site (str or list of strings):
 |          a valid site is '01585200' or ['01585200', '01646502']. Default is
 |          None. If site is not specified, you will need to select sites using
 |          stateCd or countyCd.
 |  
 |      service (str):
 |          can either be 'iv' or 'dv' for instantaneous or daily data.
 |          'dv'(default): daily values. Mean value for an entire day.
 |          'iv': instantaneous value measured at this time. Also known
 |          as 'Real-time data'. Can be measured as often as every
 |          five minutes by the USGS. 15 minutes is more typical.
 |  
 |      start_date (str):
 |         should take on the form 'yyyy-mm-dd'
 |  
 |      end_date (str):
 |          should take on the form 'yyyy-mm-dd'
 |  
 |      stateCd (str):
 |          a valid two-letter state postal abbreviation, such as 'MD'. Default
 |          is None. Selects all stations in this state. Because this type of
 |          site selection returns a large number of sites, you should limit
 |          the amount of data requested for each site.
 |  
 |      countyCd (str or list of strings):
 |          a valid county FIPS code. Default is None. Requests all stations
 |          within the county or list of counties. See https://en.wikipedia.org/wiki/FIPS_county_code
 |          for an explanation of FIPS codes.
 |  
 |      bBox (str, list, or tuple):
 |          a set of coordinates that defines a bounding box.
 |              * Coordinates are in decimal degrees.
 |              * Longitude values are negative (west of the prime meridian).
 |              * Latitude values are positive (north of the equator).
 |              * comma-delimited, no spaces, if provided as a string.
 |              * The order of the boundaries should be: "West,South,East,North"
 |              * Example: "-83.000000,36.500000,-81.000000,38.500000"
 |  
 |      parameterCd (str or list of strings):
 |          NWIS parameter code. Usually a five digit code. Default is 'all'.
 |          A valid code can also be given as a list: parameterCd=['00060','00065']
 |          This will request data for this parameter.
 |  
 |              * if value is 'all', or no value is submitted, then NWIS will                 return every parameter collected at this site. (default option)
 |              * stage: '00065'
 |              * discharge: '00060'
 |              * not all sites collect all parameters!
 |              * See https://nwis.waterdata.usgs.gov/usa/nwis/pmcodes for full list
 |  
 |      period (str):
 |          NWIS period code. Default is None.
 |              * Format is "PxxD", where xx is the number of days before                 today, with a maximum of 999 days accepted.
 |              * Either use start_date or period, but not both.
 |  
 |  Method resolution order:
 |      NWIS
 |      Station
 |      builtins.object
 |  
 |  Methods defined here:
 |  
 |  __init__(self, site=None, service='dv', start_date=None, end_date=None, stateCd=None, countyCd=None, bBox=None, parameterCd='all', period=None, file=None)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |  
 |  __repr__(self)
 |      Return repr(self).
 |  
 |  df(self, *args)
 |      Return a subset of columns from the dataframe.
 |      
 |      Args:
 |          '': If no args are provided, the entire dataframe will be returned.
 |      
 |          str 'all': the entire dataframe will be returned.
 |      
 |          str 'data': all of the parameters will be returned, with no flags.
 |      
 |          str 'flags': Only the _qualifier flags will be returned. Unless the             flags arg is provided, only data columns will be returned. Visit             https://waterdata.usgs.gov/usa/nwis/uv?codes_help#dv_cd1 to see a             more complete listing of possible codes.
 |      
 |          str 'discharge' or 'q': discharge columns ('00060') will be returned.
 |      
 |          str 'stage': Gauge height columns ('00065') will be returned.
 |      
 |          int any five digit number: any matching parameter columns will be returned. '00065' returns stage, for example.
 |      
 |          int any eight to twelve digit number: any matching stations will be returned.
 |  
 |  get_data(self)
 |  
 |  read(self, file)
 |  
 |  save(self, file)
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from Station:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  ----------------------------------------------------------------------
 |  Data and other attributes inherited from Station:
 |  
 |  station_dict = {}



In [9]:

    
# Use the dir() function to see what sort of methods you have available to you,
# or type hf.NWIS.<TAB> to see the same list.
dir(hf.NWIS)









    Out[9]:





['__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 'df',
 'get_data',
 'read',
 'save',
 'station_dict']