Saving Data

Once you request data, it is possible to save it locally to a file. For this task, HydroFunctions uses Parquet, a compact file format for storing large datasets. Parquet is efficient: file sizes are small and can be read quickly. Parquet is great for large files, becaues it is possible to access parts of the file without reading the entire file.

Hydrofunctions allows you to save your data to a parquet file automatically by providing a file name as a parameter to the NWIS object:



In [1]:

    
import hydrofunctions as hf



In [2]:

    
new = hf.NWIS('01585200', 'dv', start_date='2018-01-01', end_date='2019-01-01', file='parquet_example.parquet')
new









    



Requested data from https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200&startDT=2018-01-01&endDT=2019-01-01
Saving data to parquet_example.parquet






    Out[2]:





USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00

Automatic file reading & writing

The first time that you make the request, hydrofunctions will save the incoming data into a new file, and you will get a message, Saving data to filename.

The second time that you make the request, hydrofunctions will read the data from the file instead of requesting it, and you will get a message, Reading data from filename.



In [3]:

    
new = hf.NWIS('01585200', 'dv', start_date='2018-01-01', end_date='2019-01-01', file='parquet_example.parquet')
new









    



Reading data from parquet_example.parquet






    Out[3]:





USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00

In effect, the local parquet file will act as a local cache for your data, reducing your network traffic.

Manual file reading & writing

It is also possible to force hydrofunctions to read or write a file by using the NWIS.read() and NWIS.save() methods.



In [5]:

    
new.save('parquet_example2.parquet')









    Out[5]:





USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00



In [6]:

    
new.read('parquet_example2.parquet')
new









    Out[6]:





USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00