This Notebook was created to help you get familiar with the commands you can use in LiPD. Follow each step and experiment as much as you'd like! Each step will help prepare you for the next step.
For this tutorial we will be using the example files found in the Github Repostiory's Examples folder.
Last Notebook Edit : 05.10.17
This guide assumes that you have already followed the installation steps and the LiPD package is installed on your computer.
In [ ]:
import lipd
In [ ]:
lipd.quit()
There are four valid file types that you may import. Use the appropriate function for the file type that you would like to read.
**NOTE**
All "read" functions accept a file path argument. If the path argument is left empty, a GUI window will appear and allow you to choose a file or choose a directory.
If you want to multi-select speciic files, but not an entire directory, use any singular read function under without a providing a path argument. Use the GUI to multi-select the files you want.
In [ ]:
# Read File - GUI
lipd.readLipd()
lipd.readExcel()
lipd.readNoaa()
# Read File - with path argument - no GUI
lipd.readLipd("/path/to/file.lpd")
lipd.readExcel("/path/to/file.xls")
lipd.readNoaa("/path/to/file.txt")
# Read Directory - GUI
lipd.readLipds()
lipd.readExcels()
lipd.readNoaas()
# Read Directory - with path argument - no GUI
lipd.readLipds("/path/to/dir/")
lipd.readExcels("/path/to/dir/")
lipd.readNoaas("/path/to/dir/")
# Read Directory - all file types - GUI
lipd.readAll()
# Read Directory - all file types - no GUI
lipd.readAll("/path/to/dir/")
Microsoft Excel spreadsheets must be converted to LiPD before any LiPD functions can be used. Use the Excel template to create an Excel file with your data. Make sure to follow the formatting guidelines and the hints noted throughout the spreadsheet.
**NOTE**
The Excel converter runs every `.xls` and `.xlsx` file currently loaded.
In [ ]:
lipd.excel()
National Oceanic and Atmospheric Administration (NOAA) text files must be converted to LiPD before any LiPD functions can be used. The converter is designed to parse data from the NOAA text template. Please insert your data in this template format to ensure a complete and accurate conversion to LiPD. Use the example file as a reference for correct formatting.
LPD to NOAA:
Converts all LiPD files into NOAA text files. Creates one NOAA text file for each data table found in the LiPD file. The LiPD file is updated to include the WDC URL that links to the corresponding NOAA dataset that it creates.
NOAA to LPD:
Converts NOAA text file into a LiPD file.
NOAA text template:
NOAA template
Example File:
khider2011b.txt
**NOTE**
The NOAA converter will run for every `.txt` file currently loaded.
These conversions are both potentially lossy conversions. We have worked hard to map data between formats, but there are some keys that remain unmapped and will not be transferred to the new file format.
In [ ]:
# Run the function
lipd.noaa()
# Choose a conversion
Which conversion?
1. LPD to NOAA
2. NOAA to LPD
The DOI updater will take your LiPD files and update them with the most recent information provided by doi.org. The updater will run once per LiPD, and will skip any LiPD files that were updated previously.
**NOTE**
`doi()` will update each LiPD file loaded after `readLipds()`
In [ ]:
lipd.doi()
In [ ]:
# Write Files - GUI
lipd.writeLipds()
# Write with path argument - No GUI
lipd.writeLipds("/path/to/dir/")
In [ ]:
import pickle
import gzip
# Read a pickle file
f = gzip.open('filename.pklz','rb')
newData = pickle.load(f)
f.close()
# Write a pickle file
yourData = {'a':'blah','b':range(10)}
f = gzip.open('filename.pklz','wb')
pickle.dump(yourData,f)
f.close()
Show functions are useful for printing data to the console. Note: Some consoles will truncate extensive output. Printing large data to the console may not work well.
showLipds()
showMetadata(filename)
showCsv(filename)
showDfs(dataframe_dictionary)
In [ ]:
odp_csv = lipd.getCsv("ODP1098B12.lpd")
In [ ]:
odp_metadata = lipd.getMetadata("ODP1098B12.lpd")
All LiPD Library data is stored in Python "objects" that allow us to manage and manipulate the data more easily. However, storing and pickling the data in this structure is not ideal for most situations. The JSON format is recommended as it is universal and easier to share.
getLibrary()
In [ ]:
D = lipd.getLibrary()
In [ ]:
%%html
<img src="./d.png" />
TimeSeries functions are useful creating, filtering, and exporting TimeSeries from the LiPD data in the workspace.
extractTs()
collapseTs(time_series)
find(expression, time_series)
**WARNING**
`collapseTs()` will overwrite the contents of the LiPD Library
In [ ]:
%%html
<img src="./tso1.png" />
<img src="./tso2.png" />
In [ ]:
time_series = lipd.extractTs()
In [ ]:
new_time_series = lipd.find("archiveType is marine sediment", time_series)
In [ ]:
new_time_series = lipd.find("geo_meanElev <= -1000 && geo_meanElev > -1100", time_series)
In [ ]:
lipd.collapseTs(time_series)
ensToDf(arrays)
lipdToDf(filename)
tsToDf(time_series, filename)
**NOTE**
After creating the data frames, calling a specific data frame variable will display the formatted data frame.
In [ ]:
dfs_lipd = lipd.lipdToDf("ODP1098B12.lpd")
In [ ]:
lipd.showDfs(dfs_lipd)
In [ ]:
dfs_lipd["metadata"]
In [ ]:
dfs_lipd["paleoData"]["ODP1098B12.Paleo1.measurementTable1.csv"]
In [ ]:
dfs_lipd["chronData"]["ODP1098B12.Chron1.measurementTable1.csv"]
**NOTE**
TimeSeries objects are not able to use the autocomplete feature for filenames. Be sure to run `showTsos()` and copy/paste the TimeSeries object of interest
In [ ]:
dfs_ts = lipd.tsToDf(time_series, "ODP1098B12_data_SST")
In [ ]:
showDfs(dfs_ts)
In [ ]:
dfs_ts["metadata"]
In [ ]:
dfs_ts["paleoData"]
In [ ]:
dfs_ts["chronData"]["ODP1098B12"]
In [ ]:
lipd.removeLipds()