Fieldmove note merger

Data collected with the Fieldmove Clino app in the field can be exported into a number of discrete .csv files. The code in this notebook:

imports these files into pandas dataframes
combines these dataframes into a single dataframe
filters the dataframe to only include data fields that are particularly relevant
exports this filtered dataframe as a LaTeX table

Import necessary modules:



In [1]:

    
import os
import pandas as pd
pd.set_option('display.max_colwidth', -1)
from dateutil.parser import parse
from numpy import zeros

ACTION REQUIRED: Set your folder path:

All output files will be created in this directory.

First time users:

comment (append a '#') all paths
using Terminal, navigate to the folder (using 'cd') that contains all Fieldmove .csv's
enter command 'pwd'
copy + paste output to 'blank'
uncomment the new path (remove the '#')

Everyone else:

comment (append a '#') all paths
uncomment (remove the '#') your particular path



In [2]:

    
# folder_path = 'blank'
# folder_path = u'/Users/Laurentia/Dropbox/research_projects/keweenawan/2015_fieldwork/Field_data/Michipicoten_data/project1.fm'
# folder_path = '/Users/yuempark/Documents/Berkeley/Ethiopia/FieldMove/project1.fm'
folder_path = '/Users/yuempark/Documents/Berkeley/Research_China/FieldMove/project1.fm'

print(folder_path)









    



/Users/yuempark/Documents/Berkeley/Research_China/FieldMove/project1.fm

Read in the data:



In [3]:

    
image = pd.read_csv(folder_path + '/image.csv')
note = pd.read_csv(folder_path + '/note.csv')
plane = pd.read_csv(folder_path + '/plane.csv')
line = pd.read_csv(folder_path + '/line.csv')

print('image.csv imported with ' + str(image.shape[0]) + ' entries.')
print('note.csv imported with ' + str(note.shape[0]) + ' entries.')
print('plane.csv imported with ' + str(plane.shape[0]) + ' entries.')
print('line.csv imported with ' + str(line.shape[0]) + ' entries.')









    



image.csv imported with 207 entries.
note.csv imported with 280 entries.
plane.csv imported with 434 entries.
line.csv imported with 2 entries.

Concatenate the 4 data files:



In [4]:

    
all_notes = pd.concat((image, note, plane, line))

Sort by time:



In [5]:

    
all_notes['time'] = pd.to_datetime(all_notes[' timedate'])
all_notes.sort_values(by='time',inplace='True')
all_notes.reset_index(inplace='True')

Output to 'all_notes.csv', without any filtering:



In [6]:

    
all_notes.to_csv(os.path.join(folder_path,r'all_notes.csv'),index=False)

print(all_notes.columns)









    



Index(['index', ' altitude', ' dataId', ' declination', ' dip', ' dipAzimuth',
       ' heading', ' horiz_precision', ' image name', ' latitude',
       ' lineationType', ' localityName', ' longitude', ' notes', ' planeType',
       ' plunge', ' plungeAzimuth', ' strike', ' timedate', ' unitId',
       ' vert_precision', ' x', ' y', ' zone', 'localityId', 'time'],
      dtype='object')

Select columns that we want in our final LaTeX document:

('localityName' is not a header in all .csv's)



In [7]:

    
if ' localityName' in all_notes.columns:
    filtered_notes = all_notes[['time',
                                ' latitude',
                                ' longitude',
                                ' localityName',
                                ' notes',
                                ' image name',
                                ' heading',
                                ' planeType',
                                ' dipAzimuth',
                                ' dip',
                                ' lineationType',
                                ' plungeAzimuth',
                                ' plunge',
                                ' unitId']]
    
else:
    fill = zeros((all_notes.shape[0],1))
    all_notes[' localityName'] = fill
    filtered_notes = all_notes[['time',
                                ' latitude',
                                ' longitude',
                                ' localityName',
                                ' notes',
                                ' image name',
                                ' heading',
                                ' planeType',
                                ' dipAzimuth',
                                ' dip',
                                ' lineationType',
                                ' plungeAzimuth',
                                ' plunge',
                                ' unitId']]
    
print(filtered_notes.columns)
print(filtered_notes.shape[1])









    



Index(['time', ' latitude', ' longitude', ' localityName', ' notes',
       ' image name', ' heading', ' planeType', ' dipAzimuth', ' dip',
       ' lineationType', ' plungeAzimuth', ' plunge', ' unitId'],
      dtype='object')
14

Output to 'all_notes_filtered.csv':



In [8]:

    
filtered_notes.to_csv(os.path.join(folder_path,r'all_notes_filtered.csv'), index=False)

Convert notes to a LaTeX format:

The 'to_latex' package doesn't handle mega-tables (like ours) well at all, so we will instead manually create a table.

First, input the necessary frontmatter (packages, etc.):



In [9]:

    
frontmatter = r'\documentclass[11pt]{article}' + '\n' +\
              r'\usepackage{amsmath}' + '\n' +\
              r'\usepackage{amsfonts}' + '\n' +\
              r'\usepackage{amssymb}' + '\n' +\
              r'\usepackage{wasysym}' + '\n' +\
              r'\usepackage{graphicx}' + '\n' +\
              r'\usepackage{pslatex}' + '\n' +\
              r'\usepackage{lscape}' + '\n' +\
              r'\usepackage[T1]{fontenc}' + '\n' +\
              r'\usepackage[latin1]{inputenc}' + '\n' +\
              r'\usepackage{fancyhdr}' + '\n' +\
              r'\usepackage{lastpage}' + '\n' +\
              r'\usepackage[english]{babel}' + '\n' +\
              r'\usepackage[usenames, dvipsnames]{color}' + '\n' +\
              r'\usepackage{color}' + '\n' +\
              r'\usepackage{booktabs}' + '\n' +\
              r'\usepackage{longtable}' + '\n' +\
              r'\usepackage{geometry}' + '\n' +\
              r'\geometry{hmargin={0.75in,0.75in},vmargin={1in,1in}}' + '\n'

ACTION REQUIRED: When prompted, enter a title for your Fieldmove Notes (e.g. Ethiopia 2015), as well as your name:



In [10]:

    
titleIn = input('Enter the title of your notes: ')
authorIn = input('Enter your name: ')
title = r'\title{Fieldmove Notes\\' + titleIn + r'}' + '\n' +\
        r'\author{' + authorIn + r'}' + '\n'









    



Enter the title of your notes: South China 2016
Enter your name: Yuem Park

Create a header for each page:



In [11]:

    
header = r'\pagestyle{fancy}' + '\n' +\
         r'\rhead {\emph{\textcolor{blue}{' + authorIn + r'}, \thepage ~of \pageref{LastPage}}}' + '\n' +\
         r'\lhead{' + titleIn + r'}' + '\n' +\
         r'\cfoot{}' + '\n' +\
         r'\renewcommand{\headrulewidth}{0.4pt}' + '\n'

Some more necessary LaTeX commands before starting the table:



In [12]:

    
start = r'\begin{document}' + '\n' +\
        r'\maketitle' + '\n' +\
        r'\begin{longtable}{llllr}' + '\n' +\
        r'\endhead' + '\n' +\
        r'\endfoot' + '\n' +\
        r'\endlastfoot' + '\n'

Necessary LaTeX commands after the table ends:



In [13]:

    
end = r'\end{longtable}' + '\n' +\
      r'\end{document}'

Now create the table:

(First, a little function to make text bold:)



In [14]:

    
def bold(string):
    out = r'\textbf{' + string + '}'
    return out

Generate the main body of the LaTeX code:

Note that certain special characters need special treatment, otherwise LaTeX will fail to compile.



In [15]:

    
body = ''

for i in range(filtered_notes.shape[0]-1):
    entry = ''
    
    entry += r'\hline' + '\n'
    entry += r'$_{' + str(i+1) + r'}$&&&&\\' + '\n'
    entry += bold('time:') + ' ' + str(filtered_notes['time'][i]) + r' & '
    entry += bold('lat:') + ' ' + str(filtered_notes[' latitude'][i]) + r' & '
    entry += bold('lon:') + ' ' + str(filtered_notes[' longitude'][i]) + r' & '
    entry += r'& '
    
    #if we have locality data, include it
    if filtered_notes[' localityName'][i] != 0:
        entry += bold('locality:') + ' ' + str(filtered_notes[' localityName'][i]) + r' \\' + '\n'
    else:
        entry += r'\\' + '\n'
    
    #always include notes
    entry += r'\multicolumn{5}{p{\linewidth}}{'
    entry += bold('note:') + ' ' + str(filtered_notes[' notes'][i]) + r'} \\' + '\n'
    
    #if we have photos, include it
    try:
        temp = filtered_notes[' image name'][i].split('_') #special character
        entry += bold('image:') + ' ' + temp[0] + r'\_' + temp[1] + r' & '
        entry += bold('heading:') + ' ' + str(round(filtered_notes[' heading'][i],1)) + r' & '
        entry += r'& & \\' + '\n'
    except:
        pass
    
    #if we have a formation name, include it
    if str(filtered_notes[' unitId'][i]) == 'nan':
        pass
    else:
        entry += bold('fm:') + ' ' + str(filtered_notes[' unitId'][i]) + r' & '
        entry += r'& & & \\' + '\n'
    
    #if we have plane data, include it
    if str(filtered_notes[' dip'][i]) == 'nan':
        pass
    else:
        entry += bold('plane:') + ' ' + str(filtered_notes[' planeType'][i]) + r' & '
        entry += bold('dip:') + ' ' + str(round(filtered_notes[' dip'][i],1)) + r' & '
        entry += bold('azimuth:') + ' ' + str(round(filtered_notes[' dipAzimuth'][i],1)) + r' & '
        entry += r'& \\' + '\n'
    
    #if we have line data, include it
    if str(filtered_notes[' plunge'][i]) == 'nan':
        pass
    else:
        entry += bold('line:') + ' ' + str(filtered_notes[' lineationType'][i]) + r' & '
        entry += bold('plunge:') + ' ' + str(round(filtered_notes[' plunge'][i],1)) + r' & '
        entry += bold('azimuth:') + ' ' + str(round(filtered_notes[' plungeAzimuth'][i],1)) + r' & '
        entry += r'& \\' + '\n'
    
    body += entry

body += r'\hline' + '\n'
    
#special characters  
body = body.replace('%',r'\%')
body = body.replace('<',r'$<$')
body = body.replace('>',r'$>$')
body = body.replace('~',r'\textasciitilde')

Combine all the strings, and prepare for output into a .tex file:



In [16]:

    
toLatex = frontmatter + header + title + start + body + end

Output to 'latexoutput.tex':



In [17]:

    
afile = open(os.path.join(folder_path,r'latexoutput.tex'), 'wt')
afile.write(toLatex)









    Out[17]:





289407

ACTION REQUIRED: Open 'latexoutput.tex' with your favourite .tex editor, and compile!

'latexoutput.pdf' is the final product.