In [225]:
import zipfile
import requests # not a dependency of engarde
import engarde.checks as ck
%matplotlib inline
pd.options.display.max_rows = 20
Just a couple quick examples.
In [213]:
# This will take a few minutes
r = requests.get("http://www.transtats.bts.gov/Download/On_Time_On_Time_Performance_2015_1.zip",
stream=True)
with open("otp-1.zip", "wb") as f:
for chunk in r.iter_content(chunk_size=1024):
f.write(chunk)
f.flush()
r.close()
z = zipfile.ZipFile("otp-1.zip")
fp = z.extract('On_Time_On_Time_Performance_2015_1.csv')
In [144]:
columns = ['FlightDate', 'Carrier', 'TailNum', 'FlightNum',
'Origin', 'OriginCityName', 'OriginStateName',
'Dest', 'DestCityName', 'DestStateName',
'DepTime', 'DepDelay', 'TaxiOut', 'WheelsOn',
'WheelsOn', 'TaxiIn', 'ArrTime', 'ArrDelay',
'Cancelled', 'Diverted', 'ActualElapsedTime',
'AirTime', 'Distance', 'CarrierDelay', 'WeatherDelay',
'NASDelay', 'SecurityDelay', 'LateAircraftDelay']
df = pd.read_csv('On_Time_On_Time_Performance_2015_1.csv', usecols=columns,
dtype={'DepTime': str})
dep_time = df.DepTime.fillna('').str.pad(4, side='left', fillchar='0')
df['ts'] = pd.to_datetime(df.FlightDate + 'T' + dep_time,
format='%Y-%m-%dT%H%M%S')
df = df.drop(['FlightDate', 'DepTime'], axis=1)
Let's suppose that down the road our probram can only handle certain carriers; an update to the data adding a new carrier would violate an assumpetion we hold. We'll use the within_set method to check our assumption
In [226]:
carriers = ['AA', 'AS', 'B6', 'DL', 'US', 'VX', 'WN', 'UA', 'NK', 'MQ', 'OO',
'EV', 'HA', 'F9']
df.pipe(ck.within_set, items={'Carrier': carriers}).Carrier.value_counts().head()
Out[226]:
Great, our assumption was true (at least for now).
Surely, we can't count on each flight having a Carrier, TailNum and FlightNum, right?
In [217]:
df.pipe(ck.none_missing, columns=['Carrier', 'TailNum', 'FlightNum'])
Note: this isn't too user-friendly yet. I'm planning to make the error messages more informative. Just haven't gotten there yet.
That said, you wouldn't really use engarde to determine whether or not those columns are always not null. Instead, you might find that for January every flight has all of those fields, assume that hold generally, only to be surprised when next month you get a flight without a tail number.
Each of your checks can also be written as a decorator on a function that returns a DataFrame.
I really like how slick this is.
Let's do a nonsense example. Suppose we want to show the counts of each Carrier, but our UI designer
worries that if things are too spread out the bar graph will look weird (again, silly example). We'll
assert that teh counts are within a comfortable range and that each count is within 3 standard deviations of the mean.
In [266]:
import engarde.decorators as ed
@ed.within_range({'Counts!': (4000, 110000)})
@ed.within_n_std(3)
def pretty_counts(df):
return df.Carrier.value_counts().to_frame(name='Counts!')
In [268]:
pretty_counts(df)
Out[268]:
No AssertionError was raised so we're all good.
I'll typically find myself with a handful of functions that define my ETL process from a flat file to
whatever the end result is. Each function takes and returns a DataFrame. This can be nicely layered into a pipeline. Using decorators allows you to make the checks without breaking up the flow of the pipeline.
In [ ]: