Influxdb is a time series database written from scratch (in go) and independent of any other database infrastructure. Using the basic influxdb python client, we create a database, a measurement and upload some time series data


In [1]:
%%bash
pip install influxdb


Collecting influxdb
  Downloading influxdb-2.12.0-py2.py3-none-any.whl (262kB)
Collecting python-dateutil>=2.0.0 (from influxdb)
  Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
Requirement already satisfied (use --upgrade to upgrade): requests>=1.0.3 in /usr/lib/python2.7/dist-packages (from influxdb)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/local/lib/python2.7/dist-packages (from influxdb)
Requirement already satisfied (use --upgrade to upgrade): pytz in /usr/lib/python2.7/dist-packages (from influxdb)
Installing collected packages: python-dateutil, influxdb
  Found existing installation: python-dateutil 1.5
    Uninstalling python-dateutil-1.5:
      Successfully uninstalled python-dateutil-1.5
Successfully installed influxdb-2.12.0 python-dateutil-2.5.3
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

In [2]:
import pandas as pd
import charts

from opengrid.library import houseprint
from influxdb import DataFrameClient


Server running in the folder /usr/local/opengrid/notebooks/WIP at 127.0.0.1:35117

In [3]:
indbclient = DataFrameClient(host='influxdb')

In [60]:
indbclient.drop_database('opengrid')
indbclient.create_database('opengrid')

In [5]:
hp = houseprint.Houseprint()
hp.sync_tmpos()


Opening connection to Houseprint sheet
Opening spreadsheets
Parsing spreadsheets
24 Sites created
24 Devices created
75 sensors created
Houseprint parsing complete
Using tmpo database from /data/.tmpo/tmpo.sqlite3

Pump all tmpo data to influxdb


In [63]:
for tpe in [#'electricity', 
            'water',
            'gas']:
    df = hp.get_data(sensortype=tpe, diff=False, resample='raw')
    for col in df:
        print("Writing data for {}, sensor {}".format(tpe, col))
        try:
            indbclient.write_points(dataframe=df[[col]].dropna(),
                                measurement=tpe,
                                database='opengrid')
        except:
            print('   Upload to influxdb failed')


Writing data for water, sensor 661684d35cd07dfc657a60b844860785
Writing data for water, sensor b28509eb97137e723995838c393d49df
Writing data for water, sensor 36dac700f81c6be9ca233e6534182ed4
Writing data for water, sensor 2923b75daf93e539e37ce5177c0008c5
Writing data for water, sensor a8054ca7865584e69a7c946aeea1e13c
Writing data for water, sensor 4e63e6b6b5bdb137658d84a861593e5c
Writing data for water, sensor 1a5c55ec176d24483572aca420ac59b5
Writing data for water, sensor c57f45b21e2ec21ab427f172478e1ca4
Writing data for water, sensor a926bc966f178fc5d507a569a5bfc3d7
Writing data for water, sensor a519ce042e8748570d3a91cfca16a747
Writing data for gas, sensor a6028457292f1866b468b63adcc82ee3
Writing data for gas, sensor 29ba25498c657eda0cdd34bf22d3f9e4
Writing data for gas, sensor 313b78fec4f845be91c328ee2f92c6d4
Writing data for gas, sensor 051a928dd04ca55e0411bd6f07e05c04
Writing data for gas, sensor d4b28740c7ee7a98f94a4d23d794af79
Writing data for gas, sensor 4fed53d0c00bbffc26b23366441578f7
Writing data for gas, sensor 1e1e43f5edb4d5e43ab721c391410cde
Writing data for gas, sensor 185ce03278a9ad44c6cab22869d26e11
Writing data for gas, sensor 0a556d75eeb1b5783d7ebfab2e309cb7
Writing data for gas, sensor d5a747b86224834f745f4c9775d70241
Writing data for gas, sensor 3d75bed61ffb84766a84c107921e32b0
Writing data for gas, sensor 1fd6e92d38ccc7ea87793119b7888eec
Writing data for gas, sensor ba14f92064072f9783baf27f45a147b4
Writing data for gas, sensor 212ce724e124fbde0fb649396375d099
Writing data for gas, sensor a5c7faf24bf77ccb13f59e5d87731524
Writing data for gas, sensor 2e2e42c92da420eba8fb9aeea78376fc
Writing data for gas, sensor 62597ad3e7b8a543db2a574bf5d4685b
Writing data for gas, sensor 33e3e6e333de61d67a40e564baad101b
Writing data for gas, sensor d23028c01bea77170d8d8ee6254a5a2c
Writing data for gas, sensor 29259ec6ef1a8dd2a2af711e86c765aa

In [11]:
df.info()


<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 2107597 entries, 2014-10-10 21:27:52+00:00 to 2016-06-05 19:47:28+00:00
Data columns (total 10 columns):
661684d35cd07dfc657a60b844860785    float64
b28509eb97137e723995838c393d49df    float64
36dac700f81c6be9ca233e6534182ed4    float64
2923b75daf93e539e37ce5177c0008c5    float64
a8054ca7865584e69a7c946aeea1e13c    float64
4e63e6b6b5bdb137658d84a861593e5c    float64
1a5c55ec176d24483572aca420ac59b5    float64
c57f45b21e2ec21ab427f172478e1ca4    float64
a926bc966f178fc5d507a569a5bfc3d7    float64
a519ce042e8748570d3a91cfca16a747    float64
dtypes: float64(10)

Time the querying of data


In [69]:
head_str = "2016-01-01 00:00:00"
head = pd.Timestamp(head_str)
tpe = 'gas'

Influxdb


In [70]:
%%timeit
df = indbclient.query("SELECT * from {} where time > '{}'".format(tpe, head_str), database='opengrid')[tpe]


1 loop, best of 3: 22.5 s per loop

In [71]:
df = indbclient.query("SELECT * from {} where time > '{}'".format(tpe, head_str), database='opengrid')[tpe]
df.info()


<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1220930 entries, 2016-01-01 00:02:49+00:00 to 2016-06-05 19:28:14+00:00
Data columns (total 13 columns):
1e1e43f5edb4d5e43ab721c391410cde    162266 non-null float64
212ce724e124fbde0fb649396375d099    184086 non-null float64
29259ec6ef1a8dd2a2af711e86c765aa    9158 non-null float64
2e2e42c92da420eba8fb9aeea78376fc    6 non-null float64
313b78fec4f845be91c328ee2f92c6d4    191608 non-null float64
4fed53d0c00bbffc26b23366441578f7    260924 non-null float64
62597ad3e7b8a543db2a574bf5d4685b    8336 non-null float64
a5c7faf24bf77ccb13f59e5d87731524    63880 non-null float64
a6028457292f1866b468b63adcc82ee3    190994 non-null float64
ba14f92064072f9783baf27f45a147b4    73392 non-null float64
d23028c01bea77170d8d8ee6254a5a2c    9570 non-null float64
d4b28740c7ee7a98f94a4d23d794af79    7396 non-null float64
d5a747b86224834f745f4c9775d70241    102464 non-null float64
dtypes: float64(13)

tmpo


In [72]:
%%timeit
df = hp.get_data(sensortype=tpe, head=head, diff=False, resample='raw')


1 loop, best of 3: 13.2 s per loop

In [73]:
df = hp.get_data(sensortype=tpe, head=head, diff=False, resample='raw')
df.info()


<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 610465 entries, 2016-01-01 00:02:49+00:00 to 2016-06-05 19:28:14+00:00
Data columns (total 20 columns):
a6028457292f1866b468b63adcc82ee3    95497 non-null float64
29ba25498c657eda0cdd34bf22d3f9e4    0 non-null float64
313b78fec4f845be91c328ee2f92c6d4    95804 non-null float64
051a928dd04ca55e0411bd6f07e05c04    0 non-null float64
d4b28740c7ee7a98f94a4d23d794af79    3698 non-null float64
4fed53d0c00bbffc26b23366441578f7    130462 non-null float64
1e1e43f5edb4d5e43ab721c391410cde    81133 non-null float64
185ce03278a9ad44c6cab22869d26e11    0 non-null float64
0a556d75eeb1b5783d7ebfab2e309cb7    0 non-null float64
d5a747b86224834f745f4c9775d70241    51232 non-null float64
3d75bed61ffb84766a84c107921e32b0    0 non-null float64
1fd6e92d38ccc7ea87793119b7888eec    0 non-null float64
ba14f92064072f9783baf27f45a147b4    36696 non-null float64
212ce724e124fbde0fb649396375d099    92043 non-null float64
a5c7faf24bf77ccb13f59e5d87731524    31940 non-null float64
2e2e42c92da420eba8fb9aeea78376fc    3 non-null float64
62597ad3e7b8a543db2a574bf5d4685b    4168 non-null float64
33e3e6e333de61d67a40e564baad101b    0 non-null float64
d23028c01bea77170d8d8ee6254a5a2c    4785 non-null float64
29259ec6ef1a8dd2a2af711e86c765aa    4579 non-null float64
dtypes: float64(20)

Conclusion

tmpo seems MORE efficient for large queries!!


In [ ]: