

In [2]:

    
%autosave 10









    














    



Autosaving every 10 seconds

Source links

Pivotal contributions

PyMADLib - Python Wrapper for MADlib
- MADlib is massively scalable machine learning platform based on PostgreSQL

PL/Python

You have a massively-parallel database.
Want to run Python, e.g. nltk, scikit-learn, directly on database.
pandas, matplotlib too, all inside database. Don't need to bring data back.
Bringing the code to the data, not vice versa.

How to scale

Could have 20-100 PostgreSQL instances, and one master distributor/gatherer of query/results
- This is Greenplum DB. http://en.wikipedia.org/wiki/Greenplum
  - Free version: http://www.ndm.net/datawarehouse/Greenplum/download-greenplum-database
PL/Python allows user-defined functions in Python, not SQL.

!!AI so if this is PL/Python on the database side, could you use pandas pandas.read_sql to execute it then return results? I guess so.

But not really the point. Really want to do everything on DB and just return a very small result.

How to deploy

Each database node needs a full Python stack deployed.
- Maintain using your favourite DevOps technique.
- Pivotal use something Fabric-like.

Distribution large amounts of data

Since you want to join or use data across tables, each maybe on different nodes, it's important to figure out how you split data depending on how you wantt o subsequently join it.

If you split by customer name, but then need it by order ID, always hitting multiple nodes.

If instead you split by customer ID, and all orders for that customer tend to live in same servers, avoid multi-node hits.

So when they go to customers ask "how do you use data? what queries do you need?" in order to speed up joins.

This "sharding" is explicit. Use "DISTRIBUTED BY (name)" to use hash of name as shard key.
Be careful that the cardinality of the output of the hash function is large enough to use all the nodes in your cluster.
When you run future SELECTs grouped by your shard key you transparently achieve parallelisation, because data lives on different nodes.

Applying parallelisation to machine learning

Just run your e.g. scipy.stats.linregress on your data, importantly GROUP BY the shard key.
Assuming correct cardinality and groups of data by hash of shard key your machine learning is naturally parallelisation.

Questions

Can PostgreSQL query planner take data moving costs into account?
- No.
- !!AI I think Hadoop vis a vis HDFS does.
What about scalability of master, does it scale to concurrent usage?
- Not really. But it's much better than moving data to your laptop.
!!AI This is an interesting contrast to Hive, which is using the Hive Query Langauage (HQL) to use SQL-like syntax to execute MapReduce jobs on Hadoop. This talk is about staying inside the SQL world yet trying to scale.
What about debugging errors, do you get tracebacks?
- There might be a way of getting stack traces, but not a good story yet.



In [ ]: