In the last section, we learned about PointerTensors, which create the underlying infrastructure we need for privacy preserving Deep Learning. In this section, we're going to see how to use these basic tools to implement our first privacy preserving deep learning algorithm, Federated Learning.
Authors:
It's a simple, powerful way to train Deep Learning models. If you think about training data, it's always the result of some sort of collection process. People (via devices) generate data by recording events in the real world. Normally, this data is aggregated to a single, central location so that you can train a machine learning model. Federated Learning turns this on its head!
Instead of bringing training data to the model (a central server), you bring the model to the training data (wherever it may live).
The idea is that this allows whoever is creating the data to own the only permanent copy, and thus maintain control over who ever has access to it. Pretty cool, eh?
Let's start by training a toy model the centralized way. This is about a simple as models get. We first need:
Note: If this API is un-familiar to you - head on over to fast.ai and take their course before continuing in this tutorial.
In [ ]:
import torch
from torch import nn
from torch import optim
In [ ]:
# A Toy Dataset
data = torch.tensor([[0,0],[0,1],[1,0],[1,1.]])
target = torch.tensor([[0],[0],[1],[1.]])
# A Toy Model
model = nn.Linear(2,1)
def train():
# Training Logic
opt = optim.SGD(params=model.parameters(),lr=0.1)
for iter in range(20):
# 1) erase previous gradients (if they exist)
opt.zero_grad()
# 2) make a prediction
pred = model(data)
# 3) calculate how much we missed
loss = ((pred - target)**2).sum()
# 4) figure out which weights caused us to miss
loss.backward()
# 5) change those weights
opt.step()
# 6) print our progress
print(loss.data)
In [ ]:
train()
And there you have it! We've trained a basic model in the conventional manner. All our data is aggregated into our local machine and we can use it to make updates to our model. Federated Learning, however, doesn't work this way. So, let's modify this example to do it the Federated Learning way!
So, what do we need:
updated training logic to do federated learning
New Training Steps:
In [ ]:
import syft as sy
hook = sy.TorchHook(torch)
In [ ]:
# create a couple workers
bob = sy.VirtualWorker(hook, id="bob")
alice = sy.VirtualWorker(hook, id="alice")
In [ ]:
# A Toy Dataset
data = torch.tensor([[0,0],[0,1],[1,0],[1,1.]], requires_grad=True)
target = torch.tensor([[0],[0],[1],[1.]], requires_grad=True)
# get pointers to training data on each worker by
# sending some training data to bob and alice
data_bob = data[0:2]
target_bob = target[0:2]
data_alice = data[2:]
target_alice = target[2:]
# Iniitalize A Toy Model
model = nn.Linear(2,1)
data_bob = data_bob.send(bob)
data_alice = data_alice.send(alice)
target_bob = target_bob.send(bob)
target_alice = target_alice.send(alice)
# organize pointers into a list
datasets = [(data_bob,target_bob),(data_alice,target_alice)]
In [ ]:
from syft.federated.floptimizer import Optims
workers = ['bob', 'alice']
optims = Optims(workers, optim=optim.Adam(params=model.parameters(),lr=0.1))
In [ ]:
def train():
# Training Logic
for iter in range(10):
# NEW) iterate through each worker's dataset
for data,target in datasets:
# NEW) send model to correct worker
model.send(data.location)
#Call the optimizer for the worker using get_optim
opt = optims.get_optim(data.location.id)
#print(data.location.id)
# 1) erase previous gradients (if they exist)
opt.zero_grad()
# 2) make a prediction
pred = model(data)
# 3) calculate how much we missed
loss = ((pred - target)**2).sum()
# 4) figure out which weights caused us to miss
loss.backward()
# 5) change those weights
opt.step()
# NEW) get model (with gradients)
model.get()
# 6) print our progress
print(loss.get()) # NEW) slight edit... need to call .get() on loss\
# federated averaging
In [ ]:
train()
And voilà! We now are training a very simple Deep Learning model using Federated Learning! We send the model to each worker, generate a new gradient, and then bring the gradient back to our local server where we update our global model. Never in this process do we ever see or request access to the underlying training data! We preserve the privacy of Bob and Alice!!!
So, while this example is a nice introduction to Federated Learning, it still has some major shortcomings. Most notably, when we call model.get() and receive the updated model from Bob or Alice, we can actually learn a lot about Bob and Alice's training data by looking at their gradients. In some cases, we can restore their training data perfectly!
So, what is there to do? Well, the first strategy people employ is to average the gradient across multiple individuals before uploading it to the central server. This strategy, however, will require some more sophisticated use of PointerTensor objects. So, in the next section, we're going to take some time to learn about more advanced pointer functionality and then we'll upgrade this Federated Learning example
Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement toward privacy preserving, decentralized ownership of AI and the AI supply chain (data), you can do so in the following ways!
The easiest way to help our community is just by starring the Repos! This helps raise awareness of the cool tools we're building.
The best way to keep up to date on the latest advancements is to join our community! You can do so by filling out the form at http://slack.openmined.org
The best way to contribute to our community is to become a code contributor! At any time you can go to PySyft GitHub Issues page and filter for "Projects". This will show you all the top level Tickets giving an overview of what projects you can join! If you don't want to join a project, but you would like to do a bit of coding, you can also look for more "one off" mini-projects by searching for GitHub issues marked "good first issue".
If you don't have time to contribute to our codebase, but would still like to lend support, you can also become a Backer on our Open Collective. All donations go toward our web hosting and other community expenses such as hackathons and meetups!
In [ ]:
In [ ]: