Recap: For Part 2 of this tutorial, we use simple version of Federated Learning to train a model. This step require sey to see gradient each data owner must to trust model owner.
Description: For this tutorial, we go show you hiw you fit use the advanced aggregation tools wey dey Part 3 to allow weights to dey aggregated by trusted "secure worker" before we go send final resulting model to the model owner (wey be us).
For this way wey we dey talk so, only secure go see who get the weights and where he come from. We go fit tell the part of the model wey we change, the problem na we no know which worker (Bob or Alice) change am, wey come create layer of privacy.
Person wey write am:
Person wey translate am:
In [ ]:
import torch
import syft as sy
import copy
hook = sy.TorchHook(torch)
from torch import nn, optim
First tin wey we go do na to create two data owners (Bob and Alice) with small amount of data. We go initialize a secure machine called "secure_worker". In practice we fit say na secure hardware he be (sometin like Intel's SGX) or make we say na trusted intermediary.
In [ ]:
# make we create couple workers
bob = sy.VirtualWorker(hook, id="bob")
alice = sy.VirtualWorker(hook, id="alice")
secure_worker = sy.VirtualWorker(hook, id="secure_worker")
# A Toy Dataset
data = torch.tensor([[0,0],[0,1],[1,0],[1,1.]], requires_grad=True)
target = torch.tensor([[0],[0],[1],[1.]], requires_grad=True)
# we go get pointers to training data on each worker if
# we send some training data to bob and alice
bobs_data = data[0:2].send(bob)
bobs_target = target[0:2].send(bob)
alices_data = data[2:].send(alice)
alices_target = target[2:].send(alice)
In [ ]:
# We go initialize A Toy Model
model = nn.Linear(2,1)
In [ ]:
bobs_model = model.copy().send(bob)
alices_model = model.copy().send(alice)
bobs_opt = optim.SGD(params=bobs_model.parameters(),lr=0.1)
alices_opt = optim.SGD(params=alices_model.parameters(),lr=0.1)
In [ ]:
In [ ]:
for i in range(10):
# We go train Bob's Model
bobs_opt.zero_grad()
bobs_pred = bobs_model(bobs_data)
bobs_loss = ((bobs_pred - bobs_target)**2).sum()
bobs_loss.backward()
bobs_opt.step()
bobs_loss = bobs_loss.get().data
# We go train Alice's Model
alices_opt.zero_grad()
alices_pred = alices_model(alices_data)
alices_loss = ((alices_pred - alices_target)**2).sum()
alices_loss.backward()
alices_opt.step()
alices_loss = alices_loss.get().data
print("Bob:" + str(bobs_loss) + " Alice:" + str(alices_loss))
Now wey every data owner get partially trained model, we go average them in a secure way. We go achieve this by instructing Alice and Bob make dem send their model to the secure (trusted) server.
Look am wella, as we go come use our API, he maen sey we go send each model DIRECTLY to the secure_worker. We never see am.
In [ ]:
alices_model.move(secure_worker)
In [ ]:
bobs_model.move(secure_worker)
Finally, the last step for this training epoch na to average Bob and Alice's trained models togeda and then we go use this to set the values for our global "model".
In [ ]:
with torch.no_grad():
model.weight.set_(((alices_model.weight.data + bobs_model.weight.data) / 2).get())
model.bias.set_(((alices_model.bias.data + bobs_model.bias.data) / 2).get())
In [ ]:
iterations = 10
worker_iters = 5
for a_iter in range(iterations):
bobs_model = model.copy().send(bob)
alices_model = model.copy().send(alice)
bobs_opt = optim.SGD(params=bobs_model.parameters(),lr=0.1)
alices_opt = optim.SGD(params=alices_model.parameters(),lr=0.1)
for wi in range(worker_iters):
# We go train Bob's Model
bobs_opt.zero_grad()
bobs_pred = bobs_model(bobs_data)
bobs_loss = ((bobs_pred - bobs_target)**2).sum()
bobs_loss.backward()
bobs_opt.step()
bobs_loss = bobs_loss.get().data
# We go train Alice's Model
alices_opt.zero_grad()
alices_pred = alices_model(alices_data)
alices_loss = ((alices_pred - alices_target)**2).sum()
alices_loss.backward()
alices_opt.step()
alices_loss = alices_loss.get().data
alices_model.move(secure_worker)
bobs_model.move(secure_worker)
with torch.no_grad():
model.weight.set_(((alices_model.weight.data + bobs_model.weight.data) / 2).get())
model.bias.set_(((alices_model.bias.data + bobs_model.bias.data) / 2).get())
print("Bob:" + str(bobs_loss) + " Alice:" + str(alices_loss))
Last last, we go make sure sey our resulting model learn wella, so we go evaluate am on top test dataset. For this proble, we go use origina data, but in practice we go use new data to understand how well the model generalizes to unseen examples.
In [ ]:
preds = model(data)
loss = ((preds - target) ** 2).sum()
In [ ]:
print(preds)
print(target)
print(loss.data)
For dis toy example, the average model go dey underfitting relative to how plaintext model trained go behave, we got fit train am and we no go expose each worker's training data. We go fit add all the updated models wey we get from each worker we dey trusted aggregator togeda. Dis aggregate go prevent data leakage to the model owner.
For the example wey we go do again, we go try to do our trusted aggregation directly with the gradients, if we do am like dis we go fit update the model with beta gradient estimates and we go arrive at a stronger model.
In [ ]:
Clap for una sef as you don finish this notebook tutorial! If you enjoy am and you wan join the movement towards privacy preserving, decentralized ownership of AI and the AI supply chain (data), follow the steps wey dey below.
The easiset way to helep our community na to star the GitHub repos! This go helep raise awareness of the tools we dey build.
To follow up bumper to bumper on how latest advancements, join our community! You can do so by filling out the form at http://slack.openmined.org
The best way to contribute to our community na to become code contributor! You fit go to PySyft GitHub Issues page and filter for "Projects". E go show you all the top level Tickets giving an overview of what projects you fit join! If you no wan join any project, but you wan code small, you fit look for more "one off" mini-projects by searching for GitHub issues marked "good first issue"
If you no get time to contribute to our codebase, but still like to lend support, you fit be a Backer on our Open Collective. All donations wey we get na for our web hosting and other community expenses such as hackathons and meetups! meetups!
In [ ]: