This notebook takes the Bitcoin Developers Reference, specifically the Bitcoin API, and demonstrates the use of the Remote Procedure Calls (RPC) via python. We employ the regression testing mode of bitcoin, regtest, to start with a clean blockchain.
The notebook orders the RPC methods to illustrate how to use the API. In the Bitcoin Developers Reference the RPC methods are in alphabetical order while the python example code below is in discovery sequence, starting with creating the genesis block, mining our first bitcoins, sending some to another user and showing updates to the blockchain throughout the process.
The notebook may be useful for beginners looking for a simple python interface to the Bitcoin API; we use the python bitcoin module created by Peter Todd for its simplicity. As well, the value to the user who is new to python is a simple step by step walk thru of the various methods in the API in an incremental introduction without the confusion of thousands of transactions in a typical block on mainnet.
We create two users on the regression test network (regtest):
We create independent peers on the same network.
Handy shell command to see current nodes:
ps aux |grep bitcoin
We could add as many nodes to regtest as we like (upper limit?) and connect them in a round-robin way so that node 1 listens to node 2... and node N listens to node 1. For our examples here, we use just two nodes in the dictionary connects.
The python script below is designed for Linux (Ubuntu) and would need to be modified for other platforms.
In [1]:
#!/bin/bash
#regtest_start_network.sh
import os
import shutil
#os.system("killall --regex bitcoin.*")
idir = os.environ['HOME']+'/regtest'
if os.path.isdir(idir):
shutil.rmtree(idir)
os.mkdir(idir)
connects = {'17591' : '17592', '17592' : '17591'}
for port in connects.keys():
adir = idir+'/'+port
os.mkdir(adir)
args = " -server -txindex=1 -listen -port=" + port + \
" -rpcuser=bitcoinrpc -rpcpassword=P0 -rpcport="+\
str(int(port)-1000) +\
" -datadir=" + adir + " -connect=localhost:" + connects[port] +\
" -regtest -pid="+port+".pid -daemon -debug"
os.system("bitcoind" + args)
Import the python bitcoin module created by Peter Todd.
In [2]:
import bitcoin
import bitcoin.rpc
import bitcoin.core
import bitcoin.core.script
We hard-code the username and passwords here for pedagogical ease. We display the blockchain information using the widely used, but deprecated getinfo() RPC.
getinfo() returns:
version - The version number of this bitcoin-qt or bitcoind program itself. Both of are equivalent. -qt is simply the graphical user interface version
protocolversion: The version of the bitcoin network protocol supported by this client (user agent software).
walletversion: The version of the wallet.dat file. Wallet.dat contains bitcoin addresses and public & private key pairs for these addresses. There is additional data on the wallet. Care must be taken to not restore from an old wallet backup. New addresses generated in the wallet since the old backup was made will not exist in the old backup! Source
balance: The total number of bitcoins held in the wallet.dat file.
blocks: The total number of blocks which constitute the shared block chain.
timeoffset: Seconds of difference between this node's "wall clock time" and the median reported by our network peers.
connections: the number of peers on the bitcoin P2P network that this node is connected to.
proxy: If using a proxy to connect to the network, listed here, otherwise blank.
difficulty: the current mining difficulty factor. Difficulty is increased as more miners and more hash compute power compete to be the next one to have a block of transactions added to the blockchain.
testnet: Boolean value (true OR false). There is a parallel bitcoin network, the testnet, where trials and experiments may be carried out without impacting the official, live bitcoin P2P network
keypoololdest: timestamp (UNIX epoch) of the oldest key in the keypool
keypoolsize: A number of addresses are kept in reserve by the client. This is the size of that reserve.
paytxfee: Specifies what fee the client is willing to pay to expedite transactions. Miners may choose to ignore transactions that do not pay a fee, and these fee-less transactions will have low priority on queue of pending transaction and may linger there for hours.
errors: This field may inform of different status conditions. Full list of error codes in source file bitcoinrpc.h (Examples: "Bitcoin not connected", "database error", "Keypool ran out"...)
In [3]:
bitcoin.SelectParams('regtest')
Bob = bitcoin.rpc.RawProxy("http://bitcoinrpc:P0@127.0.0.1:16591")
Mary = bitcoin.rpc.RawProxy("http://bitcoinrpc:P0@127.0.0.1:16592")
info = Bob.getinfo()
for key in info.keys():
print key + ' : ' + str(info[key])
The following RCPs are used to determine the network attributes. We highlight the informational RPCs and leave the AddNode and GetAddedNodeInfo for later investigation.
In [4]:
getnetworkinfo = Bob.getnetworkinfo()
print '\ngetnetworkinfo\n'
print getnetworkinfo
getpeerinfo = Bob.getpeerinfo()
print '\ngetpeerinfo\n'
print getpeerinfo
getconnectioncount = Bob.getconnectioncount()
print '\ngetconnectioncount\n'
print getconnectioncount
getnettotals = Bob.getnettotals()
print '\ngetnettotals\n'
print getnettotals
In [5]:
blockchaininfo = Bob.getblockchaininfo()
getblockcount = Bob.getblockcount()
getbestblockhash = Bob.getbestblockhash()
getdifficulty = Bob.getdifficulty()
getchaintips = Bob.getchaintips()
getmempoolinfo = Bob.getmempoolinfo()
print '\nblockchaininfo\n'
print blockchaininfo
print '\ngetblockcount ' + str(getblockcount)
print '\ngetbestblockhash\n'
print getbestblockhash
print '\ngetdifficulty ' + str(getdifficulty)
print '\ngetchaintips\n'
print getchaintips
print '\ngetmempoolinfo\n'
print getmempoolinfo
print '\n\n'
bestblockhash = blockchaininfo['bestblockhash']
blocks = blockchaininfo['blocks']
print '\nblocks = ' + str(blocks)
print '\nbestblockhash = ' + str(bestblockhash)
In [6]:
## N.B. our balance is zero in the genesis block
print 'Initial balance, before any mining ' + str(Bob.getbalance())
Bob.generate(101)
print 'Balance after mining 101 blocks ' + str(Bob.getbalance())
Now let's look at the block structure, rerunning code from above. Note that the block height is now 101. There is one transaction for 50BTC. Note that this is a coinbase transaction with just a vout and no vin.
How to make serialized transaction?
In [7]:
getblockcount = Bob.getblockcount()
print '\ngetblockcount = ' + str(getblockcount)
getblockhash = Bob.getblockhash(getblockcount)
print '\ngetblockhash = ' + str(getblockhash)
print '\ngetblock\n'
getblock = Bob.getblock(getblockhash)
print getblock
tx = getblock['tx']
print '\n' + str(len(tx)) + ' Transactions\n'
print tx
for i in range(len(tx)):
print '\nSerialized Transaction #' + str(i) +'\n'
serializedTX = Bob.getrawtransaction(tx[i],0)
print serializedTX
print '\nRaw Transaction\n'
rawTX = Bob.getrawtransaction(tx[i],1)
print rawTX
print '\nDecoded Transaction\n'
decodedTX = Bob.decoderawtransaction(serializedTX)
print decodedTX
Now that we have a raw transaction, let's look at the details. As noted, the first 100 blocks in regtest are blank, necessary for mining our first coinbase reward of 50BTC. This shows in block 101 with a single transaction, denoted rawTX below. rawTX is a JSON object with:
We veirfy that the serialized transaction hex is the same as the "hex" entry in the transaction. We note that the txid is repeated in the raw transaction. We also note that the confirmations = 1 (no further mining has been done on this block), that there is a single transaction (the mining of 50btc) and the blocktime and time are the same, while the locktime is the beginning of the epoch.
Note that an address has been created, in general an array of addresses, each corresponding to a transaction output (vout). Here, a new address has been created corresponding to the miner's address for the 50BTC reward.
In [8]:
from datetime import datetime as dt
#import pytz
print rawTX
print '\n\n'
print 'blockhash = ' + str(rawTX['blockhash']) + '\n'
for i in range(len(rawTX['vout'])):
spk = rawTX['vout'][i]['scriptPubKey']
print 'vout ' + str(i) + ' : ' + str(spk) + '\n'
for field in spk.keys():
#['reqSigs','hex','addresses','asm','type']:
print 'vout ' + str(i) + ' ' + field + ' : ' + str(spk[field])
print 'vout ' + str(i) + ' value : ' + str(rawTX['vout'][i]['value'])
print 'vout ' + str(i) + ' n : ' + str(rawTX['vout'][i]['n'])
print '\nserialized hex = ' + str(rawTX['hex'])
print 'Is serialized hex == rawTX["hex"]? ' + str(rawTX['hex']==serializedTX) + '\n'
for i in range(len(rawTX['vin'])):
spk = rawTX['vin'][i]
print 'vin ' + str(i) + ' : ' + str(spk) + '\n'
for field in spk.keys():
#['reqSigs','hex','addresses','asm','type']:
print 'vin ' + str(i) + ' ' + field + ' : ' + str(spk[field])
print '\n'
for field in ['txid','blocktime','version','confirmations','time','locktime']:
if field in ['blocktime','time','locktime']:
print field + ' = ' + str(rawTX[field]) +\
' ' + dt.fromtimestamp(rawTX[field]).strftime('%Y-%m-%d:%H%M%S')
else:
print field + ' = ' + str(rawTX[field])
Let's create an address and send some bitcoins there. Some things to notice:
In [12]:
print 'Mary\'s balance = ' + str(Mary.getbalance())
#print 'Mary\'s peers: '
#print Mary.getpeerinfo()
getnewaddress = Mary.getnewaddress()
print '\nNew address ' + str(getnewaddress)
print '\nMary\'s address has received how many BTC? ' +\
str(Mary.getreceivedbyaddress(getnewaddress,0))
##have Bob (proxy) send 25 bitcoins to Mary
txid = Bob.sendtoaddress(getnewaddress,25)
In [13]:
getmempoolinfo = Bob.getmempoolinfo()
getrawmempool = Bob.getrawmempool(True)
print '\ngetmempoolinfo ' + str(getmempoolinfo)
print '\ngetrawmempool'
print getrawmempool
print '\n'
for key in getrawmempool.keys():
for field in getrawmempool[key].keys():
print str(field) + ' : ' + str(getrawmempool[key][field])
#print '\ntxid from sendtoaddress output ' + str(txid)
print '\nIs the send txid the same as memory pool txid? ****' +\
str(txid == getrawmempool.keys()[0]) + '****'
print '\nMary\'s balance before mining = ' + str(Mary.getbalance())
print 'Bob\'s balance before mining = ' + str(Bob.getbalance())
##how can I see transaction details before mining?
print '\nMemory Pool Raw Transaction Data\n'
import pprint
pprint.pprint(Bob.getrawtransaction(txid,1))
##N.B. no transaction on the blockchain yet!!!
Bob mines 6 more blocks, after the fifth, the 25BTC sent to Mary is confirmed and shows in her balance. Note that the transaction created in block 101, when Bob sent Mary 25BTC shows up in the first block mined, but the balance isn't updated until some number of blocks have been processed.
In [14]:
for i in range(7):
Bob.generate(1)
getblockcount = Bob.getblockcount()
getblockhash = Bob.getblockhash(getblockcount)
getblock = Bob.getblock(getblockhash)
print 'Block #' + str(getblockcount) + ' Mary\'s balance ' + str(Mary.getbalance())
print 'txids ' + str(getblock['tx'])
In [15]:
print ' Mary\'s balance ' + str(Mary.getbalance())
Note: the wallet RPCs are only available if Bitcoin Core was built with wallet support, which is the default.
In [16]:
print '\nBob\'s Wallet\n'
wallet= Bob.getwalletinfo()
for key in wallet.keys():
print key + '\t' + str(wallet[key])
print '\nMary\'s Wallet\n'
wallet= Mary.getwalletinfo()
for key in wallet.keys():
print key + '\t' + str(wallet[key])
In [17]:
import time
print '\nMary has ' + str(len(Mary.listtransactions())) + ' transactions from Bob\'s largesse'
print '\nMary\'s first address has received how many BTC? ' +\
str(Mary.getreceivedbyaddress( Mary.listtransactions()[0]['address'],0))
#str(Mary.getreceivedbyaddress('mmT3ER6w98jZAKwtTZrr3DSxrchS7fGxKW',0))
print '\nBob has ' + str(len(Bob.listtransactions())) + ' transactions from all that mining'
#print Bob.listtransactions()
##let's send Mary some more of Bob's bitcoins so we can see her unconfirmed balance
getnewaddress = Mary.getnewaddress()
print '\nNew address ' + str(getnewaddress)
##have Bob (proxy) send 0.5 bitcoins to Mary
txid = Bob.sendtoaddress(getnewaddress,0.5)
time.sleep(2)
print '\nMary\'s unconfirmed balance ' + str(Mary.getunconfirmedbalance())
print '\nMary\'s confirmed balance ' + str(Mary.getbalance())
##let's mine 6 blocks
Bob.generate(6)
time.sleep(2)
print 'After Bob\'s mining'
print '\nMary\'s unconfirmed balance ' + str(Mary.getunconfirmedbalance())
print '\nMary\'s confirmed balance ' + str(Mary.getbalance())
Permitting receiving and spending of satoshis is the only essential feature of wallet software—but a particular wallet program doesn’t need to do both things. Two wallet programs can work together, one program distributing public keys in order to receive satoshis and another program signing transactions spending those satoshis.
Wallet programs also need to interact with the peer-to-peer network to get information from the block chain and to broadcast new transactions. However, the programs which distribute public keys or sign transactions don’t need to interact with the peer-to-peer network themselves.
This leaves us with three necessary, but separable, parts of a wallet system: a public key distribution program, a signing program, and a networked program. In the subsections below, we will describe common combinations of these parts.
The simplest wallet is a program which performs all three functions: it generates private keys, derives the corresponding public keys, helps distribute those public keys as necessary, monitors for outputs spent to those public keys, creates and signs transactions spending those outputs, and broadcasts the signed transactions.
Full-Service Wallets
As of this writing, almost all popular wallets can be used as full-service wallets.
The main advantage of full-service wallets is that they are easy to use. A single program does everything the user needs to receive and spend satoshis.
The main disadvantage of full-service wallets is that they store the private keys on a device connected to the Internet. The compromise of such devices is a common occurrence, and an Internet connection makes it easy to transmit private keys from a compromised device to an attacker.
In [ ]:
In [ ]:
This code determines whether a transaction is included in a Merkle Tree. From Luke Dashjr
In [ ]:
# Eloipool - Python Bitcoin pool server
# Copyright (C) 2011-2012 Luke Dashjr <luke-jr+eloipool@utopios.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# original code https://github.com/Crypto-Expert/stratum-mining/blob/master/lib/merkletree.py
# http://code.runnable.com/U3jqtyYUmAUxtsSS/bitcoin-block-merkle-root-python
from hashlib import sha256
from util import doublesha
class MerkleTree:
def __init__(self, data, detailed=False):
self.data = data
self.recalculate(detailed)
self._hash_steps = None
def recalculate(self, detailed=False):
L = self.data
steps = []
if detailed:
detail = []
PreL = []
StartL = 0
else:
detail = None
PreL = [None]
StartL = 2
Ll = len(L)
if detailed or Ll > 1:
while True:
if detailed:
detail += L
if Ll == 1:
break
steps.append(L[1])
if Ll % 2:
L += [L[-1]]
L = PreL + [doublesha(L[i] + L[i + 1]) for i in range(StartL, Ll, 2)]
Ll = len(L)
self._steps = steps
self.detail = detail
def hash_steps(self):
if self._hash_steps == None:
self._hash_steps = doublesha(''.join(self._steps))
return self._hash_steps
def withFirst(self, f):
steps = self._steps
for s in steps:
f = doublesha(f + s)
return f
def merkleRoot(self):
return self.withFirst(self.data[0])
# MerkleTree tests
def _test():
import binascii
import time
mt = MerkleTree([None] + [binascii.unhexlify(a) for a in [
'999d2c8bb6bda0bf784d9ebeb631d711dbbbfe1bc006ea13d6ad0d6a2649a971',
'3f92594d5a3d7b4df29d7dd7c46a0dac39a96e751ba0fc9bab5435ea5e22a19d',
'a5633f03855f541d8e60a6340fc491d49709dc821f3acb571956a856637adcb6',
'28d97c850eaf917a4c76c02474b05b70a197eaefb468d21c22ed110afe8ec9e0',
]])
assert(
b'82293f182d5db07d08acf334a5a907012bbb9990851557ac0ec028116081bd5a' ==
binascii.b2a_hex(mt.withFirst(binascii.unhexlify('d43b669fb42cfa84695b844c0402d410213faa4f3e66cb7248f688ff19d5e5f7')))
)
print '82293f182d5db07d08acf334a5a907012bbb9990851557ac0ec028116081bd5a'
txes = [binascii.unhexlify(a) for a in [
'd43b669fb42cfa84695b844c0402d410213faa4f3e66cb7248f688ff19d5e5f7',
'999d2c8bb6bda0bf784d9ebeb631d711dbbbfe1bc006ea13d6ad0d6a2649a971',
'3f92594d5a3d7b4df29d7dd7c46a0dac39a96e751ba0fc9bab5435ea5e22a19d',
'a5633f03855f541d8e60a6340fc491d49709dc821f3acb571956a856637adcb6',
'28d97c850eaf917a4c76c02474b05b70a197eaefb468d21c22ed110afe8ec9e0',
]]
s = time.time()
mt = MerkleTree(txes)
for x in range(100):
y = int('d43b669fb42cfa84695b844c0402d410213faa4f3e66cb7248f688ff19d5e5f7', 16)
#y += x
coinbasehash = binascii.unhexlify("%x" % y)
x = binascii.b2a_hex(mt.withFirst(coinbasehash))
print x
print time.time() - s
if __name__ == '__main__':
_test()
This code snippet illustrates a simplified proof-of-work algorithm not used by miners; by incementing the nonce and trying difficulties from 1-31 bits (2 - 2^32). Note that this runs for a long time, more than 20 minutes on a 4GB-RAM Ubuntu box.
From Mastering Bitcoin
In [ ]:
#!/usr/bin/env python
# example of proof-of-work algorithm
import hashlib
import time
max_nonce = 2 ** 32 # 4 billion
def proof_of_work(header, difficulty_bits):
# calculate the difficulty target
target = 2 ** (256-difficulty_bits)
for nonce in xrange(max_nonce):
hash_result = hashlib.sha256(str(header)+str(nonce)).hexdigest()
# check if this is a valid result, below the target
if long(hash_result, 16) < target:
print "Success with nonce %d" % nonce
print "Hash is %s" % hash_result
return (hash_result,nonce)
print "Failed after %d (max_nonce) tries" % nonce
return nonce
if __name__ == '__main__':
nonce = 0
hash_result = ''
# difficulty from 0 to 31 bits
for difficulty_bits in xrange(32):
difficulty = 2 ** difficulty_bits
print "Difficulty: %ld (%d bits)" % (difficulty, difficulty_bits)
print "Starting search..."
# checkpoint the current time
start_time = time.time()
# make a new block which includes the hash from the previous block
# we fake a block of transactions - just a string
new_block = 'test block with transactions' + hash_result
# find a valid nonce for the new block
(hash_result, nonce) = proof_of_work(new_block, difficulty_bits)
# checkpoint how long it took to find a result
end_time = time.time()
elapsed_time = end_time - start_time
print "Elapsed Time: %.4f seconds" % elapsed_time
if elapsed_time > 0:
# estimate the hashes per second
hash_power = float(long(nonce)/elapsed_time)
print "Hashing Power: %ld hashes per second" % hash_power
In [ ]: