Basic Ultrastorage: Storage unit

For reproducibility.


In [24]:
import IPython
IPython.__version__


Out[24]:
'3.0.0'

In [25]:
import ultrastorage
ultrastorage.__version__


Out[25]:
'0.0.1'

Creating storage units

Import related modules.


In [26]:
from ultrastorage.storageunit import StorageUnit
from ultrastorage.item import Item

Create a storage unit named myStorageUnit with capacity 100 and 2 CPUs.


In [27]:
storage_unit = StorageUnit(capacity=100, cpu=2, name="myStorageUnit")
print(storage_unit)


(capacity=100,count=0,cpu=2,free_space=100.0,load=0.0,name=myStorageUnit,reserved_memory=0.0,size=0)

Display various data of this storage unit.


In [28]:
# total capacity
print("capacity={}".format(storage_unit.capacity))

# number of items
print("count={}".format(storage_unit.count()))

# number of cpy (i.e., number of parallel operations)
print("cpu={}".format(storage_unit.cpu))

# free space (i.e., capacity minus total size minus set aside)
print("free_space={}".format(storage_unit.free_space()))

# load (i.e., total size divided by capacity)
print("load={}".format(storage_unit.load()))

# name
print("name={}".format(storage_unit.name))

# reserved memory
print("reserved_memory={}".format(storage_unit.reserved_memory))

# total size (i.e., sum of the size of the items)
print("size={}".format(storage_unit.size()))


capacity=100
count=0
cpu=2
free_space=100.0
load=0.0
name=myStorageUnit
reserved_memory=0.0
size=0

In [29]:
storage_unit.is_empty()


Out[29]:
True

The capacity of the storage unit is a mandatory field. The default number of CPUs is 1 and the storage unit is given a name in case it is not given.


In [30]:
print(StorageUnit(100))


(capacity=100,count=0,cpu=1,free_space=100.0,load=0.0,name=StorageUnit-4378231864,reserved_memory=0.0,size=0)

Note that, once created, it is not possible to modify the capacity and the number of CPUs of the storage unit. Also, it is not possible to rename a storage unit.

Adding items


In [31]:
storage_unit = StorageUnit(capacity=100, cpu=2, name="myStorageUnit")
print(storage_unit)


(capacity=100,count=0,cpu=2,free_space=100.0,load=0.0,name=myStorageUnit,reserved_memory=0.0,size=0)

Add 9 items with sizes 1, 2, 3, 4, 5, 6, 7, 8 and 9 to the storage unit.


In [32]:
for i in range(1,10):
    storage_unit.add_item(Item(i))
    print("{} {}".format(i, storage_unit))
    
storage_unit.is_empty()


1 (capacity=100,count=1,cpu=2,free_space=99.0,load=0.01,name=myStorageUnit,reserved_memory=0.0,size=1)
2 (capacity=100,count=2,cpu=2,free_space=97.0,load=0.03,name=myStorageUnit,reserved_memory=0.0,size=3)
3 (capacity=100,count=3,cpu=2,free_space=94.0,load=0.06,name=myStorageUnit,reserved_memory=0.0,size=6)
4 (capacity=100,count=4,cpu=2,free_space=90.0,load=0.1,name=myStorageUnit,reserved_memory=0.0,size=10)
5 (capacity=100,count=5,cpu=2,free_space=85.0,load=0.15,name=myStorageUnit,reserved_memory=0.0,size=15)
6 (capacity=100,count=6,cpu=2,free_space=79.0,load=0.21,name=myStorageUnit,reserved_memory=0.0,size=21)
7 (capacity=100,count=7,cpu=2,free_space=72.0,load=0.28,name=myStorageUnit,reserved_memory=0.0,size=28)
8 (capacity=100,count=8,cpu=2,free_space=64.0,load=0.36,name=myStorageUnit,reserved_memory=0.0,size=36)
9 (capacity=100,count=9,cpu=2,free_space=55.0,load=0.45,name=myStorageUnit,reserved_memory=0.0,size=45)
Out[32]:
False

In [33]:
print("len={}".format(len(storage_unit)))
print("is_empty={}".format(storage_unit.is_empty()))


len=9
is_empty=False

Visiting a storage unit

Visinting a storage unit reduces to visiting the stored items. Notice that a storage unit does visit the items in some arbitrary order.


In [34]:
",".join(str(item) for item in storage_unit)


Out[34]:
'8,1,4,7,3,5,2,6,9'

Some specific iterators.


In [35]:
# items by descending size
",".join(str(item) for item in storage_unit.items_by_descending_size())


Out[35]:
'9,8,7,6,5,4,3,2,1'

In [36]:
# items by ascending size
",".join(str(item) for item in storage_unit.items_by_ascending_size())


Out[36]:
'1,2,3,4,5,6,7,8,9'

One can get randomly (uniform) choosen item.


In [37]:
item = storage_unit.random_item()
print("item={}".format(item))


item=7

Deleting items


In [38]:
storage_unit = StorageUnit(capacity=100, cpu=2, name="myStorageUnit")
print(storage_unit)


(capacity=100,count=0,cpu=2,free_space=100.0,load=0.0,name=myStorageUnit,reserved_memory=0.0,size=0)

One can only delete items using item references and not item sizes. After all it would be silly to try to delete a file giving its size only.


In [39]:
items = [Item(i) for i in range(1,5)]
storage_unit.add_items(items)
",".join(str(item) for item in storage_unit)


Out[39]:
'1,2,4,3'

In [40]:
storage_unit.delete_item(items[2])
",".join(str(item) for item in storage_unit)


Out[40]:
'1,2,4'

Deleting an item that is not stored in the storage unit raises an exception.


In [41]:
storage_unit.delete_item(Item(5))


---------------------------------------------------------------------------
StorageUnitException                      Traceback (most recent call last)
<ipython-input-41-9a9c5b77e731> in <module>()
----> 1 storage_unit.delete_item(Item(5))

/Users/vialette/Documents/git-data/ultrastorage/ultrastorage/storageunit/storageunit.py in delete_item(self, item)
    200 
    201         """
--> 202         self.release_item(item)
    203 
    204     def retain_item(self, item):

/Users/vialette/Documents/git-data/ultrastorage/ultrastorage/storageunit/storageunit.py in release_item(self, item)
    234         """
    235         if item not in self._items:
--> 236             raise StorageUnitException("no such item '{}'".format(item))
    237 
    238         self._items[item].release()

StorageUnitException: no such item '5'

More on adding and deleting items


In [42]:
storage_unit = StorageUnit(capacity=100, cpu=2, name="myStorageUnit")
print(storage_unit)


(capacity=100,count=0,cpu=2,free_space=100.0,load=0.0,name=myStorageUnit,reserved_memory=0.0,size=0)

A storage unit is not a multi set, you cannot add the same item twice.


In [43]:
item = Item(1)
storage_unit.add_item(item)
storage_unit.add_item(item)


---------------------------------------------------------------------------
StorageUnitException                      Traceback (most recent call last)
<ipython-input-43-5c3961838cc4> in <module>()
      1 item = Item(1)
      2 storage_unit.add_item(item)
----> 3 storage_unit.add_item(item)

/Users/vialette/Documents/git-data/ultrastorage/ultrastorage/storageunit/storageunit.py in add_item(self, item)
    257 
    258         if item in self._items:
--> 259             raise StorageUnitException("overwrite item '{}'".format(item))
    260 
    261         self._items[item] = StorageUnit.ItemAttributes()

StorageUnitException: overwrite item '1'

But, of course, distinct items with the same size can be added.


In [44]:
storage_unit.add_item(Item(2))
storage_unit.add_item(Item(2))
print(",".join(str(item) for item in storage_unit))


2,2,1

For convenience, you can add several items with a single function.


In [45]:
items = [Item(i) for i in range(2,6)]
storage_unit.add_items(items)
print(storage_unit)
print(",".join(str(item) for item in storage_unit))


(capacity=100,count=7,cpu=2,free_space=81.0,load=0.19,name=myStorageUnit,reserved_memory=0.0,size=19)
2,3,2,2,5,4,1

Items are managed using a reference count like strategy. The storage unit maintains a reference count for each item, allowing the user to retain the item (i.e., increment the associated reference count by 1) and to release the item (i.e., decremenent the associated reference count by 1). Deleting an item reduces to releasing an item. An item cannot be effectively deleted unless its reference count drops to 0.


In [46]:
storage_unit = StorageUnit(capacity=100, cpu=2, name="myStorageUnit")
item = Item(1)
storage_unit.add_item(item)
storage_unit.retain_item(item)
storage_unit.delete_item(item)
print(",".join(str(item) for item in storage_unit))


1

In [47]:
storage_unit.delete_item(item) # or storage_unit.release_item(item)
storage_unit.is_empty()


Out[47]:
True

Reserving memory

Setting aside to freeze some available space that is not considered as free space. This is particularly useful in case a monitoring system is copying a large item to a storage unit.


In [48]:
storage_unit = StorageUnit(capacity=100, cpu=2, name="myStorageUnit")
print("storage uit={}".format(storage_unit))
print("reserved memory={}".format(storage_unit.reserved_memory))


storage uit=(capacity=100,count=0,cpu=2,free_space=100.0,load=0.0,name=myStorageUnit,reserved_memory=0.0,size=0)
reserved memory=0.0

In [49]:
storage_unit.reserve_memory(50)
print("storage unit={}".format(storage_unit))
print("reserved memory={}".format(storage_unit.reserved_memory))


storage unit=(capacity=100,count=0,cpu=2,free_space=50.0,load=0.0,name=myStorageUnit,reserved_memory=50.0,size=0)
reserved memory=50.0

Whereas there is no item in this storage unit with capacity $100$, the free space takes into account the reserved memory/


In [50]:
storage_unit.free_space()


Out[50]:
50.0

In [51]:
storage_unit.capacity


Out[51]:
100

In [52]:
storage_unit.reserved_memory


Out[52]:
50.0

In the current situration, it is thus not possible to add any item whose size is larger than 50.


In [53]:
storage_unit.add_item(Item(51))


---------------------------------------------------------------------------
StorageUnitException                      Traceback (most recent call last)
<ipython-input-53-27cc05ffe0fd> in <module>()
----> 1 storage_unit.add_item(Item(51))

/Users/vialette/Documents/git-data/ultrastorage/ultrastorage/storageunit/storageunit.py in add_item(self, item)
    254         """
    255         if item > self.free_space():
--> 256             raise StorageUnitException("storage unit overflow")
    257 
    258         if item in self._items:

StorageUnitException: storage unit overflow

One can restore (i.e., return back) the reserved memory. Note that the only constraint is to restore at most the reserved memory.


In [54]:
storage_unit.restore_memory(1)
print("storage unit={}".format(storage_unit))
storage_unit.add_item(Item(51))
print("storage unit={}".format(storage_unit))
storage_unit.restore_memory(storage_unit.reserved_memory)
print("storage unit={}".format(storage_unit))


storage unit=(capacity=100,count=0,cpu=2,free_space=51.0,load=0.0,name=myStorageUnit,reserved_memory=49.0,size=0)
storage unit=(capacity=100,count=1,cpu=2,free_space=0.0,load=0.51,name=myStorageUnit,reserved_memory=49.0,size=51)
storage unit=(capacity=100,count=1,cpu=2,free_space=49.0,load=0.51,name=myStorageUnit,reserved_memory=0.0,size=51)