Keeping the Last N Items

Problem

You want to keep a limited history of the last few items seen during iteration or during some other kind of processing.

Solution

Keeping a limited history is a perfect use for a collections.deque.
Using deque(maxlen=N) creates a fixed-sized queue. When new items are added and the queue is full, the oldest item is automatically removed

Example 1



In [1]:

    
from collections import deque

q = deque(maxlen=3) 
q.append(1)
q.append(2)
q.append(3)
q









    Out[1]:





deque([1, 2, 3])



In [7]:

    
q.append(4)
q









    Out[7]:





deque([2, 3, 4])



In [8]:

    
q.append(5)
q









    Out[8]:





deque([3, 4, 5])

Example 2

The following code performs a simple text match on a sequence of lines and yields the matching line along with the previous N lines of context when found



In [34]:

    
import os

file_dir = os.path.dirname(os.path.realpath('__file__'))
filename = os.path.abspath(os.path.join(file_dir, "..", "code/src/1/keeping_the_last_n_items/somefile.txt"))



In [36]:

    
!head $filename



In [37]:

    
from collections import deque

def search(lines, pattern, history=5):
    previous_lines = deque(maxlen=history) 
    for line in lines:
        if pattern in line:
            yield line, previous_lines
        previous_lines.append(line)
        
# Example use on a file
if __name__ == '__main__':
    with open(filename) as f:
        for line, prevlines in search(f, 'python', 5):
            for pline in prevlines:
                print(pline, end='') 
            print(line, end='') 
            print('-'*20)









    



Keeping a limited history is a perfect use for a `collections.deque`.
For example, the following code performs a simple text match on a
sequence of lines and prints the matching line along with the previous
N lines of context when found:

[source,python]
--------------------
        previous_lines.append(line)

# Example use on a file
if __name__ == '__main__':
    with open('somefile.txt') as f:
         search(f, 'python', 5)
--------------------