Worker Pools

In this example we’ll look at how to implement a worker pool using goroutines and channels.



In [1]:

    
import (
    "fmt"
    "time"
)

Worker Function

Here’s the worker, of which we’ll run several concurrent instances. These workers will receive work on the jobs channel and send the corresponding results on results. We’ll sleep a second per job to simulate an expensive task.



In [2]:

    
func worker(id int, jobs <-chan int, results chan<- int) {
    for j := range jobs {
        fmt.Println("worker", id, "processing job", j)
        time.Sleep(time.Second)
        results <- j * 2
    }
}

Utilize Workers

In order to use our pool of workers we need to send them work and collect their results. We make 2 channels for this.



In [3]:

    
jobs := make(chan int, 100)
results := make(chan int, 100)









    Out[3]:





(chan int)(0xc42006f500)

Next, start up 3 workers, initially blocked because there are no jobs yet:



In [4]:

    
for w := 1; w <= 3; w++ {
    go worker(w, jobs, results)
}

Send 9 jobs and then close that channel to indicate that’s all the work we have:



In [5]:

    
for j := 1; j <= 9; j++ {
    jobs <- j
}
close(jobs)

Collect Results

Finally we collect all the results of the work:



In [6]:

    
for a := 1; a <= 9; a++ {
    <-results
}









    Out[6]:





worker 3 processing job 1
worker 1 processing job 2
worker 2 processing job 3
worker 3 processing job 5
worker 1 processing job 6
worker 2 processing job 4
worker 2 processing job 7
worker 1 processing job 8
worker 3 processing job 9

Our running program shows the 9 jobs being executed by various workers. The program only takes about 3 seconds despite doing about 9 seconds of total work because there are 3 workers operating concurrently.