~/Gather Scatter Pattern in Go Explained

Sep 14, 2019


The gather scatter pattern, also called scatter gather, is a parallel computation strategy in Go which divides work into independent tasks using goroutines, then collects the results after all are done.

Scatter Phase
Work is split across multiple goroutines. Each goroutine processes a chunk of data independently.

Gather Phase
Results from all goroutines are combined. Commonly, a channel is used for goroutines to report their result to a central goroutine that aggregates the data.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

package main

import (
    "fmt"
    "sync"
)

func process(chunk int, out chan<- int, wg *sync.WaitGroup) {
    defer wg.Done()
    out <- chunk * chunk
}

func main() {
    data := []int{2, 4, 6, 8}
    out := make(chan int, len(data))
    var wg sync.WaitGroup

    // Scatter
    for _, v := range data {
        wg.Add(1)
        go process(v, out, &wg)
    }

    // Gather
    wg.Wait()
    close(out)

    var results []int
    for r := range out {
        results = append(results, r)
    }

    fmt.Println("Squares:", results)
}

This pattern is useful for parallelizing CPU-bound or IO-bound tasks such as web scraping, batch processing, and parallel API calls.

Links

Tags: [concurrency]