Tutorial: Running parallel stages

This tutorial will take 5 minutes.

Installation

To start off, you need to have togomak installed on your system. If you haven't installed it yet, check out the GitHub Releases (opens in a new tab) to find an appropriate version for your system, or you could take a look at the installation too.

Project Initialization

Create a new directory for your Togomak project and navigate to it in your terminal.

mkdir tutorial-parallel-stages
cd tutorial-parallel-stages

Initializing

Create a file called togomak.hcl, this is a bare minimum pipeline that is required for togomak to identify it as a valid pipeline.

togomak.hcl

togomak {
  version = 2
}

You may give a shot executing it now,

INFO[0000] togomak (version=dev)
INFO[0000] took 0s

Adding stages

Now, let's create your first Togomak stage. Add the following code to your togomak.hcl file:

togomak.hcl

 
stage "hello_world" {
  script = <<-EOT
  for i in apple orange mango; do 
    echo Hello, World! $i
    sleep 1
  done
  EOT 
}
 
stage "another_stage" {
  script = <<-EOT
  for i in go rust python; do 
    echo I use $i
    sleep 1
  done
  EOT 
}

ℹ️

<<-EOT is called a Heredoc. If you are not familiar with this syntax, you should definitely check out Indented Heredocs. It's going to be handy.

Run your pipeline

You can run your pipeline, by either using togomak run or just togomak

togomak

You will notice that, although another_stage was defined after hello_stage, both of them runs in parallel. This is because togomak is parallel by default.

INFO[0000] togomak (version=dev)
INFO[0000] [stage=hello_world] [+] hello_world
INFO[0000] [stage=another_stage] [+] another_stage
INFO[0000] [stage=hello_world] Hello, World! apple
INFO[0000] [stage=another_stage] I use go
INFO[0001] [stage=another_stage] I use rust
INFO[0001] [stage=hello_world] Hello, World! orange
INFO[0002] [stage=hello_world] Hello, World! mango
INFO[0002] [stage=another_stage] I use python
INFO[0003] took 3.009s

You will notice that the entire pipeline execution took 3 seconds. This is intuitively because both stages which was supposed to take 3 seconds ran in parallel.

Running one stage after another

Now, we will modify the pipeline such that the stage, stage.another_stage runs exactly after stage.hello_world completes, here is our new pipeline:

togomak.hcl

togomak {
  version = 2
}
 
stage "hello_world" {
  script = <<-EOT
  for i in apple orange mango; do 
    echo Hello, World! $i
    sleep 1
  done
  EOT 
}
 
stage "another_stage" {
  depends_on = [stage.hello_world]
  script = <<-EOT
  for i in go rust python; do 
    echo I use $i
    sleep 1
  done
  EOT 
}

Notice the new depends_on parameter? This implies that stage.another_stage depends on the completion of stage.hello_world. Let's see how this turns out when we run togomak.

togomak

INFO[0000] togomak (version=dev)
INFO[0000] [stage=hello_world] [+] hello_world
INFO[0000] [stage=hello_world] Hello, World! apple
INFO[0001] [stage=hello_world] Hello, World! orange
INFO[0002] [stage=hello_world] Hello, World! mango
INFO[0003] [stage=another_stage] [+] another_stage
INFO[0003] [stage=another_stage] I use go
INFO[0004] [stage=another_stage] I use rust
INFO[0005] [stage=another_stage] I use python
INFO[0006] took 6.02s

Yep, now it ran one after another. The time taken, 6 seconds proves it too.

Run a specific stage

You can run specific stages.

togomak stage.another_stage

INFO[0000] togomak (version=dev)
INFO[0000] [stage=hello_world] [+] hello_world (skipped)
INFO[0000] [stage=another_stage] [+] another_stage (overriden)
INFO[0000] [stage=another_stage] I use go
INFO[0001] [stage=another_stage] I use rust
INFO[0002] [stage=another_stage] I use python
INFO[0003] took 3.006s

🎉 Congratulations

You just learnt about how the Togomak dependency resolution and default parallelism works. Note that, Togomak uses Topological Sort (opens in a new tab) to figure out the dependency tree.

Creating Your First Module Documentation