Published on

An InfoSec Architect's First Taste of Temporal

Authors

It's the spring of 2005. I'm staring at a computer monitor the size of a small refrigerator, the kind that makes a thunk sound when you turn it on. My mission? Installing Red Hat 9(Not RHEL!) and learning this magical command called cron. It felt like peak technology—telling the computer to do something later, automatically!

Fast forward a decade or two, and I'm a professional developer. cron is still around, but now I'm also writing code to handle the chaos between microservices. This is where I met my white whale, one of the most terrifying bugs of my career. I was working on a backend service for an app with over a billion daily active users (no pressure, right?). In a moment of what I thought was pure genius, I added some retry logic inside a try...catch block. The problem? Another bug in my retry code caused it to... throw another exception.

The result was an infinite loop of failures, like trying to put out a fire with a gasoline-filled water gun. It was a horrifying lesson: manually writing robust retry and recovery logic is a dangerous, soul-crushing business.

Recently, I was brainstorming how to build a cost-effective secrets management system. I stumbled upon a blog post from Uber about their system, and a tool called Cadence was the star of the show. This led me to its successor, Temporal, and I decided to build a small demo to see if it could finally save me from my retry-logic nightmares.

And oh boy, it can.

So, What is Temporal?

Imagine Temporal is a super-smart, patient, and forget-proof project manager. This manager doesn't do the actual work (your code does that), but it orchestrates, supervises, and remembers every single detail of a complex project, ensuring it gets done perfectly, no matter what disasters happen along the way.

For our scenario, the "complex project" is this: Safely rotate a database password for a critical service deployed across multiple regions.

Let's see how our new project manager, Temporal, handles this job.

Example : The "Cross-Region Key Rotation" Workflow

Our goal: Rotate the database password for "Payment Service A," which runs in both Azure East US and Azure West US. The whole process might take 10 minutes, and failure is not an option.

Step 1: Kicking Off the Project (Scheduling)

The Old Way: You’d set up an external cron job to trigger a script every 90 days. Hope the server running cron doesn't go down!

The Temporal Way: You define a Workflow and simply tell it, "Run yourself every 90 days." Temporal has powerful, built-in scheduling. It will reliably kick off the project right on time.

Step 2: Generating the New Password (Executing a Task)

The project manager (Temporal) gives the first command: "Go generate a new password." (This calls a Secret Provider API).

This task might fail because the network decided to take a coffee break.

Temporal's Superpower (Automatic Retries): You can configure the task and tell Temporal: "If this fails, wait 5 seconds and try again. Do this up to 3 times." Temporal will diligently follow your orders. You don't have to write a single try-catch-retry loop.

Step 3: Distributing the New Password (State Persistence)

This is the critical part. Our manager now has the new password and needs to give it to two different teams:

  • Write the new password to the East US Key Vault.
  • Write the same new password to the West US Key Vault.

Disaster Strikes! Imagine the server running your code crashes right after it successfully writes to the East US vault!

The Old Way: When your program restarts, it has amnesia. It has no idea what it was doing. It might generate a new password (yikes!) or just sit there, confused, leaving your systems in an inconsistent state. You’d need to build a complex state-tracking table in a database just to remember "Step 3a is done, but 3b is not."

Temporal's Superpower (State Persistence / Flawless Memory): After every successful step, Temporal automatically saves a snapshot of the workflow's progress. When the server comes back online, a new worker picks up the job and Temporal says, "Ah, I remember! The new password was generated and already sent to East US. My next task is to send it to West US." This guarantees the workflow runs exactly once without missing a beat or redoing work.

Step 4: Waiting for Confirmation (Long-Running & Timeouts)

After distributing the password, the project manager needs to tell the deployment platform to restart the apps and then wait for them to report "new password test successful."

This waiting period could be 5-10 minutes.

The Old Way: A typical web service or script would time out or die trying to "pause" for that long waiting for a callback.

Temporal's Superpower (Long-Running Workflows): Temporal workflows are designed to run for a long time. A workflow can happily pause for minutes, hours, or even days waiting for an external signal, all without consuming any compute resources.

Temporal's Superpower (Timeouts): You can also tell Temporal, "If you don't hear back in 15 minutes, consider the mission a failure." This prevents the process from getting stuck forever.

Step 5: Handling Failure (Recovery & Compensation)

Disaster Strikes Again! The app reports back: "New password test FAILED!"

The Old Way: You'd need to write a tangled mess of if-else and catch blocks to manually roll back the changes. This "compensation" logic (like calling an API to delete the new password version from the Key Vaults) is incredibly error-prone.

Temporal's Superpower (Failure Recovery): You can define a "cleanup plan" or a Saga pattern right in your workflow. When Temporal receives the failure signal, it automatically executes your pre-defined rollback steps, like deleting the new secrets from both Key Vaults, restoring everything to its initial state, and sending an alert to the on-call engineer.

Let's Build It! A Hands-On Temporal Project

Step Zero: Prep Your Dev Environment

Please follow the official docs for Temporal and Vault to get them set up. I'm running my local environment in WSL on Windows.(Ask AI for the setup steps, please:p)

Once ready, start the Temporal and Vault services.

Step One: Create the Go Project and Install Dependencies

Now, let's set up our project structure:

ss@ss:~$ tree temporal-vault-demo/
temporal-vault-demo/
├── go.mod
├── go.sum
├── starter
│   └── main.go
├── worker
│   └── main.go
└── workflow.go

Step Two: Write the Workflow and Activities (The Core Logic)

In your temporal-vault-demo directory, create a new file named workflow.go. This is where we'll define the master plan for our project manager.

In Temporal, any action that has a side effect (like an API call, database write, or reading from a file) should be in an Activity. The Workflow is the orchestrator that calls these Activities in the correct order.

workflow.go Code:

package temporal_vault_demo

import (
	"context"
	"crypto/rand"
	"fmt"
	"math/big"
	"time"

	"github.com/hashicorp/vault/api"
	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/workflow"
)

// ========================================================================
// 1. Activities - The actual "doers" of work
// ========================================================================

type Activities struct {
	VaultClient *api.Client
}

// GeneratePassword is an activity that creates a random password.
// We should use KMS service in real world
func (a *Activities) GeneratePassword(ctx context.Context) (string, error) {
	activity.GetLogger(ctx).Info("Generating a new password...")
	chars := "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*"
	length := 16
	password := ""
	for i := 0; i < length; i++ {
		n, err := rand.Int(rand.Reader, big.NewInt(int64(len(chars))))
		if err != nil {
			return "", err
		}
		password += string(chars[n.Int64()])
	}
	return password, nil
}

// WriteSecretToVault is an activity that writes a secret to a given path in Vault.
// We should use the APIs from Cloud Service Provider to write it to, like Azure Key Vault
func (a *Activities) WriteSecretToVault(ctx context.Context, path string, secretValue string) error {
	logger := activity.GetLogger(ctx)
	logger.Info("Writing secret to Vault path.", "Path", path)

	secretData := map[string]interface{}{
		"password": secretValue,
	}

	_, err := a.VaultClient.KVv2("secret").Put(ctx, path, secretData)
	if err != nil {
		logger.Error("Failed to write secret to Vault.", "Path", path, "Error", err)
		return err
	}
	logger.Info("Successfully wrote secret to Vault.", "Path", path)
	return nil
}

// ========================================================================
// 2. Workflow - The orchestrator, the project manager
// ========================================================================

func SecretDistributionWorkflow(ctx workflow.Context) error {
	// Set timeout options for our activities.
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)
	logger := workflow.GetLogger(ctx)
	logger.Info("Secret Distribution Workflow started.")

	// Step 1: Call the activity to generate a new password.
	var newPassword string
	err := workflow.ExecuteActivity(ctx, "GeneratePassword").Get(ctx, &newPassword)
	if err != nil {
		logger.Error("Failed to generate password.", "Error", err)
		return err
	}
	logger.Info("Successfully generated a new password.")

	// Step 2: Distribute the password to both "regions" concurrently.
	// We use an error channel to collect results from our concurrent tasks.
	errorChan := workflow.NewChannel(ctx)

	// A note on concurrency: We must use workflow.Go, not a standard Go routine.
	// This allows Temporal to manage the concurrency in a deterministic way
	// that can be safely replayed if the worker crashes.

	// Concurrent Task A: Write to East US
	workflow.Go(ctx, func(ctx workflow.Context) {
		err := workflow.ExecuteActivity(ctx, "WriteSecretToVault", "db/east-us", newPassword).Get(ctx, nil)
		if err != nil {
			errorChan.Send(ctx, fmt.Errorf("failed to write to east-us: %w", err))
		} else {
			errorChan.Send(ctx, nil) // Send nil for success
		}
	})

	// Concurrent Task B: Write to West US
	workflow.Go(ctx, func(ctx workflow.Context) {
		err := workflow.ExecuteActivity(ctx, "WriteSecretToVault", "db/west-us", newPassword).Get(ctx, nil)
		if err != nil {
			errorChan.Send(ctx, fmt.Errorf("failed to write to west-us: %w", err))
		} else {
			errorChan.Send(ctx, nil) // Send nil for success
		}
	})

	// Wait for both concurrent tasks to finish.
	var errs []error
	for i := 0; i < 2; i++ {
		var errResult error
		errorChan.Receive(ctx, &errResult)
		if errResult != nil {
			errs = append(errs, errResult)
		}
	}

	if len(errs) > 0 {
		logger.Error("One or more distribution tasks failed.", "Errors", errs)
		// This is where you would add compensation logic,
		// like an activity to delete the secrets that were already written.
		return fmt.Errorf("workflow failed with errors: %v", errs)
	}

	logger.Info("Workflow completed successfully! Secret distributed to all regions.")
	return nil
}

Step Three: Create and Run the Worker

The Worker is a background process that connects to the Temporal server. It listens for tasks and executes the workflow and activity code we just wrote.

Create a new folder worker and place a main.go file inside it.

worker/main.go Code:

package main

import (
	"log"

	"github.com/hashicorp/vault/api"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"

	// Import the workflow and activities from our root package
	temporalvault "temporal-vault-demo"
)

func main() {
	// 1. Create the Temporal client.
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create Temporal client", err)
	}
	defer c.Close()

	// 2. Create the Vault client.
	// It's configured to automatically pick up the VAULT_ADDR and VAULT_TOKEN
	// environment variables, which is super handy for local dev.
	vaultClient, err := api.NewClient(api.DefaultConfig())
	if err != nil {
		log.Fatalln("Unable to create Vault client", err)
	}

	// 3. Create a new Worker.
	// "secret-distribution-task-queue" is the name of the list our Starter
	// will place jobs on.
	w := worker.New(c, "secret-distribution-task-queue", worker.Options{})

	// 4. Register our Workflow and Activities with the Worker.
	activities := &temporalvault.Activities{VaultClient: vaultClient}
	w.RegisterWorkflow(temporalvault.SecretDistributionWorkflow)
	w.RegisterActivity(activities.GeneratePassword)
	w.RegisterActivity(activities.WriteSecretToVault)

	// 5. Start the Worker! It will now poll for tasks.
	log.Println("Starting Worker...")
	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start Worker", err)
	}
}

A friendly reminder: the import "temporal-vault-demo" line should match the module name from your go mod init command.

Step Four: Create and Run the Starter

The Starter is a simple, one-off program whose only job is to tell Temporal, "Hey, please kick off a new SecretDistributionWorkflow for me."

Create another new folder starter, and add a main.go file to it.

starter/main.go Code:

package main

import (
	"context"
	"log"

	"go.temporal.io/sdk/client"

	temporalvault "temporal-vault-demo"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create Temporal client", err)
	}
	defer c.Close()

	// Define the options for our workflow.
	// WorkflowID is a unique business-level ID for this specific run.
	// TaskQueue must match the name the Worker is listening on.
	workflowOptions := client.StartWorkflowOptions{
		ID:        "secret-distribution-workflow-1",
		TaskQueue: "secret-distribution-task-queue",
	}

	// Start the workflow! This is an asynchronous operation.
	log.Println("Starting Secret Distribution Workflow...")
	we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, temporalvault.SecretDistributionWorkflow)
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	}

	log.Printf("Started workflow - WorkflowID: %s, RunID: %s", we.GetID(), we.GetRunID())

	// Now, we'll wait for the workflow to complete.
	log.Println("Waiting for workflow to complete...")
	err = we.Get(context.Background(), nil)
	if err != nil {
		log.Fatalln("Workflow failed", err)
	}

	log.Println("Workflow completed successfully!")
}

Step Five: Run and Verify!

You'll need two separate terminal windows (in addition to the ones running Temporal and Vault).

Terminal 1 - Run the Worker:

ss@ss:~/temporal-vault-demo$ go run worker/main.go
2025/07/08 21:12:33 INFO  No logger configured for temporal client. Created default one.
2025/07/08 21:12:33 Starting Worker...
2025/07/08 21:12:33 INFO  Started Worker Namespace default TaskQueue secret-distribution-task-queue WorkerID 160626@ss@
...

Terminal 2 - Run the Starter (after the Worker is running):

ss@ss:~/temporal-vault-demo$ go run starter/main.go
2025/07/08 21:12:56 INFO  No logger configured for temporal client. Created default one.
2025/07/08 21:12:56 Starting Secret Distribution Workflow...
2025/07/08 21:12:56 Started workflow - WorkflowID: secret-distribution-workflow-1, RunID: 0197ea2b-0293-719a-9e21-c937d191147b
2025/07/08 21:12:56 Waiting for workflow to complete...
2025/07/08 21:12:56 Workflow completed successfully!

Observation and Verification:

Now for the moment of truth. You can use the vault CLI in your terminal to see if the secrets were actually written.

ss@ss:~/temporal-vault-demo$ vault kv get secret/db/east-us
===== Secret Path =====
secret/data/db/east-us

======= Metadata =======
Key                Value
---                -----
created_time       2025-07-08T13:12:56.746048772Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    GiAd@lEQ0lSaxdSP

ss@ss:~/temporal-vault-demo$ vault kv get secret/db/west-us
===== Secret Path =====
secret/data/db/west-us

======= Metadata =======
Key                Value
---                -----
created_time       2025-07-08T13:12:56.746342827Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    GiAd@lEQ0lSaxdSP

Success! The same password in both locations. And if you check out the Temporal Web UI at http://localhost:8233, you'll see a beautiful record of our completed workflow.

It lets you focus on your business logic, while the robot project manager handles the messy reality of distributed systems. And trust me, after my try...catch...fail incident, that's a promotion we could all use.