engineering
golang
dynamodb

2 Feb 2025

Go and DynamoDB: A Test-Driven Approach

I tend to use Go for most of my projects these days, having made the transition from Python a little over a year ago. I also tend to use AWS, and that includes DynamoDB. One of the standout advantages to using Python with DynamoDB is moto, a package that lets you mock AWS services (including DynamoDB) for your unittests. I'm a major proponent of test-driven development and in a world where AI generated code is becoming more and more frequent, solid testing practices become more and more important. moto became such a staple of my development flow that I became reluctant to use anything other than Python when developing with AWS DynamoDB. However, as I began using Go more and more, I needed to get a testing framework in place. So, I set out to try and find some parity with moto.

In Go, I wanted something that resembles the following

package main

import "testing"

func TestFetchResource(t *testing.T) {
    table := NewDynamoTestTable()
    // create new database instance seeded with test data
    InitializeTestingTable(table)

    got := FetchResource(table, "12345")
    if got.Foo != 5 {
        t.Fatalf("expected Foo to be 5: got %d", got.Foo)
    }
}

func TestExample1(t *testing.T) {
    table := NewDynamoTestTable()
    // create new database instance seeded with test data
    InitializeTestingTable(table)

    ...
}

func TestExample2(t *testing.T) {
    table := NewDynamoTestTable()
    // create new database instance seeded with test data
    InitializeTestingTable(table)

    ...
}

where the InitializeTestingTable table function creates a mocked DynamoDB table seeded with some pre-defined test data that I can then use to run all my tests against. Critically, I wanted the mocked database to be initiated at the start of each test. This is important because it allows all of my tests to run independently of each other, effectively ensuring that all database changes are scoped to the individual test.

In this blog article, I'm going to walk you through the testing setup that I put together to facilitate test-driven development with Go and DynamoDB. The end result is a mocked DynamoDB table that is seeded from a local JSON file as each test is initiated, giving you a testing environment that is clean, manageable and most importantly of all, consistent and repeatable. I'm also going to briefly cover some good design patterns for writing testable Go code as its required to make this possible. Particularly important is the concept of dependency injection, but more on that later.

Prerequisites

The dependency list is fairly small. Spoiler alert, I did have to utilise Docker to get everything working in the end (more on that in the next section). I'm using the following

  • Go version 1.23.4
  • Docker version 27.4.0

I'm also using the Go AWS v2 client. You'll need to use go get to install the following packages

  • github.com/aws/aws-sdk-go-v2
  • github.com/aws/aws-sdk-go-v2/config
  • github.com/aws/aws-sdk-go-v2/service/dynamodb
  • github.com/aws/aws-sdk-go-v2/service/dynamodb/types
  • github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue

Step 1: Code Design and Source Code

In order for this testing strategy to work, you need to make sure that your code is written using a dependency injection pattern. This is good practice in most cases anyway, and if you aren't familiar with this style of coding, a good description can be found here. For what we are doing here, dependency injection just means that any functions that require a database client are going to require the database client as an argument as opposed to creating the connections themselves. This is good practice for a number of reasons, but its important for what I'm doing here because it allows us to configure the database connection outside the scope of the function. This lets me manage the database configuration in my test. I'll give an example of what this looks like in practice shortly.

First, I'm going to define a DynamoDB interface that's going to execute any database operations in my main codebase.

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)

type Resource struct {
    Id string  `json:"id"`
    Foo int    `json:"foo"`
    Bar string `json:"bar"`
}

type Table interface {
    GetResource(id string) (Resource, error)
}


type DynamoTable struct {
    Client *dynamodb.Client
    TableName string
}


// GetResource retrieves a resource from the DynamoDB table based on the provided ID.
// It constructs the primary key (PK) and sort key (SK) using the given ID and queries the table.
// If the item is found, it unmarshals the result into a Resource object and returns it.
// If the item is not found or an error occurs during the query, it returns an error.
//
// Parameters:
// - id: The ID of the resource to retrieve.
//
// Returns:
// - Resource: The retrieved resource.
// - error: An error if the resource is not found or if there is an issue with the query.
func (table *DynamoTable) GetResource(id string) (Resource, error) {

    response, err := table.Client.GetItem(context.TODO(), &dynamodb.GetItemInput{
        TableName: &table.TableName,
        Key: map[string]types.AttributeValue{
            "PK": &types.AttributeValueMemberS{
                Value: fmt.Sprintf("ID#%s", id),
            },
            "SK": &types.AttributeValueMemberS{
                Value: fmt.Sprintf("ID#%s", id),
            },
        },
    })

    var resource Resource
    if err != nil {
        return resource, err
    }

    if response.Item == nil {
        return resource, fmt.Errorf("Resource not found")

    } else {
        err = attributevalue.UnmarshalMap(response.Item, &resource)
        return resource, err
    }
}



// NewDynamoTable creates a new DynamoTable instance with the specified table name.
// It initializes the AWS configuration and creates a DynamoDB client.
//
// Parameters:
// - tableName: The name of the DynamoDB table.
//
// Returns:
// - A pointer to a DynamoTable instance.
func NewDynamoTable(tableName string) *DynamoTable {
    cfg, _ := config.LoadDefaultConfig(context.TODO())
    // Create an Amazon S3 service client
    client := dynamodb.NewFromConfig(cfg)

    return &DynamoTable{
        Client: client,
        TableName: tableName,
    }
}

There is some wiggle room on how you can implement this, but the crucial part is that the AWS dynamodb.Client instance can be configured externally and passed down as an argument. In this case, its set as a property of the DynamoTable struct so that I can configure a connection to a local database in the scope of a test, and then create a DynamoTable interface from the configured connection. This allows me to point the codebase at a locally running DynamoDB server without making any code changes.

Once you have your DynamoDB interface setup, write the rest of your functions in such a way that they take a configured DynamoTable struct as an argument. For example, consider the following two functions

func Example1(table Table, id string) {
    r, err := table.GetResource(id)
    if err != nil {
        panic(err)
    }

    // do something with resource here
    ...
}

func Example2(id string) {
    table := NewDynamoTable()

    r, err := table.GetResource(id)
    if err != nil {
        panic(err)
    }

    // do something with resource here
    ...
}

Both functionally produce the same output. However, Example1 follows a dependency injection pattern because the database connection is provided, while Example2 does not. This is an important difference to understand. Example2 cannot be tested easily because it creates its own database connection. Example1 on the other hand uses whatever database connection is passed down to it. This allows you to point Example1 at a local testing database without changing the behaviour of the code.

Step 2: Database Setup

When testing with Python, moto implements a DynamoDB interface that it exposes for clients to connect to. Under the hood, moto intercepts AWS API calls and routes them to its own in-memory interfaces. A lot of the core functionality in moto relies on monkey patching, which is problematic since monkey patching is not a pattern that works well (if at all) in Go.

In theory, it would be possible to implement an interface that mocks the AWS DynamoDB Go client. However, this is a monumental task. A far easier solutions is to run a DynamoDB server locally using Docker. While this does introduce an external dependency, its arguably much less complex to implement than maintaining a fully-mocked DynamoDB interface.

To run a DynamoDB server locally

$ docker run -p 8000:8000 amazon/dynamodb-local

If you've already created the container previously, start it instead using

$ docker start {container-name}

Do I need a volume attached to persist data?

As I mentioned in the introduction, the database is going to be setup and seeded with data from a local JSON file when each test is initiated. This means that its completely irrelevant what table(s) exist and what data is in the tables when the tests start. Each test setup will create the tables it needs, clear out any existing data, and seed the database with a fresh dataset.

Step 3: Define Seed Data and Table Schema

At the start of each test, the local DynamoDB service is going to be cleared, and seeded with initial test data. The data itself is going to be read from a local JSON file. This provides a clean interface to manage the data that is being seeded. While you can add any seed data that you want, the code that I'm going to be writing to seed the database requires that the JSON file contains an array of objects. All of the objects present in the JSON array are going to be loaded into the DynamoDB table at test initiation time. An example file my look like the following.

[
  {
    "PK": "ID#1234",
    "SK": "ID#1234",
    "id": "1234",
    "foo": 1,
    "bar": "testing"
  },
  {
    "PK": "ID#2345",
    "SK": "ID#2345",
    "id": "2345",
    "foo": 1,
    "bar": "testing"
  },
  {
    "PK": "ID#3456",
    "SK": "ID#3456",
    "id": "3456",
    "foo": 1,
    "bar": "testing"
  },
  {
    "PK": "ID#45678",
    "SK": "ID#45678",
    "id": "45678",
    "foo": 1,
    "bar": "testing"
  },
  {
    "PK": "ID#56789",
    "SK": "ID#56789",
    "id": "56789",
    "foo": 1,
    "bar": "testing"
  }
]

In addition to the data, I'm also going to define the DynamoDB table schema in a JSON file. The important thing is to make sure that the JSON schema matches the structure of the CreateTable AWS request. A basic table structure with a global secondary index looks like the following.

{
  "TableName": "testing-table",
  "AttributeDefinitions": [
    {
      "AttributeName": "PK",
      "AttributeType": "S"
    },
    {
      "AttributeName": "SK",
      "AttributeType": "S"
    },
    {
      "AttributeName": "GSI1PK",
      "AttributeType": "S"
    },
    {
      "AttributeName": "GSI1SK",
      "AttributeType": "S"
    }
  ],
  "KeySchema": [
    {
      "AttributeName": "PK",
      "KeyType": "HASH"
    },
    {
      "AttributeName": "SK",
      "KeyType": "RANGE"
    }
  ],
  "GlobalSecondaryIndexes": [
    {
      "IndexName": "GSI1",
      "KeySchema": [
        {
          "AttributeName": "GSI1PK",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "GSI1SK",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 5,
        "WriteCapacityUnits": 5
      }
    }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 10,
    "WriteCapacityUnits": 10
  }
}

The full documentation for the CreateTable input can be found here.

In all downstream code snippets, I'm going to assume that the data seed file is located at initial_db_items.json and that the table schema is at table_schema.json. Make sure to change the code where appropriate if you are using different file paths.

Step 4: Define Functions to Setup Testing Database

Before any actual tests can be written, the functions to initiate the database need to be defined. Just as a reminder, the final test suite should look something like the following.

import "testing"

func TestExample1(t *testing.T) {
    table := NewDynamoTestTable()
    // create new database instance seeded with test data
    InitializeTestingTable(table)

    ...
}

func TestExample2(t *testing.T) {
    table := NewDynamoTestTable()
    // create new database instance seeded with test data
    InitializeTestingTable(table)

    ...
}

func TestExample3(t *testing.T) {
    table := NewDynamoTestTable()
    // create new database instance seeded with test data
    InitializeTestingTable(table)

    ...
}

The InitiateDynamoDB function is going to be called at the start of every test, and its going to do the following jobs

  1. Create the DynamoDB table if it doesn't exist
  2. Delete any existing data
  3. Load the seed data from the local JSON file
  4. Seed the database

First, we need to be able to create a database client that connects to the locally running Docker container. This is done as follows (ignore the LoadTableSchema function for the time being, this is defined in the next step).

package main

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "os"
    "strings"
    "sync"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)

// NewDynamoTestTable creates a new DynamoTable for testing purposes with the specified table name.
// It loads the default AWS configuration and sets up a local DynamoDB endpoint for testing.
//
// Parameters:
// - tableName: The name of the DynamoDB table.
//
// Returns:
// - *DynamoTable: A pointer to the DynamoTable struct configured for testing.
func NewDynamoTestTable() *DynamoTable {
    schema, err := LoadTableSchema()
    if err != nil {
        panic(err)
    }

    cfg, _ := config.LoadDefaultConfig(context.TODO())

    endpoint := "http://localhost:8000" // NOTE: change this if your DB container is listening on a different port
    client := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
        o.BaseEndpoint = aws.String(endpoint)
    })

    return &DynamoTable{
        TableName: schema.TableName,
        Client: client,
    }
}

Notice that I'm still returning a DynamoTable type defined earlier in the persistence layer. This is why the dependency injection pattern is so important. Because the configured AWS client is a property of the DynamoTable struct, I can create a DynamoTable instance that connects to the local database in my test, and then use it in my function calls without changing any of the source code. This would not be possible if the database connections were being created in the function calls themselves.

Next, lets add some code to load the table schema from the local JSON file and create a DynamoDB table.

// LoadTableSchema loads a DynamoDB table schema from a JSON file named "table_schema.json".
// It returns a dynamodb.CreateTableInput object representing the schema and an error if any occurs during the process.
// The function performs the following steps:
// 1. Opens the "table_schema.json" file.
// 2. Reads the contents of the file.
// 3. Unmarshals the JSON contents into a dynamodb.CreateTableInput object.
// 4. Returns the schema and any error encountered during the process.
func LoadTableSchema() (dynamodb.CreateTableInput, error) {
    var schema dynamodb.CreateTableInput

    file, err := os.Open("table_schema.json") // NOTE: change this if your schema is on a different path.
    if err != nil {
        return schema, err
    }
    defer file.Close()

    contents, err := io.ReadAll(file)
    if err != nil {
        return schema, err
    }

    if err := json.Unmarshal(contents, &schema); err != nil {
        return schema, err
    }
    return schema, nil
}

// CreateTestingTable creates a DynamoDB table based on the provided schema.
// If the table already exists, it clears the database and returns no error.
//
// Parameters:
//   - table: A pointer to a DynamoTable which contains the DynamoDB client.
//   - schema: A dynamodb.CreateTableInput struct that defines the schema for the table.
//
// Returns:
//   - error: An error if the table creation fails for reasons other than the table already existing.
func CreateTestingTable(table *DynamoTable, schema dynamodb.CreateTableInput) error {
    _, err := table.Client.CreateTable(context.TODO(), &schema)

    var exists *types.ResourceInUseException
    // if the table already exists, clear database
    // and return no error
    if errors.As(err, &exists) {
        return nil
    } else {
        return err
    }
}

Then, lets add the seeding code. This requires several functions.

  1. GetKeysToDelete - this function will scan the DynamoDB table and extract a list of keys to delete
  2. ClearTableContents - this function will take a list of keys and delete from from the DynamoDB table
  3. SeedTable - this function will load the seed data from the local JSON file and write it to the database
// GetKeysToDelete retrieves the keys to delete from a DynamoDB table based on the provided schema.
// It scans the table and extracts the keys specified in the schema.
//
// Parameters:
//   - table: A pointer to a DynamoTable struct that contains the DynamoDB client and table name.
//   - schema: A dynamodb.CreateTableInput struct that defines the schema of the table.
//
// Returns:
//   - A slice of maps, where each map represents a key to delete with attribute names as keys and AttributeValue as values.
//   - An error if the scan operation fails or any other issue occurs.
func GetKeysToDelete(table *DynamoTable, schema dynamodb.CreateTableInput) ([]map[string]types.AttributeValue, error) {
    // get the keys from the schema
    schemaKeys := []string{}
    for _, attr := range schema.KeySchema {
        schemaKeys = append(schemaKeys, *attr.AttributeName)
    }
    projectionExp := strings.Join(schemaKeys, ", ")

    // scan the table to get all keys
    response, err := table.Client.Scan(context.TODO(), &dynamodb.ScanInput{
        TableName:            aws.String(table.TableName),
        ProjectionExpression: aws.String(projectionExp),
    })
    if err != nil {
        return nil, err
    }

    // extract keys from scan results
    keys := []map[string]types.AttributeValue{}
    for _, item := range response.Items {

        key := map[string]types.AttributeValue{}
        for _, k := range schemaKeys {
            key[k] = item[k]
        }
        keys = append(keys, key)
    }
    return keys, nil
}


// ClearTableContents deletes multiple items from a DynamoDB table.
// It takes a DynamoTable pointer and a slice of keys to delete.
// Each key is a map of attribute names to AttributeValue.
// The function performs the deletions concurrently and returns a slice of errors, if any.
//
// Parameters:
//   - table: A pointer to the DynamoTable from which items will be deleted.
//   - keys: A slice of maps, where each map represents a key of an item to be deleted.
//
// Returns:
//   - A slice of errors encountered during the deletion process.
func ClearTableContents(table *DynamoTable, keys []map[string]types.AttributeValue) []error {
    errors := make(chan error, len(keys))
    var wg sync.WaitGroup
    wg.Add(len(keys))

    // delete keys from dynamodb table in coroutine
    for _, key := range keys {
        go func(k map[string]types.AttributeValue) {
            defer wg.Done()
            params := dynamodb.DeleteItemInput{
                TableName: &table.TableName,
                Key:       k,
            }
            _, err := table.Client.DeleteItem(context.TODO(), &params)
            if err != nil {
                errors <- err
            }
        }(key)
    }
    wg.Wait()

    close(errors)

    errorsList := []error{}
    // collect errors and return
    if len(errors) > 0 {
        for err := range errors {
            println(fmt.Sprintf("error deleting data row: %+v", err))
            errorsList = append(errorsList, err)
        }

    }
    return errorsList
}


// SeedTable reads seed data from a JSON file and inserts it into the specified DynamoDB table.
// The JSON file should contain an array of items, where each item is a map of attribute names to values.
//
// Parameters:
//   - table: A pointer to a DynamoTable struct that contains the DynamoDB client and table name.
//
// Returns:
//   - error: An error if any step of reading, parsing, or inserting the data fails, otherwise nil.
func SeedTable(table *DynamoTable) error {
    // read seed data from file
    file, err := os.Open("initial_db_items.json") // NOTE: change to your seed data file
    if err != nil {
        return err
    }
    defer file.Close()

    contents, err := io.ReadAll(file)
    if err != nil {
        return err
    }
    // parse JSON contents
    var items []map[string]interface{}
    if err := json.Unmarshal(contents, &items); err != nil {
        return err
    }

    for _, item := range items {
        // encode item into DynamoDB type from JSON
        encoded, err := attributevalue.MarshalMap(item)
        if err != nil {
            return err
        }
        // insert item into table
        _, err = table.Client.PutItem(context.TODO(), &dynamodb.PutItemInput{
            TableName: &table.TableName,
            Item:      encoded,
        })
        if err != nil {
            return err
        }
    }
    return nil
}

Finally, lets wrap everything into a single utility function that we can call in our tests.

// InitializeTestingTable sets up a DynamoDB table for testing purposes.
// It performs the following steps:
// 1. Loads the table schema.
// 2. Creates a testing table using the loaded schema.
// 3. Retrieves keys of items to be deleted from the table.
// 4. Clears the table contents by deleting the items.
// 5. Seeds the table with initial data.
//
// If any step encounters an error, the function will panic.
//
// Parameters:
// - table: A pointer to the DynamoTable that will be used for testing.
func InitializeTestingTable(table *DynamoTable) {
    schema, err := LoadTableSchema()
    if err != nil {
        panic(err)
    }

    if err := CreateTestingTable(table, schema); err != nil {
        panic(err)
    }

    keys, err := GetKeysToDelete(table, schema)
    if err != nil {
        panic(err)
    }

    errors := ClearTableContents(table, keys)
    if len(errors) > 0 {
        for i, err := range errors {
            println(fmt.Sprintf("error deleting data row %d: %+v", i, err))
        }
        panic(fmt.Sprintf("errors deleting data rows: %+v", errors))
    }

    if err := SeedTable(table); err != nil {
        panic(err)
    }
}

Step 5: Integration into Tests

With all the above code in place, the test can be setup using the structure that we initially wanted

package main

import "testing"

func TestGetItem(t *testing.T) {
    // initiate testing table and seed data
    table := NewDynamoTestTable()
    InitializeTestingTable(table)

    resource, err := table.GetResource("12345")
    if err != nil {
        t.Fatalf("failed to get resource: %v", err)
    }

    if resource.Id != "12345" {
        t.Fatalf("expected resource ID to be 12345, got %s", resource.Id)
    }

    if resource.Foo != 5 {
        t.Fatalf("expected resource Foo to be 5, got %d", resource.Foo)
    }
}

func TestUpdateItem(t *testing.T) {
    // initiate testing table and seed data
    table := NewDynamoTestTable()
    InitializeTestingTable(table)

    ...
}

func TestDeleteItem(t *testing.T) {
    // initiate testing table and seed data
    table := NewDynamoTestTable()
    InitializeTestingTable(table)

    ...
}

And thats it. Since the table is configured and seeded with each test iteration, all tests that you write in this manner are entirely self contained, and any database changes made are scoped to the test.

Final Notes

That's its for today folks. If you enjoyed this article, consider following us on LinkedIn to get updates on future content and what we are working on, or send me a connection request directly. Don't forget to check out the rest of our content and stay tuned for weekly blog posts on tech and business.