Module 5: DynamoDB Index Design

Introduction to DynamoDB Indexes

In Module 3, you learned about DynamoDB table design and primary keys. In this module, we'll dive deeper into DynamoDB by exploring secondary indexes, which provide additional ways to query your data beyond what's possible with just the primary key.

Secondary indexes in DynamoDB allow you to query the data in your table using an alternate key, in addition to queries against the primary key. DynamoDB supports two types of secondary indexes:

• Global Secondary Index (GSI): An index with a partition key and optional sort key that can be different from those on the base table
• Local Secondary Index (LSI): An index that has the same partition key as the base table, but a different sort key

In this module, we'll focus primarily on Global Secondary Indexes (GSIs) as they are more flexible and commonly used.

Understanding Global Secondary Indexes

What is a GSI?

A Global Secondary Index (GSI) is a data structure that contains a subset of attributes from a table, along with an alternate key to support Query operations. The index is "global" because queries on the index can span all of the data in the base table, across all partitions.

You can think of a GSI as a separate table that is automatically maintained by DynamoDB. When you add, update, or delete an item in the base table, DynamoDB automatically propagates these changes to any GSIs on that table.

Key Features of GSIs

• Can have different partition and sort keys than the base table
• Every GSI must have a partition key, and can optionally have a sort key
• GSIs are eventually consistent (unlike the base table, which can be strongly consistent)
• You can project attributes from the base table into the index
• GSIs have their own provisioned throughput settings
• You can create up to 20 GSIs per table

Benefits of Using GSIs

GSIs provide several advantages:

• Flexible querying: Query your data by different attributes
• Improved performance: Queries using indexes are generally faster than scans
• Multiple access patterns: Support different application access patterns with a single table
• Automatic maintenance: DynamoDB keeps indexes in sync with the base table

When to Use GSIs

GSIs are particularly useful when:

• You need to query data using attributes that aren't part of the table's primary key
• You have multiple access patterns that require different key structures
• You need to optimize specific query patterns for performance
• You want to avoid expensive scan operations

Designing Effective GSIs

Attribute Projections

When creating a GSI, you must specify which attributes from the base table should be "projected" (copied) into the index. You have three projection options:

• KEYS_ONLY: Only the index and primary key attributes are projected
• INCLUDE: The index and primary keys plus specific attributes you specify
• ALL: All attributes from the base table are projected

Choosing the right projection type involves balancing storage costs and query performance:

• More projected attributes = higher storage costs but fewer need for additional queries
• Fewer projected attributes = lower storage costs but potentially more queries needed

GSI Overloading

One powerful technique with GSIs is "overloading" them to support multiple access patterns. This involves using a GSI for multiple purposes by carefully designing your attribute values.

For example, consider a social media application where you need to:

• Get all posts by a user
• Get all posts with a specific hashtag

You could create a GSI with:

• Partition key: GSI_PK (which could be either "USER#username" or "HASHTAG#tag")
• Sort key: CreatedAt (timestamp when the item was created)

This allows you to query for all posts by a user or all posts with a specific hashtag using the same index.

GSI Write Sharding

When you have a "hot" partition key (one that receives a disproportionate amount of traffic), you can use write sharding to distribute the load across multiple partitions. This involves adding a random suffix to your partition key to spread the writes across partitions.

Common GSI Design Patterns

Here are some common patterns for using GSIs effectively:

• Inverted indexes: Swapping the table's sort key as the GSI's partition key
• Sparse indexes: Creating indexes that only contain items with specific attributes
• Aggregation: Using GSIs to support aggregations like counts or summaries
• Time-based access patterns: Using time attributes as GSI sort keys

Creating GSIs with CloudFormation

In your project, you'll use AWS CloudFormation to create and configure GSIs. Here's an example of how to define a GSI in a CloudFormation template:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "MyTable": {
      "Type": "AWS::DynamoDB::Table",
      "Properties": {
        "TableName": "MyTable",
        "BillingMode": "PAY_PER_REQUEST",
        "AttributeDefinitions": [
          {
            "AttributeName": "PK",
            "AttributeType": "S"
          },
          {
            "AttributeName": "SK",
            "AttributeType": "S"
          },
          {
            "AttributeName": "GSI1PK",
            "AttributeType": "S"
          },
          {
            "AttributeName": "GSI1SK",
            "AttributeType": "S"
          }
        ],
        "KeySchema": [
          {
            "AttributeName": "PK",
            "KeyType": "HASH"
          },
          {
            "AttributeName": "SK",
            "KeyType": "RANGE"
          }
        ],
        "GlobalSecondaryIndexes": [
          {
            "IndexName": "GSI1",
            "KeySchema": [
              {
                "AttributeName": "GSI1PK",
                "KeyType": "HASH"
              },
              {
                "AttributeName": "GSI1SK",
                "KeyType": "RANGE"
              }
            ],
            "Projection": {
              "ProjectionType": "ALL"
            }
          }
        ]
      }
    }
  }
}

In this example, we've defined a table with a composite primary key (PK and SK) and a GSI with its own composite key (GSI1PK and GSI1SK). The GSI projects all attributes from the base table.

Querying GSIs

To query a GSI in Java, you use the DynamoDB query() operation with the index name specified. Here's an example:

QueryRequest queryRequest = new QueryRequest()
    .withTableName("MyTable")
    .withIndexName("GSI1")
    .withKeyConditionExpression("GSI1PK = :pk")
    .withExpressionAttributeValues(Map.of(":pk", new AttributeValue("USER#johndoe")));

QueryResult result = dynamoDbClient.query(queryRequest);
List<Map<String, AttributeValue>> items = result.getItems();

This query retrieves all items from the GSI1 index where the GSI1PK is "USER#johndoe". The query will return all attributes that were projected into the index.

Query Best Practices

When querying GSIs, keep these best practices in mind:

• Always try to use key conditions to filter data at the database level
• Only project the attributes you need to minimize storage costs
• Be aware of eventual consistency in GSIs
• Monitor GSI performance and adjust as needed
• Use query pagination for large result sets

Key Topics

Guided Project

In the guided project for this module, you'll apply what you've learned about DynamoDB Index Design to enhance your Learn and Be Curious project with GSIs.

Project: Adding GSIs to Your DynamoDB Tables

This project will guide you through designing and implementing GSIs for your project's DynamoDB tables to support additional access patterns.

Additional Resources