M YUSUF

Backend Developer

Building a Go Lambda to Process SQS Messages: A Complete Guide

2025-08-26

This example will cover everything you need to know to get a Go (Golang) Lambda function running that processes messages from an Amazon SQS queue. We'll go through the handler code, project setup, build process, deployment, and testing.

1. The Go Code (main.go)

This code defines a Lambda handler that is triggered by an SQS event. It iterates through the messages in the batch, logs the content of each message, and then completes successfully. Create a new directory for your project (e.g., go-sqs-lambda) and place the following code in a file named main.go.


// main.go
package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

// Handler is the main Lambda function handler.
// It receives an SQS event which can contain one or more messages.
func Handler(ctx context.Context, sqsEvent events.SQSEvent) error {
    // An SQS event can contain multiple records (messages) in a single batch.
    // We should loop through all of them.
    if len(sqsEvent.Records) == 0 {
        log.Println("No records found in SQS event. Exiting.")
        return nil
    }

    for _, message := range sqsEvent.Records {
        fmt.Printf("Processing message with ID: %s\n", message.MessageId)
        fmt.Printf("Message Body: %s\n", message.Body)

        // Here you would add your actual business logic to process the message.
        // For example, you might parse the JSON body, save data to a database,
        // or call another API.

        log.Printf("Successfully processed message ID: %s", message.MessageId)
    }

    // If the handler returns nil, the Lambda service will automatically
    // delete the successfully processed messages from the SQS queue.
    // If it returns an error, the messages in the batch will be returned
    // to the queue for another attempt (based on your queue's configuration).
    return nil
}

func main() {
    // The lambda.Start function is from the aws-lambda-go package.
    // It takes our handler function and starts the Lambda execution loop.
    lambda.Start(Handler)
}

2. Project Setup and Dependencies

You need to initialize a Go module and get the necessary AWS Lambda packages. In your terminal, navigate to your project directory and run:


# Initialize a Go module
go mod init go-sqs-lambda

# Get the AWS SDK packages for Lambda and SQS events
go get github.com/aws/aws-lambda-go/lambda
go get github.com/aws/aws-lambda-go/events

3. Build the Deployment Package

AWS Lambda runs on a Linux environment. You need to compile your Go code for a Linux architecture and then zip it into a deployment package.


# Build the Go binary for Linux
# The output file 'bootstrap' is the standard name for the executable in the 'provided.al2' runtime.
GOOS=linux GOARCH=amd64 go build -o bootstrap main.go

# Create a zip file containing the executable
zip deployment.zip bootstrap

You will now have a deployment.zip file ready to be uploaded to AWS Lambda.

4. Deployment to AWS

Step 4.1: Create an IAM Role

Your Lambda function needs permission to interact with other AWS services.

  1. Go to the IAM console in AWS.
  2. Navigate to Roles and click Create role.
  3. For "Trusted entity type", select AWS service.
  4. For "Use case", select Lambda. Click Next.
  5. Search for and attach the following managed policy: AWSLambdaSQSQueueExecutionRole. This policy grants permissions to read from SQS and write logs to CloudWatch.
  6. Click Next, give your role a name (e.g., LambdaSqsProcessorRole), and click Create role.

Step 4.2: Create the SQS Queue

  1. Go to the Simple Queue Service (SQS) console.
  2. Click Create queue.
  3. Keep the type as Standard and give it a name (e.g., my-processing-queue).
  4. Leave the default settings for now and click Create queue.

Step 4.3: Create the Lambda Function

  1. Go to the AWS Lambda console and click Create function.
  2. Select Author from scratch.
  3. Function name: my-sqs-processor
  4. Runtime: Select Provide your own bootstrap on Amazon Linux 2 (this is provided.al2). Go is a compiled language, so we use this custom runtime.
  5. Architecture: x86_64
  6. Permissions: Expand "Change default execution role", select Use an existing role, and choose the LambdaSqsProcessorRole you created earlier.
  7. Click Create function.

Step 4.4: Upload Code and Configure Handler

  1. In the function's "Code source" panel, click Upload from and select .zip file.
  2. Upload the deployment.zip file you created.
  3. Go to Configuration > General configuration > Edit.
  4. Under Runtime settings, ensure the Handler is set to bootstrap. This tells Lambda to execute the file named bootstrap. Click Save.

Step 4.5: Add the SQS Trigger

  1. In your Lambda function's main page, click Add trigger.
  2. Select SQS as the trigger source.
  3. SQS queue: Choose the my-processing-queue you created.
  4. You can leave the Batch size at its default (10). This means the Lambda will be invoked with up to 10 messages at a time.
  5. Click Add.

5. Testing the Function

  1. Navigate to the SQS console and select your queue.
  2. Click on Send and receive messages.
  3. In the "Message body" text box, enter a simple message, like { "orderId": "12345" }.
  4. Click Send message.
  5. Go back to your Lambda function's page, click the Monitor tab, then click View CloudWatch logs.
  6. You should see a new log stream with output confirming that your message was processed.

What happens if the handler returns an error?

This is a critical aspect of building robust event-driven systems. When your Go Lambda handler returns an error, it signals to the AWS Lambda service that the processing of the incoming batch of messages has failed. The entire batch is considered failed, and the messages will reappear in the queue after the Visibility Timeout expires for an automatic retry. To prevent infinite loops with messages that always fail ("poison pills"), you should always configure a Dead-Letter Queue (DLQ) on your source SQS queue.

Advanced Solution: Report Batch Item Failures

The "all or nothing" batch behavior can be inefficient. To solve this, AWS provides a feature called "Report batch item failures". When you enable this in your Lambda trigger's configuration, you can change your function's return signature to tell SQS exactly which messages in the batch failed. SQS will then only return the failed messages to the queue for a retry.


// main_with_partial_failure.go
package main

import (
    "context"
    "fmt"
    "log"
    "strings"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

// The return type is now events.SQSBatchResponse
func Handler(ctx context.Context, sqsEvent events.SQSEvent) (events.SQSBatchResponse, error) {
    var failedMessageIDs []events.SQSBatchItemFailure

    for _, message := range sqsEvent.Records {
        fmt.Printf("Processing message with ID: %s\n", message.MessageId)

        if strings.Contains(message.Body, "fail") {
            // Add the ID of the failed message to our list
            failure := events.SQSBatchItemFailure{ItemIdentifier: message.MessageId}
            failedMessageIDs = append(failedMessageIDs, failure)
        } else {
            // Process the message successfully
            log.Printf("Successfully processed message ID: %s", message.MessageId)
        }
    }

    // Return a response object containing the list of failed message IDs
    return events.SQSBatchResponse{BatchItemFailures: failedMessageIDs}, nil
}

func main() {
    lambda.Start(Handler)
}

To use this, you must update your function code as shown above and enable the "Report batch item failures" option in the AWS Lambda Console under your function's SQS trigger configuration. Now, only the messages you explicitly identify will be retried, making your function much more efficient and resilient.