David - Musings of an SRE

Writing a Serverless Python Microservice with AWS Lambda and AWS API Gateway

Serverless Architectures are the rage nowadays. In this article, I’ll attempt to walkthrough the process of writing a Serverless Microservice with AWS Lambda and Amazon API Gateway.

For people who know me, know that I’m not much of a Pythonista, but doing this blog post helped me to better appreciate Python and its awesome standard libraries and while I could have done this in NodeJS, this was a perfect example to pick up Python while doing something new.

Onto the article.

Introduction to the Serverless Architecture.

First off, the name itself can be quite the misnomer. A Serverless Architecture doesn’t mean that you are no longer running servers. But rather, it just means that you no longer need to worry about them that much so that you can focus on the applications that you’re building itself. With everything service-based, you won’t need to worry about scaling the instances that would normally run your different applications.

Lambda

AWS Lambda is Amazon’s answer to the serverless question. Lambda is a service which allows you to write NodeJS/Python/Java functions that live in their cloud which can then be invoked via a variety of sources.

The most common way of invoking a lambda function is through the use of the AWS events (ie. a new file being added into an S3 Bucket) or nowadays in this microservices world that we’re living in, from an API endpoint.

Being able to exposing a lambda function as an API endpoint is one of the most exciting thing to come out of AWS in recent years imho. This allows developers to create really tiny microservices that performs just a single task without having to worry about how to deploy them on hardware.

To give a better example about how we can use AWS Lambda, I’ll walk you through a quick example of writing a Python microservice that takes in a PBKDF2 hash, a plaintext password and an optional digest, and returns a true/false (boolean in a string) if there’s a match between the hash and the plaintext password.

Pre-requisite

  • An existing and verified AWS account
  • Python > 2.7

Lets start by first creating a lambda function.

We’re going to write our own lambda function from scratch, so get your python environment ready. Create a new directory called hello_world and cd into it.

$ mkdir hello_world
$ cd hello_world

All AWS Lambda functions requires a function that takes in an “event” and “context” parameter. I’ll explain what they are in more detail below but for now, create a new python file and add the following:

# index.py
from __future__ import print_function # from __future__ helps ensure compatibility with Python 2 or 3.

import json

print('Loading function')


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event['key1'])
    print("value2 = " + event['key2'])
    print("value3 = " + event['key3'])
    return event['key1']  # Echo back the first key value
    #raise Exception('Something went wrong')

Note, you can actually replace lambda_handler with any name that you choose.

What’s event? The event object is the parameter that is used to pass in data into the handler. This is how you can get information into the function for processing. Commonly, it is a dict type object but it can also be of any Python type.

What’s context? The context object contains all information about the lambda service running your function and can return information like:

  • Remaining time before Lambda timeouts your function
  • Memory limits
  • Cloudwatch log group etc.

So what can we do now?

So with those preconditions in mind, let’s start writing our microservice. To save the trouble, just copy and paste the following code:

# index.py
from __future__ import print_function
import httplib,urllib
from passlib.hash import pbkdf2_sha256
from passlib.hash import pbkdf2_sha512
from passlib.hash import pbkdf2_sha1
import json


def lambda_handler(event, context):
    digest = event['digest']
    hash_pass = event['hash_pass']
    password = event['password']

    if digest == "sha256":
        verification = pbkdf2_sha256.verify(password, hash_pass)
    elif digest == "sha512":
        verification = pbkdf2_sha512.verify(password, hash_pass)
    else:
        verification = pbkdf2_sha1.verify(password, hash_pass)
    return verification

So I am expecting an dict event parameter that contains 3 keys whose values will be used to determine how we can verify an hash with a plaintext password.

event = {
  "digest": '',
  "hash_pass": '',
  "password": ''
}

With those assumptions in mind, let us now deploy this simple service onto AWS Lambda.

As we’re using an external library, we need to make sure that we also provide them to Lambda for use.

Download the dependencies onto the project root.

$ pip install passlib -t .

This should create a new directory called passlib in your project root.

Zip up all the files

$ zip -r lambda.zip *

Upload them onto AWS Lambda via AWS Console.

  1. Logon to AWS Management Console.
  2. Make sure you’re in one of the supported regions. For this example, just hit “Tokyo”.
  3. Select AWS Lambda in your AWS Dashboard
  4. Create a Lambda function
  5. Search and select “hello-world-python”
  6. Under “Configure function > Lambda function code”, select “Upload a .ZIP file” and upload lambda.zip
  7. Under “Lambda function handler and role”, your Handler should be in the following format <name_of_file>.<function_name>
  8. This would correspond to “index.lambda_handler” as our filename is index.py
  9. Create a new Role, “* Basic execution role”. You should be brought to a new page.
  10. Select IAM Role “lambda_basic_execution” and hit “Allow”
  11. You should be brought back to the previous page. You can hit next at this stage to finalize your function but take note of the following parameters:

Memory (MB) tells Lambda how much memory to use to run your function. Note that this does affect pricing and you should check out the AWS Lambda function for more details. For the purpose of this example, we can use the lowest available memory setting of 128MB.

Timeout tells Lambda how much time should elapse before it kills the script. This is important as in Lambda, you are charged for script duration and amount of memory used. I find a sensible 5 seconds to be good enough.

With that, you’ve created your very first Lambda function!

Exposing your Microservice to the Public

With your little password verification microservice done, you need a way to allow people to use it. Amazon API Gateway works great by providing you with a publicly accessible endpoint that you can define and direct to your created Lambda function.

Note: You can direct your API endpoint to other resources besides Lambda. For a whole list of available resources, do check out the Amazon API Gateway page for more details.

  1. Logon to AWS Management Console.
  2. Use the “Tokyo” region for easier coordination.
  3. Select API Gateway in your AWS Dashboard
  4. Hit the “Create API” button
  5. In API Name, fill in “Password API” or any name that you choose.
  6. You should now see a Resource called “/” being created.

Resources is the Rails equivalent of RESTFUL controllers. It is what your users access to consume your actions.

  1. Hit the “Actions” button and “Create Method”, you should see a dropdown appear just below “/” on the left of the API Gateway Dashboard.
  2. Select “POST” and hit the ’tick’ button next to it
  3. Under “Integration type” select “Lambda Function”
  4. In “Lambda Region” select “ap-northeast-1” to select the Tokyo Region
  5. Under “Lambda Function”, type in the name of the function you created above in the Lambda section.
  6. Select OK and you’re almost done!
  7. One final step is to make sure that all parameters that get sent to the API is properly forwarded to the Lambda function
  8. You should be at a page that shows the API flow right now, click on the link “Integration Request”
  9. Click on “Body Mapping Templates” to expand that section
  10. Click on “Add mapping template”
  11. Type in “application/json” under Content-Type and click on the ’tick'
  12. Now, at the top of the page, you should see a “Deploy API”, hit that
  13. In “Deployment stage” just select [New Stage] and type “production” under Stage name
  14. Hit the “Deploy” button.
  15. If everything is successful, you’ll see an Invoke URL appear. Copy and paste this URL. This is your public API.

Now if you send a POST to your Invoke URL with a body like the following, you should receive a response from your Lambda function that returns a true/false string.

{
  'digest': 'sha512',
  'hash_pass': 'hash_digest',
  'password': 'password_to_check'
}

What’s next?

You can fork, modify and submit pull requests on the microservice at its Github Repo

I’d recommend you also checkout Serverless a framework that was previously known as JAWS (Javascript Amazon Web Services). They are doing really cool stuff in this space.

References: