Ingesting a Webhook with AWS API Gateway and RudderStack

By Max Werner, published: 2021-07-30, Updated: 2021-11-13

AWS & RudderStack Logos

This post will show you step-by-step how to setup a simple way to ingest an incoming webhook with RudderStack, using a AWS Lambda Function, hooked up to the AWS API Gateway, with built-in API key security!

Screenshot of AWS API Gateway Landing Page

Overview

  1. Create a Lambda function placeholder
  2. Create an AWS API Gateway setup
  3. Creating an usage plan and API Key setup
  4. The Lambda Function Setup
  5. Calling the API
  6. Relax

Step 1: Create a Lambda Function Placeholder

Simply head over to the Lambda section in AWS and create a new function. For now all we need is a placeholder, so select “Create new function” and “Author from scratch”. I’ll be using Node JS (version 14.x as of writing this post) as the language for the function.

No need to do anything with the function, we just need it to exist for now :)

Step 2: Creating the AWS API Gateway Setup

Head on over to the API Gateway section of AWS and hit “Create API”. It should look something like this:

Screenshot of AWS API Gateway Creation Page

We’ll be making a REST API, well not really but it comes with a bunch of goodies that will come in handy later. In the next dialog choose “REST”, “New API”, a name, and “Edge optimized” as the Endpoint type like so:

Screenshot of AWS API Gateway Resource Menu

Now we got a blank API template. Time to add a resource. For our purposes here, effectively an endpoint as it will become part of the URL to call later. Give it a name and resource path (that’s the URL we’ll call later). Do NOT check “proxy resource”. Enable CORS if you need to call the endpoint via client side browser JS (we don’t since it will be a webhook ingestion endpoint).

Screenshot of AWS API Gateway Resource Creation Details Page
Screenshot of AWS API Gateway Method Menu

Next we’ll create a method. Generally speaking webhooks are always sent as POST requests, so we’ll make one for that resource we just created. This is where the magic starts. Choose “Lambda Function” as the "integration type" and point it at the placeholder lambda you created in step 1.

Screenshot of AWS API Gateway Method Setup Page
Screenshot of AWS API Gateway Method Overview Page

So now that we have a POST endpoint, that will call a Lambda function, time to secure it a bit. Click on “Method Request” and configure the method to require an API key. Don’t forget to hit the little check mark next to “true”, otherwise it won’t save.

Screenshot of AWS API Gateway Method Security Page
Screenshot of AWS API Gateway Usage Plan Page

Step 3: Creating an Usage Plan and API Key

Alright, now the method requires an API key, let’s make one. In the left hand menu go to “Usage Plans” and hit “create”. In the following dialog, you can set throttling and quota limits, which is a good idea to do to avoid getting huge bills if the key is compromised.

Screenshot of AWS API Gateway API Deployment Menu

Usage plans only work with API Stages, which is AWS’s fancy way of saying that you need to deploy the API first. So head back to the API section, select your API and hit deploy. Select “[New Stage]” unless you already have one, I’m naming mine v1 and hit deploy. You’ll see the generated endpoint URL, with the v1 base path in my case. Your resource is accessible after that path. So if your resource is called webhook-xyz your full URL would be https://r4b4gvdfe9.execute-api.us-east-2.amazonaws.com/v1/webhook-xyz. Of course it will just give you 403 errors since it requires an API key, so back to it!

Screenshot of AWS API Gateway Deployment Page
Screenshot of AWS API Gateway API Base URL
Screenshot of AWS API Gateway Usage Plan Details Page

Back in the usage plan you created earlier, simply add the API stage you just deployed like so:

Screenshot of AWS API Gateway API Key Generation Page

Now all that’s left here is to create the API key and add it to the usage plan. So head over to the API key section on the left hand menu, name and generate an API key and add it to the usage plan.

Screenshot of AWS API Gateway API Key to Usage Plan Settings Page

That’s it, the API key and usage plan will protect your endpoint which is already hooked up to the demo lambda function. You can test it out by sending a POST request to that URL. The way you include the API key in your request is the X-API-KEY header. This will return the lambda default Hello from Lambda!.

Step 4: The Lambda Function Setup

This part of course varies from service to service that you want to ingest but the rough outline is this:

  1. Create a local dev folder for the function
  2. Add the RudderStack Node SDK
  3. Write a couple of bash scripts to build and package the function
  4. Upload it to Lambda

I’ve prepared the boilerplate for this and published it on github. Enjoy!

https://github.com/maximum-pixels/rudderstack-awslambda-boilerplate

Simply run the numbered bash scripts in order and upload the resulting ZIP file to the Lambda function we created initially.

Step 5: Calling the API

As mentioned above you can manually call the API with the URL AWS gives you (in this tutorial case https://r4b4gvdfe9.execute-api.us-east-2.amazonaws.com/v1/webhook-xyz ) by providing it with the X-API-KEY header for authentication.

If your specific webhook service does not let you set extra headers, you can skip the API key parts of this tutorial. Adding other authentication methods is out of the scope of this post.

Step 6: Relax

I hope this helps you in your data engineering endeavors!