Ingesting a Webhook with AWS API Gateway and RudderStack

By Max Werner

On Jul 30, 2021

Updated Nov 13, 2021

Cloud Technologies
AWS API Gateway, Lambda, RudderStack Logos


  1. Create a Lambda function placeholder
  2. Create an AWS API Gateway setup
  3. Creating an usage plan and API Key setup
  4. The Lambda Function Setup
  5. Calling the API
  6. Relax

Step 1: Create a Lambda Function Placeholder

Simply head over to the Lambda section in AWS and create a new function. For now all we need is a placeholder, so select “Create new function” and “Author from scratch”. I’ll be using Node JS (version 14.x as of writing this post) as the language for the function.

No need to do anything with the function, we just need it to exist for now :) Step 2: Creating the AWS API Gateway Setup

Head on over to the API Gateway section of AWS and hit “Create API”. It should look something like this:

AWS API Gateway Overview screen

We’ll be making a REST API, well not really but it comes with a bunch of goodies that will come in handy later. In the next dialog choose REST, New API, a name, and Edge optimized as the Endpoint type like so:

AWS API Gateway New API Configuration Screen

Now we got a blank API template. Time to add a resource. For our purposes here, effectively an endpoint as it will become part of the URL to call later. Give it a name and resource path (that’s the URL we’ll call later). Do NOT check proxy resource. Enable CORS if you need to call the endpoint via client side browser JS (we don’t since it will be a webhook ingestion endpoint).

AWS API Gateway New Resource Menu

AWS API Gateway Resource Config Menu

Next we’ll create a method. Generally speaking webhooks are always sent as POST requests, so we’ll make one for that resource we just created. This is where the magic starts. Choose Lambda Function as the integration type and point it at the placeholder lambda you created in step 1.

AWS API Gateway Create Method Menu

AWS API Gateway Method Config Menu

So now that we have a POST endpoint, that will call a Lambda function, time to secure it a bit. Click on “Method Request” and configure the method to require an API key. Don’t forget to hit the little check mark next to “true”, otherwise it won’t save.

AWS API Gateway Method Authentication

AWS API Gateway Method Authentication Details

Step 3: Creating an Usage Plan and API Key

Alright, now the method requires an API key, let’s make one. In the left hand menu go to “Usage Plans” and hit “create”. In the following dialog, you can set throttling and quota limits, which is a good idea to do to avoid getting huge bills if the key is compromised.

AWS API Gateway Usage Plans Menu

Usage plans only work with API Stages, which is AWS’s fancy way of saying that you need to deploy the API first. So head back to the API section, select your API and hit deploy. Select “[New Stage]” unless you already have one, I’m naming mine v1 and hit deploy.

AWS API Gateway Deploy API Menu

AWS API Gateway Deploy API Config

You’ll see the generated endpoint URL, with the v1 base path in my case. Your resource is accessible after that path. So if your resource is called webhook-xyz your full URL would be Of course it will just give you 403 errors since it requires an API key, so back to it!

AWS API Gateway Deployed API URL

Back in the usage plan you created earlier, simply add the API stage you just deployed like so:

AWS API Gateway Add API Stage

Now all that’s left here is to create the API key and add it to the usage plan. So head over to the API key section on the left hand menu, name and generate an API key and add it to the usage plan.

AWS API Gateway Create API Key

AWS API Gateway Associate API key with Plan

That’s it, the API key and usage plan will protect your endpoint which is already hooked up to the demo lambda function. You can test it out by sending a POST request to that URL. The way you include the API key in your request is the X-API-KEY header. This will return the lambda default Hello from Lambda!.

Step 4: The Lambda Function Setup

This part of course varies from service to service that you want to ingest but the rough outline is this:

  1. Create a local dev folder for the function
  2. Add the RudderStack Node SDK
  3. Write a couple of bash scripts to build and package the function
  4. Upload it to Lambda

I’ve prepared the boilerplate for this and published it on github. Enjoy!

Simply run the numbered bash scripts in order and upload the resulting ZIP file to the Lambda function we created initially.

Step 5: Calling the API

As mentioned above you can manually call the API with the URL AWS gives you (in this tutorial case ) by providing it with the X-API-KEY header for authentication.

If your specific webhook service does not let you set extra headers, you can skip the API key parts of this tutorial. Adding other authentication methods is out of the scope of this post.

Step 6: Relax

I hope this helps you in your data engineering endeavors!

© 2024 Obsessive Analytics Consulting. All rights reserved.