Overview

1. Create a Lambda function placeholder
2. Create an AWS API Gateway setup
3. Creating an usage plan and API Key setup
4. The Lambda Function Setup
5. Calling the API
6. Relax

Step 1: Create a Lambda Function Placeholder

Simply head over to the Lambda section in AWS and create a new function. For now all we need is a placeholder, so select “Create new function” and “Author from scratch”. I’ll be using Node JS (version 14.x as of writing this post) as the language for the function.

No need to do anything with the function, we just need it to exist for now :) Step 2: Creating the AWS API Gateway Setup

Head on over to the API Gateway section of AWS and hit “Create API”. It should look something like this:

We’ll be making a REST API, well not really but it comes with a bunch of goodies that will come in handy later. In the next dialog choose REST, New API, a name, and Edge optimized as the Endpoint type like so:

Now we got a blank API template. Time to add a resource. For our purposes here, effectively an endpoint as it will become part of the URL to call later. Give it a name and resource path (that’s the URL we’ll call later). Do NOT check proxy resource. Enable CORS if you need to call the endpoint via client side browser JS (we don’t since it will be a webhook ingestion endpoint).

Next we’ll create a method. Generally speaking webhooks are always sent as POST requests, so we’ll make one for that resource we just created. This is where the magic starts. Choose Lambda Function as the integration type and point it at the placeholder lambda you created in step 1.

So now that we have a POST endpoint, that will call a Lambda function, time to secure it a bit. Click on “Method Request” and configure the method to require an API key. Don’t forget to hit the little check mark next to “true”, otherwise it won’t save.

Step 3: Creating an Usage Plan and API Key

Alright, now the method requires an API key, let’s make one. In the left hand menu go to “Usage Plans” and hit “create”. In the following dialog, you can set throttling and quota limits, which is a good idea to do to avoid getting huge bills if the key is compromised.

Usage plans only work with API Stages, which is AWS’s fancy way of saying that you need to deploy the API first. So head back to the API section, select your API and hit deploy. Select “[New Stage]” unless you already have one, I’m naming mine v1 and hit deploy.

You’ll see the generated endpoint URL, with the v1 base path in my case. Your resource is accessible after that path. So if your resource is called webhook-xyz your full URL would be https://r4b4gvdfe9.execute-api.us-east-2.amazonaws.com/v1/webhook-xyz. Of course it will just give you 403 errors since it requires an API key, so back to it!

Back in the usage plan you created earlier, simply add the API stage you just deployed like so:

Now all that’s left here is to create the API key and add it to the usage plan. So head over to the API key section on the left hand menu, name and generate an API key and add it to the usage plan.

That’s it, the API key and usage plan will protect your endpoint which is already hooked up to the demo lambda function. You can test it out by sending a POST request to that URL. The way you include the API key in your request is the X-API-KEY header. This will return the lambda default Hello from Lambda!.

Step 4: The Lambda Function Setup

This part of course varies from service to service that you want to ingest but the rough outline is this:

1. Create a local dev folder for the function
2. Add the RudderStack Node SDK
3. Write a couple of bash scripts to build and package the function
4. Upload it to Lambda

I’ve prepared the boilerplate for this and published it on github. Enjoy!

https://github.com/maximum-pixels/rudderstack-awslambda-boilerplate

Simply run the numbered bash scripts in order and upload the resulting ZIP file to the Lambda function we created initially.

Step 5: Calling the API

As mentioned above you can manually call the API with the URL AWS gives you (in this tutorial case https://r4b4gvdfe9.execute-api.us-east-2.amazonaws.com/v1/webhook-xyz ) by providing it with the X-API-KEY header for authentication.

If your specific webhook service does not let you set extra headers, you can skip the API key parts of this tutorial. Adding other authentication methods is out of the scope of this post.

Step 6: Relax

I hope this helps you in your data engineering endeavors!