How To: Access Your AWS VPC-based Elasticsearch Cluster Locally

AWS recently announced that their Elasticsearch Service now supports VPC, which is awesome, for a number of reasons:

1. No more signing every request

Remember this?

javascriptCopy code
let creds = new AWS.SharedIniFileCredentials({ profile: esProfile });
let signer = new AWS.Signers.V4(req, "es");
signer.addAuthorization(creds, new Date());

Every request had to be signed with AWS's SigV4 so that the Elasticsearch endpoint could be properly authorized. That meant additional code to sign all your requests, and additional time for the endpoint to decode it. It might only be a few milliseconds of extra processing time, but those can add up. Now we can call our VPC Elasticsearch endpoint with a simple HTTP request.

2. No need to set up NATs or Internet Gateways

If your apps don't require outgoing access to the Internet, there is no longer a need to set up NATs and IGs to access your Elasticsearch cluster. This saves you both complexity and money by not needing to maintain the extra configurations. Especially in a multi-availability zone deployment.

3. More secure, no more publicly available URLs protected by weak IP restrictions

Elasticsearch has no built-in security, so we used to simply restrict access to our EC2 instances that were running ES using security groups. AWS's Elasticsearch Service, however, only allowed for a publicly accessible URL, requiring additional levels of security to authorize access, like signing the request. This meant managing your cluster locally from the command line, or accessing Kibana, required you to compromise security by authorizing specific IP addresses to have access to the cluster. This was a terrible idea and opened up huge security risks. VPC-based ES clusters are no longer publicly accessible, which closes that security hole.

Accessing Your Elasticsearch Cluster Locally

All this new VPC stuff is great, but if you read the ES documentation you probably noticed this:

To access the default installation of Kibana for a domain that resides within a VPC, users must first connect to the VPC. This process varies by network configuration, but likely involves connecting to a VPN or corporate network.

This is also true if you want to access your ES cluster from the command line. The URL is no longer publicly accessible, and in fact, routes to an internal VPC IP address. If you already have a VPN solution that allows you to connect to your VPC, then configuring your security groups correctly should work. However, if you don't have a VPN configured, you can solve your problem using a simple SSH tunnel with port forwarding.

Step 1:

You need to have an EC2 instance running in the same VPC as your Elasticsearch cluster. If you don't, fire up a micro Linux instance with a secure key pair.

NOTE: Make sure your instance's security group has access to the Elasticsearch cluster and that your Elasticsearch cluster's access policy uses the "Do not require signing request with IAM credential" template.

Step 2:

Create an entry in your SSH config file (~/.ssh/config on a Mac):

defaultCopy code
# Elasticsearch Tunnel
Host estunnel
HostName 12.34.56.78 # your server's public IP address
User ec2-user
IdentitiesOnly yes
IdentityFile ~/.ssh/MY-KEY.pem
LocalForward 9200 vpc-YOUR-ES-CLUSTER.us-east-1.es.amazonaws.com:443

NOTE: The "HostName" should be your instance's PUBLIC IP address or DNS. "User" should be your Linux distro's default user (ec2-user if using Amazon Linux).

Step 3:

Run ssh estunnel -N from the command line

Step 4:

localhost:9200 should now be forwarded to your secure Elasticsearch cluster.

Access via a web browser, ignore the invalid SSL certificate:

Search: https://localhost:9200

Kibana: https://localhost:9200/\_plugin/kibana

Access via cURL, be sure to use the -k option to ignore the security certificate:

curl -k https://localhost:9200/

Access programmatically (example in Node.js). Be sure to use the corresponding option to ignore SSL certificates (as in strictSSL below):

javascriptCopy code
const REQUEST = require("request-promise");

const options = {
  method: "GET",
  uri: "https://localhost:9200/",
  strictSSL: false,
  json: true,
};

REQUEST(options).then((res) => {
  console.log(res);
});

And that's it! Now you can take advantage of the benefits of VPC-based Elasticsearch clusters and still maintain your local development workflows.