How to “Watch” Your Amazon S3 Repository with Voyager HQ

Automatically sync your search index with Amazon S3

Voyager Search’s newest release, Vose, is packed with REST APIs to integrate with external applications. One such REST API provides the ability to “scan” a repository for changes and update your Voyager index with those changes. With this API, we can automatically keep our search index synced with our S3 buckets utilizing Amazon Web Services (AWS) Lambda functions and S3 Bucket triggers.

In a matter of minutes, we can add an AWS Lambda Function that calls our REST API on our ec2 instance running Voyager Search software. S3 allows us to trigger our new Lambda Function when anything is added to our Bucket. And, voila! We now have our new S3 “watch” feature. And, it’s a self-contained microservice. No changes to the core Voyager software are needed to add this new feature! Cool, right?!

Ready to set yours up? Let’s do it! The process only takes a few short minutes.

Let’s assume we’re already using AWS and your Voyager instance is already running on ec2 and that it’s connected to an existing S3 bucket. We’re just going to add our new “watch” feature using an AWS Lambda Function.

Step 1:  Add a Lambda Function

First, select Services then Compute then Lambda.

Next, click Create function.

Then, we’ll create your function. We’re going to use Node.js here, but you can use the language of your choice.

Before creating the function, we’ll need to choose your Role as seen below.

Choose your role and then click Create function.

2. Add a Trigger to your Lambda Function

See screenshot below.

3. Add your Javascript code to call the REST API  

This is performing a simple HTTP PUT to our ec2 instance running Voyager, which will Scan our S3 Repository for changes and index only the changes so the new content is searchable.

Here we’ve hardcoded the HTTP options for simplicity sake, but we could also easily make these environment variables.

See full code here (it’s only 20ish lines of code without the AWS handler wrapper function).

4. Save it. That’s it!

Now, let’s test it.  

Step 1. Configure a new test event

Step 2. Choose event template and Create

Step 3. Run it! It works! Optional: Click logs to confirm/see what was logged.

Now, let’s really test it by manually adding something to our S3 Bucket and checking the results.

Choose Services then S3 and choose your bucket (assuming you know this part since you have a bucket).


I’ve uploaded the files above and more. Let’s hit our ec2 instance and see if we get them in our search results. Note: You’ll need to install Voyager Search software on your ec2 instance to do this step.

Success! The documents are now searchable and wee can see them in the search results below.

What’s Next

Watching your S3 Repository is just the beginning. Vose provides many new and interesting features as well as REST APIs that make running enterprise search a breeze to set up and augment, making it easier than ever to find data throughout the enterprise, whether in the cloud or on premise.

Web Design and Web Development by Buildable