Fulltext Search DynamoDB

Question

Following situation:

I´m storing elements in a DyanmoDb for my customers. HashKey is a Element ID and Range Key is the customer ID. In addition to these fields I´m storing an array of strings -> tags (e.g. ["Pets", "House"]) and a multiline text.

I want to provide a search function in my application, where the user can type a free text or select tags and get all related elements.

In my opinion a plain DB query is not the correct solution. I was playing around with CloudSearch, but I´m not really sure if this is the correct solution, because everytime the user adds a tag the index must be updated...

I hope you have some hints for me.

Anselmi

Answer 1

DynamoDB is now integrated with Elasticsearch, enabling you to perform full-text queries on your data.

https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-dynamodb-elasticsearch-integration/

DynamoDB streams are used to keep the search index up-to-date.

Thong

Thanks for that link. Will check it and share my results – Anselmi

Hey @SnowMax, do you have results to share? How was your experience with Dynamo + ES? Thanks – Revolving

@RenatoByrro EXPENSIVE – Redan

Answer 2

You can use an instant-search engine like Typesense to search through data in your DynamoDB table:

https://github.com/typesense/typesense

There's also ElasticSearch, but it has a steep learning curve and can become a beast to manage, given the number of features and configuration options it supports.

At a high level:

1. Turn on DynamoDB streams
2. Setup an AWS Lambda trigger to listen to these change events
3. Write code inside your lambda function to index data into Typesense:

def lambda_handler(event, context):
    client = typesense.Client({
        'nodes': [{
            'host': '<Endpoint URL>',
            'port': '<Port Number>',
            'protocol': 'https',
        }],
        'api_key': '<API Key>',
        'connection_timeout_seconds': 2
    })

    processed = 0
    for record in event['Records']:
        ddb_record = record['dynamodb']
        if record['eventName'] == 'REMOVE':
            res = client.collections['<collection-name>'].documents[str(ddb_record['OldImage']['id']['N'])].delete()
        else:
            document = ddb_record['NewImage'] # format your document here and the use upsert function to index it.
            res = client.collections['<collection-name>'].upsert(document)
            print(res)
        processed = processed + 1
        print('Successfully processed {} records'.format(processed))
    return processed

Here's a detailed article from Typesense's docs on how to do this: https://typesense.org/docs/0.19.0/guide/dynamodb-full-text-search.html

Adduct

Answer 3

DynamoDB just added PartiQL, a SQL-compatible language for querying data. You can use the contains() function to find a value within a set (or a substring): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ql-functions.contains.html

Thebes

You could do this, but it's equivalent to doing filtering like today. Without an index on the particular column, it's going to still perform a full table scan. PartiQL is only a SQL-like language for accessing the data but doesn't add any new functionality to DynamoDB. – Marketplace

Answer 4

In your specific case you need Elastic search. But you can do wildcard text search on sort-key,

/* Return all of the songs by an artist, matching first part of title */

SELECT * FROM Music
WHERE Artist='No One You Know' AND SongTitle LIKE 'Call%'; 

/* Return all of the songs by an artist, with a particular word in the title...
...but only if the price is less than 1.00 */

SELECT * FROM Music
WHERE Artist='No One You Know' AND SongTitle LIKE '%Today%'
AND Price < 1.00;

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SQLtoNoSQL.ReadData.Query.html

Caulis

Answer 5

This is the advantage of using dynamodb as a 'managed service' by aws. You get multiple components managed apart from the managed nosql db. If you are using the 'downloaded' version of dynamodb then you need to ' build your own ' elasticcluster and index the data in dynamodb .

Spraggins