How can I retrieve all searchable (not deleted) documents in Amazon cloudsearch

Question

I want to retrieve all my searchable document from cloudsearch

I tried to do a negative search like that:

search-[mySearchEndPoint].cloudsearch.amazonaws.com/2011-02-01/search?bq=(not keywords: '!!!testtest!!!')

It work's but it also return all the deleted documents.

So how can I get all active document only?

Palacios

Hi. have you found a solution for the issue? – Tedie

No, not yet. I don't think it's possible – Palacios

Answer 1

The key thing to know is that CloudSearch doesn't really delete. Instead, the "delete" function retains IDs in the index, but clears all fields in those deleted docs, including setting uint fields to 0. This works fine for positive queries, which will match no text in the cleared, "deleted" docs.

A workaround is to add a uint field to your docs, called 'updated' below, to use as a filter for queries that might return deleted IDs, such as negative queries.

(The samples below uses the Boto interface library for CloudSearch, with many steps omitted for brevity.)

When you add docs, set the field to the current timestamp

doc['updated'] = now_utc  # unix time in seconds; useful for 'version' also.
doc_service.add(id, now_utc, doc)
conn.commit()

when you delete, CloudSearch sets uint fields to 0:

doc_service.delete(id, now_utc)
conn.commit()
# CloudSearch sets doc's 'updated' field = 0

Now you can distinguish between deleted and active docs in a negative query. The samples below are searching a test index with 86 docs, about half of them deleted.

# negative query that shows both active and deleted IDs
neg_query = "title:'-foobar'"
results = search_service.search(bq=neg_query)
results.hits  # 86 docs in a test index

# deleted items
deleted_query = "updated:0"
results = search_service.search(bq=deleted_query)
results.hits  # 46 of them have been deleted

# negative, filtered query that lists only active
filtered_query = "(and updated:1.. title:'-foobar')"
results = search_service.search(bq=filtered_query)
results.hits  # 40 active docs

Morganmorgana

Answer 2

I think you can do that like this:

search-[mySearchEndPoint].cloudsearch.amazonaws.com/2011-02-01/search?bq=-impossibleTermToSearch

Attention to the '-' in the begin of the term

Sheehan

Thanks. I tried that too, with the same result that it return me the deleted documents too – Palacios