How to modify AWS EventBridge Rule to use AND instead of OR filter logic?

Question

I want to trigger an AWS lambda function via EventBridge every time an S3 Object is created in an S3 bucket called "mybucket", but ONLY if its name/key ends with a ".csv"-suffix AND if it was created within the "in"-folder of that bucket. The EventBridge Rule that I currently have is this:

{
  "detail-type": ["Object Created"],
  "source": ["aws.s3"],
  "detail": {
    "bucket": {
      "name": ["mybucket"]
    },
    "object": {
      "key": [{
        "suffix": ".csv"
      }, {
        "prefix": "in/"
      }]
    }
  }
}

I would actually expect this rule to work the correct way BUT it is not, instead it behaves as if there was an OR relation between the suffix and prefix filter conditions. As I understand the AWS Documentation (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns-content-based-filtering.html#eb-filtering-complex-example) the above rule should define an AND relation between the suffix and prefix filter conditions similar to this example given in the documentation:

{
  "time": [ { "prefix": "2017-10-02" } ],
  "detail": {
    "state": [ { "anything-but": "initializing" } ],
    "c-count": [ { "numeric": [ ">", 0, "<=", 5 ] } ],
    "d-count": [ { "numeric": [ "<", 10 ] } ],
    "x-limit": [ { "anything-but": [ 100, 200, 300 ] } ]
  }
}

Whereas an OR relation would require an extra $or-syntax as in this example given in the documentation:

{
  "detail": {
    "$or": [
      { "c-count": [ { "numeric": [ ">", 0, "<=", 5 ] } ] },
      { "d-count": [ { "numeric": [ "<", 10 ] } ] },
      { "x-limit": [ { "numeric": [ "=", 3.018e2 ] } ] }
    ]
  }
}

So, why is my rule behaving as if there was an OR relation between the suffix and prefix conditions? And what do I need to change to get it working the way I want?

Kelby

S3 object key starts with leading slash, try changing in/ to /in/. – Samuele

@AnkushJain S3 keys and prefixes don't typically start with forward slash. – Blanding

@Blanding May be I am wrong. Thanks for confirming. – Samuele

It is as @Blanding stated. The s3 object keys are not starting with a forward slash. I can actually see the object keys in the logged events, and they look like: "in/ipsumlorem...". My problem is not that the prefix or suffix filter rules of the "Object Created" events are not working at all but that they are not working correctly in combination. I'm getting an OR behavior where I would expect an AND behavior. – Kelby

@Kelby Can you please try [{ "suffix": ".csv" , "prefix": "in/" }] ? One item in an array instead 2 items. – Samuele

@AnkushJain I am using AWS Cloudformation to deploy my infrastructure. I have tried your solution but Cloudformation fails with the following error: Event pattern is not valid. Reason: Only one key allowed in match expression at [Source: (String)"{"detail-type":["Object Created"],"source":["aws.s3"],"detail":{"bucket":{"name":["mybucket"]},"object":{"key":[{"prefix":"in/","suffix":".csv"}]}}}"; line: 1, column: 151] (Service: AmazonCloudWatchEvents; Status Code: 400; Error Code: InvalidEventPatternException; Request ID: ...; Proxy: null) – Kelby

Answer 1

This is not possible at the moment

There are two ways to setup something that "look like and operator` and do not throw syntax errors, but they will work differently:

1. Two keys with different filter (as proposed by Peter Bouwdewijn) - the latter filter will overwrite the former, so it will only filter by suffix, prefix will be completely ignored:

   "key": [{"prefix": "example/directory/"}],
   "key": [{"suffix": ".png"}]
   

   Input "key": "other/directory/image.png" will be valid
2. Provide two filter objects in the same array - they will act as OR operator:

   "key": [{"prefix": "example/directory/"}, {"suffix": ".png"}]
   

   Both inputs "key": "other/directory/image.png" and "key": "example/directory/not_image.txt" will be valid

See Content-based filtering and Arrays pages of the AWS EventBridge documentation for more info

Boundless

Answer 2

Its possible now with wildcards, your just update your key:

{
  "detail-type": ["Object Created"],
  "source": ["aws.s3"],
  "detail": {
    "bucket": {
      "name": ["mybucket"]
    },
    "object": {
      "key": [{
        "wildcard": "in/*.csv"
      }]
    }
  }
}

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns-content-based-filtering.html#eb-filtering-wildcard-matching

Elite

Answer 3

Sounds like the exact problem I am facing. I found something in IBM docs: https://www.ibm.com/docs/en/dsm?topic=csqcson-forwarding-objectcreated-notifications-sqs-queue-by-using-amazon-eventbridge

There they state to repeat the key

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["<example-bucket>"]
    },
    "object": {
      "key": [{
        "prefix": "example/directory/"
      }],
      "key": [{
        "suffix": ".png"
      }]
    }
  }
}

This is very counter intuitive and even goes against the docs from AWS. I have not tried it yet.

Holsinger

Any way to filter out suffix in a anything-but? "SourceIdentifier": [{ "anything-but": { "suffix": "-test" } }] – Success