Sadly, this isn't something I've discovered myself. However, it's hard to figure out and info hard to come by. You setup AWS EventBridge(formerly CloudWatch Events) rule with CloudWatch log group as a target by any means except web console. Events start flowing in and then - nothing. All you get are failed invocations, resp. FailedInvocation metric going up. EventBridge seems to be hard to debug since there are no logs what so ever which is supported here. Quote:

I just confirmed from internal cloudwatch team, cloudwatch do not provide any logs for failed invocation. Apart from the failedinvocation metrics, there is no logging avaialble from cloudwatch side. As mentioned, you need to rely on lambda logs or resources logs.

Failed invocation can be caused by event pattern, input transformer, input template, IAM policy in case of Lambda and ECS - you name it. Or it can be caused by a missing CloudWatch log resource policy.

data "aws_iam_policy_document" "trust_events_to_store_log_event" {
  statement {
    sid = "TrustEventsToStoreLogEvent"

    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = [
      "arn:aws:logs:us-east-1:111111111111:log-group:/aws/events/*:*"
    ]

    principals {
      identifiers = [
        "delivery.logs.amazonaws.com",
        "events.amazonaws.com"
      ]
      type = "Service"
    }
  }

  version = "2012-10-17"
}

resource "aws_cloudwatch_log_resource_policy" "trust_events_to_store_log_event" {
  policy_document = data.aws_iam_policy_document.trust_events_to_store_log_event.json
  policy_name     = "TrustEventsToStoreLogEvent"
}

As it turns out such policy is added(behind the scenes, of course) when you setup things through web console. And as far as I know these policies aren't visible anywhere in web console(ok, I haven't figured out where). One can list these policies through AWS CLI by aws logs describe-resource-policies, though. Once this policy is set, even by conducting some tests in the early life of an AWS account, through web console things magically work. However, it can be very confusing when things don't work and all you know it must be some kind of magic, but you don't know what kind.

See this GitHub issue for more info.

— Zdenek Styblik 2022/05/14 09:08