Update existing Log Group using CloudFormation

Question

I have a lambda which has a log group, say LG-1, for which retention is set to Never Expire (default). I need to change this Never Expire to 1 month. I am doing this using CloudFormation. As the log group already exists, when I am trying to deploy my lambda again with the changes in template as :

LambdaFunctionLogGroup:
Type: 'AWS::Logs::LogGroup'
DependsOn: MyLambda
Properties:
  RetentionInDays: 30
  LogGroupName: !Join 
    - ''
    - - /aws/lambda/
      - !Ref MyLambda

the update is failing with error :

[LogGroup Name] already exists.

One possible solution is to delete the log group and then again create it with new changes as shown above which works perfectly well.

But I need to do it without deleting the log group as it will result in the deletion of all the previous logs that I have.

Is there any workaround which is possible ?

Viperish

Answer 1

@ttulka answered:

".. it is impossible to manipulate resources from CF which already exist out of the stack."

But actually the problem is more general than that and applies to resources created inside of the stack. It has to do with AWS CloudFormation resource "Replacement policy". For some resources the way CloudFormation "updates" the resource is to create a new resource, then delete the old resource (this is called the "Replacement" update policy). This means there is a period of time where you've got two resources of the same type with many of the same properties existing at the same time. But if a certain resource property has to be unique, the two resource can't exist at the same time if they have the same value for this property, so ... CloudFormation blows up.

AWS::Logs::LogGroup.LogGroupName property is one such property. AWS::CloudWatch::Alarm.AlarmName is another example.

A work around is to unset the name so that a random name is used, perform an update, then set the name back to it's predictable fixed value and update again.

---

Rant: It's an annoying problem that really shouldn't exist. I.e. AWS CF should be smart enough to not have to use this weird clunky resource replacement implementation. But ... that's AWS CF for you ...

Devoir

Answer 2

I think it is impossible to manipulate resources from CF which already exist out of the stack.

One workaround would be to change the name of the Lambda like my-lambda-v2 to keep the old log group together with the new one.

After one month you can delete the old one.

Measles

@Measles the log group that is initially present was created through CF only but as a part of lambda execution policy using logs::CreateLogGroup and not AWS::Logs::LogGroup. – Viperish

@Viperish If your CF template doesn't include the resource AWS::Logs::LogGroup, it means, the log group was created by the lambda itself with its very first execution, NOT by CF. – Measles

@Measles Oh I see. So basically it was my lambda that was creating the log group and not CF ! Thanks :) – Viperish

@Measles Exactly. You can accept my answer if it helped. – Measles

Answer 3

Use customresource Backed lambda within your cloudformation template. The custom resource would be triggered automatically the first time and update your retention policy of the existing log group. If you need it you custom resource lambda to be triggered every time, then use a templating engine like jinja2.

import boto3

client = boto3.client('logs')
response = client.put_retention_policy(
    logGroupName='string',
    retentionInDays=123
)

You can basically make your CF template do (almost) anything you want using Custom Resource

More information (Boto3, you can find corresponding SDK for the language you use) - https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/logs.html#CloudWatchLogs.Client.put_retention_policy

EDIT: Within the CloudFormation Template, it would look something like the following:

  LogRetentionSetFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src
      Handler: set_retention_period.handler
      Role: !GetAtt LambdaRole.Arn
      DeploymentPreference:
        Type: AllAtOnce

  PermissionForLogRetentionSetup:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      FunctionName:
        Fn::GetAtt: [ LogRetentionSetFunction, Arn ]
      Principal: lambda.amazonaws.com

  InvokeLambdaFunctionToSetLogRetention:
    DependsOn: [PermissionForLogRetentionSetup]
    Type: Custom::SetLogRetention
    Properties:
      ServiceToken: !GetAtt LogRetentionSetFunction.Arn
      StackName: !Ref AWS::StackName
      AnyVariable: "Choose whatever you want to send"     
      Tags:
        'owner': !Ref owner
        'task': !Ref task

The lambda function would have the code which sets up the log retention as per the code which I already specified before.

For more information, please google "custom resource backed lambda". Also to get you a head start I have added the ink below: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html

Dogwatch

Can you explain it in terms of CF template not code ? I did not get "Use customresource Backed lambda within your cloudformation template". What custom resource ? – Viperish

Please refer to the edit to my originally provided answer! – Dogwatch

1. I will have to do this change in the existing lambda template or create a new ? 2. So there will be a new lambda that will be created and that lambda will be triggered by old lamba's CF template and new lambda will update old lambdas retention ? I am finding this custom resource really difficult to understand and implement – Viperish

1. You can add this in the same template where you have your lambda, you wouldn't have to create anything new. 2. Exactly, custom resources are triggered once (unless you add do something to change resource name dynamically) 3. Custom resources (Lambda backed) gives you the freedom to perform anything which you can't do using CloudFormation directly. Let me know if you have any specific questions regarding the implementation of this solution. – Dogwatch

Answer 4

The Log Group's retention can be updated by importing the Lambda's Log Group to the stack.

Note:

• It shouldn't have a retention set yet. You will do it later.

• The import resource must include a DeletionPolicy attribute.

Once imported, you can deploy your new template with the desired retention period set.

Auld

Answer 5

You can use the construct that I published, that automatically sets a defined Retention when new LogGroups (even those unmaintained by CF) are created.

See here: https://constructs.dev/packages/cloudwatch-retention-setter/

Indigenous