Function not found after manually deleting a function in a SAM CloudFormation stack

Question

I am using sam deploy to deploy lambda function and API gateway. It works fine but it doesn't work after I manually deleted the lambda function via AWS console. I got below error:

"ResourceStatusReason": "Function not found: 
  arn:aws:lambda:ap-southeast-2:286334053171:function:polaroid (Service: 
  AWSLambdaInternal; Status Code: 404; Error Code: ResourceNotFoundException;
  Request ID: b431cbfc-7772-11e9-8022-1b92fa2cfa9e)

What is the proper way to delete the lambda and do a refresh deployment? If this happens, how can I force SAM to create the missing lambda function?

My lambda in template yaml looks like:

...
Resources:
  PolaroidFunction:
    Type: AWS::Serverless::Function 
    Properties:
      FunctionName: test
      CodeUri: ./lambdas
      Handler: lib/index.fun
      Runtime: nodejs8.10
      Events:
        polaroid:
          Type: Api 
          Properties:
            Path: /test
            Method: post
...

Lipson

Why delete it? To "refresh" it just do sam package/sam deploy, without manually deleting anything, and if there are any changes to your function they will be deployed. – Mop

I know that but I already deleted it. What can I do to bring it back? – Lipson

If you manually recreate that lambda with the same name, that should sort it. – Mop

Is there an automatic way to sort it? The deploy script is written in CI and the lambda is shared with the team. I dont want CI fail because any developer delete the lambda manually. I have tried --force switch but it doesn't seem to work. – Lipson

Answer 1

I guess you already learnt the hard way that you should never manually delete resources managed by SAM or CloudFormation.

In general, if you just want to change the function, you can just call sam build and sam deploy, and the new version of it will be deployed. There is no need to delete anything. If you need a more advanced workflow, you will need to read blog posts. There is no one right way to do this.

To fix your immediate problem however, here is what you can do.¹

Firstly, you need to get the generated AWS CloudFormation template:

▶ aws cloudformation get-template --stack-name HelloWorld \
    --template-stage Processed --query TemplateBody | cfn-flip -y > processed.yml

Next, you need to comment out the function in the processed.yml file you just created, and also comment out the Lambda Permissions that refer to it. Save a backup of the original processed.yml file too.

Also, update any other template references to it if possible with the actual values CloudFormation computed when you built the stack, by getting them from your AWS console. For example, if you had references to ${HelloWorldFunction.Arn} you might have to update those references in the template with a string like arn:aws:lambda:ap-southeast-2:123456789012:function:HelloWorld-HelloWorldFunction-1NJGQI7GEAUM1.

Next, validate the template using AWS CloudFormation commands:

▶ aws cloudformation validate-template --template-body file://processed.yml
{
    "CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::Role]",                                                                         
    "Description": "sam-app\nSample SAM Template for sam-app\n",
    "Parameters": [],
    "Capabilities": [
        "CAPABILITY_IAM"
    ]
}

Next, you will update the stack using this modified template. By updating the stack this way, you get your template and real state to be back in sync from CloudFormation's point of view:

▶ aws cloudformation update-stack --template-body file://processed.yml --stack-name HelloWorld --capabilities CAPABILITY_IAM                        
{
    "StackId": "arn:aws:cloudformation:ap-southeast-2:885164491973:stack/HelloWorld/af2c6810-7884-11e9-9bb3-068b1a8e1450"
}

If all goes well, your stack goes into UPDATE_COMPLETE state. Great!

Finally, uncomment all the resources you commented out, and restore all the original values. Then update stack a second time, and your stack should be restored to its original state.

See also:

• AWS Knowledge Base, 2016, 2019, How do I update an AWS CloudFormation stack that's failing because of a resource that I manually deleted?.
• More on the cfn-flip utility, if you don't have it.

---

¹ Note that I tested this method using the default HelloWorld Python 2.7 example that ships with SAM.

Niall

Would it also be reasonable to simply delete the stack in the CloudFormation console, and then once that's done do a new serverless deploy? – Belligerence

Answer 2

So as suggested in other answers I deleted the function manually from the console.

I was deploying the stack from CDK

The solution

• comment the lambda function code (in cdk) of the function I deleted manually.
• Deploy stack
• Un-comment the code and deploy again

Gorgerin

And works for serveless.com framework, too! – Adrienadriena

Answer 3

I had a similar issue. In my case I had deleted the Lambda as an experiment while trying to reset the TRIM_HORIZON to get it to reprocess old events in a DynamoDB Stream.

I found a simpler solution:

Go into the CloudFormation Console and delete the deployed Stack.

sam deploy works fine again after that.

Watchband

Thanks, this is a lot simpler than the accepted answer. Worked for me. – Jitterbug

Don't do this in production. – Hesychast

Yes, the OP should put a disclaimer in the post that "don't do it in production", it will delete your entire system. Also, if you have s3 buckets in your stack, it would create new problems. – Gorgerin

Answer 4

If you want to avoid deleting the stack and deploying it back again, or avoid aligning the CloudFormation template file, perhaps you can just align the resources in AWS to the template file.

That means, if you deleted a certain Lambda (for example) that was created initially from the template file, just create the same Lambda MANUALLY in AWS (either GUI or aws cli). Now run 'sam deploy' again - you should be aligned.

Now remove the Lambda definition from the template file and deploy again - the Lambda should be removed and the CloudFormation will be aligned.

Quechuan