chef-client cannot find cookbooks using berkshelf

Question

I'm new to chef and ran into problems with berkshelf and chef-client. I want to have my own cookbook with dependencies and apply it. My initial cookbook looks like this:

.
├── Berksfile
├── Berksfile.lock
├── chefignore
├── client.rb
├── Gemfile
├── Gemfile.lock
├── metadata.rb
├── README.md
└── recipes
    └── default.rb

#./Berksfile
source 'https://supermarket.getchef.com'

metadata

cookbook 'znc'


#./client.rb
cookbook_path '~/.berkshelf/cookbooks'

and when I run sudo bundle exec chef-client -z -o znc --config=client.rb the chef-client cannot find the cookbook:

Starting Chef Client, version 11.16.4
[2014-10-25T15:34:59+02:00] WARN: Run List override has been provided.
[2014-10-25T15:34:59+02:00] WARN: Original Run List: []
[2014-10-25T15:34:59+02:00] WARN: Overridden Run List: [recipe[znc]]
resolving cookbooks for run list: ["znc"]

================================================================================
Error Resolving Cookbooks for Run List:
================================================================================

Missing Cookbooks:
------------------
No such cookbook: znc

Expanded Run List:
------------------
* znc


Running handlers:
[2014-10-25T15:34:59+02:00] ERROR: Running exception handlers
Running handlers complete
[2014-10-25T15:34:59+02:00] ERROR: Exception handlers complete
[2014-10-25T15:34:59+02:00] FATAL: Stacktrace dumped to /home/sebastian/.chef/local-mode-cache/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.474758165 seconds
[2014-10-25T15:34:59+02:00] ERROR: 412 "Precondition Failed "
[2014-10-25T15:34:59+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

also note:

ls ~/.berkshelf/cookbooks 
build-essential-2.1.2  znc-0.0.1

Any suggestions why the cookbooks cannot be found?

EDIT:

Thanks for the quick answer. The solution was as John Bellone said to bundle exec berks vendor and change my client.rb configuration to:

# ./client.rb
cookbook_path File.dirname(File.expand_path(__FILE__)) + '/berks-cookbooks'

Obelia

Answer 1

There are a few directories that Chef Client will look for cookbooks by default. One of which is the $CWD/cookbooks which we can take advantage of using the berks vendor cookbooks command. I find it much easier to do this during my testing (especially with an automated process) as it ensures that I get a fresh copy of the cookbooks and not a cached, stale one.

If you are looking to simply get an environment up and running I would suggest using a Vagrantfile with the vagrant-berkshelf plugin. This will automatically vendor all of the cookbooks in a cache directory and upload it to the guest machine. It will even work if the provider that you are using is a cloud one, e.g. Amazon, Backspace, as it will use rsync instead of the shared directories.

Skylight

Thanks for explaining about the default $CWD/cookbooks. I was getting the same No such cookbook error but for different reason than the question here. I had a typo in the folder name. It was cookbook in singular instead of plural! (embarrassing) – Hukill

Answer 2

I'm using the following script, this allows me to use a git repo with my cookbooks that depend on cookbooks in the supermarket but without a chef server. Put below content in a file called local-chef-client at the same level as your cookboks dir. You can then converge the local machine with ./local-chef-client "recipe[some::default]" for instance.

#!/bin/bash
cmd="berks vendor -b $(pwd)/cookbooks/Berksfile"
pushd /tmp
rm -rf cookbooks
$cmd
mv berks-cookbooks cookbooks
sudo chef-client -z  -r "$@"
popd

Mcgann

You can also take advantage of the fact that berks vendor accepts a path to pull them to and just put the desired directory name at the end of your command. You can also avoid having to name the recipe with messy brackets and quotes by using -o cookbook_name and letting chef-client find the right cookbooks and run the default recipe. This may only work if your git repo has a parent folder called cookbooks. – Transistorize

Answer 3

All credit to @revau.lt's answer as I only made a couple tweaks for my own use. This may require a certain version of Berkshelf, but all the documentation I've seen implies you can use "PATH" at the end to tell it where to put the dependencies. This also explicitly cleans up the $tmpdir after since some of the files end up owned as root from the run.

I run this with the path to any chef cookbook repo with ./local-chef-client my_cookbook_name or ./local-chef-client path/to/chef-repos/specific_repo where the path gets resolved by pwd -P and cookbook is the basename of the $1 referenced in the script.

#!/bin/bash -x
: ${1:?"Usage: $0 ./path/to/cookbook [doit]"}
case $1 in
 .) cookbook=$(basename $(pwd -P $1));repodir=$(dirname $(pwd -P $1));;
 *) cookbook=$(basename $1);repodir=$(pwd -P $1);;
esac
case $2 in
 doit) whyrun="";;
 *) whyrun="-W ";;
esac
realcookbook=$(grep '^name' $repodir/$cookbook/metadata.rb | sed "s/'//g" | awk '{ print $2 }')
# echo "cookbook is $cookbook"
# echo "realcookbook is $realcookbook"
# echo "repodir is $repodir"
cmd="berks vendor -b $repodir/$cookbook/Berksfile cookbooks"
tmpdir=$(mktemp -d)
pushd $tmpdir
$cmd
sudo chef-client $whyrun -z -o "$realcookbook"
popd
sudo rm -rf $tmpdir
if [ ! "${whyrun}x" == "x" ]; then echo "Append 'doit' after the recipe name to really apply, eg $0 $1 doit"; fi

This should handle the case that somebody runs it from INSIDE the repo, or pretty much any path to the repo. I'll have to write some BATS tests someday to prove that, but for now it works splendidly.

I updated it to handle folders that don't match the cookbook name in case the repository name doesn't match or somebody cloned to a custom path. I actually had somebody try this under my direction and then had to dig in to Chef a bit and find out why it didn't work, turns out the folder name matters a little less than the recipe/cookbook name.

Transistorize

As an addendum to this, if you want to run a cookbook on a remote server that has chef-client but you don't want to install the whole ChefDK you can simply berks package && scp cookbooks-*.tar.gz user@remoteHost:~ && ssh user@remoteHost then on the host run tar xvf cookbooks-*.tar.gz && sudo chef-client -z -W -o recipeName and if you like the output remove the -W to really do it. – Transistorize

Answer 4

For local development, local-mode looks for config.rb or knife.rb, starting in the following order $KNIFE_HOME, $PWD, .chef/, and finally $HOME/.chef/*.

• source: lib/chef-config/workstation_config_loader.rb in chef-config-13.8.5 gem called from load_config_file() method from client.rbin chef-13.8.5 gem

The cookbook_path of '~/.berkshelf/cookbooks' is invalid for two reasons, (1) first you can only use ~ from the command line, ENV['HOME'] from within ruby, and (2) that location has versioned cookbooks, e.g. foo-0.4.6, which won't resolve when a chef run is looking for foo.

For local mode, the cookbooks must be downloaded in saved somewhere, such as a local berks-cookbooks dir after running berks vendor. If you use the current chef13 patterns using metadata.rb, then you have to fetch all the references to supermarket cookbooks from your cookbooks' metadata, put it in a main Berksfile, then vendor cookbooks, and later reference this new berks-cookbooks from your configuration (knife.rb).

Metadata with Berks Pattern

In your own cookbooks, metadata.rb as depends lines, example:

depends 'hostsfile', '~> 3.0.1'

The Berksfile would reference that using something like this:

source 'https://supermarket.chef.io'
metadata

Create Master Berksfile

You need to create a Berksfile that references all of your dependencies in your cookbooks/ directory. I use a shell script at my chef-repo level that looks like this:

#!/usr/bin/env bash
cat <<-EOF > Berksfile
source 'https://supermarket.chef.io'
$(grep -Rh 'depends' cookbooks/* | sed 's/depends/cookbook/')
EOF

berks vendor

This creates a berks-cookbooks at the same level.

Configuration

Now you need a config.rb or a knife.rb that references both cookbook locations for local development.

chef_repo = File.join(File.dirname(__FILE__), '/path/to/chef-repo')
site_cookbooks = "#{chef_repo}/cookbooks"
berks_cookbooks = "#{chef_repo}/berks-cookbooks"
cookbook_path [site_cookbooks, berks_cookbooks]

Optional (local dev vs. chef server)

As a final tip, there are many methods to keep your local dev vs. chef server workflows separated. Personally, I use git submodule, where chef-repo is in its own git repo, with it's own .chef (which is in .gitignore as it has credentials to use a Chef Server).

In my development environment, I have a sub directory ${orgname}-chef-repo and my local dev .chef/knife.rb configured to use berks-cookbooks. When I cd ${orgname}-chef-repo, I run the shell script, to create the Berksfile and berks-cookbooks dir, and then run berks install that uploads all the dependencies to the server using local credentials. Then cd .. back to my local dev environment and do local dev stuff with vagrant, knife-zero, and other tools.

Vermis