Can Selenium and Chrome buildpack installs be cached on HerokuCI?

Asked 14/11, 2018 at 23:17 Answered 5/12, 2018 at 18:42

I've just started using HerokuCI. I have a few Selenium tests so I've added buildpacks in my app.json.

{
  "buildpacks": [
    { "url": "heroku/ruby" }
  ],
  "environments": {
    "test": {
      "addons": ["heroku-postgresql:in-dyno", "heroku-redis:in-dyno"],
      "buildpacks": [
        { "url": "heroku/ruby" },
        { "url": "https://github.com/notvad/heroku-buildpack-selenium" },
        { "url": "https://github.com/heroku/heroku-buildpack-chromedriver" },
        { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" }
      ],
      "env": {
        "REDIS_PROVIDER": "REDIS_URL",
        "RAILS_ENV": "test",
        "RACK_ENV": "test",
        "MALLOC_ARENA_MAX": 2,
        "LANG": "en_US.UTF-8",
        "TEST_API": 1
      },
      "scripts": {
        "test-setup": "./bin/setup-cc-test-reporter",
        "test": "./bin/test-with-cc-test-reporter"
      }
    }
  }
}

These keep getting rebuilt along with all their dependencies.

-----> Selenium app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
-----> Updating apt caches
       Hit:1 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
       Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
       Hit:3 http://archive.ubuntu.com/ubuntu bionic-security InRelease
       Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
       Reading package lists...
-----> Fetching .debs for xserver-xorg-core libgl1-mesa-dri xvfb libglide3 xfs unzip
       Reading package lists...
       Building dependency tree...
       Package xfs is not available, but is referred to by another package.
       This may mean that the package is missing, has been obsoleted, or
       is only available from another source

E: Package 'xfs' has no installation candidate
-----> Installing adwaita-icon-theme_3.28.0-1ubuntu1_all.deb
-----> Installing at-spi2-core_2.28.0-1_amd64.deb

...and so on...

-----> Downloading and installing Selenium
-----> Writing profile script
-----> chromedriver app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
bash: /tmp/buildpacks/f55d0db8af2b8c70f2c0e028ef5430955852d9e4f07a06e333e54eb5264a81b9d1e77d1cc02851b4c7da30f6da47b13959e18a24c630212b899920d05f32a566/export: No such file or directory
-----> Looking up latest chromedriver version...
-----> Downloading chromedriver v2.43...
Archive:  /tmp/chromedriver.zip
  inflating: /app/.chromedriver/bin/chromedriver  
-----> Creating chromedriver export scripts...
-----> Google Chrome app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
bash: /tmp/buildpacks/f55d0db8af2b8c70f2c0e028ef5430955852d9e4f07a06e333e54eb5264a81b9d1e77d1cc02851b4c7da30f6da47b13959e18a24c630212b899920d05f32a566/export: No such file or directory
bash: /tmp/buildpacks/27353aa3b5edb2f8361186206e863f5d481f02805d02f0efdec439f3d30349a4fbdad2951af33a0492023fb0d90d26dafda6fe76f1b2d34f68564a98545f7022/export: No such file or directory
-----> Installing Google Chrome from the stable channel.
-----> Updating apt caches
       Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
       Hit:2 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
       Hit:3 http://archive.ubuntu.com/ubuntu bionic-security InRelease
       Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
       Reading package lists...
-----> Fetching .debs for gconf-service

...and so on...

This is taking up most of the test run time.

Can these installs be cached?

Cirillo answered 14/11, 2018 at 23:17 Comment(16)

What is you CLI to launch the process? – Biedermeier 29/11, 2018 at 13:43

@Bsquare Sorry, I'm not sure I understand what you're asking. – Cirillo 29/11, 2018 at 16:26

I mean, what is the command you launch to get this behaviour? – Biedermeier 29/11, 2018 at 16:27

@Bsquare I don't have direct control. It's launched as a Github PR hook provided by Heroku. I don't control provisioning the CI dyno directly. Instead it's by specifying buildpacks in a configuration file. See devcenter.heroku.com/articles/… – Cirillo 29/11, 2018 at 16:41

Ok, I'm going to analyse that ;) – Biedermeier 29/11, 2018 at 17:0

@Bsquare Thanks. It might be worth looking into the individual build packs themselves. github.com/notvad/heroku-buildpack-selenium for example. And how buildpacks are made, particularly the caching option. Thank you for your help. – Cirillo 29/11, 2018 at 17:21

DO you have any news? I'm going to work on it tomorrow ;) – Biedermeier 3/12, 2018 at 22:34

@Bsquare Nope. I haven't touched the config. – Cirillo 4/12, 2018 at 6:11

What is your github project? I'd like to check the corresponding hook. – Biedermeier 5/12, 2018 at 8:36

In addition, can you provide the whole contents of app.json? – Biedermeier 5/12, 2018 at 8:43

@Bsquare I've added the whole app.json. I'll comment on your answer. – Cirillo 5/12, 2018 at 19:34

@Bsquare Sorry, I can't show you the project. It's private. The hook is kolkrabbi.heroku.com/hooks/github, "application/json", and "Send me everything". It was set up by Heroku. I suspect all it's doing is informing Heroku there's a new commit to pull and deploy. – Cirillo 5/12, 2018 at 19:47

I think I don't have access to your link, because I got the following json: {"id":"not_found","message":"Not found."} – Biedermeier 6/12, 2018 at 8:46

@Bsquare That's expected. It's the service hook Heroku registered on Github. See devcenter.heroku.com/articles/… – Cirillo 6/12, 2018 at 16:52

@Cirillo Hi, it's been a long time, but were you able to find a solution ? – Bukhara 14/9, 2021 at 9:25

@Bukhara No, I have not. – Cirillo 14/9, 2021 at 13:5

If you have the caches enabled as described here it does not seem like there is much more you can do to cache the installs. If this is a serious problem you could always point to local packages that you cloned from the remote repos to speed things up.

Thursday answered 5/12, 2018 at 18:42 Comment(2)

I don't think I have a way to enable caches, my understanding is they should just work. I don't call bin/compile. I just supply the buildpack URL and my understanding is Heroku is supposed to do the rest. Is there a problem with the buildpack? – Cirillo 5/12, 2018 at 19:36

the documentation i linked to says If the build pack does intend to use a cache, it should create the CACHE_DIR directory if it doesn’t exist. i assume if caching was intended it would have cached it – Thursday 5/12, 2018 at 20:16

-1

After various search and try, I bring good news.

The key issue is your errors in log files, like:

bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory

This auto-generated directory is obviously a cache directory one, which is generated automatically when not fixed.

Fore instance, this can be confirmed inside the heroku-buildpack-selenium buildpack, defined in your app.json file:

# parse and derive params
BUILD_DIR=$1
CACHE_DIR=$2
LP_DIR=`cd $(dirname $0); cd ..; pwd`

It is confirmed by this official documentation I found:

The contents of CACHE_DIR will be persisted between builds. You can cache the results of long processes like dependency resolution here to speed up future builds.

So the key solution is to define this CACHE_DIR variable within your installation; so I guess it is either directly in your app.json file, or directly in your GitHub Hook.

Let me know if you need further help.

Aristarchus answered 5/12, 2018 at 9:32 Comment(4)

It looks like CACHE_DIR isn't an environment variable but a variable inside bin/compile that's populated from its 2nd argument. I have no control over that. All I do is supply the buildpack URL. I guess I could try defining a CACHE_DIR environment variable and see what happens. I've also purged the cache just to see if that helps. I'll let you know next deploy, and I'll reup the bounty. – Cirillo 5/12, 2018 at 19:43

Yes it is the second argument used when launching bin/compile but according to the documentation I think it is Heroku which specify this argument with an environment variable having the same name. – Biedermeier 5/12, 2018 at 21:38

Could you copy/paste No example files which are under .git/hook in a clone of your repository? – Biedermeier 6/12, 2018 at 7:31

There are no Git hooks in this project. The only way Git and Github are involved is Heroku is notified there's a new commit on master to deploy. – Cirillo 6/12, 2018 at 18:32

Recommended topics

Hot tags