AWS CodeBuild buildspec.yml get all files and subfolders recursively

Question

I'm trying to use AWS CodeBuild to get all files and subfolders inside a nested public folder and deploy to an S3 bucket using CodePipeline. I was able to hook them all together but struggling to configure the buildspec.yml file to get the output I want.

My folder structure:

<path>/public/

├── 404.html
├── css
│   ├── ...
├── fonts
│   ├── bootstrap
│   │   ├── ...
│   ├── icomoon
│   │   ├── icomoon
│   │   │   ├── ...
│   └── simple-line-icons
│       ├── ...
├── images
│   ├── ...
├── index.html
├── index.xml
├── js
│   ├── ...
└── tags
    └── index.xml

I need to put everything (including the subfolders) inside the public folder into the root of an S3 bucket.

So far I've tried following the docs here, here and here. I've tried using:

• **/* to get everything recursively inside the folder public but the S3 bucket will have the path to the folder so the index.html is not in the root.
• discard-paths: yes to remove the path to the public folder but inside the S3 bucket, all files are there in the root, no sub-folder structure is kept.
• base-directory: as described here.
• artifacts: secondary-artifacts: artifact1: files: - directory/file artifact2: files: - directory/file2 to keep the folder structure but my build failed.
• Different combinations of all the syntaxes above but my build just failed.

Hohenstaufen

you need files straight after artifacts too, regardless of secondary artifacts, otherwise it will always fail – Headliner

Answer 1

I've figured out a way to work around it. I don't think it's the best way but it works. I hope you can find this solution useful buildspec.yml:

version: 0.2

phases:
  build:
    commands:
      - mkdir build-output
      - cp -R <path>/public/ build-output
  post_build:
    commands:
      - mv build-output/**/* ./
      - mv build-output/* ./
      - rm -R build-output *.yml LICENSE README* .git*
artifacts:
  files:
    - '**/*'

In words:

• I copy everything inside the nested <path>/public folder out to a folder called ./build-output.

• Then in post_build I move everything out of folder build-output.

• Delete files pulled from my GitHub's repo which aren't needed to host the static website in the S3 bucket

• Then I get the result I want: all files inside public in the root of the S3 bucket with the right folder tree.

Update:

• You can also use my buildspec.yml example file here. Since the example is out of the scope of this question, I don't paste the code here.
• I explain in details here.

Hohenstaufen

A comment about symlinks: I've tried to use this approach, but my files had symlinks and codebuild lose them all when zipping the files. forums.aws.amazon.com/thread.jspa?threadID=245780 In my solution, I had to create a tarball myself, unfortunately. – Knackwurst

this answer seems promising, but when I check my S3, I still received all files instead of specific build directory. Do you know what's wrong? – Barbiturism

Answer 2

TL;DR You probably don't need to use "discard-paths: yes". So, drop "discard-paths: yes" and instead use "base-directory" with "**/*" for glob pattern. Read on for more reasoning why.

So yeah, as it usually happens, the culprit was pretty trivial. In my case the directory structure looked like this:

dist/
|-- favicon.ico
|-- index.html
|-- js
|   `-- app.js
`-- revision.txt

So in order for the artifact to contain everything from the "dist" directory whilst preserving the nested directories structure, the buildspec had to look like this:

artifacts:
  files:
    - '**/*'
  base-directory: 'dist'

Essentially "base-directory" parameter behaves as unix's "cd" command, so at first CodeBuild jumps to that directory and only after that resolves the glob pattern, in this case - **/*. If you also happen to use "discard-paths" then what happens is that, presumably, CodeBuild will change current directory to "dist", resolve the files structure correctly but then remove all path prefixes from the resolved paths, thus the result will be a plain directory structure with no nesting.

Hope that helps!

Mena

Removing 'discard-paths' made it work for me. – Impoverished

Answer 3

I was running into a similar issue and after many permutations below syntax worked for me.

artifacts:
  files:
    - './**/*'
  base-directory: public
  name: websitename

Catmint

Answer 4

This worked for me.

    /
    buildspec.yml
    /public
        index.html
        /js
        /img
        appspec.yml
    /src

in buildspec.yml

    artifacts:
    files:
      - 'public/*'
    discard-paths: yes

Parquet

as @Butiri Dan, mentioned, adding discard-paths: yes helped me with corretto11. Because the runtime was expecting the jar file in the root of zip file according to the logs. – Parrisch

Answer 5

For those who are facing this issue while using CodeBuid in CodePipline and have tried the correct configuration for buildspec.yml file, the culprit for me was on the deploy stage, where I had set as Input Artifact the Source artifact instead of the Build artifact.

Vassalize

Answer 6

For me a very simple construct like this was enough:

artifacts:
  files:
    - 'dist/**/*'

Impressionism