Using PREMIRRORS in Bitbake configuration

Question

How do I use PREMIRRORS in Bitbake local configurations or recipes?

I want to provide my own download locations for some slow or inaccessible third-party URLs, but the official PREMIRRORS documentation is vague and lacks examples.

Necropolis

Answer 1

Note: These results are based on experimentation with Yocto 2.3, but probably apply to 2.5 as well.

A simple example

Suppose that your recipe file contains this target URL:

SRC_URI = "http://download.example.com:8080/foo/bar/baz-1.0.tar.gz"

Then in your local.conf, you can define your custom download location as:

PREMIRRORS_prepend = "http://download\.example\.com:8080/.* http://my-mirror.example.com/copies/\n"

In this default case (with no special placeholders) Bitbake does not include the additional /foo/bar path elements, and instead tries to download just the filename from http://my-mirror.example.com/copies/baz-1.0.tar.gz

Advanced examples

These samples use special predefined placeholders, which are detailed in the next section.

HTTP/HTTPS with same file structure

• Recipe: SRC_URI = "https://example.com:1234/foo/bar.zip"
• Setting: PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/PATH\n"
• Attempts: http://mirror.local/foo/bar.zip

HTTP/HTTPS with flat structure

• Recipe: SRC_URI = "https://example.com:1234/foo/bar.zip"
• Setting: PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/MIRRORNAME\n"
• Attempts: http://mirror.local/example.com.1234.foo.bar.zip

Just switch the hostname

• Recipe: SRC_URI = "ftp://example.com:1234/foo/bar.zip"
• Setting: PREMIRRORS_prepend = "(\w+)://example\.com(:\d+)?/.* TYPE://mirror.local/PATH\n"
• Attempts: ftp://mirrors.local/foo/bar.zip

Placeholders in replacement URI

PREMIRRORS parses all matched URIs and provides five special placeholder values in the target URI. Supposing the matched URI is http://host.example.com:1234/foo/bar/baz.txt:

TYPE          https
HOST          host.example.com%3A1234
PATH          foo/bar/baz.txt
BASENAME      baz.txt
MIRRORNAME    host.example.com.1234.foo.bar.baz.txt

Altering the PREMIRRORS variable

The PREMIRRORS variable consists of series of lines (separated by \n) each with a regular expression to match a URI, and then a replacement string, with both portions separated by a space.

Bitbake tries them in order of appearance, and you generally want your private mirrors to take priority, so prepend onto PREMIRRORS, ex:

PREMIRRORS_prepend = "http://original/location/.* http://alternate/location/\n"

What file should I edit?

You can add entries to PREMIRRORS inside your bitbake recipes, but it is not recommended, since a major use of PREMIRRORS is for people reusing your recipe in some other context or location.

Instead, you can put it inside your local.conf file in an existing build directory. Alternately, edit the source template which Poky script use when creating a new local.conf in a fresh build-directory.

Other questions

What about SOURCE_MIRROR_URL?

The SOURCE_MIRROR_URL is a quick way to add a series of PREMIRROR entries for all supported protocols. For example, this setting:

INHERIT += "own-mirrors"    
SOURCE_MIRROR_URL = "TYPE://mirror.local/PATH"

is the same as writing:

PREMIRRORS_prepend = "\
cvs://.*/.*     TYPE://mirror.local/PATH \
svn://.*/.*     TYPE://mirror.local/PATH \
git://.*/.*     TYPE://mirror.local/PATH \
gitsm://.*/.*   TYPE://mirror.local/PATH \
hg://.*/.*      TYPE://mirror.local/PATH \ 
bzr://.*/.*     TYPE://mirror.local/PATH \
p4://.*/.*      TYPE://mirror.local/PATH \
osc://.*/.*     TYPE://mirror.local/PATH \ 
https?$://.*/.* TYPE://mirror.local/PATH \
ftp://.*/.*     TYPE://mirror.local/PATH \
npm://.*/?.*    TYPE://mirror.local/PATH \
"

It seems the INHERIT+SOURCE_MIRROR_URL directives will still work if used in your local.conf (as opposed to a particular recipe.) However, Bitbake will emit warnings, so it may not be the intended use-case. Ex:

WARNING: Invalid protocol in PREMIRRORS: ('cvs://.*/.*', 'TYPE://mirror.local/PATH')

How can I check and debug my settings?

The -D debug flag will cause bitbake to emit information about what URLs it attempts to download from. You can also use -C do_fetch, which will force it to try the fetch step and re-download anything needed for the given recipe.

bitbake -D -C do_fetch software-recipe-name-here

Here's some example debug output, showing the PREMIRROR URL it attempts to access:

DEBUG: some-software-1.0 do_fetch: Trying PREMIRRORS
DEBUG: some-software-1.0 do_fetch: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /home/user/build_foo/DL_DIR 'http://mirror.local/path/to/the/filename.ext

If you need to experiment and run bitbake many times, it will be faster to temporarily put your new PREMIRRORS_prepend directive into a particular test-recipe, as opposed to modifying the local.conf. This is because Bitbake won't need to re-parse all the other recipes whenever you change it.

What if I want to isolate a port-number, e.g. http://host:123/foo?

Apparently there's no easy way to get the 123 on its own. While PREMIRRORS allows you to match with regular expressions, it does not seem to support using captured text from the match inside the replacement URI.

The port number is present inside HOST and MIRRORNAME, but there's no standard mechanism to split those values apart.

Necropolis

Answer 2

Follow up. What if I want to isolate a port-number, e.g. http://host:123/foo?

Maybe you can use captured. Like:

org: "http://somewhere.org:1234/somedir1/somedir2/somefile_1.2.3.tar.gz"

reg: "http://somewhere.org(:\d+)?/.*"

sub: "http://somewhere2.org\\1/somedir3"
    
result: "http://somewhere2.org:1234/somedir3/somefile_1.2.3.tar.gz"

Lindon

"Maybe you can"? Has this been tested? If so, with which version? – Necropolis