The lack of the preservation was probably an oversight when --mount was added for RUN.
I added a test that fails without the modification and succeeds with.
Fixes#4375
Signed-off-by: Marcus Watkins <mwatkins@mitre.org>
Before this commit buildah produces warning for UnsetBuildArgs at the
top level while preparing the build in `imagebuildah/build` hence it
makes it hard to honor certain logic like `do not produce warning for
stages which are not used` therefore move `warnOnUnsetBuildArgs` to
`imagebuldah/executor` so we can warn while processing each stage.
Following commit will prevent warning unnecessarily for stages which are
completely skipped.
Closes: https://github.com/containers/buildah/issues/4303
Signed-off-by: Aditya R <arajan@redhat.com>
Podman adds an Error: to every error message. So starting an error
message with "error" ends up being reported to the user as
Error: error ...
This patch removes the stutter.
Also ioutil.ReadFile errors report the Path, so wrapping the err message
with the path causes a stutter.
Signed-off-by: Daniel J Walsh dwalsh@redhat.com
[NO NEW TESTS NEEDED]
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
In multi-stage builds buildah will skip stages which are unused (i.e
stages which don't contribute anything to target stage directly or
indirectly) however in certain cases users need to process these unused
stages hence add support for `--skip-unused-stages` which allows users
to control this behaviour.
Ref: https://github.com/GoogleContainerTools/kaniko#flag---skip-unused-stages
Closes: https://github.com/containers/buildah/issues/4243
Signed-off-by: Aditya R <arajan@redhat.com>
As per buildkit backend documentation when using the BuildKit backend, docker build searches for
a `.dockerignore` file relative to the Dockerfile name. For example, running `docker build -f myapp.Dockerfile .`
will first look for an ignore file named `myapp.Dockerfile.dockerignore`. If such a file is not found, the
`.dockerignore` file is used if present. Using a Dockerfile based `.dockerignore` is useful if a project contains
multiple Dockerfiles that expect to ignore different sets of files.
Following PR introduces this buildkit feature parity.
Closes: https://github.com/containers/buildah/issues/4236
Signed-off-by: Aditya R <arajan@redhat.com>
`build` or `bud` now supports a new flag `--cache-ttl` which accepts
duration and allows end users to ignore cache images which are not under
the specified duration.
Following flag is useful for setups/platforms which heavily relies on
`--layer` and buildah caching but want recompute certain `RUN` steps
after specified duration to make sure specific `RUN` steps are always
updated. Example `RUN dnf update` or `RUN dnf install`
Closes: https://github.com/containers/buildah/issues/4160
Somewhat similar to kaniko's: https://github.com/GoogleContainerTools/kaniko#--cache-ttl-duration
Signed-off-by: Aditya R <arajan@redhat.com>
Following commit
* Initiates `cacheKey` or `layerKey` for intermediate images generated
for layers.
* Allows end users to upload cached layers with `cacheKey` to remote
sources using `--cache-to`. `--cache-to` is a optional flag to be used
with `buildah build` which publishes cached layers to remote sources.
* Allows end users to use cached layers from `remote` sources with
`--cache-from`. `--cache-from` is a optional flag to be used with
`buildah build` and it pulls cached layers from remote sources in a step
by step manner only if is a valid cache hit.
Example
* Populate cache source or use cached layers if already present
```bash
buildah build -t test --layers --cache-to registry/myrepo/cache --cache-from registry/myrepo/cache .
```
Future:
* `cacheKey` or `layerKey` model is only being used when working with
remote sources however local cache lookup can be also optimized if its
is altered to use `cacheKey` model instead of iterating through all the
images in local storage. As discussed here
References:
* Feature is quite similar to `kaniko`'s `--cache-repo`: https://github.com/GoogleContainerTools/kaniko#--cache-repo
Closes: issues#620
Signed-off-by: Aditya R <arajan@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
While performing mult-stage builds buildah does not needs to computed
and run each and every stage it can skip stages which are not used by
`target` stage.
Following commit adds `dependencyTree/dependencyMap` to executor which
calculates and populates the dependencyTree for all the stages and marks
if a particular stage is used by `Target/Final` stage directly or
indirectly.
```Dockerfile
// buildah should skip this stage
FROM alpine
RUN echo "unwanted stage"
FROM alpine as one
RUN echo "wanted stage"
// buildah should skip this stage
FROM alpune
RUN echo "another unwanted stage"
FROM one
RUN echo "wanted stage"
```
Signed-off-by: Aditya R <arajan@redhat.com>
As builds got more complicated, the ability to only access files from one location became quite limiting. With `multi-stage` builds where you can `copy` files from other parts of the Containerfile by adding the `--from` flag and pointing it to the name of another Containerfile stage or a remote image.
The new named build context feature is an extension of this pattern. You can now define additional build contexts when running the build command, give them a name, and then access them inside a Dockerfile the same way you previously did with build stages.
Additional build contexts can be defined with a new `--build-context [name]=[value]` flag. The key component defines the name for your build context and the value can be:
```console
Local directory – e.g. --build-context project2=../path/to/project2/src
HTTP URL to a tarball – e.g. --build-context src=https://example.org/releases/src.tar
Container image – Define with a docker-image:// prefix, e.g. --build-context alpine=docker-image://alpine:3.15, ( also supports docker://, container-image:// )
```
On the Containerfile side, you can reference the build context on all commands that accept the “from” parameter. Here’s how that might look:
```Dockerfile
FROM [name]
COPY --from=[name] ...
RUN --mount=from=[name] …
```
The value of [name] is matched with the following priority order:
* Named build context defined with `--build-context [name]=..`
* Stage defined with `AS [name]` inside Dockerfile
* Remote image `[name]` in a container registry
Added Features
* Pinning images for `FROM` and `COPY`
* Specifying multiple buildcontexts from different projects
and using them with `--from` in `ADD` and `COPY` directive
* Override a Remote Dependency with a Local One.
* Using additional context from external `Tar`
Signed-off-by: Aditya R <arajan@redhat.com>
Teach `buildah build` and `buildah config` about the OS version and
features fields. We don't tend to use them on Linux, but they're in the
OCI and Docker config structures, so we need to be able to preserve and
manipulate their values, much as we do for variant information.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a `--env` option to `buildah build` that functions similarly to the
`buildah config --env` option, to complement `buildah build`'s
`--unsetenv` option.
Document that `buildah config`'s `--env` function fetches the current
value for a variable when the name is supplied, but no `=` or value
follows it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
While processing `FROM <token> as final` executor populates baseMap as
it is without resolving or processing for any ARG values. Following
commit ensures that we process resolve any ARG variables with ARG values
while populating baseMap so it can be used later to check if stage is
reused.
Fixes: https://github.com/containers/buildah/issues/3939
Signed-off-by: Aditya R <arajan@redhat.com>
Allows end-users to export final build content or rootfs to external formats.
By default, a local container image is created from the build result. The --output (or -o) flag allows you to override this behavior, and a specify a custom exporter. For example, custom exporters allow you to export the build artifacts as files on the local filesystem instead of a Container image, which can be useful for generating local binaries, code generation etc.
The value for --output is a CSV-formatted string defining the exporter type and options. Currently, local and tar exporters are supported. The local exporter writes the resulting build files to a directory on the client side. The tar exporter is similar but writes the files as a single tarball (.tar).
```console
buildah build --output type=local,dest=dir .
buildah build --output type=tar,dest=rootfs.tar .
buildah build -o dir .
```
Reference: https://docs.docker.com/engine/reference/commandline/build/#custom-build-outputs
Signed-off-by: Aditya R <arajan@redhat.com>
Users want to turn off addition of /etc/hosts file while building
container images, this would allow them to customize the /etc/hosts
file within the image.
Fixes: https://github.com/containers/buildah/issues/3808
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Make sure that, if we're given neither a concurrent jobs count nor a
semaphore to use, we create a semaphore. Otherwise, each platform in a
multi-platform build will run with maximum concurrency, which isn't the
desired default. Our CLI always passes in a Jobs count, so it hasn't
been affected by this problem.
[NO NEW TESTS NEEDED]
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Fix the following warning:
> imagebuildah/executor.go:307:6: S1033: unnecessary guard around call to delete (gosimple)
> if _, stillUnused := exec.unusedArgs[list[0]]; stillUnused {
> ^
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Buildah `builds` are composed of various `stages`. That is each `stage` in a build is a `container`.
As of now buildah does not provides any `processLabel` and `mountLabel` to the stage containers which
results in `c/storage` assigning random generated `selinux` labels to each stage.
If each stage has its own unqiue `processLabel` and `mountLabel` stages cannot be mounted across each other
in the same build as `selinux` prevents that.
We get the `processLabel` and `mountLabel` of the first stage created by the executor ( label assigned by `c/stoage` )
and share it with all the other subsequent stages of the same build.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Podman uses the new netavark network stack. Buildah should be able to do
the same. Both projects should use the same networking code which was
move to c/common/libnetwork. The new network interface can use either
CNI or netvavark. Using the same code for podman and buildah is
important to ensure that both use the same backend. Mixing CNI and
netavark is not supported.
This also fixes some outstanding CNI issues, e.g. buildah trying to
connect all cni networks.
[NO NEW TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This idea of this PR is to allow Podman to add a Podman
prefix to containers. This would allow it to keep track
of containers created by Podman and make it easier to remove
them when it wants to remove all.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Run secrets can now be created from an environment variable. The
environment variable is read and is briefly stored as a file on /dev/shm
when it's being used, and the file is removed after the RUN command is
finished.
Fixes: #3524
Signed-off-by: Ashley Cui <acui@redhat.com>
Users have accidently had a .containerignore or .dockerignore
file in their context directly which blocked all content.
Currently we report that no globs matched, but do not
identify where the globs came from.
This change is an attempt to add this data to the error.
Example: https://github.com/containers/buildah/issues/3318
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
If the build for a single stage fails, we break out of the loop that's
iterating through all of the stages over in its own goroutine, and start
cleaning up after the stages that were already completed.
Because the function that launched that goroutine also calls its cleanup
function in non-error cases, the cleanup function sets the map that's
used to keep track of what needs to be cleaned up to `nil` after the
function finishes iterating through the map, so that we won't try to
clean up (a given thing that needs to be cleaned up) more than once.
Because the loop that's iterating through all of the stages is running
in its own goroutine, it doesn't stop when the function that started it
returns in error cases, so it would still attempt to build subsequent
stages. Have it check for cases where the map variable has already been
cleared, or if one of the stages that it's already run returned an
error. If the function that it calls to build the stage, using the map
variable as a parameter, is already running at that point, it'll have a
non-`nil` map, so it won't crash, but it might not be cleaned up
correctly, either.
If such a stage finishes, either successfully or with an error, the
goroutine would try to pass the result back to its parent(?) goroutine
over a channel that was no longer being read from, and it would stall,
never releasing the jobs semaphore. Because we started sharing that
semaphore across multiple-platform builds, builds for other platforms
would stall completely, and the whole build would stall. Make the
results channel into a buffered channel to allow it to not stall there.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we're told to add built images to a manifest list, manipulate the
list ourselves, so that if we're creating a list, we won't have a
partially-populated list if some of the builds fail.
This also lets us include all of the platform information (including
variant info, which we can't sniff out after the fact) that we were
given when we started building the images.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When the terminatedStage map has its entry for a stage set, make sure
that we set the corresponding entry in the imageMap while holding the
mutex for it, eliminating the sliver of time when the first is set but
the second isn't, which could bite StageExecutor.Execute(), which waits
for the first and then reads the second.
Make terminatedStage record the error if a stage doesn't complete
successfully, and have waitForStage() return that error if it's set.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Move multiple-platform build juggling logic from the CLI wrapper
directly into the imagebuildah package, to make using it easier for
packages that consume us as a library.
This requires reading Dockerfiles into byte slices so that we can
re-parse them for each per-platform build, rather than parsing them
directly, as we used to, since building modifies the parsed tree.
When building for multiple platforms, prefix progress log messages with
the platform description.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Allow ssh socket from host or certain ssh keys to be exposed to a
certain RUN instruction, but not any other instructions, as well as not
showing up in the final image.
This is done by spawining a new agent from buildah and mounting
the listening socket inside the run. SSH_AUTH_SOCK inside the container
will be set to the socket mountpoint. The defualt mountpoint is
/run/buildkit/ssh_agent.{i}
Signed-off-by: Ashley Cui <acui@redhat.com>
Add a pkg/parse.PlatformsFromOptions() which understands a "variant"
value as an optional third value in an OS/ARCH[/VARIANT] argument value,
which accepts a comma-separated list of them, and which returns a list
of platforms.
Teach "from" and "pull" about the --platform option and add integration
tests for them, warning if --platform was given multiple values.
Add a define.BuildOptions.JobSemaphore which an imagebuildah executor
will use in preference to one that it might allocate for itself.
In main(), allocate a JobSemaphore if the number of jobs is not 0 (which
we treat as "unlimited", and continue to allow executors to do).
In addManifest(), take a lock on the manifest list's image ID so that we
don't overwrite changes that another thread might be making while we're
attempting to make changes to it. In main(), create an empty list if
the list doesn't already exist before we start down this path, so that
we don't get two threads trying to create that manifest list at the same
time later on. Two processes could still try to create the same list
twice, but it's an incremental improvement.
Finally, if we've been given multiple platforms to build for, run their
builds concurrently and gather up their results.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In prepare(), don't check if the image name that it's passed is a
pseudonym for the result of a stage in the Dockerfile. Its callers
already did that.
When execute() knows that the image it's told to use as a base is a
pseudonym for the result of another stage in the Dockerfile, force the
pull policy to "never" to prevent an error when --pull-always=true.
Make imagebuildah.Mount a type alias instead of its own type, since we
never needed it to be a distinct type.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When vendoring into Podman, Buildah is output log-rusage on
the server side of podman-remote. It is using os.Stdout rather
then options.Out by default.
[NO TESTS NEEDED] This can only be tested on the Podman side,
existing tests should show if this breaks anything.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Currently the rusage is reported to stdout and rather cumbersome to parse. The
new flag rusage-logfile can be used to specify a file to which the log will be
written instead.
Signed-off-by: Dan Čermák <dcermak@suse.com>
* Don't try to count COMMIT as a step; it's very confusing and doesn't
match the behavior of traditional docker build.
* Include the step count for the stage, which is easy if we're not
trying to predict COMMIT, which we don't always do, because we don't
always have to, in multi-stage builds.
* In multi-stage builds, prefix the stage number and stage count, which
is fun to see when we're building stages in parallel.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We want Info, Warning and Debug logrus messages to be writen to the
buildah stderr. this way when podman-remote is using build, it will
get the messages back on the client side.
[NO TESTS NEEDED] Since this will be tested in Podman.
Fixes: https://github.com/containers/buildah/issues/3214
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Update Buildah to the latest libimage. Migrating Podman over to
libimage entailed a number of fixes and changes to libimage which
we need to account for in Buildah.
Most notably:
* `(*Runtime).LookupImage()` now returns `storage.ErrImageUnknown`
instead of `nil` in case no matching image is found.
* `(*Runtime).LookupImage()` now does quite a bit more work finding
a local image and will also look at the repotags (or digests) of
all local images if needed.
* The signature of `(*Runtime).RemoveImages()` was changed and now
returns a slice of reports and errors. The reports aggregate the
data of a removed image which allows the function to be used by
`podman image prune` which is also interested in the size of the
removed data. The slice of errors is also needed in Podman which
needs to have a closer look at _all_ rmi errors in order to determine
the appropriate exit code (Docker compat).
* `libimage/types` has been removed. Pull policies have been merged
into already existing logic in `pkg/config`.
Please refer to containers/podman/pull/10147 for a more detailed
changelog.
[NO NEW TESTS NEEDED]
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Add support for secrets. Secrets is a two-part flag that allows secret files to
be accessed for a certain RUN instruction, but not any other
instructions, as well as now showing up in the final image.
Signed-off-by: Ashley Cui <acui@redhat.com>
Restore the push and pull API that commit dcd2a92e56 removed.
These changes would break vendoring into openshift/builder due
to build errors.
For the same reason, restore `util.FindImage` and `util.AddImageNames`
but deprecate the `findRegistry` argument.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Move all code related handling container image over to the new
`libimage` package in containers/common. The new package is an
attempt to consolidate the code across the containers tools under the
github.com/containers umbrella.
The new `libimage` packages provides functionality to perform all kinds
of operations for managing images such as local lookups, pushing,
pulling, listing, removing, etc.
The following packages have been moved over the containers/common:
`manifests` -> `common/image/manifests`
`pkg/manifests` -> `common/pkg/manifests`
`pkg/supplemented` -> `common/pkg/supplemented`
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Buildah currently handles multiple tags when building, but does not
report it to the user by default. This reports the tags back to the
user.
Removed some unused code from commit.go, that would blow up if a user
specified AdditionalTags to the commit command, even though this is not
exposed to the user currently. In a previous try to fix this, the
removed code was causing breakage, and I did not see a real purpose in
the code.
Fixes: https://github.com/containers/buildah/issues/3084
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This PR removes the pkg/auth which brings in docker/docker
since it really is not needed, and was only there to help users
discover the settings of where the authfile was, when the environment
variables were set. Would almost never be of any value.
Move imagebuildah.BuildOptions to define.BuildOptions
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Prefix the image ID with the hash prefix when using `--iidfile` to be
compatible with Docker. The absence of the hash can cause
docker-compose to error out.
Reported-in: github.com/containers/podman/issues/9260
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Buildah bud --manifest XYZ was not working.
The manifest was never created. This PR Finishes
the plumbing and allows users to create a manifest
while building an image in one single command.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
These changes will allow the "FROM" statement in a Containerfile
to be overridden with a new `--from` option. If I have this Dockerfile.fedora
```
FROM fedora
```
This command will instead build an alpine image:
```
STEP 1: FROM alpine
Completed short name "alpine" with unqualified-search registries (origin: /etc/containers/registries.conf)
Getting image source signatures
Copying blob 188c0c94c7c5 done
Copying config d6e46aa247 done
Writing manifest to image destination
Storing signatures
STEP 2: COMMIT tom
--> d6e46aa2470
d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
```
Addresses: #2404
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Allow users to override location of the .dockerignore file.
If user specified an --ignorefile flag, buildah will read the
file and pass in the exclude lines based on the .dockerignore
contents.
Also add a --contextdir flag to buildah copy and buildah add to
specify where the context directory is located. For these two
commands the --ignorefile flag requires the --contextdir flag.
When the --ignorefile flag is passed in, the .dockerignore files
in the context directory will be ignored.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Cache the information we look up about images when evaluating them as
potential build cache matches, in an effort to reduce the number of
NewImage() calls we make, because they can be expensive.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Marcus Watkins
Aditya R
Daniel J Walsh
Sascha Grunert
Nalin Dahyabhai
Kir Kolyshkin
Paul Holzinger
Ashley Cui
openshift-ci[bot]
Dan Čermák
Valentin Rothberg
OpenShift Merge Robot
caiges
TomSweeneyRedHat