Allow cache mounts (RUN --mount=type=cache) to refer to other stages or
additional build contexts.
Update the build-check-cve-2024-9675 integration test to use different
directories for its main build context and the additional build context
that it uses for its final run.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Signed-off-by: David Shea <dshea@redhat.com>
Support splitting build logs for multi-arch builds ( `--platform` ) by
using `--logfile` and `--logsplit` which splits generated log file in the
convention `${file}_${platform_os}_${platform_arch}`.
Closes: https://github.com/containers/buildah/issues/3960
Signed-off-by: Aditya R <arajan@redhat.com>
As per `OCI` `image-spec` setting history is optional so buildah must
support these use-cases via `--omit-history=true` some of the use-cases
are discussed below.
* There are use-cases when image-spec contributes to the image size by
adding build history to image-spec in certain use-cases this image
history is not required by the end-users but there is no way to remove
it. See: https://github.com/containers/buildah/issues/3513
* Certain build tools don't honor setting build history in image spec
however buildah does not allow processing images without build history
since setting build history is still optional as per `OCI` image-spec
( https://github.com/opencontainers/image-spec/blob/main/config.md#properties)
so buildah must support such use-cases.
Closes: https://github.com/containers/buildah/issues/4025
Closes: https://github.com/containers/buildah/issues/3513
Signed-off-by: Aditya R <arajan@redhat.com>
As builds got more complicated, the ability to only access files from one location became quite limiting. With `multi-stage` builds where you can `copy` files from other parts of the Containerfile by adding the `--from` flag and pointing it to the name of another Containerfile stage or a remote image.
The new named build context feature is an extension of this pattern. You can now define additional build contexts when running the build command, give them a name, and then access them inside a Dockerfile the same way you previously did with build stages.
Additional build contexts can be defined with a new `--build-context [name]=[value]` flag. The key component defines the name for your build context and the value can be:
```console
Local directory – e.g. --build-context project2=../path/to/project2/src
HTTP URL to a tarball – e.g. --build-context src=https://example.org/releases/src.tar
Container image – Define with a docker-image:// prefix, e.g. --build-context alpine=docker-image://alpine:3.15, ( also supports docker://, container-image:// )
```
On the Containerfile side, you can reference the build context on all commands that accept the “from” parameter. Here’s how that might look:
```Dockerfile
FROM [name]
COPY --from=[name] ...
RUN --mount=from=[name] …
```
The value of [name] is matched with the following priority order:
* Named build context defined with `--build-context [name]=..`
* Stage defined with `AS [name]` inside Dockerfile
* Remote image `[name]` in a container registry
Added Features
* Pinning images for `FROM` and `COPY`
* Specifying multiple buildcontexts from different projects
and using them with `--from` in `ADD` and `COPY` directive
* Override a Remote Dependency with a Local One.
* Using additional context from external `Tar`
Signed-off-by: Aditya R <arajan@redhat.com>
* Last step of a squashed build produces a layer which contains contents
from other layers as well. So last step of last stage should be always
computed and never fetched from build cache, while other steps and
stages don't have this restriction.
* Refactor: drop squashToggle since its not useful
SquashToggle is not useful anymore since we don't alter
s.executor.squash instead we pass squash as bool to `s.commit(`
Signed-off-by: Aditya R <arajan@redhat.com>
The recent update to the storage library switched its error wrapping
from using github.com/pkg/errors to the standard library's fmt.Errorf()
%w specifier, which errors.Cause() can't unwrap. Use errors.Is() to
check for specific errors from the storage library.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Restore podman's CLI compatibility with `docker build` by making its
`--label` and `--annotate` options create empty labels and annotations
when they're given a name but no `=` or value.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
To prevent concurrent read/write between maps perform deep copy of args
before we spawn a goroutine.
[NO NEW TESTS NEEDED]
[NO TESTS NEEDED]
Signed-off-by: Aditya R <arajan@redhat.com>
Teach `buildah build` and `buildah config` about the OS version and
features fields. We don't tend to use them on Linux, but they're in the
OCI and Docker config structures, so we need to be able to preserve and
manipulate their values, much as we do for variant information.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Have the handling of the `--annotation` and `--label` options for
`buildah build` explicitly remove values when they're not given an "="
before an empty value, instead of setting them to empty values. The
final result doesn't seem to care about the distinction.
Allow the value of the `io.buildah.version` label to be manually
specified on the `buildah build` command line if someone really wants to
override the value that we would otherwise set by default.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a `--env` option to `buildah build` that functions similarly to the
`buildah config --env` option, to complement `buildah build`'s
`--unsetenv` option.
Document that `buildah config`'s `--env` function fetches the current
value for a variable when the name is supplied, but no `=` or value
follows it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Prevent fatal concurrent read/write over options.Args by concurrent
multi-arch builds and concurrent stages
[NO TESTS NEEDED]
[NO NEW TESTS NEEDED]
Signed-off-by: Aditya R <arajan@redhat.com>
Automatically set correct value of TARGETPLATFORM if one or more
`--platform` are selected for the build. This ensures that behaviour
of buildah is equivalent to `buildkit` for the args `$BUILDPLATFORM` and
`$TARGETPLATFORM`, `$TARGETOS`, `$TARGETARCH` and `$TARGETVARIANT`
Where $BUILDPLATFORM is set always native to host and $TARGETPLATFORM
is foreign selected via `--platform`.
Example
```console
$ buildah build --platform linux/arm64 .
```
```Dockerfile
FROM --platform=$BUILDPLATFORM alpine
ARG TARGETPLATFORM
ARG BUILDPLATFORM
RUN echo "I'm compiling for $TARGETPLATFORM on $BUILDPLATFORM"
```
Signed-off-by: Aditya R <arajan@redhat.com>
While processing `FROM <token> as final` executor populates baseMap as
it is without resolving or processing for any ARG values. Following
commit ensures that we process resolve any ARG variables with ARG values
while populating baseMap so it can be used later to check if stage is
reused.
Fixes: https://github.com/containers/buildah/issues/3939
Signed-off-by: Aditya R <arajan@redhat.com>
Allows end-users to export final build content or rootfs to external formats.
By default, a local container image is created from the build result. The --output (or -o) flag allows you to override this behavior, and a specify a custom exporter. For example, custom exporters allow you to export the build artifacts as files on the local filesystem instead of a Container image, which can be useful for generating local binaries, code generation etc.
The value for --output is a CSV-formatted string defining the exporter type and options. Currently, local and tar exporters are supported. The local exporter writes the resulting build files to a directory on the client side. The tar exporter is similar but writes the files as a single tarball (.tar).
```console
buildah build --output type=local,dest=dir .
buildah build --output type=tar,dest=rootfs.tar .
buildah build -o dir .
```
Reference: https://docs.docker.com/engine/reference/commandline/build/#custom-build-outputs
Signed-off-by: Aditya R <arajan@redhat.com>
Allow end users to remove default identity labels if they want to.
Since there are instances where images can be reproduced across version
hence users must have option to suppress default labels.
Closes: https://github.com/containers/buildah/issues/3826
Signed-off-by: Aditya R <arajan@redhat.com>
Users want to turn off addition of /etc/hosts file while building
container images, this would allow them to customize the /etc/hosts
file within the image.
Fixes: https://github.com/containers/buildah/issues/3808
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
If base image has a healthconfig and child does not overrides it then
docker formatted builds must inherit healthconfig from base image.
Following commit ensures that we are in parity with docker for docker
formatted builds.
Signed-off-by: Aditya R <arajan@redhat.com>
`Buildkit/Docker` adds variables in pre-allowlist for e.g `proxy
variables` to OCI/Docker history only if user explicitly specifies them in
Dockerfile using `ARG`.
By default variables in pre-allowlist e.g `proxy variables` will be used
normally but will not be leaked into `docker/OCI` history of images.
A test for following behviour is added with this commit and similar test
can be verified against `Docker/Buildkit`
Closes: https://github.com/containers/buildah/issues/2937
Signed-off-by: Aditya R <arajan@redhat.com>
Allows end users to configure executor's `OS`, `ARCH`,`VARIANT` via inline
`--platform`.
Usage
```
FROM --platform=linux/arm64 alpine
RUN uname -a
```
While this allows executor to pull base images with custom
`OS`, `ARCH`, `VARIANT` it still allows end-users to tag images with
different format if they need to.
Signed-off-by: Aditya R <arajan@redhat.com>
Make sure that, if we're given neither a concurrent jobs count nor a
semaphore to use, we create a semaphore. Otherwise, each platform in a
multi-platform build will run with maximum concurrency, which isn't the
desired default. Our CLI always passes in a Jobs count, so it hasn't
been affected by this problem.
[NO NEW TESTS NEEDED]
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Fix the following warning:
> imagebuildah/executor.go:307:6: S1033: unnecessary guard around call to delete (gosimple)
> if _, stillUnused := exec.unusedArgs[list[0]]; stillUnused {
> ^
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
If you run `buildah bud --platform windows/` the os is ignored at the
moment and it uses the host os. We should still use the os from the
platform in this case. This fixes an issue with podman-remote build
where it is possible that only the os is set in the platform string.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Buildah `builds` are composed of various `stages`. That is each `stage` in a build is a `container`.
As of now buildah does not provides any `processLabel` and `mountLabel` to the stage containers which
results in `c/storage` assigning random generated `selinux` labels to each stage.
If each stage has its own unqiue `processLabel` and `mountLabel` stages cannot be mounted across each other
in the same build as `selinux` prevents that.
We get the `processLabel` and `mountLabel` of the first stage created by the executor ( label assigned by `c/stoage` )
and share it with all the other subsequent stages of the same build.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Following commit adds buildkit like support for `from` field to `--mount=type=bind`
and `--mount=type=cache` so images and stage can be used as mount source.
Usage looks like
```dockerfile
RUN --mount=type=bind,source=.,from=<your-image>,target=/path ls /path
```
and
```dockerfile
RUN --mount=type=cache,from=<your-image>,target=/path ls /path
```
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Podman uses the new netavark network stack. Buildah should be able to do
the same. Both projects should use the same networking code which was
move to c/common/libnetwork. The new network interface can use either
CNI or netvavark. Using the same code for podman and buildah is
important to ensure that both use the same backend. Mixing CNI and
netavark is not supported.
This also fixes some outstanding CNI issues, e.g. buildah trying to
connect all cni networks.
[NO NEW TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This idea of this PR is to allow Podman to add a Podman
prefix to containers. This would allow it to keep track
of containers created by Podman and make it easier to remove
them when it wants to remove all.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Re-use all the layers from cache for squashed case and commit as late as possible
Current implementation of marking for re-use of cache `commits` on every
`stage` when used with `--squashed` however we should try to re-use as
many layers are possible if `--layers` is specified and `commit` only on
`last instruction` of `last stage` to perform final squash.
Also treat all other layers as if they are being written for regular
build and write them cache.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Currently we are only wiring the logger into run_linux.go
Not into the Config section.
This PR is needed in order to update vendor in Podman.
https://github.com/containers/podman/pull/12375
[NO NEW TESTS NEEDED] Tests will be done in Podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
It seems docker never writes ARG values to build history and only writes
values with RUN statements. As a result it can utilize caching better if
assgnment of an ARG value is never used.
Instance
```Dockerfile
FROM alpine
ARG FOO
ENV FOO=bat
RUN echo $FOO
```
In above example --build-arg FOO=value has no significance as value
is always overriden by ENV.
Following PR makes sure buildah mimics a similar behaviour.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Run secrets can now be created from an environment variable. The
environment variable is read and is briefly stored as a file on /dev/shm
when it's being used, and the file is removed after the RUN command is
finished.
Fixes: #3524
Signed-off-by: Ashley Cui <acui@redhat.com>
Users have accidently had a .containerignore or .dockerignore
file in their context directly which blocked all content.
Currently we report that no globs matched, but do not
identify where the globs came from.
This change is an attempt to add this data to the error.
Example: https://github.com/containers/buildah/issues/3318
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Make sure that the CommonBuildOpts field is set when nil.
Found the nil deref while working in Podman.
[NO NEW TESTS NEEDED]
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Following commit adds support for using buildkit like
`--mount=type=bind` with `RUN` statements. Mounts created by `--mount`
are transient in nature and only scoped to current RUN statements.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Add a --all-platforms that instructs the builder to build for the
intersection of all platforms for which the build's base images are
available. Returns an error if any of them aren't references to
manifest lists. We've learned that we can't really trust architecture
and OS information stored in image config blobs, so we don't try to
salvage that case.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If the build for a single stage fails, we break out of the loop that's
iterating through all of the stages over in its own goroutine, and start
cleaning up after the stages that were already completed.
Because the function that launched that goroutine also calls its cleanup
function in non-error cases, the cleanup function sets the map that's
used to keep track of what needs to be cleaned up to `nil` after the
function finishes iterating through the map, so that we won't try to
clean up (a given thing that needs to be cleaned up) more than once.
Because the loop that's iterating through all of the stages is running
in its own goroutine, it doesn't stop when the function that started it
returns in error cases, so it would still attempt to build subsequent
stages. Have it check for cases where the map variable has already been
cleared, or if one of the stages that it's already run returned an
error. If the function that it calls to build the stage, using the map
variable as a parameter, is already running at that point, it'll have a
non-`nil` map, so it won't crash, but it might not be cleaned up
correctly, either.
If such a stage finishes, either successfully or with an error, the
goroutine would try to pass the result back to its parent(?) goroutine
over a channel that was no longer being read from, and it would stall,
never releasing the jobs semaphore. Because we started sharing that
semaphore across multiple-platform builds, builds for other platforms
would stall completely, and the whole build would stall. Make the
results channel into a buffered channel to allow it to not stall there.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Make sure label modification behaviour stays similar to docker i.e
new step is added when labels are modified and leave caching of layers
to executor
Advantages.
* Layers will be automatically cached if needed.
* If Labels are modified , new step will be built and cached if
configured instead of forcing pre-built cached layers.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
When we're told to add built images to a manifest list, manipulate the
list ourselves, so that if we're creating a list, we won't have a
partially-populated list if some of the builds fail.
This also lets us include all of the platform information (including
variant info, which we can't sniff out after the fact) that we were
given when we started building the images.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When the terminatedStage map has its entry for a stage set, make sure
that we set the corresponding entry in the imageMap while holding the
mutex for it, eliminating the sliver of time when the first is set but
the second isn't, which could bite StageExecutor.Execute(), which waits
for the first and then reads the second.
Make terminatedStage record the error if a stage doesn't complete
successfully, and have waitForStage() return that error if it's set.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Move multiple-platform build juggling logic from the CLI wrapper
directly into the imagebuildah package, to make using it easier for
packages that consume us as a library.
This requires reading Dockerfiles into byte slices so that we can
re-parse them for each per-platform build, rather than parsing them
directly, as we used to, since building modifies the parsed tree.
When building for multiple platforms, prefix progress log messages with
the platform description.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Allow ssh socket from host or certain ssh keys to be exposed to a
certain RUN instruction, but not any other instructions, as well as not
showing up in the final image.
This is done by spawining a new agent from buildah and mounting
the listening socket inside the run. SSH_AUTH_SOCK inside the container
will be set to the socket mountpoint. The defualt mountpoint is
/run/buildkit/ssh_agent.{i}
Signed-off-by: Ashley Cui <acui@redhat.com>
Add a pkg/parse.PlatformsFromOptions() which understands a "variant"
value as an optional third value in an OS/ARCH[/VARIANT] argument value,
which accepts a comma-separated list of them, and which returns a list
of platforms.
Teach "from" and "pull" about the --platform option and add integration
tests for them, warning if --platform was given multiple values.
Add a define.BuildOptions.JobSemaphore which an imagebuildah executor
will use in preference to one that it might allocate for itself.
In main(), allocate a JobSemaphore if the number of jobs is not 0 (which
we treat as "unlimited", and continue to allow executors to do).
In addManifest(), take a lock on the manifest list's image ID so that we
don't overwrite changes that another thread might be making while we're
attempting to make changes to it. In main(), create an empty list if
the list doesn't already exist before we start down this path, so that
we don't get two threads trying to create that manifest list at the same
time later on. Two processes could still try to create the same list
twice, but it's an incremental improvement.
Finally, if we've been given multiple platforms to build for, run their
builds concurrently and gather up their results.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In prepare(), don't check if the image name that it's passed is a
pseudonym for the result of a stage in the Dockerfile. Its callers
already did that.
When execute() knows that the image it's told to use as a base is a
pseudonym for the result of another stage in the Dockerfile, force the
pull policy to "never" to prevent an error when --pull-always=true.
Make imagebuildah.Mount a type alias instead of its own type, since we
never needed it to be a distinct type.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Nalin Dahyabhai
Aditya R
Daniel J Walsh
Giuseppe Scrivano
Valentin Rothberg
Kir Kolyshkin
Paul Holzinger
Ashley Cui
Valentin Rothberg
openshift-ci[bot]