When we're told to add built images to a manifest list, manipulate the
list ourselves, so that if we're creating a list, we won't have a
partially-populated list if some of the builds fail.
This also lets us include all of the platform information (including
variant info, which we can't sniff out after the fact) that we were
given when we started building the images.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When the terminatedStage map has its entry for a stage set, make sure
that we set the corresponding entry in the imageMap while holding the
mutex for it, eliminating the sliver of time when the first is set but
the second isn't, which could bite StageExecutor.Execute(), which waits
for the first and then reads the second.
Make terminatedStage record the error if a stage doesn't complete
successfully, and have waitForStage() return that error if it's set.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Move multiple-platform build juggling logic from the CLI wrapper
directly into the imagebuildah package, to make using it easier for
packages that consume us as a library.
This requires reading Dockerfiles into byte slices so that we can
re-parse them for each per-platform build, rather than parsing them
directly, as we used to, since building modifies the parsed tree.
When building for multiple platforms, prefix progress log messages with
the platform description.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Allow ssh socket from host or certain ssh keys to be exposed to a
certain RUN instruction, but not any other instructions, as well as not
showing up in the final image.
This is done by spawining a new agent from buildah and mounting
the listening socket inside the run. SSH_AUTH_SOCK inside the container
will be set to the socket mountpoint. The defualt mountpoint is
/run/buildkit/ssh_agent.{i}
Signed-off-by: Ashley Cui <acui@redhat.com>
Add a pkg/parse.PlatformsFromOptions() which understands a "variant"
value as an optional third value in an OS/ARCH[/VARIANT] argument value,
which accepts a comma-separated list of them, and which returns a list
of platforms.
Teach "from" and "pull" about the --platform option and add integration
tests for them, warning if --platform was given multiple values.
Add a define.BuildOptions.JobSemaphore which an imagebuildah executor
will use in preference to one that it might allocate for itself.
In main(), allocate a JobSemaphore if the number of jobs is not 0 (which
we treat as "unlimited", and continue to allow executors to do).
In addManifest(), take a lock on the manifest list's image ID so that we
don't overwrite changes that another thread might be making while we're
attempting to make changes to it. In main(), create an empty list if
the list doesn't already exist before we start down this path, so that
we don't get two threads trying to create that manifest list at the same
time later on. Two processes could still try to create the same list
twice, but it's an incremental improvement.
Finally, if we've been given multiple platforms to build for, run their
builds concurrently and gather up their results.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In prepare(), don't check if the image name that it's passed is a
pseudonym for the result of a stage in the Dockerfile. Its callers
already did that.
When execute() knows that the image it's told to use as a base is a
pseudonym for the result of another stage in the Dockerfile, force the
pull policy to "never" to prevent an error when --pull-always=true.
Make imagebuildah.Mount a type alias instead of its own type, since we
never needed it to be a distinct type.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When vendoring into Podman, Buildah is output log-rusage on
the server side of podman-remote. It is using os.Stdout rather
then options.Out by default.
[NO TESTS NEEDED] This can only be tested on the Podman side,
existing tests should show if this breaks anything.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Remove some redundancy in preprocessContainerfileContents(), have it
search the $PATH for "cpp", and have it log warning messages containing
errors output by the preprocessor using the specified logrus logger.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Currently the rusage is reported to stdout and rather cumbersome to parse. The
new flag rusage-logfile can be used to specify a file to which the log will be
written instead.
Signed-off-by: Dan Čermák <dcermak@suse.com>
* Don't try to count COMMIT as a step; it's very confusing and doesn't
match the behavior of traditional docker build.
* Include the step count for the stage, which is easy if we're not
trying to predict COMMIT, which we don't always do, because we don't
always have to, in multi-stage builds.
* In multi-stage builds, prefix the stage number and stage count, which
is fun to see when we're building stages in parallel.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Currently the /usr/bin/cpp will blow up if a user adds a
comment to a containerfile that is not a preprocessor.
Since the Containerfile.in could include other Containerfile
which may have comments, begining with `#` this can cause
problems.
If we just warn on these errors, we can successfully process
all of the containerfiles.
Fixes: https://github.com/containers/buildah/issues/3229
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We want Info, Warning and Debug logrus messages to be writen to the
buildah stderr. this way when podman-remote is using build, it will
get the messages back on the client side.
[NO TESTS NEEDED] Since this will be tested in Podman.
Fixes: https://github.com/containers/buildah/issues/3214
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Fix a regression in `buildah bud`. In case of a cache hit or an
intermediate image, the pull policy should be set to "never" so that we
enforce a local-only lookup without even attempting to reach out to a
registry.
The regression was detected in Podman which defaults to --layers=true
which ultimately broke when setting --pull or --pull-always. In case of
a cache hit, the image reference refers to a local image which conflicts
with an "always" pull policy.
Also extend an existing bud test to make sure we don't regress on it in
the future.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We re-tag images produced for Dockerfiles which contain no instructions
by "copying" them from their current names to their destination names,
letting the lower storage libraries deduplicate them into the same image
record.
When the source included signatures, this would break because
containers-storage can't guarantee that the compressed version of the
blob it will produce for a given layer will have the same digest that
the version referenced by the manifest had, so the image library would
refuse to "copy" them.
When the source and destination are the same, though, the
RemoveSignatures option doesn't cause the signatures to be deleted, but
it does bypass that check in the image library, so toggling it on works
around the problem.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Update Buildah to the latest libimage. Migrating Podman over to
libimage entailed a number of fixes and changes to libimage which
we need to account for in Buildah.
Most notably:
* `(*Runtime).LookupImage()` now returns `storage.ErrImageUnknown`
instead of `nil` in case no matching image is found.
* `(*Runtime).LookupImage()` now does quite a bit more work finding
a local image and will also look at the repotags (or digests) of
all local images if needed.
* The signature of `(*Runtime).RemoveImages()` was changed and now
returns a slice of reports and errors. The reports aggregate the
data of a removed image which allows the function to be used by
`podman image prune` which is also interested in the size of the
removed data. The slice of errors is also needed in Podman which
needs to have a closer look at _all_ rmi errors in order to determine
the appropriate exit code (Docker compat).
* `libimage/types` has been removed. Pull policies have been merged
into already existing logic in `pkg/config`.
Please refer to containers/podman/pull/10147 for a more detailed
changelog.
[NO NEW TESTS NEEDED]
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When building with non-overlay drivers, we archive the contents of
volumes before RUN instructions and restore their contents when the RUN
instruction is finished.
The buildah.Run() ("buildah run") logic defaults to creating a copy of
the contents of built-in volumes and bind-mounting them when running
subcommands, but does not restore the contents afterward because the
rootfs's contents aren't changed. The rootfs's contents do get changed
by buildah.Add() ("buildah add/copy"), and the volumes go out of sync
with the rootfs because we don't really know what to do then.
The save/restore logic in imagebuildah for non-overlay paths wasn't
adding mounts to the list used by buildah.Run(), so it was continuing to
set up bind mounts even in cases where our cache was invalidated, which
produced this kind of loss of synchronization. The overlay path wasn't
affected because it adds overlay mounts to the list we pass to
buildah.Run(), and buildah.Run() doesn't override mount points. So to
avoid this for the VFS methods, add bind mounts to the rootfs, which
we're updating anyway. They're basically redundant, but they work
around the problem.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Saving a volume cache using VFS had the side-effect of creating the
volume's directory, so have the overlay path for it do so as well.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
resolveSymlink() was parsing the combined stdout and stderr from its
child process, which would include warning messages the storage library
printed to stderr during the child process's init() functions.
Instead of fixing just that, drop resolveSymlink() and replace it with a
call to copier.Eval().
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add support for secrets. Secrets is a two-part flag that allows secret files to
be accessed for a certain RUN instruction, but not any other
instructions, as well as now showing up in the final image.
Signed-off-by: Ashley Cui <acui@redhat.com>
Restore the push and pull API that commit dcd2a92e56 removed.
These changes would break vendoring into openshift/builder due
to build errors.
For the same reason, restore `util.FindImage` and `util.AddImageNames`
but deprecate the `findRegistry` argument.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Move all code related handling container image over to the new
`libimage` package in containers/common. The new package is an
attempt to consolidate the code across the containers tools under the
github.com/containers umbrella.
The new `libimage` packages provides functionality to perform all kinds
of operations for managing images such as local lookups, pushing,
pulling, listing, removing, etc.
The following packages have been moved over the containers/common:
`manifests` -> `common/image/manifests`
`pkg/manifests` -> `common/pkg/manifests`
`pkg/supplemented` -> `common/pkg/supplemented`
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
In Docker if you are copying more then one object, and
one of them is successful, then the command is successful. Currently in
buildah each glob has to be successful. This PR matches Buildah to
Docker.
Fixes: https://github.com/containers/podman/issues/9594
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Buildah currently handles multiple tags when building, but does not
report it to the user by default. This reports the tags back to the
user.
Removed some unused code from commit.go, that would blow up if a user
specified AdditionalTags to the commit command, even though this is not
exposed to the user currently. In a previous try to fix this, the
removed code was causing breakage, and I did not see a real purpose in
the code.
Fixes: https://github.com/containers/buildah/issues/3084
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
If a Containerfile had an ARG specified with a value and it was
referenced later in a multifrom scenario, a Warning would be raised
unless there was a `--build-arg` option specified for the argument.
This change removes the warning if the ARG has a value specified
in the Containerfile. We still need the warning however, if it
is not specified via a `--build-arg` or within the Containerfile.
Fixes: #3020
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Currently if the user specifies a --file path
and path is a directory, we only append on Dockerfile.
This PR searches for Containerfile and then Dockerfile.
Fixes: https://github.com/containers/buildah/issues/3078
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When we're using the overlay driver (which means we know overlay is
available), use it to make volumes appear to be writeable during RUN
instructions instead of saving/restoring their contents.
This avoids having to copy the contents of the volume directory before
each RUN instruction, and having to remove and extract the contents
after each RUN instruction, which should be faster, particularly if the
amount of content in that volume location is large.
For empty directories, it will at least avoid adding an "opaque"
notation for the directory in a layer that might otherwise be empty.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This PR removes the pkg/auth which brings in docker/docker
since it really is not needed, and was only there to help users
discover the settings of where the authfile was, when the environment
variables were set. Would almost never be of any value.
Move imagebuildah.BuildOptions to define.BuildOptions
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Currently if the buildah image has a label that was built with
an older version of buildah, we don't update to the latest version.
This PR Will cause the new version to use the current label.
Fixes:https://github.com/containers/buildah/issues/3035
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This fixes a regression introduced in
9b299588c0.
ib.Run() is only really needed in the ARG step. On all the other steps,
it can cause potentially expensive commands to be executed unecessarily.
Closes https://github.com/containers/buildah/issues/2992
Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Prefix the image ID with the hash prefix when using `--iidfile` to be
compatible with Docker. The absence of the hash can cause
docker-compose to error out.
Reported-in: github.com/containers/podman/issues/9260
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Buildah bud --manifest XYZ was not working.
The manifest was never created. This PR Finishes
the plumbing and allows users to create a manifest
while building an image in one single command.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Fix the check on build args to be the length of the map
and not whether the map is nil. The nil check was causing
the cache layer to not be used as it would give a false
result.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Check whether the ARG in the containerfile is changed by
either the --build-arg flag or local environment and use
the cached layer or rebuild the layer accordingly.
Add tests for this use case as well.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
These changes will allow the "FROM" statement in a Containerfile
to be overridden with a new `--from` option. If I have this Dockerfile.fedora
```
FROM fedora
```
This command will instead build an alpine image:
```
STEP 1: FROM alpine
Completed short name "alpine" with unqualified-search registries (origin: /etc/containers/registries.conf)
Getting image source signatures
Copying blob 188c0c94c7c5 done
Copying config d6e46aa247 done
Writing manifest to image destination
Storing signatures
STEP 2: COMMIT tom
--> d6e46aa2470
d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
```
Addresses: #2404
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Allow users to override location of the .dockerignore file.
If user specified an --ignorefile flag, buildah will read the
file and pass in the exclude lines based on the .dockerignore
contents.
Also add a --contextdir flag to buildah copy and buildah add to
specify where the context directory is located. For these two
commands the --ignorefile flag requires the --contextdir flag.
When the --ignorefile flag is passed in, the .dockerignore files
in the context directory will be ignored.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Always handle RUN instructions with no pseudo terminal, which matches
what I see with docker build 19.03. Interactive 'buildah run' will
still have the same default behavior.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When attempting to read the history of an image in order to try to check
if it's suitable as a cache entry for an image we're building, if we
fail to read its history, ignore the image instead of failing.
If the image was pulled into local storage for a different architecture,
or it's a list with no entry in it for the local architecture, or if
it's a list which references an image for the local architecture that we
haven't pulled down, we'll get an error back from the image library, and
we don't want to fail because of that.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Nalin Dahyabhai
openshift-ci[bot]
Ashley Cui
Daniel J Walsh
Dan Čermák
OpenShift Merge Robot
Valentin Rothberg
TomSweeneyRedHat
Jakub Guzik
caiges
Antonio Terceiro
Urvashi Mohnani