Buildah `builds` are composed of various `stages`. That is each `stage` in a build is a `container`.
As of now buildah does not provides any `processLabel` and `mountLabel` to the stage containers which
results in `c/storage` assigning random generated `selinux` labels to each stage.
If each stage has its own unqiue `processLabel` and `mountLabel` stages cannot be mounted across each other
in the same build as `selinux` prevents that.
We get the `processLabel` and `mountLabel` of the first stage created by the executor ( label assigned by `c/stoage` )
and share it with all the other subsequent stages of the same build.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Podman uses the new netavark network stack. Buildah should be able to do
the same. Both projects should use the same networking code which was
move to c/common/libnetwork. The new network interface can use either
CNI or netvavark. Using the same code for podman and buildah is
important to ensure that both use the same backend. Mixing CNI and
netavark is not supported.
This also fixes some outstanding CNI issues, e.g. buildah trying to
connect all cni networks.
[NO NEW TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
This idea of this PR is to allow Podman to add a Podman
prefix to containers. This would allow it to keep track
of containers created by Podman and make it easier to remove
them when it wants to remove all.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Currently we are only wiring the logger into run_linux.go
Not into the Config section.
This PR is needed in order to update vendor in Podman.
https://github.com/containers/podman/pull/12375
[NO NEW TESTS NEEDED] Tests will be done in Podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add a pkg/parse.PlatformsFromOptions() which understands a "variant"
value as an optional third value in an OS/ARCH[/VARIANT] argument value,
which accepts a comma-separated list of them, and which returns a list
of platforms.
Teach "from" and "pull" about the --platform option and add integration
tests for them, warning if --platform was given multiple values.
Add a define.BuildOptions.JobSemaphore which an imagebuildah executor
will use in preference to one that it might allocate for itself.
In main(), allocate a JobSemaphore if the number of jobs is not 0 (which
we treat as "unlimited", and continue to allow executors to do).
In addManifest(), take a lock on the manifest list's image ID so that we
don't overwrite changes that another thread might be making while we're
attempting to make changes to it. In main(), create an empty list if
the list doesn't already exist before we start down this path, so that
we don't get two threads trying to create that manifest list at the same
time later on. Two processes could still try to create the same list
twice, but it's an incremental improvement.
Finally, if we've been given multiple platforms to build for, run their
builds concurrently and gather up their results.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Since opencontainers/image-spec/pull/822/ the OCI spec supports two new
annotations to set the fully-qualified name and the digest of the base
image.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Update Buildah to the latest libimage. Migrating Podman over to
libimage entailed a number of fixes and changes to libimage which
we need to account for in Buildah.
Most notably:
* `(*Runtime).LookupImage()` now returns `storage.ErrImageUnknown`
instead of `nil` in case no matching image is found.
* `(*Runtime).LookupImage()` now does quite a bit more work finding
a local image and will also look at the repotags (or digests) of
all local images if needed.
* The signature of `(*Runtime).RemoveImages()` was changed and now
returns a slice of reports and errors. The reports aggregate the
data of a removed image which allows the function to be used by
`podman image prune` which is also interested in the size of the
removed data. The slice of errors is also needed in Podman which
needs to have a closer look at _all_ rmi errors in order to determine
the appropriate exit code (Docker compat).
* `libimage/types` has been removed. Pull policies have been merged
into already existing logic in `pkg/config`.
Please refer to containers/podman/pull/10147 for a more detailed
changelog.
[NO NEW TESTS NEEDED]
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Move all code related handling container image over to the new
`libimage` package in containers/common. The new package is an
attempt to consolidate the code across the containers tools under the
github.com/containers umbrella.
The new `libimage` packages provides functionality to perform all kinds
of operations for managing images such as local lookups, pushing,
pulling, listing, removing, etc.
The following packages have been moved over the containers/common:
`manifests` -> `common/image/manifests`
`pkg/manifests` -> `common/pkg/manifests`
`pkg/supplemented` -> `common/pkg/supplemented`
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Move the platform check into `resolveLocalImage` and inspect each of the
candidates in the given order. This preserves the local lookup order
and fixes a bug reported in containers/podman/issues/9915.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Currently if you pull one image and then pullifmissing with
a different --arch, Buildah does not pull the different arch,
even though the arch is missing.
This PR checks the existing image to see if it matches the arch, os
and variant of the specified image, before using the local image.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
overlay supports a new mount option "volatile" that reduces I/O by
ignoring fsync and syncfs requests.
Enable it for Buildah containers as they are not supposed to survive a
machine crash and they are thrown away if the build fails.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Detect local-image lookups by digest. Those clearly refer to local
images only, so we must not proceed to remote lookups.
Note that the specifed digest refers to an image ID and not to the
digest of an image's manifest.
Fixes: #2836
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Some pull policies require to first look up a local image and compare
that to the remote counter part. When looking up the remote image, we
need to make sure to use the name of the local image, if it exists.
This fixes a bug where a short name resolved to an image with the
"localhost/" prefix. This prefix is only used for local image look ups
via `shortnames.ResolveLocally`. Hence, when looking up the remote
counter part, we must preserve this prefix.
Fixes: #2904
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Short-name aliasing was introduced as an opt-in preview by enabling an
environment variable. Now, as we're preparing for the Podman 3.0
release, we can enable short-name aliasing by default. Opting out can
be done by configuring the `registries.conf` config file.
Please refer to the following blog post for more details:
https://www.redhat.com/sysadmin/container-image-short-names
**NOTE**
This *may* justify a major version bump of Buildah. The default mode is
permissive, so scripts and builds will continue working as before.
However, `buildah pull` will now prompt users.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
If PullOptions/BuilderOptions/CommitOptions/PushOptions includes a
MaxRetries value other than 0, retry operations except for (currently)
connection-refused, authentication, and no-such-repository/no-such-tag
errors, at a default-but-configurable interval of 5 seconds.
Set the default for `buildah pull/from/commit/push` to 3 retries at 2
second intervals.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This is a rework of Qi Wang's patches.
Import package pkg/config from containers/common to read containers.conf
This patch allows users to specify default values stored in containers.conf
that will modify the behaviour of buildah tool.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #2011
Approved by: TomSweeneyRedHat
(Replaces #1873 as it had lint issues that were timing out tests that I couldn't
track down easily)
Prior to this fix, if someone did `buildah bud --pull=false .` and the image in
the Containerfile's FROM statement was not local, the build would fail. The same
build on Docker will succeed. In Docker, when `--pull` is set to false, it only
pulls the image from the registry if there was not one locally. Buildah would never
pull the image and if the image was not locally available, it would throw an error.
In certain Kubernetes environments, this was especially troublesome.
To retain the old `--pull=false` functionality, I've created a new `--pull-never`
option that fails if an image is not locally available just like the old
`--pull=false` option used to do.
In addition, if there was a newer version of the image on the repository than
the one locally, the `--pull=true` option would not pull the image as it should
have, this corrects that.
Changes both the from and bud commands.
Addresses: #1675
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Closes: #1959
Approved by: rhatdan
When the base image or an image that we're inspecting is a reference to
a manifest list, resolve it to a runnable image instance, then try to
read the configuration blob from the runnable image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1960
Approved by: TomSweeneyRedHat
Some Dockerfiles (fuse-overlay) require additional devices to be in the
build environment.
This patch allows the user to specify additional devices.
Also I noticed that CapAdd and CapDrop was not working in buildah bud situations,
so this patch also fixes this.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1820
Approved by: @TomSweeneyRedHat
Record the digest of the base image's manifest, if there is a base
image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1724
Approved by: rhatdan
Overlay mounts allow buildah bud and buildah from to
specify a directory on the disk that will be mounted
as an overlay into the container, where the overlay can be written to
but when the RUN or buildah run exits, the modified files will dissapear.
The basic idea is to be able to mount cache from the disk for things like yum/dnf/apt
to be able to be used and modified in the contianer on a run command, but to be
kept fresh for each RUN.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1560
Approved by: giuseppe
Miloslav had some good comments on a previous commit.
https://github.com/containers/buildah/pull/1411
These changes address his issues by removing them.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1412
Approved by: mtrmac
Currently rootless podman attempts to write to /var/lib/containers/cache
and fails. This causes us to repeatedly push images that have already been
pushed. This cache directory should be relative to the location of containers/storage
and not always stored in the same directory.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1411
Approved by: TomSweeneyRedHat
Use a typed value, to hopefully decrease further temptation to process strings
manually, and to avoid the unnecessary alltransports.ParseImageName which
resolveImage has already called.
This may change the strings used in some error/debug messages, which
now use transports.ImageName instead of the original input; the strings
should by definition have the same semantics.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
This only moves the code, does not modify it at all; a separate
commit to make review easier.
pullImage eventually computes the same value anyway, so this
should not change behavior. We will soon remove the redundant
value in pullImage.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
It should always be redundant with the reference itself; so,
use srcRef.StringWithinTransport() in the cases where we do
need to understand and hard-code the string syntax, after all.
Also improve the oci: format parsing a bit, to be robust
against including an image name.
NOTE: This might change the semantics a bit because StringWithinTransport
does not guarantee preserving the original string (e.g. paths
tend to be normalized not to contain symlinks). Using local paths
as docker/distribution image names is conceptually so problematic
that this seems worth the code cleanup - but I might be wrong.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
ResolveName now guarantees that the transport, if it exists,
is not a part of the image name; the semantics is no longer
ambiguous, so use the value only as expected.
This could possibly fix incorrect handling of some strings
(pull docker://dir:localpath), and the debug log will no longer
contain "error parsing image name %q as given, trying with transport" for every
name parsing attempt.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
Use the right variable to make sure transport and image are
colon-separated in error reports.
Changes user-visible strings.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
This case was originally here for options.Transport, which
no longer exists; and the previous commit has made it impossible
for transport == "" to reach this code path.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
When ResolveName has already determined that the value is an
ID (prefix), and returned the full ID, rely on that knowledge
and don't try at all to pull the image from a 'remote transport ""';
also, don't try to match strings that are already known not to be
ID prefixes, or that are known to use a different transport, against
local storage.
Should not change behavior, except possibly in theoretical
inconsistency cases when store.Image(knownImageID) fails; the code
now does not report other unrelated errors on the transport == ""
path below.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1361
Approved by: rhatdan
For some reason, the CI does not report any of these; on macOS
I see many more reports (including complaints about the standard
library), this only cleans up the trivial cases.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #1365
Approved by: rhatdan
Change references to Transfer to transfer to make it internal only.
It should be determined from the image specification and only determined
in one place.
Make buildah.Pull use registries.conf
Currently buildah pull does not resolve images based on registries.conf
This does not match the behaviour of buildah from or buildah bud
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1319
Approved by: rhatdan
Add API hooks for designating locations to be used as blob caches when
pulling and pushing images. When we commit read-only copies of
container layers for use in images, if we're using blob caching, store a
copy of the layer in the blob cache directory so that it can be found.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1149
Approved by: rhatdan
Also speed up container name selection by making sure the container name
is not chosen before trying it out.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
ReserveSELinuxLabels() checks if an error returned by OpenBuilder() is a
does-not-exist error, but OpenBuilder() returns wrapped errors now, and
it wasn't checking the root cause error.
When newBuilder() fails, check the right error value when deciding
whether or not deleting the partially-constructed container failed.
OpenBuildersByPath() shouldn't choke on non-buildah containers, so have
it handle does-not-exist errors the same way OpenAllBuilders() does.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1109
Approved by: rhatdan
Instead of just using multierror to collect the error strings (and hope that they
contain enough context about the attempted image names), explicitly collect pairs of
(attempted image name, error).
Then, report an appropriate error text depending on the failures and environment.
Notably, if there was only one attempt (e.g. a fully-qualified name), just report
the error with minimal context, instead of adding extra "1 error occurred:\n\n".
If search registries were used and empty, note that in the error message (now
for real).
Also, make sure we return _some_ error if util.ResolveName ever returned
an empty list for osome reason.
It seems fairly attractive now to split resolveImage into the outer loop
and a separate function for making one attempt, to consolidate the
repetitive failures = append(...) code. OTOH that would force us to add
another return value as a "fatal" indication. Maybe later.
NOTABLY CHANGES BEHAVIOR:
- Changes the error format.
- Restores more detailed error reporting if we have no search registries,
but had multiple candidates anyway, which was given up in the previous commit.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #909
Approved by: rhatdan
Use the value now returned by util.ResolveImage instead of trying to
recompute it.
Then drop the no longer used getRegistries. (It might be reasonable
to split that part of util.ResolveImage to make it shorter; but it should
not ideally have any independent second-guessing callers. So, just
keep the inlined one instead; that way we certainly don't break it.)
Also drop the no longer used hasRegistry.
CHANGES BEHAVIOR:
- Most notably, the "short name but no search registries" code
has been broken for some time; pullImage was called with
localhost/$shortname, which was a qualified name, so the
specialized error handling was never attempted.
- Temporarily, the error handling in the "short name but no
search registries" code trigers even if there were actually
valid values to try (in practice there is always localhost/$shortname,
and possibly also options.Registry/$shortname). The next commit
will improve it again.
- We now have more legitimate access to the original short name,
so include it in the error message (it was technically available
before, but using it was awkward).
NOTE: registriesConfPath is computed using the sysregistries
package, but actual access happens using the sysregistriesv2 package.
That should be cleaned up eventually.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #909
Approved by: rhatdan
pullImage now tries (incorrectly) to redundantly compute the same
value as part of error handling. So, return the actually used
data in util.ResolveName.
The computed value is not used yet, so should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #909
Approved by: rhatdan
resolveImage already either returns an image, or an error; never (nil, nil, nil),
so this is dead code; drop it.
Should not change behavior.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Closes: #909
Approved by: rhatdan
Aditya Rajan
Paul Holzinger
Daniel J Walsh
Nalin Dahyabhai
Valentin Rothberg
OpenShift Merge Robot
Giuseppe Scrivano
Brandon Lum
Boaz Shuster
TomSweeneyRedHat
Miloslav Trmač