When suppressing what we think are pulled-up directories at commit-time,
only do that when we're _not_ squashing the image, in which case we
really do need to output it into the one layer that our output image
will have.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When Ensure() tells us about the parent directories of a mountpoint
target that it created for us, add them to the list of items that we'll
exclude from the commit unless something else causes them to be
modified.
The "native" overlay diff method just walks the diff directory, so we
can get directories in the committed layer that look exactly like the
ones in the previous layer, depending on the storage driver. And while
it's barely noticeable, we can still control for it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When building or committing an image in OCI format, default to setting
the org.opencontainers.image.created annotation to the value used in the
image's config blob for the image's creation date. The behavior can be
controlled using the new --created-annotation flag.
Add --annotation and --unsetannotation flags to `buildah commit` which
mimic the same flags for `buildah build`.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When RUN requires us to create the target for a mountpoint, make note of
it and any parent directories that needed to be created, and filter them
out when generating a layer diff or --output data.
The exceptions will be directories that the conformance tests confirm
that BuildKit also leaves behind, though for compatibility with the
classic builder, we have to make that conditional.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an OmitLayerHistoryEntry field to CommitOptions, which more or less
causes us to reproduce our base image, except with PrependedEmptyLayers,
AppendedEmptyLayers, PrependedLinkedLayers, AppendedLinkedLayers, and
config changes still added in.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Without this change, specifying `Compression: imagebuildah.Zstd` in
`imagebuildah`'s `BuildOptions fails, so it is not possible to push
cache to a registry with zstd compression.
Note this is only applicable to OCI manifests.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
When using either --source-date-epoch or --timestamp, make sure that
handling of RUN instructions uses a defined hostname if possible, and
commits using a reference to a static container name.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Use $SOURCE_DATE_EPOCH as the default for the --source-date-epoch flag
to the "build" CLI.
When a source-date-epoch is set, we'll use it when writing new history
entries, force timestamps in data written for --output to the specified
timestamp, and populate a "SOURCE_DATE_EPOCH" ARG that we treat as
always being set, and which we don't complain about being left unused.
By default, this will not affect timestamps in newly-added layers.
Add a --rewrite-timestamp flag, which "clamps" timestamps in newly-added
layers to not be later than the --source-date-epoch value if the
--source-date-epoch flag is set, but has no effect otherwise.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a --source-date-epoch flag, defaulting to $SOURCE_DATE_EPOCH if set,
which sets the created-on date and the timestamp for the new history
entries, but does not default to modifying the timestamps on contents in
new layers.
Add a --rewrite-timestamp flag, which "clamps" timestamps in the new
layers to not be later than the --source-date-epoch value if both
the --rewrite-timestamp and --source-date-epoch flags were set.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
* Refactor NewImageSource to add a manifest type abstraction
Currently, NewImageSource creates a Docker schema2 manifest and an OCI
manifest at the same time. This precludes functionality that isn't
supported by both manifest types, for example zstd compression.
Refactoring this to create only the desired manifest type solves this
and also cleans up the code by separating manifest-type-specific code
into distinct implementations of a "manifest builder".
See discussion in https://github.com/containers/buildah/pull/5452.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
* Review feedback
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
* Review feedback, round 2
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
---------
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
Co-authored-by: flouthoc <flouthoc@users.noreply.github.com>
This option is set from CommitOptions.HistoryTimestamp, which
corresponds to the buildah option '--timestamp', and therefore is off
by default.
If this option is not given, we can save ourselves one layer of
copying (tar filtering is a tar.Reader/tar.Writer connnected with io.Pipe())
Signed-off-by: Han-Wen Nienhuys <hanwen@engflow.com>
As of
e024854ba3,
Uname/Gname fields are not populated on Unix. On Windows, the golang
tar package leaves the Uname/Gname fields empty.
Signed-off-by: Han-Wen Nienhuys <hanwen@engflow.com>
These experimental packages are now available in the Go standard
library since Go 1.21:
1. golang.org/x/exp/slices -> slices [1]
2. golang.org/x/exp/maps -> maps [2]
[1]: https://go.dev/doc/go1.21#slices
[2]: https://go.dev/doc/go1.21#maps
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add API for adding arbitrary layers at commit-time via CommitOptions,
and via methods of the Builder type.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If we're prepending history entries before the one for "this" commit,
make sure the "FROM $baseimage" comment gets set on the first history
entry that we add, not just the one goes with this (maybe) layer diff.
In layers=false mode, the output was so, so confusing otherwise.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Make setting the Parent field in the config blob of a docker format
image optional (yes, we're bringing it back!), since it no longer
appears to be set by newer versions of docker build.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Use slices.Clone() and maps.Clone() instead of our own non-generic
functions. We have to be more careful in a couple of places where we
set items in maps which aren't unconditionally initialized.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Merge the two tar filters, if we need two, that we use when committing
an image. Try to improve passing of error information from the writing
end of a pipe to the reader, so that it can be reported better.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a --sbom flag to `buildah build` and `buildah commit` which will
scan the rootfs and specified context directories to build SPDX or
CycloneDX SBOMs and lists of package URLs.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If the parent image has layers but no history, force our own omitHistory
setting on.
The alternative is to create a history that only explains the presence
of some of the layers in our output image, which looks broken to
everyone who might consume that image, including ourselves if we try to
use it as a base image later.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we have extra files to add to the image, handle them by adding them
to the upper overlay layer before creating the plaintext filesystem
image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a flag to `buildah commit` which allows adding arbitrary files to
the image while we're committing it. When not squashing, they'll take
the form of a second new layer.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When using the working container's rootfs to populate a plaintext disk
image with mkfs, instead of writing .krun_config.json to the rootfs and
then removing it afterward (since we don't want it to show up if the
same working container is later committed to non confidential-workload
image), mount an overlay filesystem using a temporary directory as the
upper and the rootfs as the lower, create the .krun_config.json file in
the overlay filesystem, and use the overlay filesystem as the source
directory for mkfs.
Add the necessary stubs to allow pkg/overlay to at least compile on
non-Linux systems. Change the naming scheme for a test so that the path
names it uses for temporary directories don't include "," or "=", which
can confuse the kernel.
Creating confidential workload images will now only be possible on Linux
systems, but we exec'd out to sevctl to read platform certificates, and
that requires kernel support with vendor firmware, so I don't know that
anyone will actually be impacted by the change.
Teach pkg/overlay.MountWithOptions() to accept `nil` as a pointer to a
struct parameter that is otherwise optional.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Ignore whether or not the final build instruction should produce a layer
if we're squashing or producing a confidential workload, when we'd still
have to produce a layer containing the contents of the base image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an OverrideChanges and an OverrideConfig field to CommitOptions,
both of which can be used to make last-minute edits to the configuration
of an image that we're committing.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Clear the docker-format rootfs and history information before we clear
other fields in the config for confidential workload cases, so that it's
easier to track that it exactly parallels what we're doing with the OCI
format data.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Projects which are using buildah as a library and set `TMPDIR` manually
can stumble upon a use-case where `TMPDIR` was set to a relative path.
Such as `export TMPDIR=.` in such case buildah will try to create a
temporary root using `Mkdirtemp` leading to a point where bundle is not
generated correctly since path was relative.
Following use case can be resolved by making sure that buildah always
converts relative path to absolute path and `GetTempDir` does it well.
Example reproducer with podman
```Dockerfile
FROM alpine
RUN echo hello
```
```console
export TMPDIR=.
podman build --no-cache -t test .
```
Expected failure
```console
STEP 1/2: FROM alpine
STEP 2/2: RUN echo hello
error running container: checking permissions on "buildah2341274198": stat buildah2341274198: no such file or directory
ERRO[0000] did not get container create message from subprocess: EOF
Error: building at STEP "RUN echo hello": while running runtime: exit status 1
```
Closes: RHEL-2598
Signed-off-by: Aditya R <arajan@redhat.com>
Add a --cw option to `buildah build` and `buildah commit`, which takes a
comma-separated list of arguments and produces an image laid out for use
as a confidential workload:
type: sev or snp
attestation_url: location of a key broker server
cpus: expected number of virtual CPUs to run with
memory: expected megabytes of memory to run with
workload_id: a distinguishing identifier for the key broker server
ignore_attestation_errors: ignore errors registering the workload
passphrase: for encrypting the disk image
slop: extra space to allocate for the disk image
At least one of attestation_url and passphrase must be specified in
order for the encrypted disk image to be decryptable at run-time. Other
arguments can be omitted. ignore_attestation_errors is intentionally
undocumented, as it's mainly used to permit some amount of testing on
systems which don't have the required hardware.
Add an `mkcw` top-level command, for converting directly from an image
to a confidential workload.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In golang 1.19, `io/ioutil` is fully deprecated preventing Buildah from
compiling. Replace all calls with equivalent calls from the `os`
package.
Signed-off-by: Chris Evich <cevich@redhat.com>
Podman adds an Error: to every error message. So starting an error
message with "error" ends up being reported to the user as
Error: error ...
This patch removes the stutter.
Also ioutil.ReadFile errors report the Path, so wrapping the err message
with the path causes a stutter.
Signed-off-by: Daniel J Walsh dwalsh@redhat.com
[NO NEW TESTS NEEDED]
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When generating layer diffs or extracting container contents, scrub the
user and group name fields in tar headers before saving them.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If errors for which os.IsExist() or os.IsNotExist() would have returned
true have been wrapped using fmt.Errorf()'s "%w" verb, os.IsExist() and
os.IsNotExist(), not having been retrofitted to use errors.Is(), will
return false.
Use errors.Is() to check if an error is an os.ErrExist or os.ErrNotExist
error instead of calling os.IsExist() or os.IsNotExist().
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
As per `OCI` `image-spec` setting history is optional so buildah must
support these use-cases via `--omit-history=true` some of the use-cases
are discussed below.
* There are use-cases when image-spec contributes to the image size by
adding build history to image-spec in certain use-cases this image
history is not required by the end-users but there is no way to remove
it. See: https://github.com/containers/buildah/issues/3513
* Certain build tools don't honor setting build history in image spec
however buildah does not allow processing images without build history
since setting build history is still optional as per `OCI` image-spec
( https://github.com/opencontainers/image-spec/blob/main/config.md#properties)
so buildah must support such use-cases.
Closes: https://github.com/containers/buildah/issues/4025
Closes: https://github.com/containers/buildah/issues/3513
Signed-off-by: Aditya R <arajan@redhat.com>
Allows end-users to export final build content or rootfs to external formats.
By default, a local container image is created from the build result. The --output (or -o) flag allows you to override this behavior, and a specify a custom exporter. For example, custom exporters allow you to export the build artifacts as files on the local filesystem instead of a Container image, which can be useful for generating local binaries, code generation etc.
The value for --output is a CSV-formatted string defining the exporter type and options. Currently, local and tar exporters are supported. The local exporter writes the resulting build files to a directory on the client side. The tar exporter is similar but writes the files as a single tarball (.tar).
```console
buildah build --output type=local,dest=dir .
buildah build --output type=tar,dest=rootfs.tar .
buildah build -o dir .
```
Reference: https://docs.docker.com/engine/reference/commandline/build/#custom-build-outputs
Signed-off-by: Aditya R <arajan@redhat.com>
In image-spec 1.0.2, the MediaType field is now available in OCI
manifests, so set it.
Update the imgtype helper to output the MediaType field as-is from OCI
image manifests instead of just always supplying the expected value.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When creating a new image we can avoid hashing before and after compression
when we know compression will not be used. This significantly improves speed
for hashing large containers with --compress=false
There should be no functional differences so tests are not
changed/added.
[NO NEW TESTS NEEDED]
Signed-off-by: Kurt Kartaltepe <kkartaltepe@google.com>
Instead of extracting layer content to a temporary file when we're
committing to transports that aren't containers-storage, record the ID
of a layer and the uncompressed size it has recorded for its contents.
When later asked for a blob, if we cached a layer ID and size, generate
the layer diff on the fly, otherwise check for a file named after the
digest of the requested blob in our cache directory location (usually
used for new layers that we're adding) and the supplemental location
(which can be supplied by a caller).
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Nalin Dahyabhai
openshift-merge-bot[bot]
Aaron Lehmann
Han-Wen Nienhuys
Eng Zer Jun
Daniel J Walsh
Aditya R
Chris Evich
Sascha Grunert
Kurt Kartaltepe