* Refactor NewImageSource to add a manifest type abstraction
Currently, NewImageSource creates a Docker schema2 manifest and an OCI
manifest at the same time. This precludes functionality that isn't
supported by both manifest types, for example zstd compression.
Refactoring this to create only the desired manifest type solves this
and also cleans up the code by separating manifest-type-specific code
into distinct implementations of a "manifest builder".
See discussion in https://github.com/containers/buildah/pull/5452.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
* Review feedback
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
* Review feedback, round 2
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
---------
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
Co-authored-by: flouthoc <flouthoc@users.noreply.github.com>
This option is set from CommitOptions.HistoryTimestamp, which
corresponds to the buildah option '--timestamp', and therefore is off
by default.
If this option is not given, we can save ourselves one layer of
copying (tar filtering is a tar.Reader/tar.Writer connnected with io.Pipe())
Signed-off-by: Han-Wen Nienhuys <hanwen@engflow.com>
As of
e024854ba3,
Uname/Gname fields are not populated on Unix. On Windows, the golang
tar package leaves the Uname/Gname fields empty.
Signed-off-by: Han-Wen Nienhuys <hanwen@engflow.com>
These experimental packages are now available in the Go standard
library since Go 1.21:
1. golang.org/x/exp/slices -> slices [1]
2. golang.org/x/exp/maps -> maps [2]
[1]: https://go.dev/doc/go1.21#slices
[2]: https://go.dev/doc/go1.21#maps
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add API for adding arbitrary layers at commit-time via CommitOptions,
and via methods of the Builder type.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If we're prepending history entries before the one for "this" commit,
make sure the "FROM $baseimage" comment gets set on the first history
entry that we add, not just the one goes with this (maybe) layer diff.
In layers=false mode, the output was so, so confusing otherwise.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Make setting the Parent field in the config blob of a docker format
image optional (yes, we're bringing it back!), since it no longer
appears to be set by newer versions of docker build.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Use slices.Clone() and maps.Clone() instead of our own non-generic
functions. We have to be more careful in a couple of places where we
set items in maps which aren't unconditionally initialized.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Merge the two tar filters, if we need two, that we use when committing
an image. Try to improve passing of error information from the writing
end of a pipe to the reader, so that it can be reported better.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a --sbom flag to `buildah build` and `buildah commit` which will
scan the rootfs and specified context directories to build SPDX or
CycloneDX SBOMs and lists of package URLs.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If the parent image has layers but no history, force our own omitHistory
setting on.
The alternative is to create a history that only explains the presence
of some of the layers in our output image, which looks broken to
everyone who might consume that image, including ourselves if we try to
use it as a base image later.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we have extra files to add to the image, handle them by adding them
to the upper overlay layer before creating the plaintext filesystem
image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a flag to `buildah commit` which allows adding arbitrary files to
the image while we're committing it. When not squashing, they'll take
the form of a second new layer.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When using the working container's rootfs to populate a plaintext disk
image with mkfs, instead of writing .krun_config.json to the rootfs and
then removing it afterward (since we don't want it to show up if the
same working container is later committed to non confidential-workload
image), mount an overlay filesystem using a temporary directory as the
upper and the rootfs as the lower, create the .krun_config.json file in
the overlay filesystem, and use the overlay filesystem as the source
directory for mkfs.
Add the necessary stubs to allow pkg/overlay to at least compile on
non-Linux systems. Change the naming scheme for a test so that the path
names it uses for temporary directories don't include "," or "=", which
can confuse the kernel.
Creating confidential workload images will now only be possible on Linux
systems, but we exec'd out to sevctl to read platform certificates, and
that requires kernel support with vendor firmware, so I don't know that
anyone will actually be impacted by the change.
Teach pkg/overlay.MountWithOptions() to accept `nil` as a pointer to a
struct parameter that is otherwise optional.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Ignore whether or not the final build instruction should produce a layer
if we're squashing or producing a confidential workload, when we'd still
have to produce a layer containing the contents of the base image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an OverrideChanges and an OverrideConfig field to CommitOptions,
both of which can be used to make last-minute edits to the configuration
of an image that we're committing.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Clear the docker-format rootfs and history information before we clear
other fields in the config for confidential workload cases, so that it's
easier to track that it exactly parallels what we're doing with the OCI
format data.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Projects which are using buildah as a library and set `TMPDIR` manually
can stumble upon a use-case where `TMPDIR` was set to a relative path.
Such as `export TMPDIR=.` in such case buildah will try to create a
temporary root using `Mkdirtemp` leading to a point where bundle is not
generated correctly since path was relative.
Following use case can be resolved by making sure that buildah always
converts relative path to absolute path and `GetTempDir` does it well.
Example reproducer with podman
```Dockerfile
FROM alpine
RUN echo hello
```
```console
export TMPDIR=.
podman build --no-cache -t test .
```
Expected failure
```console
STEP 1/2: FROM alpine
STEP 2/2: RUN echo hello
error running container: checking permissions on "buildah2341274198": stat buildah2341274198: no such file or directory
ERRO[0000] did not get container create message from subprocess: EOF
Error: building at STEP "RUN echo hello": while running runtime: exit status 1
```
Closes: RHEL-2598
Signed-off-by: Aditya R <arajan@redhat.com>
Add a --cw option to `buildah build` and `buildah commit`, which takes a
comma-separated list of arguments and produces an image laid out for use
as a confidential workload:
type: sev or snp
attestation_url: location of a key broker server
cpus: expected number of virtual CPUs to run with
memory: expected megabytes of memory to run with
workload_id: a distinguishing identifier for the key broker server
ignore_attestation_errors: ignore errors registering the workload
passphrase: for encrypting the disk image
slop: extra space to allocate for the disk image
At least one of attestation_url and passphrase must be specified in
order for the encrypted disk image to be decryptable at run-time. Other
arguments can be omitted. ignore_attestation_errors is intentionally
undocumented, as it's mainly used to permit some amount of testing on
systems which don't have the required hardware.
Add an `mkcw` top-level command, for converting directly from an image
to a confidential workload.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In golang 1.19, `io/ioutil` is fully deprecated preventing Buildah from
compiling. Replace all calls with equivalent calls from the `os`
package.
Signed-off-by: Chris Evich <cevich@redhat.com>
Podman adds an Error: to every error message. So starting an error
message with "error" ends up being reported to the user as
Error: error ...
This patch removes the stutter.
Also ioutil.ReadFile errors report the Path, so wrapping the err message
with the path causes a stutter.
Signed-off-by: Daniel J Walsh dwalsh@redhat.com
[NO NEW TESTS NEEDED]
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When generating layer diffs or extracting container contents, scrub the
user and group name fields in tar headers before saving them.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If errors for which os.IsExist() or os.IsNotExist() would have returned
true have been wrapped using fmt.Errorf()'s "%w" verb, os.IsExist() and
os.IsNotExist(), not having been retrofitted to use errors.Is(), will
return false.
Use errors.Is() to check if an error is an os.ErrExist or os.ErrNotExist
error instead of calling os.IsExist() or os.IsNotExist().
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
As per `OCI` `image-spec` setting history is optional so buildah must
support these use-cases via `--omit-history=true` some of the use-cases
are discussed below.
* There are use-cases when image-spec contributes to the image size by
adding build history to image-spec in certain use-cases this image
history is not required by the end-users but there is no way to remove
it. See: https://github.com/containers/buildah/issues/3513
* Certain build tools don't honor setting build history in image spec
however buildah does not allow processing images without build history
since setting build history is still optional as per `OCI` image-spec
( https://github.com/opencontainers/image-spec/blob/main/config.md#properties)
so buildah must support such use-cases.
Closes: https://github.com/containers/buildah/issues/4025
Closes: https://github.com/containers/buildah/issues/3513
Signed-off-by: Aditya R <arajan@redhat.com>
Allows end-users to export final build content or rootfs to external formats.
By default, a local container image is created from the build result. The --output (or -o) flag allows you to override this behavior, and a specify a custom exporter. For example, custom exporters allow you to export the build artifacts as files on the local filesystem instead of a Container image, which can be useful for generating local binaries, code generation etc.
The value for --output is a CSV-formatted string defining the exporter type and options. Currently, local and tar exporters are supported. The local exporter writes the resulting build files to a directory on the client side. The tar exporter is similar but writes the files as a single tarball (.tar).
```console
buildah build --output type=local,dest=dir .
buildah build --output type=tar,dest=rootfs.tar .
buildah build -o dir .
```
Reference: https://docs.docker.com/engine/reference/commandline/build/#custom-build-outputs
Signed-off-by: Aditya R <arajan@redhat.com>
In image-spec 1.0.2, the MediaType field is now available in OCI
manifests, so set it.
Update the imgtype helper to output the MediaType field as-is from OCI
image manifests instead of just always supplying the expected value.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When creating a new image we can avoid hashing before and after compression
when we know compression will not be used. This significantly improves speed
for hashing large containers with --compress=false
There should be no functional differences so tests are not
changed/added.
[NO NEW TESTS NEEDED]
Signed-off-by: Kurt Kartaltepe <kkartaltepe@google.com>
Instead of extracting layer content to a temporary file when we're
committing to transports that aren't containers-storage, record the ID
of a layer and the uncompressed size it has recorded for its contents.
When later asked for a blob, if we cached a layer ID and size, generate
the layer diff on the fly, otherwise check for a file named after the
digest of the requested blob in our cache directory location (usually
used for new layers that we're adding) and the supplemental location
(which can be supplied by a caller).
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Remove the parent image's ID from the config blob when we're squashing
the image, since after squashing, we share no layers or history with
what was once our base image, and leaving it set triggers verification
errors in registries that expect consistency between parent IDs and
perhaps layers and history.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Golangs os.* functions return the name of the file/directory they
fail to use. We should not wrap these errors with the file/directory
to use names, causes stuttering when the user sees the errors, and looks
bad having huge error messages.
Since this is just code cleanup, existing tests should handle the
changes.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
every error from copier.Get was previously lost and not propagated
back. That caused the caller to believe the operation was successful
and lead to incomplete images on errors.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This PR removes the pkg/auth which brings in docker/docker
since it really is not needed, and was only there to help users
discover the settings of where the authfile was, when the environment
variables were set. Would almost never be of any value.
Move imagebuildah.BuildOptions to define.BuildOptions
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add the name of the base image being used by the build
in the comments of the first layer created.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
If UID and GID mappings are specified, the container has a
mapped-layer, whose diffID is not computed when created.
Committing the image fails due to lack of diffID. This fix
computes diffID at creating an image source if a layer
doesn't have a diffID (UncompressedDigest).
This fix also tests if a container with UID and GID mappings
can be committed.
Signed-off-by: Hironori Shiina <shiina.hironori@fujitsu.com>
Golang built in functions like os.Create and others print the name of
the file system object when they fail. Wrapping them a second time
with the file system object, makes the error message look like crap
when reported to the user.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Avoid forcing the timestamps on content in new layers to the current
time when a timestamp is not specified.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Aaron Lehmann
Han-Wen Nienhuys
Eng Zer Jun
Nalin Dahyabhai
openshift-merge-bot[bot]
Daniel J Walsh
Aditya R
Chris Evich
Sascha Grunert
Kurt Kartaltepe
Giuseppe Scrivano
Ashley
Diego
Hironori Shiina