Following commit adds buildkit like support for `from` field to `--mount=type=bind`
and `--mount=type=cache` so images and stage can be used as mount source.
Usage looks like
```dockerfile
RUN --mount=type=bind,source=.,from=<your-image>,target=/path ls /path
```
and
```dockerfile
RUN --mount=type=cache,from=<your-image>,target=/path ls /path
```
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Podman uses the new netavark network stack. Buildah should be able to do
the same. Both projects should use the same networking code which was
move to c/common/libnetwork. The new network interface can use either
CNI or netvavark. Using the same code for podman and buildah is
important to ensure that both use the same backend. Mixing CNI and
netavark is not supported.
This also fixes some outstanding CNI issues, e.g. buildah trying to
connect all cni networks.
[NO NEW TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Do not configure the netns in the runtime child process, this removes
the need to send the network options to the child. This will be needed
for the new libnetwork network interface which cannot be transfered as
json.
To synchronize this between the child and parent we use two pipe pairs.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
First move podman/pkg/cgroups into Buildah.
Only set resources to nil on non cgroupsv2 systems in rootless mode.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Buildah is mainly building with Host Namespaces, this changes most
namespaces to be private matching Podman and using containers.conf
Fixes: https://github.com/containers/buildah/issues/3634
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Label stdio pipes to ensure that processes we run can read through
/dev/stdin and write through the /dev/stdout and /dev/stderr links.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This commit makes sure we honor NetworkDisabled in `runs` if
`--network=none` was specified while building the container and no
configuration was specified in while running the container.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Run secrets can now be created from an environment variable. The
environment variable is read and is briefly stored as a file on /dev/shm
when it's being used, and the file is removed after the RUN command is
finished.
Fixes: #3524
Signed-off-by: Ashley Cui <acui@redhat.com>
In linux, directory can contains colon.
Add support to mount path contains colon.
buildah run --volume /root/a\\🅱️/root/test:O
Signed-off-by: chenk008 <kongchen28@gmail.com>
Also switch default hostname from truncated ContainerID to Container
name.
It makes more sense that a user would know the name of the container
versus the generated id, so we should use this as a default.
Fixes: https://github.com/containers/buildah/issues/3509
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Following PR inroduces a new mount type=cache in parity to buildkit
which allows users to share persistant cache between different builds.
Allowing users to cache content generated by bussiness logic or enhance
build performance by caching components across builds.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Fixed a bug where buildah bud mounted secrets permissions were incorrect due to a decimal/octal conversion error. buildah bud mounted secrets now have a default permission of 400. Fixes#3557
Signed-off-by: Ashley Cui <acui@redhat.com>
Following PR adds supports for buildkit like `--mount=type=tmpfs` which
allows end users to mount a chunk of volatile memory instead of a persistent storage device.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Following commit adds support for using buildkit like
`--mount=type=bind` with `RUN` statements. Mounts created by `--mount`
are transient in nature and only scoped to current RUN statements.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Users are not seeing Warnings when doing podman --remote build.
We need to wire the logrus messages all the way through the system.
Fixes: https://github.com/containers/buildah/issues/3537
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Following commit ensures that we cleanup dangling `/run` after every RUN
command and make sure that it does not persists on physical image. Ensure
parity with how docker behaves with `.dockerenv`.
Signed-off-by: Aditya Rajan <arajan@redhat.com>
Check if `go test` supports the -race flag on the build platform, and if
so, use it for unit tests instead of just assuming that it's always
available.
Use sync/atomic to safely use a uint32 instead of a bool to keep track
of whether or not the process we started for RUN has stopped.
[NO NEW TESTS NEEDED]
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
The golang.org/x/crypto/ssh/terminal package has been deprecated and
replaced upstream by golang.org/x/term, so switch to that. It's a
simple 1:1 replacement.
[NO NEW TESTS NEEDED]
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we're told to add built images to a manifest list, manipulate the
list ourselves, so that if we're creating a list, we won't have a
partially-populated list if some of the builds fail.
This also lets us include all of the platform information (including
variant info, which we can't sniff out after the fact) that we were
given when we started building the images.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Allow ssh socket from host or certain ssh keys to be exposed to a
certain RUN instruction, but not any other instructions, as well as not
showing up in the final image.
This is done by spawining a new agent from buildah and mounting
the listening socket inside the run. SSH_AUTH_SOCK inside the container
will be set to the socket mountpoint. The defualt mountpoint is
/run/buildkit/ssh_agent.{i}
Signed-off-by: Ashley Cui <acui@redhat.com>
Make sure to use the correct resolv.conf in case of systemd-resolved.
Also filter out 127... nameserver when run in private netns.
Fixes#2780
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
[NO TESTS NEEDED] since I've no idea how to force it.
Reported-in: containers/podman/issues/11148
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Fixes: https://github.com/containers/buildah/issues/3152
Sometimes after running buildah run in a terminal, commands executed afterwards which try to read from stdin fail with EAGAIN. This is because it sets O_NONBLOCK on the FD:
[NO TESTS NEEDED] Since I don't know how to test this.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
As part of the podman-machine work, we need to stop distributing
the default CNI configuration that has traditionally been
supplied by Podman, 87-podman.conflist. Buildah (as root) also
uses this file, and will be unable to set up container networking
without it.
The replacement we've introduced is the ability to automatically
create said configuration file using a package in
containers/common. This has already been added to Podman, and
this PR adds it to Buildah.
[NO TESTS NEEDED] as there's no real easy way to verify that
Buildah made the network - no explicit command for listing
networks.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
We want Info, Warning and Debug logrus messages to be writen to the
buildah stderr. this way when podman-remote is using build, it will
get the messages back on the client side.
[NO TESTS NEEDED] Since this will be tested in Podman.
Fixes: https://github.com/containers/buildah/issues/3214
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Ed has found situations where the container exits, before we can check
the state causing a failure, where I think we can complete successfully.
Fixes: https://github.com/containers/buildah/issues/3113
[NO TESTS NEEDED] since I have no way to generate this race condition.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Mention that errors we relay from the runtime are from the runtime, to
hopefully reduce confusion when troubleshooting.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add support for secrets. Secrets is a two-part flag that allows secret files to
be accessed for a certain RUN instruction, but not any other
instructions, as well as now showing up in the final image.
Signed-off-by: Ashley Cui <acui@redhat.com>
Check if a builtin volume is a symlink. If it is,
follow the symlink and ensure that the destination
exists.
Add tests for symlink and no symlink case.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Currently depending on the sort order of mount points, we can overmount
a volume specified from the user. Podman has a function sortMount that
sorts all mounts based on destination directory to ensure all mounts
show up. This PR moves the function from Podman to Buildah. Once merged
I will change Podman to use the buildah function.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Expand conformance test coverage for COPY --chown, and correct our
behavior when the argument is a single number: instead of assuming the
number is the UID and GID = 0, the GID should be the same as the UID.
This means that we can't quite use the same function for parsing the
arguments for COPY --chown and USER, as they evidently have different
defaults for the case where the argument is a single number.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This PR removes the pkg/auth which brings in docker/docker
since it really is not needed, and was only there to help users
discover the settings of where the authfile was, when the environment
variables were set. Would almost never be of any value.
Move imagebuildah.BuildOptions to define.BuildOptions
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We have been asked to leak some container information
and image information into the container to be used
by certain tools. (Toolbox and others)
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
pkg/secrets has been moved and slightly been rewrittin
in containers/common, along with pkg/umask.
Convert Buildah to use these packages rather then internal
packages.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
I can't remember why we disabled UTS namespaces for rootless isolation,
but it doesn't look necessary.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Make sure the directory we're creating is explicitly below the chroot we
want to create it under.
Fix a similar incorrect call in imagebuildah.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
It is not entirely correct to always ignore EEXIST here. It should only
be ignored in one special case: when a working directory already exists,
and is an absolute symlink to another directory under container root.
MkdirAll reports an error because the symlink is broken in the host
context (without chroot).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This subtle bug keeps lurking in because error checking for `Mkdir()`
and `MkdirAll()` is slightly different wrt `EEXIST`/`IsExist`:
- for `Mkdir()`, `IsExist` error should (usually) be ignored
(unless you want to make sure directory was not there before)
as it means "the destination directory was already there";
- for `MkdirAll()`, `IsExist` error should NEVER be ignored.
This commit removes ignoring the IsExist error, as it should not
be ignored.
[v2: skip patching (*Builder).Run]
For more details, a quote from opencontainers/runc PR #162:
-quote-
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is
returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
> [1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
-end-quote-
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Golang built in functions like os.Create and others print the name of
the file system object when they fail. Wrapping them a second time
with the file system object, makes the error message look like crap
when reported to the user.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
What `go tool dist list` says the toolchain supports changes, so this
change removes these attempted cross-compile build targets.
* GOOS=darwin, GOARCH unspecified
* GOOS=darwin, GOARCH=386
Replace our use of slices of
github.com/opencontainers/runc/libcontainer/configs.Device structures
with a locally-defined type alias so that we can avoid importing the
package on non-Unixy systems. The result is not going to be a very
useful binary on non-Linux systems, but it helps ensure that our
subpackages won't break compilation for other projects who consume us as
a library.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
The hardwired default for containers.conf now includes a TERM variable,
and passing it through to commands that we "RUN" during a build can
subtly cause the resulting image to be different from one that `docker
build` would create, so stop using it there.
When a runtime runs the image we eventually produce, it'll consult the
configuration file, so the variable will still be set, even when it
isn't set in the image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2560: Rework ADD/COPY to use the copier package r=giuseppe a=nalind
#### What type of PR is this?
/kind failing-test
#### What this PR does / why we need it:
Use the copier package for ADD, COPY, and for ensuring that a specified directory exists in the working container's rootfs. This should improve our speed when a .dockerignore file is being used, and correctness all around.
When evaluating cache for content that's being copied/added in, switch from (digest the data, check for a cache entry, then maybe copy the data and create the new layer) to (copy the data and create the new layer, digesting as we go, check for a cache entry, either commit or discard the new layer). This reduces the amount of data that we read from disk, which helps when the data being read doesn't all fit in the kernel's cache, and if we end up keeping the result, costs us nothing compared to what we were doing before.
#### How to verify it
More of our conformance tests should pass, and existing tests should continue to pass. (We do update a couple of integration tests, but I believe they were expecting incorrect behavior before.)
#### Which issue(s) this PR fixes:
Fixes#574.
Fixes#2476.
#### Special notes for your reviewer:
This is another part of #2480.
#### Does this PR introduce a user-facing change?
```
ADD and COPY should be faster when a .dockerignore file is being used.
```
Co-authored-by: Nalin Dahyabhai <nalin@redhat.com>
Use the copier package to rework how we handle ADD and COPY.
When evaluating cache for content that's being copied/added in, switch
from (digest the data, check for a cache entry, then maybe copy the data
and create the new layer) to (copy the data and create the new layer,
digesting as we go, check for a cache entry, either commit or discard
the new layer).
Use the copier package for ADD, COPY, and for ensuring that a specified
directory exists in the working container's rootfs.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
If there's no network present in the container, don't bind
the /etc/hosts file as any changes to from the caller will
be lost when run exits. Ditto the /etc/resolv.conf file.
Addresses: #2478
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Move the conformance tests from ginkgo to using the default testing
package and github.com/stretchr/testify/require, preserving the existing
tests and adding more.
Add conformance tests to our Cirrus configuration, currently marked as
an allowed failure.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2465: Deduplicate environment variables r=rhatdan a=nalind
#### What type of PR is this?
> /kind cleanup
#### What this PR does / why we need it:
When combining lists of environment variables read from base images with defaults supplied from our own configuration, ensure that the resulting environment we produce only contains one value for any given variable. While adding variables to a runtime spec using `github.com/opencontainers/runtime-tools/generate.Generator.AddProcessEnv()` ensures that later values in the list override values that occur earlier, we shouldn't be depending on that.
#### How to verify it
#### Which issue(s) this PR fixes:
None
#### Special notes for your reviewer:
Related to https://github.com/openshift/imagebuilder/pull/169.
#### Does this PR introduce a user-facing change?
```
None
```
Co-authored-by: Nalin Dahyabhai <nalin@redhat.com>
When combining lists of environment variables, or environment variables
combined with build arguments, always deduplicate sets of values.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
fix a race condition where the container process could exit before the
runtime sends the signal, causing the command to fail.
Part of: https://github.com/containers/crun/issues/422
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The correct args are already contained in the RunOptions.
They were resolved by the imagebuilder library's Step
(edcddd8483/builder.go (L311-L317))
function
This was previously adding args that should not have been accessible
in the current stage as they were not first referenced by an ARG
command
Signed-off-by: Nick Carboni <ncarboni@redhat.com>
do not attempt again to poll the file if it is not pollable and the
the errno is not syscall.EINTR or syscall.EAGAIN.
Closes: https://github.com/containers/buildah/issues/2194
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
After determining the type of runtime to use,
either "runc" or "crun" dependent upon the system, search
the list of that type of runtime in the containers.conf
file. It includes the location of those runtimes in a
number of different architectures. Once found, set the
runtime to use to that value.
Fixes: #2113
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
When building the slice of environment variables to add to the
configuration for a container that we're about to run, in case of
conflicts, we want the values from the base image or working container
to override the global defaults, and we want values that were passed to
us through the API to override them both.
In cases of conflicts, values which occur later in the slice override
values which occurred earlier, so we want to add them in this order:
* values from containers.conf
* values from the base image or working container
* values passed to us through the API
We previously applied the containers.conf defaults after the base image
or working container's value, and that meant that containers.conf's
values always took precedence over the values in the image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In porting containers.conf to libpod, we found that buildah needed
to handle the containers.conf on the server side rather then from
the CLI.
Since the `podman-remote build` would probably not have the same content
as containers.conf on the server, the processesing of the defaults needs
to be handled in imagebuildah. The CapAdd and CapDrop values need to be
passed in.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This is a rework of Qi Wang's patches.
Import package pkg/config from containers/common to read containers.conf
This patch allows users to specify default values stored in containers.conf
that will modify the behaviour of buildah tool.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #2011
Approved by: TomSweeneyRedHat
Podman uses the overlay mounts differently then in buildah. Specifically the
overlay mount points can be used over and over again when starting and stopping
the container. Since the paths are backed into the contianer config, we have
to be able to cleanout just the Upper and Merged directory rather then destroying
and recreating the overlay directories on each container start.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1822
Approved by: vrothberg
If host is running in fips mode, then RHEL8.2 and beyond container images
will come with a directory /usr/share/crypto-policies/back-ends/FIPS.
This directory needs to be bind mounted over /etc/crypto-policies/back-ends in
order to make all tools in the container follow the FIPS Mode rules.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We have moved share code from buildah, podman and others into containers/common.
Specifically for this PR we are moving to use containers/common/pkg/unshare and
containers/common/pkg/cgroups.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #2010
Approved by: QiWang19
If disableFips is set, then we don't mount the FIPS
secret even if the machine is in FIPs mode. This is
to help users run workloads that are not fips compliant
in openshift even if their machine is in FIPs mode.
This is needed in CRI-O.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Some Dockerfiles (fuse-overlay) require additional devices to be in the
build environment.
This patch allows the user to specify additional devices.
Also I noticed that CapAdd and CapDrop was not working in buildah bud situations,
so this patch also fixes this.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1820
Approved by: @TomSweeneyRedHat
Container processes want to check for the existence of this file
to determine if they are running inside of a container.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add a DryRun flag to AddAndCopyOptions, so that we can "copy" content to
digest it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1792
Approved by: TomSweeneyRedHat
fix the detection code for running in a user namespace. When buildah
is running in rootless mode, a user namespace is automatically created
even if there are no mappings configured.
Closes: https://github.com/containers/libpod/issues/2972
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #1775
Approved by: rhatdan
This commit enabled the `unparam` linter and applies all reported issues.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Closes: #1719
Approved by: rhatdan
Aditya Rajan
Daniel J Walsh
Paul Holzinger
Nalin Dahyabhai
Giuseppe Scrivano
OpenShift Merge Robot
Ashley Cui
chenk008
openshift-ci[bot]
Valentin Rothberg
Matthew Heon
Urvashi Mohnani
Eduardo Vega
Josh Soref
bors[bot]
TomSweeneyRedHat
dependabot-preview[bot]
Brandon Lum
Nick Carboni
Sascha Grunert