Add a conformance test for cases where an intermediate stage mounts the
contents of a previous stage in a read-write fashion and modifies it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an OverrideChanges and an OverrideConfig field to CommitOptions,
both of which can be used to make last-minute edits to the configuration
of an image that we're committing.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Following PR is a attempt to add `Heredoc` support to buildah.
Once this PR is merged buildah is supposed to honor heredoc syntax while
processing containerfiles
Expected syntax to work
```Dockerfile
FROM docker.io/library/python:latest
RUN <<EOF
echo "Hello" >> /hello
echo "World!" >> /hello
EOF
RUN python3 <<EOF
with open("/hello", "w") as f:
print("Hello", file=f)
print("Something", file=f)
EOF
RUN ls -a
RUN cat hello
```
Signed-off-by: Aditya R <arajan@redhat.com>
When uploading a context directory to dockerd, generate the archive
ourselves so that we can force the ownerships in it to 0:0, which
BuildKit seems to expect clients to do.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Co-authored-by: flouthoc <flouthoc@users.noreply.github.com>
go-dockerclient gained the ability to let us ask for a build kicked off
using its API to be done using BuildKit, so we don't have to work around
that by calling the Docker client package any more when doing
conformance testing.
The go-dockerclient method also reports errors in a way that's easier
for us to consume, which we didn't have fully debugged on the other code
path.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Use require.NoErrorf() as a check instead of the more general
require.Nil(), which was both less specific and didn't expect a format
string.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We handle --label command line arguments by appending LABEL instructions
to the Dockerfile contents before we parse it. Previously, we were
appending a separate line for each label-value pair. Consolidate them
for the sake of tools that arbitrarily limit the length of histories
that they're willing to accept in images (boo!).
Add a similar implementation for --env command line arguments.
Previously, we'd set them in the initial configuration for each stage
and also set them at commit-time, and that potentially overrode any
values that were explicitly in the stage itself, and which would have
affected RUN instructions. Remove the set-at-commit-time logic so that
the history reflects what ends up in the image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In golang 1.19, `io/ioutil` is fully deprecated preventing Buildah from
compiling. Replace all calls with equivalent calls from the `os`
package.
Signed-off-by: Chris Evich <cevich@redhat.com>
When noting that a non-symlink has setuid/setgid/sticky bits, switch
from using "syscall" package bits and syscall.Chmod() to using "os"
package bits and os.Chmod(), and if the item's a directory, record the
updated mode information in the "directoryModes" map that we'll use to
reset its permissions later.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Podman adds an Error: to every error message. So starting an error
message with "error" ends up being reported to the user as
Error: error ...
This patch removes the stutter.
Also ioutil.ReadFile errors report the Path, so wrapping the err message
with the path causes a stutter.
Signed-off-by: Daniel J Walsh dwalsh@redhat.com
[NO NEW TESTS NEEDED]
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
The test passes currently, but testing the reverse of the
replace-symlink-with-directory case seems like a good idea.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When checking if something that we want to overwrite with a directory is
already a directory or not, use lstat instead of stat. If it's a
symbolic link, it's not a directory.
This is a subtle behavior change, but it's in line with docker build.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
The API flag that we'd need to set to in a build request to get dockerd
use BuildKit instead of the classic docker builder isn't available in
go-dockerclient, so add a second docker-based path that uses the API
types and client library, which the conformance tests were already
pulling in as indirect dependencies, but skip tests that set the flag if
we're not on the current client version.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When processing a directory tree, only descend into a directory that is
marked for exclusion if its path is literally a prefix of an exception
pattern.
Subtly, but in a way that's compatible with docker, this means that if
we exclude directory "subdir", but we've been told to also include
"**/file" (with an exclusion pattern of "!**/file"), we won't descend
into "subdir" and find a file named "subdir/file", because "**/file"
doesn't start with "subdir/".
More generally, exclusion patterns that start with "!" which include any
wildcards before their final component technically won't be treated
correctly.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Modify how we process exceptions in conformance tests so that we don't
unintentionally disable an attribute comparison (such as "mtime") on
items below a directory when we try to skip that attribute comparison on
the directory itself.
Fix some incorrect specifications for filesystem differences that we're
supposed to ignore, and use the -t flag in more places where we RUN the
`touch` command to create files that end up in the final image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Sometime around `docker.io 20.10.2-0ubuntu1~20.04.2` several error
message strings were updated from `file not found` to `file does not
exist`. This breaks conformance testing. Fix this by adding in
the new error message.
Signed-off-by: Chris Evich <cevich@redhat.com>
When SELinux is enabled, add the :Z mount flag to transient mounts that
we use in the tests that use transient mounts.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In Docker if you are copying more then one object, and
one of them is successful, then the command is successful. Currently in
buildah each glob has to be successful. This PR matches Buildah to
Docker.
Fixes: https://github.com/containers/podman/issues/9594
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add conformance tests that ensure that when we COPY a subdirectory
that's specified as an absolute path, that we don't regress on treating
the absolute path as still being relative to the build context's
location.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Check that COPY --from doesn't do something different when given a
symbolic link as its source than it would without the --from.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Expand conformance test coverage for COPY --chown, and correct our
behavior when the argument is a single number: instead of assuming the
number is the UID and GID = 0, the GID should be the same as the UID.
This means that we can't quite use the same function for parsing the
arguments for COPY --chown and USER, as they evidently have different
defaults for the case where the argument is a single number.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we're using the overlay driver (which means we know overlay is
available), use it to make volumes appear to be writeable during RUN
instructions instead of saving/restoring their contents.
This avoids having to copy the contents of the volume directory before
each RUN instruction, and having to remove and extract the contents
after each RUN instruction, which should be faster, particularly if the
amount of content in that volume location is large.
For empty directories, it will at least avoid adding an "opaque"
notation for the directory in a layer that might otherwise be empty.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This PR removes the pkg/auth which brings in docker/docker
since it really is not needed, and was only there to help users
discover the settings of where the authfile was, when the environment
variables were set. Would almost never be of any value.
Move imagebuildah.BuildOptions to define.BuildOptions
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Always create the destination directory first when ADDing or COPYing
content into a container, then extract contents into it using the
destination directory as the chroot instead of the container's root
directory.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Current version of Docker, has a bug we believe, that is
creating top level directories when using ADD and COPY of
tar balls. Basically the directories end up with a 777
permissions, Buildah creates these with 755, which we
believe is correct, and matches older versions of Docker.
We need to revert this patch once we have a version of Docker
that creates these direcories with the corret permissions.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Prior to Docker 20.10, failed COPY/ADD instructions result in error
messages containing the string `no such file or directory`. This was
changed in docker to a string containing `file not found`. Since
the conformance tests potentially need to work with multiple docker
versions, they need to be sensitive to both errors.
Signed-off-by: Chris Evich <cevich@redhat.com>
Improve handling of cases where extracting an archive requires us to
replace a directory with something that is not a directory, or vice-
versa:
* when replacing a directory with something that isn't a directory,
remove the directory even if it has contents
* don't fail when replacing something that isn't a directory with a
directory
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Ignore PAX global headers when extracting archives, like the archive
package does, instead of erroring out.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When a directly-named (or globbed) source directory for ADD or COPY is
marked for exclusion by .dockerignore, check if its name is a prefix for
any exceptions in the .dockerignore file, and if it is, check the
directory for things we need to include anyway.
This will miss exceptions where the pattern uses a wildcard for anything
but the final component.
When adding items, count items that are actually passed over the tar
pipe, rather than items scanned, so that we can correctly diagnose not
having found anything that we needed to copy under a directory that
would otherwise have been excluded.
In copierHandlerGet(), just don't discount any globbed directories that
are excluded.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When ADD was used to ADD a directory, the contents of archives that we
found inside of it were incorrectly being expanded at the destination
for the ADD.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we carried over the linting configuration from podman, we carried
over the list of checkers that were disabled for podman's sake, even
ones that don't complain about the code in this repository.
Make trivial changes to make gosimple happy.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When extracting archives that are added using ADD, don't override
permissions and ownership information. We regressed on this when we
switched to using the copier package to handle them.
Add a conformance test to prevent regressions on this.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
COPY --from was incorrectly discarding ownership information on files
copied from other layers, which unlike content copied from the build
context, should not default to being owned by 0:0.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Use the copier package to rework how we handle ADD and COPY.
When evaluating cache for content that's being copied/added in, switch
from (digest the data, check for a cache entry, then maybe copy the data
and create the new layer) to (copy the data and create the new layer,
digesting as we go, check for a cache entry, either commit or discard
the new layer).
Use the copier package for ADD, COPY, and for ensuring that a specified
directory exists in the working container's rootfs.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When ADDing an archive file, the destination file-name is blanked out to ensure
that archive contents are extracted in-place. However, when COPYing an
archive-file, we don’t want to blank out the destination filename.
Fixes: #2549
Signed-off-by: Sebastian Reuße <seb@wirrsal.net>
Ignore the buildah.BuilderIdentityAnnotation label when comparing images
that we build with images built using other tools, which of course don't
automatically set that label.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Nalin Dahyabhai
Aditya R
Daniel J Walsh
Chris Evich
Eng Zer Jun
Sascha Grunert
Giuseppe Scrivano
Kir Kolyshkin
Josh Soref
OpenShift Merge Robot
Sebastian Reuße