When generating archives, clear user and group names to keep up with
recent changes to the storage library.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Have Ensure() also return the parent directories of items that it
created, along with information about them that can be used to filter
them out of the layer at commit-time.
This modifies the signature of Ensure(), but it was added in 1.41.0, and
shouldn't (yet) have any external users.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
The `filepath.Glob` function does not provide deterministic output. In order to achieve a reproducible build, files must be copied in a deterministic manner, and `filepath.Glob` did not guarantee this. Other functions such as `filepath.Walk` and `os.ReadDir` return deterministic output. So copying files to the image is done in the same order each time.
Fixes: https://issues.redhat.com/browse/RUN-2661
Signed-off-by: Jan Rodák <hony.com@seznam.cz>
Add copier.Ensure() and copier.ConditionalRemove(), for controlling the
permissions and datestamps we set on multiple items we create with one
call (along with any parents), and selectively removing multiple items
with one call.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a --timestamp flag to the "add" and "copy" CLIs, along with a
corresponding field in AddAndCopyOptions.
When a timestamp is set, we'll force the timestamp on data copied in to
be the specified value while reading it, so that the content will have
the specified datestamp in the rootfs and when the image is committed.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Handle glob patterns with "**" path components by expanding "**" to the
set of subdirectories and calling filepath.Glob() on the results.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Fixes golangci-lint message:
`tar.TypeRegA has been deprecated since Go 1.11 and an alternative has
been available since Go 1.1: Use TypeReg instead.`
Signed-off-by: Chris Evich <cevich@redhat.com>
In golang 1.19, `io/ioutil` is fully deprecated preventing Buildah from
compiling. Replace all calls with equivalent calls from the `os`
package.
Signed-off-by: Chris Evich <cevich@redhat.com>
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
If errors for which os.IsExist() or os.IsNotExist() would have returned
true have been wrapped using fmt.Errorf()'s "%w" verb, os.IsExist() and
os.IsNotExist(), not having been retrofitted to use errors.Is(), will
return false.
Use errors.Is() to check if an error is an os.ErrExist or os.ErrNotExist
error instead of calling os.IsExist() or os.IsNotExist().
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Similar to the `NoOverwriteDirNonDir` one, add an option that disables
non-directories from being overwritten by directories.
Required-for: containers/podman/issues/14420
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Add StripSetuidBit/StripSetgidBit/StripStickyBit flags to
copier.PutOptions, that are interpreted similarly to their counterparts
in copier.GetOptions.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Try to improve our ability to write to directories that aren't
writable. If we encounter an EPERM error while attempting to create an
item, attempt to temporarily make writable the directory that we're
writing the item to, and restore its permissions on our way out.
The error usually isn't seen when run as UID 0, whether in a user
namespace or not, which is usually how we're called, but running the
unit tests as an unprivileged user will verify it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Handing Put() an archive which uses host IDs while also including ID
maps for mapping from container IDs to host IDs is a mistake, and the
tests have been failing when run by non-root users since storage v1.38
started flagging it as such.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Should speed up most walks escpecially if they don't need to
stat every directory entry.
[ NO NEW TESTS NEEDED]
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When checking if something that we want to overwrite with a directory is
already a directory or not, use lstat instead of stat. If it's a
symbolic link, it's not a directory.
This is a subtle behavior change, but it's in line with docker build.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we attempt to remove a directory to make way for a non-directory as
part of extracting content, use RemoveAll() instead of Remove().
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When processing a directory tree, only descend into a directory that is
marked for exclusion if its path is literally a prefix of an exception
pattern.
Subtly, but in a way that's compatible with docker, this means that if
we exclude directory "subdir", but we've been told to also include
"**/file" (with an exclusion pattern of "!**/file"), we won't descend
into "subdir" and find a file named "subdir/file", because "**/file"
doesn't start with "subdir/".
More generally, exclusion patterns that start with "!" which include any
wildcards before their final component technically won't be treated
correctly.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add an IgnoreUnreadable flag to copier.GetOptions to suppress errors
from copier.Get() that would pass the os.IsPermission() test, if they're
encountered while attempting to read files or descend into directories.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Always create the destination directory first when ADDing or COPYing
content into a container, then extract contents into it using the
destination directory as the chroot instead of the container's root
directory.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a NoDerefSymlinks flag to force items that are matched to the Globs
we're given to be treated as symlinks, rather than dereferencing them as
we would need to do for sources for ADD or COPY.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add copier.Eval(), for expanding paths using symbolic links in a
chrooted scope, without failing if a path component doesn't exist.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When attempting to handle renames, we'd fail to correctly handle renames
of prefixes of a given item's path because of a string handling error,
and add a unit test for the rename logic (finally).
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Prior to 1.16, when ADDing contents to a working container, if we were
being run by an unprivileged user using a user namespace, content that
was a device node would be ignored.
Add a flag in copier.PutOptions that tells copier.Put() to ignore
entries that are either a device, or a hard link to a device.
Make buildah.Add() set the IgnoreDevices flag in PutOptions when
libcontainer says we're running in a user namespace.
Together, these two changes should restore the earlier behavior.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Fall back to non-chroot behavior on Unixy systems when we're not started
as UID 0. Break the unit tests that exercise chroot functionality into
a separate file so that we don't even try to run those cases on non-Unix
OSes.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a flag for controlling overwriting-directories-with-non-directories
behavior in PutOptions, and fields for controlling name mappings to both
GetOptions and PutOptions.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When a directly-named (or globbed) source directory for ADD or COPY is
marked for exclusion by .dockerignore, check if its name is a prefix for
any exceptions in the .dockerignore file, and if it is, check the
directory for things we need to include anyway.
This will miss exceptions where the pattern uses a wildcard for anything
but the final component.
When adding items, count items that are actually passed over the tar
pipe, rather than items scanned, so that we can correctly diagnose not
having found anything that we needed to copy under a directory that
would otherwise have been excluded.
In copierHandlerGet(), just don't discount any globbed directories that
are excluded.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add a function for doing Mkdir-possibly-in-a-chroot, for ensuring that a
directory exists without having to possibly create it directly from
outside of a chroot.
Make use of filepath.ToSlash() and filepath.FromSlash() where it's
appropriate.
Add more unit tests.
Address some review comments:
* Check for ERANGE instead of E2BIG errors from llistxattr() and
lgetxattr() as indicators that the buffer we passed to them is too
small.
* Factor an `isRelevantXattr` helper out of Lgetxattrs() and
Lsetxattrs().
* Drop our getcwd() function in favor of using os.Getwd().
* Adjust the comment describing the GetOptions.KeepDirectoryNames field.
* Clean hdr.Name before attempting to compute where an item being
extracted should go.
* When writing items to the filesystem, create with 0?00 permissions,
set ownership, then set the correct permissions.
* Merge StatResponse.Error, GetResponse.Error, and PutResponse.Error
into Response.Error.
* When reading items from the filesystem, if a glob matches multiple
items, and one of the items is excluded, continue checking the other
items.
* Make sure we always Wait() on a child process if we spawned one.
* Clean up the cleanup logic for pipes that we use to communicate with a
child process.
* Clean up the kill-the-child-process logic we call when we encounter an
error communicating with the child process.
* Drop the separate Options structure, use helper methods to simplify
pulling out the right ID maps and exclusions list for the request.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
For the sake of conformance tests, callers need to be able to strip
setuid and setgid bits from contents being copied from the build context
while leaving the sticky bit intact. Split the StripSetidBits option
for copier.Get() into three separate flags (StripSetuidBit /
StripSetgidBit / StripStickyBit).
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add new primitives for reading and writing data, represented as tar
streams, from and to possibly-chrooted directories via subprocesses.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Nalin Dahyabhai
Jan Rodák
openshift-merge-bot[bot]
Kir Kolyshkin
Chris Evich
Eng Zer Jun
Sascha Grunert
Valentin Rothberg
OpenShift Merge Robot
Daniel J Walsh