In Docker if you are copying more then one object, and
one of them is successful, then the command is successful. Currently in
buildah each glob has to be successful. This PR matches Buildah to
Docker.
Fixes: https://github.com/containers/podman/issues/9594
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Expand conformance test coverage for COPY --chown, and correct our
behavior when the argument is a single number: instead of assuming the
number is the UID and GID = 0, the GID should be the same as the UID.
This means that we can't quite use the same function for parsing the
arguments for COPY --chown and USER, as they evidently have different
defaults for the case where the argument is a single number.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Always create the destination directory first when ADDing or COPYing
content into a container, then extract contents into it using the
destination directory as the chroot instead of the container's root
directory.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When a COPY command was being targeted at an empty new directory,
the copy would fail. It could be "worked around" by putting a dummy
file into the directory.
Addresses: #2964
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When a COPY command was being targeted at an empty new directory,
the copy would fail. It could be "worked around" by putting a dummy
file into the directory.
Addresses: #2964
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Prior to 1.16, when ADDing contents to a working container, if we were
being run by an unprivileged user using a user namespace, content that
was a device node would be ignored.
Add a flag in copier.PutOptions that tells copier.Put() to ignore
entries that are either a device, or a hard link to a device.
Make buildah.Add() set the IgnoreDevices flag in PutOptions when
libcontainer says we're running in a user namespace.
Together, these two changes should restore the earlier behavior.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Incorporate the value of the --chown option when digesting content added
from a URL, so that build caching will take it into account. Add tests
to ensure that layer caching takes into account the changed ownership of
items that we're adding or copying.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Golang built in functions like os.Create and others print the name of
the file system object when they fail. Wrapping them a second time
with the file system object, makes the error message look like crap
when reported to the user.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When a directly-named (or globbed) source directory for ADD or COPY is
marked for exclusion by .dockerignore, check if its name is a prefix for
any exceptions in the .dockerignore file, and if it is, check the
directory for things we need to include anyway.
This will miss exceptions where the pattern uses a wildcard for anything
but the final component.
When adding items, count items that are actually passed over the tar
pipe, rather than items scanned, so that we can correctly diagnose not
having found anything that we needed to copy under a directory that
would otherwise have been excluded.
In copierHandlerGet(), just don't discount any globbed directories that
are excluded.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When extracting archives that are added using ADD, don't override
permissions and ownership information. We regressed on this when we
switched to using the copier package to handle them.
Add a conformance test to prevent regressions on this.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Add() is supposed to handle relative paths when the ContextDir value
passed to it is not set, but it hasn't been doing that correctly since
it was overhauled. Correct it to recognize relative paths at the
function start, when ContextDir is not set, by combining them with the
current directory to convert them to absolute paths.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
COPY --from was incorrectly discarding ownership information on files
copied from other layers, which unlike content copied from the build
context, should not default to being owned by 0:0.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Use the copier package to rework how we handle ADD and COPY.
When evaluating cache for content that's being copied/added in, switch
from (digest the data, check for a cache entry, then maybe copy the data
and create the new layer) to (copy the data and create the new layer,
digesting as we go, check for a cache entry, either commit or discard
the new layer).
Use the copier package for ADD, COPY, and for ensuring that a specified
directory exists in the working container's rootfs.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Don't start digesting the contents of any file that we end up skipping.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #2072
Approved by: giuseppe
When generating the list of exclusions to process .dockerignore
contents, don't include .dockerignore if we don't have a .dockerignore
file in the context directory. That way, if the file doesn't exist, and
the caller didn't pass in any patterns, we get no patterns instead of
just one ".dockerignore" pattern, and we can hit the faster copy path.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #2072
Approved by: giuseppe
We are always adding .dockerignore to the pattern list, if this is the only pattern
then no patterns we added to the list, and we should return nil.
This is causing a major slowdown in buildah, since it is not using the optimized tar for
copying.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #2070
Approved by: saschagrunert
fix bug that buildah does not check .dockerignore file if source of COPY instruction is a filepath. This patch will skip the path if the overall result of the dockerignore matcher is true.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Closes: #2064
Approved by: rhatdan
In #1715 we added an additional valid path for `!excludes.Exclusions()`
when considering a full copy of the source directory to the destination
via `ADD`/`COPY`. This breaks the functionality of the `.dockerignore`
file, because `PatternMatcher.Exclusions` is true if any of the defined
patterns contain an exclusion (the pattern starts with `!`). This means
if we have no Exclusions defined at all, the `.dockerignore` behavior
does not work any more.
To still have the optimization in, we now skip whole directories if the
patterns to exclude match in a pure way.
Introduced by: https://github.com/containers/buildah/pull/1715
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Closes: #1914
Approved by: rhatdan
Use digests of the added content in history entries that we create for
ADD and COPY instructions, tightening up cache checking just a little
bit more.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1792
Approved by: TomSweeneyRedHat
Add a DryRun flag to AddAndCopyOptions, so that we can "copy" content to
digest it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1792
Approved by: TomSweeneyRedHat
Teach copyFileWithTar() about symbolic links and directories, and use it
to produce tar data to feed to untar() instead of special-casing them.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1785
Approved by: vrothberg
This commit enabled the `unparam` linter and applies all reported issues.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Closes: #1719
Approved by: rhatdan
when there are excludes defined (such as from .dockerignore), we take
a slow path and walk each file in the directory. If the files doesn't
match any exclusion pattern then it is copied into the container.
This is slow as each file requires buildah to re-exec and copy it from
a chroot environment.
When there are no excludes defined we can take a faster path and copy
the entire directory as a single re-exec operation.
Closes: https://github.com/containers/buildah/issues/1714
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #1715
Approved by: rhatdan
Fix handling of ID mapping for COPY: when copying from other containers,
use their mappings, and when copying from the host, use host mappings.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1630
Approved by: TomSweeneyRedHat
on 32-bit systems, we had type mismatches when creating a
syscall.timespec.
resolves#1629
Signed-off-by: baude <bbaude@redhat.com>
Closes: #1631
Approved by: rhatdan
Use a fileutiles.PatternMatcher, which works better for this case than
filepath.Match, to check if an item that we're copying is marked for
exclusion by .dockerignore.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1626
Approved by: TomSweeneyRedHat
If we fail to create a symbolic link because the destination already
exists, attempt to remove the destination.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1623
Approved by: giuseppe
Checks to see if the $HOME envvar has been set
and if not, trys to set it as best as possible.
Fixes: #1592
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Closes: #1594
Approved by: rhatdan
When filtering copying/adding operations to comply with a .dockerignore
file, we should directly handle copying directories and symbolic links.
This update still returns an error when attempting to copy
non-directory, non-regular items.
Make DockerIgnoreHelper() private, since it's not being used by packages
that use us, and I expect it to be removed when this gets refactored.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1583
Approved by: rhatdan
Fix the destination file path if .dockerignore is not empty. Avoid copying the source directory level into the container.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Closes: #1533
Approved by: rhatdan
my former PR #1403 didn't take into account the directory path in .dockerignore. `buildah bud` should be able to exclude the whole folder and keep
some files under the folder according to the patterns in .dockerignore.
```
// .dockerignore
test*
!*/test1*
```
should exclude `./testdir/test2.txt` but copy `./testdir/test1.txt`
Signed-off-by: Qi Wang <qiwan@redhat.com>
Closes: #1478
Approved by: rhatdan
We don't want to vendor anything from libpod into Buildah.
We want to switch this around. Moving pkg content from libpod
to Buildah allows us to fix this.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1400
Approved by: giuseppe
export fields in DockerIgnore struct for implementing the dockerignore in libpod
Signed-off-by: Qi Wang <qiwan@redhat.com>
Closes: #1431
Approved by: rhatdan
Exclude(or copy,add) files match with patterns in .dockerignore when bud with ADD or COPY in Dockerfile.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Closes: #1403
Approved by: TomSweeneyRedHat
Make sure that when attempting to diagnose an error, if we encounter an
error during the diagnostic attempt, we return the original error rather
than the error encountered in trying to diagnose it. Log that one.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1072
Approved by: rhatdan
Users expect to be able to use Symbolic links in the
ADD and COPY commands in Dockerfiles. We need to evaluate
these and pull in the correct content.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1007
Approved by: nalind
Break runSetupIntermediateMountNamespace() into its own package.
Move stringInSlice(), getHostIDs(), and getHostRootIDs() into the util
subdirectory and export them.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #796
Approved by: rhatdan
Add a field to AddOrCopyOptions that can take an io.Writer, more often a
hash.Hash returned by digest.Digester's Hash() method, to calculate a
sum over what we add or copy.
Make the help output summarizing the arguments that "buildah add" and
"buildah copy" accept more closely match their man pages.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #766
Approved by: rhatdan
In Run(), when the userspec doesn't specify a group, if the specified
user has supplemental group memberships, pass them along to the runtime.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #700
Approved by: rhatdan
Use ID mapping information when setting permissions on content that we
add to the container, and on secrets that we copy in, on pipes that we
use for stdio, and when extracting the whole filesystem as a "layer".
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #700
Approved by: rhatdan
Add options to the CLI that specify which cgroups we execute "run"
commands under, and controlling how we set up namespaces for them.
Pass them down to Builders that we create, and allow them to be
overridden by options passed to Builder.Run().
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #700
Approved by: rhatdan
Daniel J Walsh
Nalin Dahyabhai
Jakub Guzik
OpenShift Merge Robot
TomSweeneyRedHat
yangfeiyu20102011
Qi Wang
Sascha Grunert
Giuseppe Scrivano
Eric Hripko
baude