Move all code related handling container image over to the new
`libimage` package in containers/common. The new package is an
attempt to consolidate the code across the containers tools under the
github.com/containers umbrella.
The new `libimage` packages provides functionality to perform all kinds
of operations for managing images such as local lookups, pushing,
pulling, listing, removing, etc.
The following packages have been moved over the containers/common:
`manifests` -> `common/image/manifests`
`pkg/manifests` -> `common/pkg/manifests`
`pkg/supplemented` -> `common/pkg/supplemented`
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Buildah currently handles multiple tags when building, but does not
report it to the user by default. This reports the tags back to the
user.
Removed some unused code from commit.go, that would blow up if a user
specified AdditionalTags to the commit command, even though this is not
exposed to the user currently. In a previous try to fix this, the
removed code was causing breakage, and I did not see a real purpose in
the code.
Fixes: https://github.com/containers/buildah/issues/3084
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
This PR removes the pkg/auth which brings in docker/docker
since it really is not needed, and was only there to help users
discover the settings of where the authfile was, when the environment
variables were set. Would almost never be of any value.
Move imagebuildah.BuildOptions to define.BuildOptions
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
We want to shrink the size of the import when importing pkg from
buildah. This should help us shrink the size of the golang bindings
in podman.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Prefix the image ID with the hash prefix when using `--iidfile` to be
compatible with Docker. The absence of the hash can cause
docker-compose to error out.
Reported-in: github.com/containers/podman/issues/9260
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Buildah bud --manifest XYZ was not working.
The manifest was never created. This PR Finishes
the plumbing and allows users to create a manifest
while building an image in one single command.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
These changes will allow the "FROM" statement in a Containerfile
to be overridden with a new `--from` option. If I have this Dockerfile.fedora
```
FROM fedora
```
This command will instead build an alpine image:
```
STEP 1: FROM alpine
Completed short name "alpine" with unqualified-search registries (origin: /etc/containers/registries.conf)
Getting image source signatures
Copying blob 188c0c94c7c5 done
Copying config d6e46aa247 done
Writing manifest to image destination
Storing signatures
STEP 2: COMMIT tom
--> d6e46aa2470
d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
```
Addresses: #2404
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Allow users to override location of the .dockerignore file.
If user specified an --ignorefile flag, buildah will read the
file and pass in the exclude lines based on the .dockerignore
contents.
Also add a --contextdir flag to buildah copy and buildah add to
specify where the context directory is located. For these two
commands the --ignorefile flag requires the --contextdir flag.
When the --ignorefile flag is passed in, the .dockerignore files
in the context directory will be ignored.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Cache the information we look up about images when evaluating them as
potential build cache matches, in an effort to reduce the number of
NewImage() calls we make, because they can be expensive.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When evaluating cache candidates during a build, only consider an image
if it's in the same format that we're attempting to build. That way, we
won't mistakenly try to use an OCI format image as a cache when we're
attempting to build an image in Docker format because we're using
configuration features specific to that format, but we forgot to specify
the format during a previous attempt.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
What `go tool dist list` says the toolchain supports changes, so this
change removes these attempted cross-compile build targets.
* GOOS=darwin, GOARCH unspecified
* GOOS=darwin, GOARCH=386
Replace our use of slices of
github.com/opencontainers/runc/libcontainer/configs.Device structures
with a locally-defined type alias so that we can avoid importing the
package on non-Unixy systems. The result is not going to be a very
useful binary on non-Linux systems, but it helps ensure that our
subpackages won't break compilation for other projects who consume us as
a library.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Update imagebuildah.StageExecutor.intermediateImageExists() to recognize
images based on scratch as viable candidates for cache images when we
ourselves are based on scratch.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
We recieved feedback on the --omit-timestamp that
users would rather specify the timestamp seconds
rather then just use EPOCH.
This PR removes --omit-timestamp from buildah bud
since this has never been released.
We also hide --omit-timestamp from buildah commit
and allow users to continue to use it, but it conflicts
with --timestamp.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add a flag to imagebuildah.BuildOptions that will log timing and i/o
information at each step of the build process, and enable it if we're
given the hidden --log-rusage flag in the CLI.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
I can not seem to get this to happen locally, but if this
happens, I believe this should fix the problem.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Currently you can only do deterministic builds with commit command
this change will cause the metadata in the container image to be
epoch 0.
Next step is to save the data in the tar balls as 0.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
The imagebuilder.Executor.imageMap map is written in one goroutine which
waits on StageExecutors, and is read from the goroutines that run the
StageExecutors, so we need to synchronize access to the map.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When checking if a name specified for a FROM instruction, or a source
for a COPY instruction, was a stage for which it needed to wait,
StageExecutor was checking the Executor's stages list, to which stages
are added only as they're started. This could cause it to mistakenly
write off a name if the corresponding stage hadn't yet been started, and
assume that the named stage was actually an image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When building multiple stages, we weren't ensuring that the image
reference we returned to our caller was a reference to the final image
we built.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When a multi-stage build uses the result of one stage as the base image
for a later stage, make the later stage wait for the earlier stage to
finish building its image before the later stage attempts to access that
image.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Make the Jobs field in imagebuildah.BuildOptions optional. When API
callers didn't set it, we were treating the default (zero) as if it
specified that every stage should be built in parallel.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Don't allocate the Executor's stages member, which is a map, from
possibly-multiple goroutines that check if it's nil and allocate it if
it is, since that's racy. Just allocate the map when we create the
structure it's part of.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2321: imagebuildah: stages shouldn't count as their base images r=rhatdan a=nalind
<!--
Thanks for sending a pull request!
Please make sure you've read and understood our contributing guidelines
(https://github.com/containers/buildah/blob/master/CONTRIBUTING.md) as well as ensuring
that all your commits are signed with `git commit -s`.
-->
#### What type of PR is this?
<!--
Please label this pull request according to what type of issue you are
addressing, especially if this is a release targeted pull request.
Uncomment only one `/kind <>` line, hit enter to put that in a new line, and
remove leading whitespace from that line:
-->
/kind bug
#### What this PR does / why we need it:
When initializing a build stage, we were recording the stage as a possible source for content from its base image, which wouldn't necessarily be true once we started processing the instructions for that stage.
We also didn't consistently enforce the rule that stages named in the "--from" argument to COPY instructions had to have been completed _before_ the stage which contains the COPY instruction, for compatibility with "docker build".
#### How to verify it
Our tests now include tests that ensure that `COPY --from=_image_` references an image, not a stage that uses it as a base.
```
FROM docker.io/library/busybox AS build
RUN rm -f /bin/paste
USER 1001
COPY --from=docker.io/library/busybox /bin/paste /test/
```
```
FROM docker.io/library/busybox AS test
RUN rm -f /bin/nl
FROM docker.io/library/alpine AS final
COPY --from=docker.io/library/busybox /bin/nl /test/
```
The tests also now include a still-expected-to-fail test that attempts to `COPY --from` the current stage:
```
FROM busybox AS test
USER 1001
FROM busybox AS build
COPY --from=test /bin/cut /test/
COPY --from=build /bin/cp /test/
COPY --from=busybox /bin/paste /test/
```
#### Which issue(s) this PR fixes:
<!--
Automatically closes linked issue when PR is merged.
Uncomment the following comment block and include the issue
number or None on one line.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`, or `None`.
-->
<!--
Fixes #
or
None
-->
None
#### Special notes for your reviewer:
#### Does this PR introduce a user-facing change?
<!--
If no, just write `None` in the release-note block below. If yes, a release note
is required: Enter your extended release note in the block below. If the PR
requires additional action from users switching to the new release, include the
string "action required".
For more information on release notes please follow the kubernetes model:
https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```
None
```
Co-authored-by: Nalin Dahyabhai <nalin@redhat.com>
When initializing a build stage, don't assume that the working container
will be useful for COPY instructions in subsequent stages that reference
the image that we're building FROM.
When locating previously-completed stages referenced in COPY
instructions in later stages, be consistent about requiring that those
stages need to have been completed before we can accept them as
alternate contexts for the sources being copied.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Previously, every build-arg was recorded in the image history
even if the arg was not accessible to that layer when the build
was run.
This commit fixes that by ensuring that args are only added to
history when they are in scope. This is tracked in the
imagebuilder.Builder struct which is now accessible through
the stage reference in the StageExecutor
Fixes#2210
Signed-off-by: Nick Carboni <ncarboni@redhat.com>
1999: pull/from/commit/push: retry on most failures r=rhatdan a=nalind
If `PullOptions`/`BuilderOptions`/`CommitOptions`/`PushOptions`/`BuildOptions` includes a `MaxRetries` value other than 0, retry operations except for (currently) connection-refused, authentication, and no-such-repository/no-such-tag errors, at a default-but-configurable interval of 5 seconds.
Set the default for `buildah pull/from/commit/push` to 3 retries at 2 second intervals.
Co-authored-by: Nalin Dahyabhai <nalin@redhat.com>
Using this Dockerfile:
```
FROM alpine AS tommer
```
Docker does:
```
Sending build context to Docker daemon 17.92kB
Step 1/1 : FROM alpine AS tommer
latest: Pulling from library/alpine
c9b1b535fdd9: Pull complete
Digest: sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d
Status: Downloaded newer image for alpine:latest
---> e7d92cdc71fe
Successfully built e7d92cdc71fe
```
While Buildah does:
```
STEP 1: FROM alpine AS tommer
STEP 2: COMMIT tom
e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a
e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a
```
Per user requests, we've over time eliminateed some of the "noisier" output that
Docker has, and that's now led to some confusion when we show an intermediate
image id (after the commit) and then as in the case, the same final
image id immediately afterwards. We display only the full final image id to
make scripting easier per other user requests.
To aid with this, this pr will cause the intermediate image to be displayed
after an `-->` like Docker and will also show the 12 character short id.
If the `--quiet` option is used, this shorter id will not be displayed, only
the final longer one.
New behavior:
```
STEP 1: FROM alpine AS tommer
STEP 2: COMMIT tom
--> e7d92cdc71f
e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a
```
Fixes: https://github.com/containers/libpod/issues/5012
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
If PullOptions/BuilderOptions/CommitOptions/PushOptions includes a
MaxRetries value other than 0, retry operations except for (currently)
connection-refused, authentication, and no-such-repository/no-such-tag
errors, at a default-but-configurable interval of 5 seconds.
Set the default for `buildah pull/from/commit/push` to 3 retries at 2
second intervals.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
In porting containers.conf to libpod, we found that buildah needed
to handle the containers.conf on the server side rather then from
the CLI.
Since the `podman-remote build` would probably not have the same content
as containers.conf on the server, the processesing of the defaults needs
to be handled in imagebuildah. The CapAdd and CapDrop values need to be
passed in.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When building images we now allow the architecture and operating system
to be overwritten via the new `--os` and `--arch` flags. This allows the
use case of packing pre-built or cross-compiled binaries into container
images with specifying the correct environment. The `--platform` flag
now takes the `os/arch` format into account as well.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
Closes: #2097
Approved by: rhatdan
Add the --sign-by option to `buildah build-using-dockerfile`,
`buildah commit`, `buildah push`, and `buildah manifest push`. Add the
`--remove-signatures` option to `buildah pull`, `buildah push`, and
`buildah manifest push`. We just pass them to the image library, which
does all of the heavy lifting.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #2085
Approved by: rhatdan
This is a rework of Qi Wang's patches.
Import package pkg/config from containers/common to read containers.conf
This patch allows users to specify default values stored in containers.conf
that will modify the behaviour of buildah tool.
Signed-off-by: Qi Wang <qiwan@redhat.com>
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #2011
Approved by: TomSweeneyRedHat
We should be outputing just the imageID when doing a buildah bud quiet.
Currently the stdout from containers is showing up on the screen.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1963
Approved by: TomSweeneyRedHat
The 'STEP' line that's printed as part of the build processing was
going to Stderr and causing some issues for OpenShift who was looking
for it. Instead print it to Stdout.
Addresses: #1772
Signed-off-by: TomSweeneyRedHat <tsweeney@redhat.com>
Closes: #1869
Approved by: rhatdan
Some Dockerfiles (fuse-overlay) require additional devices to be in the
build environment.
This patch allows the user to specify additional devices.
Also I noticed that CapAdd and CapDrop was not working in buildah bud situations,
so this patch also fixes this.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Closes: #1820
Approved by: @TomSweeneyRedHat
Use digests of the added content in history entries that we create for
ADD and COPY instructions, tightening up cache checking just a little
bit more.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1792
Approved by: TomSweeneyRedHat
Fix detection of when we need to have an up-to-date copy of a stage's
root filesystem for later stages. ADD and COPY instructions aren't the
first lines in their stages, so we can't stop parsing a stage's lines
after its first line.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
Closes: #1786
Approved by: TomSweeneyRedHat
Valentin Rothberg
Daniel J Walsh
caiges
TomSweeneyRedHat
Nalin Dahyabhai
Giuseppe Scrivano
Brandon Lum
bors[bot]
Nick Carboni
Sascha Grunert
Miloslav Trmač
WanLinghao