Tom's Pants backends
This is the documentation for my Pants backend, which I use for my own projects - especially managing my own cloud, but also building open-source software.
Please see the documentation for more information!
Kustomize backend for Pants
Warning This plugin is in development. No stability is guaranteed! Contributions welcome.
This backends implements targets for kustomize templates.
- kustomize for overlaying state ontop of raw kubernetes files
Planned and missing features
- Key/secret/... generation from built artifacts
Targets
There's currently one target.
kustomize
A code-generation target for converting a bundle of kubernetes files into a single multi-docuent YAML file with state injected from other Pants targets.
kustomize(
name="kustomize",
sources=[
"deployment.yaml",
"server.py",
"service.yaml",
"namespace.yaml",
"kustomization.yaml",
],
dependencies=[":bin"],
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
sources | Resources used by this target | Required |
dependencies | Targets to package and pass to the build context, as well as bases | [] |
decsription | A description of the target | |
tags | List of tags | [] |
For dependencies, the builder will replace labels in the kustomization.yaml with the path of the built package.
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[Unreleased]
- Add
tailor
support - Import
git
to sandbox during expansion to support github downloads- submodule check seems to fail so it might help to add
?submodules=false
to the link to disable it
- submodule check seems to fail so it might help to add
[0.3.0] - 2023-11-19
- Target pants version is now 2.18.0
[0.2.0] - 2023-06-18
- Target pants version is now 2.16.0, with support for 2.15.0.
[0.1.2] - 2022-12-06
- Add support for specializing how other packages are injected into Kustomize files.
[0.1.1] - 2022-11-17
- Move to new repository
[0.1.0] - 2022-11-13
Initial release.
Kubernetes backend for Pants
Warning This plugin is in development. No stability is guaranteed! Contributions welcome.
This backends implements targets for working with Kubernetes clusters using raw YAML.
- kubectl for cluster operations
Targets
There's currently three targets for pants-backend-k8s
:
k8s_source
A file that can be passed into other k8s fields that are not source fields. For example, k8s_object.template
.
k8s_source(
name="namespace.yaml",
source="namespace.yaml",
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
source | The raw file | Required |
decsription | A description of the target | |
tags | List of tags | [] |
This'll eventually be automated like other rules once a suitable heuristic for generation with tailor is found. PRs welcome!
k8s_object
Input for a kubernetes command, either generated via kustomize
or via k8s_source
.
k8s_object(
name="k8s",
description="the chat backend"
template=[":kustomize"],
namespace="chat-app",
cluster="prod",
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
template | The target to act on | Required |
namespace | Namespace to target | Optional, will use default kubectl namespace |
cluster | cluster to target | Optional, will use default kubectl cluster |
decsription | A description of the target | |
tags | List of tags | [] |
k8s_object
is a generator for kubernetes
target parametrized by the potential commands that are available: apply
,
create
, get
, describe
, replace
, and delete
.
k8s_objects
A collection of kubernetes objects that should be managed together.
k8s_objects(
name="my-service",
description="all components of service-x"
objects=[":namespace", ":deployment"],
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
objects | k8s_object targets that should be managed | Required |
decsription | A description of the target | |
tags | List of tags | [] |
Like k8s_object
, k8s_objects
is a generator for parametrized targets for the commands that are available: apply
,
create
, get
, describe
, replace
, and delete
.
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Unreleased
- Kubeconfig files are no longer automatically picked up from the host. In order to support scripted
provisioning and local configuration files, all targets now take a
kubeconfig
field. This can point to eitherkubeconfig
target which uses a straight source or a generated, or ahost_kubeconfig
target which will attempt to load from~/.kube/config
. Both of these also allow you to specify default namespaces, contexts, clusters and users. All these fields can now also be specified on the object, whereas only cluster/context could before.
[0.3.0] - 2023-11-19
- Target pants version is now 2.18.0
[0.2.0] - 2023-06-18
- Target pants version is now 2.16.0, with support for 2.15.0.
[0.1.1] - 2022-11-17
- Move to new repository
[0.1.0] - 2022-11-12
- Initial release.
OCI backend for Pants
Warning This plugin is in development. No stability is guaranteed! Contributions welcome.
This is a backend implementing support for building OCI images in pants; running them, and publishing them to container registries. To do this, this plugin uses three different tools:
Planned and missing features
- Currently there's no support for pulling tags, as that would break determinism
- Multi-platform SHA/.sig is untested/unsupported
- skopeo doesn't support MacOS, preventing pulling and pushing images.
- No "in-container" build steps
Targets
There's six targets currently implemented, of which five are generic:
oci_pull_image
oci_pull_images
oci_image_build
oci_image_empty
oci_build_layer
And one with some special language semantics:
oci_python_image
- this is the same asoci_image_build
, but will prefer to set the entrypoint to.pex
files.
oci_pull_image
Pull an image from a repository with a specific digest.
oci_pull_image(
name="base-python",
repository="docker.io/library/python",
sha="b78b777208be08edd8f297035cdfbacddb45170ad778fd643c792ee045187e39"
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
repository | Fully qualified repository name | Required |
sha | The digest of the image, minus the @sha: prefix. | Required |
anonymous | Whether to pull the image anonymously. | false |
decsription | A description of the target | |
tags | List of tags | [] |
oci_pull_images
Pull multiple shas for an image, generating a target for each. In the below example, we'd get the targets :python#slim
and :python#buster
.
oci_pull_image(
name="python",
repository="docker.io/library/python",
variants={
"slim": "f8fbb2370c6314c806b2ddbec8d94375987e16bc122379bef979c6fc5e962920",
"buster": "97c123c899c8c9ca46248f4002ec4173322e0a1086b386efefac163c64967ba2"
}
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
repository | Fully qualified repository name | Required |
variants | Dictionary with local tags to the remote sha | Required |
anonymous | Whether to pull the image anonymously | false |
decsription | A description of the target | |
tags | List of tags | [] |
oci_build_image
Build an image with the provided packages embedded.
oci_image_build(
name="my-server",
base=":python#slim",
repository="my-registry.example.com/a-namespace/an-image",
tag="latest",
packages=[":my_pex"]
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
base | The base image to use. Matches the FROM directive in a Dockerfile | Required |
packages | Packaged targets to include. The first element will be used as the entrypoint. | [] |
repository | Fully qualified repository name | Required when publishing |
tag | Remote tag to use | Required when publishing |
decsription | A description of the target | |
tags | List of tags | [] |
oci_python_image
Build a Python image with the provided packages embedded.
oci_python_image(
name="my-server",
base=":python#slim",
repository="my-registry.example.com/a-namespace/an-image",
main="/app/server/start.py",
tag="latest",
packages=[":my_pex"]
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
base | The base image to use. Matches the FROM directive in a Dockerfile | Required |
packages | Packaged targets to include. The first element will be used as the entrypoint. | [] |
python_main | The main file to run | The last .pex in the dependency list |
repository | Fully qualified repository name | Required when publishing |
tag | Remote tag to use | Required when publishing |
decsription | A description of the target | |
tags | List of tags | [] |
oci_image_empty
An empty base image with no contents at all. This is declared as //:empty
automatically, but you can use this to create new targets.
oci_image_empty(
name="empty",
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
decsription | A description of the target | |
tags | List of tags | [] |
oci_build_layer
Run an image command, and capture the configured output into a layer artifact, that can be injected into other images. This matches the COPY --from
workflows.
oci_build_layer( name="layer" base=[":rust-1-70"], packages=[":files"], env=['RUSTC_OPTS=...'], command=['cd /my-package && cargo build --release'], outputs=['/my-package/target/release/my-package'], )
| Argument | Meaning | Default value |
|---------------|--------------------------------------------------------------------------------|--------------------------------------------------------|
| `name` | The target name | Same as any other target, which is the directory name |
| `packages` | Packaged targets to include. The first element will be used as the entrypoint. | `[]` |
| `env` | Environment variables to set. Does not support interpolation. | `[]` |
| `outputs` | Paths to capture into the built layer. | `[]` |
| `exclude` | Globs to not include in the output. | `[]` |
| `decsription` | A description of the target | |
| `output_path` | The output path during `pants package` | A variant generated from the target name and directory |
| `tags` | List of tags | `[]` |
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
0.6.1 - 2024-03-28
- Fix a bug where the plugin would try to publish OCI images without a repository.
- Will now default to "latest" as a tag if none is specified.
0.6.0 - 2024-03-21
- Adding layers to an image is changing. As a preparatory step, there's now an
oci_layer
target. Add these to thelayers
field of anoci_image_build
, instead of dependencies. These will process a bit better, and gives more control over what goes into each layer. - Mac support has improved. In order to provide determinism, the
gtar
(GNU tar) binary has to be available. - Both umoci and skopeo are now supported on Mac M1 and x86_64. This plugin does not support image run steps on Mac still.
- When pulling multi-arch images you can now specify
os
andarchitecture
on theoci_pull_image
target - Fix a bug where OCI layer building would fail if no dependencies were specified
0.5.0 - 2023-11-19
- Fix a crash when image pulling fails
- Add
nightly
version forumoci
with experimental support for Mac - Add
v1.13.3
version forskopeo
with experimental support for Mac
0.4.0 - 2023-06-18
-
Improved support for very large layers > 2GB. A lot of layers will now be compressed in transit. This adds some overhead later when injecting them into image, but fixes some crashes inside Pants.
-
Improved support for multi-stage builds (
COPY --from
)There is now support for building artifacts in one container and copying them into a new container. To do this, use
oci_build_layer
, and configure the output files and directories. The files and data will be inserted verbatim into the downstream container. -
Adds support for empty base images (
FROM scratch
)This change enables you to use
base=["//:empty"]
to start from a completely empty container. This can be useful to produce containers with statically linked binaries that require no runtime environment at all.- To change the target name, set
[oci].empty_image_target
inpants.toml
.
- To change the target name, set
0.3.1 - 2023-03-16
- Handle files when building layers
0.3.0 - 2023-03-14
- [breaking] Change output format when publishing
0.2.0 - 2023-02-10
- [bugfix] Ensure layer tars has no metadata or user information
- [bugfix] Properly pass args to runc
- [bugfix] Fix log output in package_oci_image
- [feature] Add proper metadata where possible
- [feature] Add support for anonymous image pulling (
anonymous=true
) - [feature] Add support for passing image SHA along to Kustomize
- [feature] Add
python_image_target
which will set entrypoint appropriately - [feature] Forward PATH, HOME, XDG_RUNTIME_DIR env variables to publish step to make credHelpers work
[0.1.1] - 2022-11-17
- Move to new repo
- Compatibility fixes for 2.15.0a0
[0.1.0] - 2022-11-13
Initial release.
MDBook backend for Pants
Warning This plugin is in development. No stability is guaranteed! Contributions welcome.
This provides a tool for building mdbook targets with pants. There is currently a single very simple rule:
md_book(
name="my-docs",
sources=["book.toml", "src/*"],
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
sources | Files included when building the book | book.toml and the src directory |
decsription | A description of the target | "" |
tags | List of tags | [] |
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[0.4.1] - 2024-04-12
- Will now use x86_64 mdbook binary on arm64, relying on Rosetta.
[0.4.0] - 2024-04-05
- Will now also include codegen sources, allowing integration with
adhoc_tool
and other generators. Note that only file and direct mdbook sources are included.
[0.3.0] - 2023-11-21
- Target pants version is now 2.18.0, with support for 2.17.0.
- Updated mdbook version to 0.4.35
- Now includes support for arm64 on Linux
[0.2.0] - 2023-06-18
- Target pants version is now 2.16.0, with support for 2.15.0.
[0.1.3] - 2022-11-18
- Raise error if
book.toml
is not found - Handle all sources in dependencies
[0.1.2] - 2022-11-17
- Merge with all other backends
- Fix README links
[0.1.1] - 2022-11-12
- Fix README.
[0.1.0] - 2022-11-12
- Initial release.
Secrets backend for Pants
Warning This plugin is in development. No stability is guaranteed! Contributions welcome.
This backends implements utilities for handling secrets.
Planned and missing features
- Setting and creating secrets via Pants
Targets
env_secret
A secret to be read from the environment.
env_secret(
name="bw_session_key",
key="BW_SESSION",
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
key | The environment variable to read. | Required |
decsription | A description of the target | |
tags | List of tags | [] |
Goals
decrypt
Decrypts and prints a secret.
BW_SESSION="..." pants decrypt //examples/bitwarden:pypi_token
22:29:42.17 [INFO] Completed: Decrypting examples/bitwarden:pypi_token
Secret examples/bitwarden:pypi_token from BitWarden: pypi-...
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[0.3.0] 2023-11-21
- Target pants version is now 2.18.0, with support for 2.17.0.
[0.2.0] - 2023-06-18
- Target pants version is now 2.16.0, with support for 2.15.0.
0.1.0 - 2023-02-10
- Initial release
BitWarden backend for Pants
Warning This plugin is in development. No stability is guaranteed! Contributions welcome.
This backends implements targets for reading BitWarden secrets.
- bw - the BitWarden CLI client
Planned and missing features
- Setting and creating secrets via Pants
Targets
bw_item
Matches one entry in your vault.
bw_item(
name="pypi",
id="386c6037-cbdd-4aa3-ba80-9ed6661f751b",
session_secret=":bw_session_key",
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
id | Item id used by this target as seen in the address bar | Required |
item_name | The name in the vault. If ambiguous this will fail. Prefer IDs. | |
session_secret | The secret to use for the BW_SESSION variable. | env["BW_SESSION"] |
decsription | A description of the target | |
tags | List of tags | [] |
bw_password
The password of an item in your vault.
bw_password(
name="pypi_password",
item=[":pypi"],
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
item | The item containing the password. | Required |
decsription | A description of the target | |
tags | List of tags | [] |
bw_field
A field from an item in your vault. These are the "Custom Fields" at the bottom of an item, not to be confused with attachments.
bw_field(
name="pypi_token",
field_name="api_token"
item=[":pypi"],
)
Argument | Meaning | Default value |
---|---|---|
name | The target name | Same as any other target, which is the directory name |
item | The item containing the password. | Required |
field_name | The item containing the password. | Required |
decsription | A description of the target | |
tags | List of tags | [] |
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[0.3.0] - 2023-11-21
- Target pants version is now 2.18.0, with support for 2.17.0.
- Default bitwarden version is now 2023.10.0
[0.2.0] - 2023-06-18
- Target pants version is now 2.16.0, with support for 2.15.0.
0.1.0 - 2023-02-10
- Initial release