EC2: metalog_add missing files from packages

The files
/usr/local/etc/dhclient.conf
/usr/local/etc/ssl/cert.pem
are created in EC2 images by installed packages, but are not being
registered in MET

EC2: metalog_add missing files from packages

The files
/usr/local/etc/dhclient.conf
/usr/local/etc/ssl/cert.pem
are created in EC2 images by installed packages, but are not being
registered in METALOG. The reason for this is not immediately clear,
but may be related to the handling of "sample" files.

Reviewed by: dch
MFC after: 5 minutes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53035

show more ...

release: Prepare EC2 cloudware images to be buildable as non-root

- Avoid chrooting to uninstall pkg, as it will not be installed in the
first place when in non-root mode.
- Add metalog entries.

release: Prepare EC2 cloudware images to be buildable as non-root

- Avoid chrooting to uninstall pkg, as it will not be installed in the
first place when in non-root mode.
- Add metalog entries.

Reviewed by: cperciva
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D52452

show more ...

EC2: Make amazon-ssm-agent optional

Move it from VM_EXTRA_PACKAGES in ec2.conf to VM_EXTRA_PACKAGES in
ec2-{base,cloud-init}.conf

Sponsored by: Amazon
Differential Revision: https://reviews.freebsd

EC2: Make amazon-ssm-agent optional

Move it from VM_EXTRA_PACKAGES in ec2.conf to VM_EXTRA_PACKAGES in
ec2-{base,cloud-init}.conf

Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46508

show more ...

EC2: Move network config into a separate function

Having the "base" FreeBSD network configuration (aka. what is used
when not using cloud-init) in ec2.conf will allow us to reuse it in
other AMIs.

EC2: Move network config into a separate function

Having the "base" FreeBSD network configuration (aka. what is used
when not using cloud-init) in ec2.conf will allow us to reuse it in
other AMIs.

Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46507

show more ...

EC2: Set dhclient_arpwait="NO"

The DHCP server in EC2 knows exactly which system should be using
which IP address (and in fact EC2 has source IP filtering on by
default) so there's no point ARPing a

EC2: Set dhclient_arpwait="NO"

The DHCP server in EC2 knows exactly which system should be using
which IP address (and in fact EC2 has source IP filtering on by
default) so there's no point ARPing an address before using it.

The preceding commits (changing the ARP wait time from 2 s to 250 ms)
and this one (eliminating the wait entirely in EC2) reduce the time
required for a newly launched FreeBSD/EC2 instance to launch by 2
seconds.

Discussed with: icattard
MFC after: 10 days
Sponsored by: Amazon

show more ...

EC2: Split off reusable configuration bits

Split ec2-base.conf into ec2-base.conf and a reusable ec2.conf,
similar to how Vagrant flavours share a common vagrant.conf.

releng/14.0 candidate.

Discu

EC2: Split off reusable configuration bits

Split ec2-base.conf into ec2-base.conf and a reusable ec2.conf,
similar to how Vagrant flavours share a common vagrant.conf.

releng/14.0 candidate.

Discussed with: gjb
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41792

show more ...

EC2: Flavour existing AMIs as "base"

Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names no

EC2: Flavour existing AMIs as "base"

Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names now include the word "base".

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41791

show more ...

release/ec2: Remove references to portsnap

We no longer install a default portsnap.conf, so the sed invocation just
generates an error.

Reviewed by: cperciva
MFC after: 1 week
Differential Revision

release/ec2: Remove references to portsnap

We no longer install a default portsnap.conf, so the sed invocation just
generates an error.

Reviewed by: cperciva
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D42003

show more ...

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

EC2: Chase awscli package rename

MFC after: 1 week

release: Add support for creating ZFS-based VM images

The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use

release: Add support for creating ZFS-based VM images

The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use
makefs to create a bootable pool populated using the same dataset layout
as bsdinstall and "poudriere image" use. The pool can be grown using
the growfs rc.d script, just as in UFS images.

This will make it easy to provide VM and cloud images with ZFS as the
root filesystem. So far I did not do extensive testing of cloud images;
I merely verified that creation of ZFS-based AWS AMIs works and allows
me to create amd64 and arm64 EC2 instances with ZFS as the root
filesystem.

Reviewed by: emaste, gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34426

show more ...

EC2: Enable ephemeral swap by default

The latest version of the ec2-scripts package includes a completely
rewritten "use EC2 ephemeral disks for swap space" script. Now that
we have something which

EC2: Enable ephemeral swap by default

The latest version of the ec2-scripts package includes a completely
rewritten "use EC2 ephemeral disks for swap space" script. Now that
we have something which works on recent versions of FreeBSD, turn it
on since it's a great way to use the ephemeral disks.

Note that the option for controlling this, ec2_ephemeral_swap_enable,
is not the same as the option (ec2_ephemeralswap_enable) used with the
previous ephemeral-swap script; this change was deliberate to avoid
astonishment for users who upgraded their ec2-scripts package and had
a setting left behind in rc.conf.

show more ...

release: Remove references to ChallengeResponseAuthentication

This sshd_config keyword was replaced by KbdInteractiveAuthentication in
openssh 8.7, though ChallengeResponseAuthentication is silently

release: Remove references to ChallengeResponseAuthentication

This sshd_config keyword was replaced by KbdInteractiveAuthentication in
openssh 8.7, though ChallengeResponseAuthentication is silently accepted
as an alias. However, this means that the code in ec2.conf which
modifies a commented-out line no longer does anything. Apply a minimal
fix.

Reviewed by: cperciva, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34400

show more ...

EC2: Turn off IPv6 DAD

Disable Duplicate Address Detection in EC2 instances. The networking
configuration in EC2, with IPv6 addresses assigned by DHCPv6 and
host egress filtering, makes "duplicate

EC2: Turn off IPv6 DAD

Disable Duplicate Address Detection in EC2 instances. The networking
configuration in EC2, with IPv6 addresses assigned by DHCPv6 and
host egress filtering, makes "duplicate addresses" impossible.

This speeds up the boot process in EC2 by 2 seconds.

Reviewed by: kp, imp, bz
MFC after: 2 weeks
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D33091

show more ...

release: allow VM_EXTRA_PACKAGES to be specified in the environment

This is useful for adding extra packages to the build of an AMI.
For example:
env VM_EXTRA_PACKAGES="zsh" make -C release ec2ami

release: allow VM_EXTRA_PACKAGES to be specified in the environment

This is useful for adding extra packages to the build of an AMI.
For example:
env VM_EXTRA_PACKAGES="zsh" make -C release ec2ami

Approved by: gjb
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

show more ...

EC2: Tell gptboot to skip its 3 second wait

Nobody is going to hit a key to tell gptboot to select another disk,
so there's no point waiting.

release: fix VMSIZE following 1ca8842f3ad9

truncate(1) is not case-sensitive with regard to setting the size
of a file. makefs(8), however, does not honor upper-case values.
Update release-specific

release: fix VMSIZE following 1ca8842f3ad9

truncate(1) is not case-sensitive with regard to setting the size
of a file. makefs(8), however, does not honor upper-case values.
Update release-specific files and the release(7) manual page to
reflect this.

MFC with: 1ca8842f3ad9
Submitted by: ehem_freebsd_m5p.com (original)
Differential Review: https://reviews.freebsd.org/D28979
Sponsored by: Rubicon Communications, LLC ("Netgate")

show more ...

Spawn the DHCPv6 client in EC2 instances via rtsold.

Prior to this commit, EC2 AMIs used a "dual-dhclient" tool which was
launched in place of dhclient and spawned both the base system dhclient
for

Spawn the DHCPv6 client in EC2 instances via rtsold.

Prior to this commit, EC2 AMIs used a "dual-dhclient" tool which was
launched in place of dhclient and spawned both the base system dhclient
for IPv4 and the ISC dhclient from ports for IPv6.

Now that rtsold supports the "M bit" (managed configuration), we can go
back to having the base system dhclient spawned normally, and provide a
script to rtsold which spawns the ISC dhclient from ports when rtsold
decides that it is appropriate.

Thanks to: bz
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva

show more ...

Bump the size of EC2 AMIs up to 5 GB.

The FreeBSD base system continues to expand. 4GB is now insufficient;
we passed 3 GB in May 2019; we passed 2 GB in August 2017. Over half
of the disk space u

Bump the size of EC2 AMIs up to 5 GB.

The FreeBSD base system continues to expand. 4GB is now insufficient;
we passed 3 GB in May 2019; we passed 2 GB in August 2017. Over half
of the disk space used is in /usr/lib/debug/.

Without this change, instances boot but are unusable, since the first
thing which breaks when VM filesystems are too small is the "pkg install"
in the VM building process.

show more ...

Make EC2 AMIs use portsnap and freebsd-update mirrors hosted in AWS

This adjusts freebsd-update.conf and portsnap.conf files in EC2 AMIs to
point at the new AWS-hosted mirror network.

Approved by:

Make EC2 AMIs use portsnap and freebsd-update mirrors hosted in AWS

This adjusts freebsd-update.conf and portsnap.conf files in EC2 AMIs to
point at the new AWS-hosted mirror network.

Approved by: re (delphij)
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D25498

show more ...

Add ebsnvme-id to EC2 AMIs and enable /dev/aws/disk

The ebsnvme-id utility exposes information about EC2 disks -- for
Elastic Block Store volumes, their volume IDs and "linux device
names", and for

Add ebsnvme-id to EC2 AMIs and enable /dev/aws/disk

The ebsnvme-id utility exposes information about EC2 disks -- for
Elastic Block Store volumes, their volume IDs and "linux device
names", and for Instance Store (aka "Ephemeral") disks, their
serial numbers.

The dev_aws_disk rc.d script and associated devd.conf rule maintains
a tree under /dev/aws/disk:
/dev/aws/disk/ebs/<volume ID>
/dev/aws/disk/linuxname/<linux device name>
/dev/aws/disk/ephemeral/<serial number>
which are symlinks to the corresponding nda or nvd devices.

MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva

show more ...

Add /etc/autofs/special_efs to EC2 AMIs

Since Amazon Elastic File System is only available within AWS, it seems
more appropriate to have this added only in EC2 AMIs rather than
"polluting" non-EC2 i

Add /etc/autofs/special_efs to EC2 AMIs

Since Amazon Elastic File System is only available within AWS, it seems
more appropriate to have this added only in EC2 AMIs rather than
"polluting" non-EC2 images with it.

Reviewed by: gjb
MFC after: 7 days
Relnotes: Amazon EFS filesystems can be automounted by enabling autofs
and placing "/efs -efs" into /etc/auto_master.
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D24791

show more ...

Set use_nvd=0 in EC2 AMIs.

FreeBSD is in the process of switching from nvd(4) to nda(4) as the disk
device front-end to NVMe. Changing the default in the kernel is tricky
since existing systems may

Set use_nvd=0 in EC2 AMIs.

FreeBSD is in the process of switching from nvd(4) to nda(4) as the disk
device front-end to NVMe. Changing the default in the kernel is tricky
since existing systems may have /dev/nvd* hard-coded e.g. in /etc/fstab;
however, there's no reason to not change the default in HEAD for *new*
systems.

At present I have no intention of MFCing this to stable branches, since
someone might reasonably expect scripts they use for launching and
configuring FreeBSD 12.1 instances to work with FreeBSD 12.2 AMIs, for
example.

Reviewed by: gjb, imp
Relnotes: NVMe disks in EC2 instances launched from 13.0 and later
now show up as nda(4) devices.
Differential Revision: https://reviews.freebsd.org/D24583

show more ...

Mount /dev while deinstalling pkg inside EC2 AMIs.

This gets rid of (harmless) warnings:
"pkg: Cannot open /dev/null:No such file or directory".

MFC after: 2 weeks

Remove /qemu from EC2 ARM AMIs

I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.

MFC after: 2

Remove /qemu from EC2 ARM AMIs

I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.

MFC after: 2 weeks

show more ...