AWS has offered A1 EC2 instances powered by Graviton processors since late 2018, and M6g powered by the newer Graviton2 processors reached GA in May 2020. I’ve been excited to use these instances, but have not been able to find many details on how to create a custom image for arm64/aarch64 from AWS or others. I was particularly interested in running CentOS 8, which still doesn’t even have an x86_64 AMI, so I set out to create an image from scratch.
Without CentOS, I found that Red Hat offers RHEL 8, so I began by starting up ami-029ba835ddd43c34f in us-east-1:
aws ec2 run-instances \
--region 'us-east-1' \
--image-id 'ami-029ba835ddd43c34f' \
--security-group-ids 'sg-xxxxxxxxxxxxxxxxx' \
--key-name 'alan' \
--instance-type 'a1.medium'
SSH into the instance as ec2-user. Let’s look at the disk:
$ fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D4C0BAF2-FA64-4781-90D2-8D9ADE3EA3A8
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 411647 409600 200M EFI System
/dev/nvme0n1p2 411648 1460223 1048576 512M Linux filesystem
/dev/nvme0n1p3 1460224 20971486 19511263 9.3G Linux filesystem
$ parted /dev/nvme0n1 print
Model: NVMe Device (nvme)
Disk /dev/nvme0n1: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 211MB 210MB fat16 EFI System Partition boot, esp
2 211MB 748MB 537MB xfs
3 748MB 10.7GB 9990MB xfs
EFI? If you search around for EC2 and (U)EFI, AWS documentation mentions that EFI isn’t used except for importing a particular kind of Windows VHD image with VM Import/Export. Here’s a Stack Overflow post reminding us that EC2 doesn’t support EFI. An FAQ about A1 instances only mentions ACPI tables. I thought I had found some details in this Creating an instance store-backed Linux AMI, but it shows commenting out the UEFI partition in /etc/fstab. We’re left to ourselves to figure it out.
Let’s look at some other AMIs available for A1 and M6g. This is Amazon Linux 2:
$ fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C459507E-A682-4B76-AF4A-09F375376D10
Device Start End Sectors Size Type
/dev/nvme0n1p1 22528 16777182 16754655 8G Linux filesystem
/dev/nvme0n1p128 2048 22527 20480 10M EFI System
Partition table entries are not in disk order.
$ parted /dev/nvme0n1 print
Model: NVMe Device (nvme)
Disk /dev/nvme0n1: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
128 1049kB 11.5MB 10.5MB fat16 EFI System Partition boot
1 11.5MB 8590MB 8578MB xfs Linux
And, Ubuntu Server 20.04:
$ fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 8 GiB, 8589934592 bytes, 16777216 sectors
Disk model: Amazon Elastic Block Store
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 47852264-02DA-40E3-B781-733CC8102697
Device Start End Sectors Size Type
/dev/nvme0n1p1 206848 16777182 16570335 7.9G Linux filesystem
/dev/nvme0n1p15 2048 204800 202753 99M EFI System
Partition table entries are not in disk order.
$ parted /dev/nvme0n1 print
Model: Amazon Elastic Block Store (nvme)
Disk /dev/nvme0n1: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
15 1049kB 105MB 104MB fat32 boot, esp
1 106MB 8590MB 8484MB ext4
So, despite no documentation about how to do create an AMI for ARM instances, I set out to create a CentOS 8 ARM image from scratch anyhow. After running through the excellent packer-chef-highperf-centos-ami project by Irving Popovetsky on an x86_64 platform, I began working out the differences for aarch64. The key takeaways are using gpt disks and, for Linux, having EFI files available at /boot/efi/EFI/.
To create a CentOS 8 ARM AMI, first launch an EC2 instance using the RHEL8 ARM AMI. Any size A1 or M6g instance will suffice. Add an additional EBS volume with at least 8 GB. Then, ssh in as ec2-user and then switch to root with sudo -i.
We’ll start by setting our chroot mount point and the secondary EBS volume as variables:
export ROOTFS=/rootfs
export DEVICE="/dev/nvme1n1"
Use parted to create the same disk layout as RHEL8:
parted --script "$DEVICE" -- \
mklabel gpt \
mkpart primary fat32 1 201MiB \
mkpart primary xfs 201MiB 713MiB \
mkpart primary xfs 713MiB -1 \
set 1 esp on
This left me with three partitions labeled “primary”. I wasn’t able to set empty labels or a label with spaces using parted --script or even parted <<HEREDOC, so I resorted to using expect for use in a non-interactive script:
rpm -q expect || dnf -y install expect
expect <<'EOS'
set device $env(DEVICE)
spawn parted "$device"
expect "(parted) "
send "name 1 'EFI System Partition'\r"
expect "(parted) "
send "name 2\r"
expect "Partition name? "
send "''\r"
expect "(parted) "
send "name 3\r"
expect "Partition name? "
send "''\r"
expect "(parted) "
send "quit\r"
expect eof
EOS
Format the partitions the same as RHEL8:
# /
mkfs.xfs -f "${DEVICE}p3"
# /boot
mkfs.xfs -f "${DEVICE}p2"
# /boot/efi
mkfs.fat -F 16 "${DEVICE}p1"
Mount these three partitions and requisite special mounts for a functional chroot:
# Chroot Mount /
mkdir -p "$ROOTFS"
mount "${DEVICE}p3" "$ROOTFS"
# Chroot Mount /boot
mkdir -p "$ROOTFS/boot"
mount "${DEVICE}p2" "$ROOTFS/boot"
# Chroot Mount /boot/efi
mkdir -p "$ROOTFS/boot/efi"
mount "${DEVICE}p1" "$ROOTFS/boot/efi"
# Special filesystems
mkdir -p "$ROOTFS/dev" "$ROOTFS/proc" "$ROOTFS/sys"
mount -o bind /dev "$ROOTFS/dev"
mount --types devpts devpts "$ROOTFS/dev/pts"
mount --types tmpfs tmpfs "$ROOTFS/dev/shm"
mount --types proc proc "$ROOTFS/proc"
mount --types sysfs sysfs "$ROOTFS/sys"
mount --types selinuxfs selinuxfs "$ROOTFS/sys/fs/selinux"
Initialize the RPM database and install centos-release-8 and centos-repos-8 for aarch64:
rpm --root="$ROOTFS" --initdb
release_pkg_latest="$( curl --silent https://mirrors.edge.kernel.org/centos/8/BaseOS/aarch64/os/Packages/ | grep --only-matching 'centos-release-8[^"]*.rpm' | sort --unique --version-sort | tail -1 )"
release_pkg_url="https://mirrors.edge.kernel.org/centos/8/BaseOS/aarch64/os/Packages/$release_pkg_latest"
repos_pkg_latest="$( curl --silent https://mirrors.edge.kernel.org/centos/8/BaseOS/aarch64/os/Packages/ | grep --only-matching 'centos-repos-8[^"]*.rpm' | sort --unique --version-sort | tail -1 )"
repos_pkg_url="https://mirrors.edge.kernel.org/centos/8/BaseOS/aarch64/os/Packages/$repos_pkg_latest"
rpm --root="$ROOTFS" --nodeps -ivh "$release_pkg_url"
rpm --root="$ROOTFS" --nodeps -ivh "$repos_pkg_url"
Run an update to make sure the packages installed successfully:
Note: use “–nogpgcheck” so users of the resulting AMI still need to confirm GPG key usage
dnf --installroot="$ROOTFS" --nogpgcheck -y update
Create the /etc/fstab file with our partitions, again, like the one from RHEL8:
cat > "${ROOTFS}/etc/fstab" <<EOF
#
# /etc/fstab
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=$( lsblk "${DEVICE}p3" --noheadings --output uuid ) / xfs defaults 0 0
UUID=$( lsblk "${DEVICE}p2" --noheadings --output uuid ) /boot xfs defaults 0 0
UUID=$( lsblk "${DEVICE}p1" --noheadings --output uuid ) /boot/efi vfat defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
EOF
From what I can tell, the file /etc/default/grub does not come from an RPM but is generated. Comparing the x86_64 versions of CentOS 7 and RHEL 7, they appear to be the same there, so I copy the one from RHEL8:
mkdir "${ROOTFS}/etc/default"
cp -av /etc/default/grub "${ROOTFS}/etc/default/grub"
I referenced create_base_ami8.sh and CentOS-7-x86_64-hvm.ks for the package selections. Because the chroot has all required mounts and both /etc/fstab and /etc/default/grub are set up, the post-installation for the kernel will take care of most of the bootloader steps without complicated manual intervention:
yum --installroot="$ROOTFS" --nogpgcheck -y install \
--exclude="iwl*firmware" \
--exclude="libertas*firmware" \
--exclude="plymouth*" \
"@Minimal Install" \
centos-gpg-keys \
cloud-init \
cloud-utils-growpart \
dracut-config-generic \
efibootmgr \
grub2 \
kernel \
shim \
yum-utils
yum --installroot="$ROOTFS" -C -y remove firewalld --setopt="clean_requirements_on_remove=1"
yum --installroot="$ROOTFS" -C -y remove linux-firmware
# This is currently failing on the dependency "timedatex" having a cpio package problem for both aarch64 and x86_64, but chrony still installs.
yum --installroot="$ROOTFS" --nogpgcheck -y install chrony || true
Complete bootloader setup and verify that grubby can detect a default kernel (Note: there is no need to run grub2-install on an EFI system):
chroot "$ROOTFS" grub2-mkconfig -o /etc/grub2-efi.cfg
chroot "$ROOTFS" grubby --default-kernel
That’s it for the EFI-specific portion of creating an AMI from scratch. Everything else is as needed, such as setting up /etc/hosts, disabling first boot, etc.
If you’d like to create CentOS 8 AMIs for both ARM and x86_64, you can find chroot-bootstrap.sh and the Packer ebs-surrogate file I wrote at gist.github.com/alanivey/68712e6172b793037fbd77ebb3112c3f.
Leave a comment at GitHub, or reach out @alanivey and please let me know if you found this helpful.