Setting up a new AMI image for fedora 11

Since its a bit of a pain to create the fedora 11 ami image for use in EC2, I thought I would share a script.

First part of the script sets up the variables used through script.

	#!/bin/sh

	EC2_MODULE="2.6.21.7-2.fc8xen-ec2-v1.0" ## set the ec2 module

	SCRIPT_DIR="/var/awsscripts" ## scripts to setup the amazon stuff, will be copied to the image

	IMAGE_DIR="/var/awsimage" ## directory to store the img file and ami image in

	RELEASEVER=11

	ARCH=i386 # x86_64

	IMAGE_SIZE=2560 ## set the image size

	## amazon stuff

	EC2_HOME=~/.ec2

	AWS_USER=<Your account number >

	AWS_ACCESS_KEY_ID=<Your access key>

	AWS_SECRET_ACCESS_KEY=<Your secret access key>

	EC2_PRIVATE_KEY=$EC2_HOME/pk-<Your private key name>.pem

	EC2_CERT=$EC2_HOME/cert-<Your public key name>.pem

	BUCKET_NAME=< s3 bucket name to store the image >

	export EC2_HOME AWS_USER AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY EC2_PRIVATE_KEY EC2_CERT BUCKET_NAME

	IMAGE_NAME=$1 ## passed in to the script

	BASE_NAME="ami-fedora$RELEASEVER-$ARCH_$IMAGE_NAME"

	IMG_FILE="$BASE_NAME.img"

	echo "Image file is: $IMG_FILE"

	echo "Base Name: $BASE_NAME"

	if [ ! -d "$IMAGE_DIR" ]; then

	mkdir $IMAGE_DIR

	fi

view raw initialimagesetupaws.sh hosted with ❤ by GitHub

In the above, you need to setup your keys and access keys for amazon. This is used later in the script to publish the image to the s3 bucket. It also assumes you have the ami tools downloaded and installed. This also assumes you are using fedora 11 to build the image (I use parallels with fedora 11 on it).

Now we need to make an img and mount to it and push files to it.

	## Setup image

	dd if=/dev/zero of="$IMAGE_DIR/$IMG_FILE" bs=1M count="$IMAGE_SIZE"

	/sbin/mke2fs -F -j "$IMAGE_DIR/$IMG_FILE"

	mkdir "/mnt/$BASE_NAME"

	mount -o loop "$IMAGE_DIR/$IMG_FILE" "/mnt/$BASE_NAME"

	mkdir "/mnt/$BASE_NAME/proc"; chmod 555 "/mnt/$BASE_NAME/proc"

	mkdir "/mnt/$BASE_NAME/etc"; chmod 755 "/mnt/$BASE_NAME/etc"

	mkdir "/mnt/$BASE_NAME/root"; chmod 750 "/mnt/$BASE_NAME/root"

	mkdir -p "/mnt/$BASE_NAME/var/lib/rpm"; chmod 755 "/mnt/$BASE_NAME/var/lib/rpm"

	mkdir -p "/mnt/$BASE_NAME/var/cache"

	mkdir -p "/mnt/$BASE_NAME/var/log"; chmod 755 "/mnt/$BASE_NAME/var/log"

	mkdir -p "/mnt/$BASE_NAME/var/lock/rpm"; chmod 755 "/mnt/$BASE_NAME/var/lock/rpm"

	mkdir -p "/mnt/$BASE_NAME/var/tmp" ; chmod ugo+rwxt "/mnt/$BASE_NAME/var/tmp"

	touch "/mnt/$BASE_NAME/etc/mtab"; chmod 644 "/mnt/$BASE_NAME/etc/mtab"

	for i in console null zero urandom; do /sbin/MAKEDEV -d "/mnt/$BASE_NAME/dev" -x $i; done

	### ---------------------------------------------------------

	## setup filesystem

	FS_STAB="/mnt/$BASE_NAME/etc/fstab"

	echo "/dev/sda1 / ext3 defaults 1 1 " >> $FS_STAB

	echo "none /dev/pts devpts gid=5,mode=620 0 0 " >> $FS_STAB

	echo "none /dev/shm tmpfs defaults 0 0 " >> $FS_STAB

	echo "none /proc proc defaults 0 0 " >> $FS_STAB

	echo "none /sys sysfs defaults 0 0 " >> $FS_STAB

	echo "/dev/sda2 /mnt ext3 defaults 1 2 " >> $FS_STAB

	echo "/dev/sda3 swap swap defaults 0 0 " >> $FS_STAB

	chmod 644 $FS_STAB

view raw setupimgdir.sh hosted with ❤ by GitHub

The above also sets up the fstab file and the directories we will use to add things to.

Next we setup the yum repo files and the start installing the fedora 11 package.

	## setup repo

	rm -f ./yum.all.repo

	DIR="/etc/yum.repos.d"

	SUFFIX="repo"



	for i in "$DIR"/*.$SUFFIX

	do

	cat ${i} >> ./yum.all.repo

	done

	sedscript="s/\$basearch/$ARCH/g"

	sed -i $sedscript ./yum.all.repo

	sedscript="s/\$releasever/$RELEASEVER/g"

	sed -i $sedscript ./yum.all.repo

	## setup fedora

	mount -t proc none "/mnt/$BASE_NAME/proc"

	yum -c yum.all.repo --installroot="/mnt/$BASE_NAME" -y groupinstall Base

	RPMS="tar wget gzip bzip2 curl ruby openssh kernel-xen kernel-xen-devel"

	yum -c yum.all.repo --installroot="/mnt/$BASE_NAME" -y install $RPMS

view raw yumreposetup.sh hosted with ❤ by GitHub

The above also installs some base rpms. Now for the amazon parts such as the module and tools.

	## kernel-xen

	if [ ! -f "ec2-modules-$EC2_MODULE.tgz" ]; then

	wget "http://ec2-downloads.s3.amazonaws.com/ec2-modules-$EC2_MODULE.tgz"

	fi

	tar zxvf "ec2-modules-$EC2_MODULE.tgz" -C "/mnt/$BASE_NAME"

	/usr/sbin/chroot "/mnt/$BASE_NAME" cp -R /lib/modules/$EC2_MODULE /lib/modules/2.6.21.7-2.fc8xen

	/usr/sbin/chroot "/mnt/$BASE_NAME" /sbin/depmod -a

	wget "http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm"

	cp ec2-ami-tools.noarch.rpm "/mnt/$BASE_NAME/var/awsscripts/"

	/usr/sbin/chroot "/mnt/$BASE_NAME" rpm -ivh --nosignature /var/awsscripts/ec2-ami-tools.noarch.rpm

view raw awskernelsetup.sh hosted with ❤ by GitHub

Now we start modifying some of the base system files to work in EC2.

	## update ssh file

	echo " GSSAPIAuthentication yes" >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"

	echo " Protocol 2" >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"

	echo " X11Forwarding yes " >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"

	echo " AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES " >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"

	echo " AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT " >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"

	echo " AcceptEnv LC_IDENTIFICATION LC_ALL " >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"

	echo " PermitRootLogin without-password" >> "/mnt/$BASE_NAME/etc/ssh/sshd_config"



	## replace networking file

	echo "NETWORKING=yes" > "/mnt/$BASE_NAME/etc/sysconfig/network"



	## replace ifcfg-eth0 file

	"DEVICE=eth0" > "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

	"BOOTPROTO=dhcp" >> "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

	"ONBOOT=yes" >> "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

	"TYPE=Ethernet" >> "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

	"USERCTL=yes" >> "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

	"PEERDNS=yes" >> "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

	"IPV6INIT=no" >> "/mnt/$BASE_NAME/etc/sysconfig/network-scripts/ifcfg-eth0"

view raw ec2tweaking.sh hosted with ❤ by GitHub

Now for general customization and including of amazon scripts, which are described below.

	## Update more packages

	yum -c yum.all.repo --installroot="/mnt/$BASE_NAME" -y erase cups ## erase printing, can do more customization here as well

	yum -c yum.all.repo --installroot="/mnt/$BASE_NAME" -y install fuse-devel fuse fuse-encfs fuse-libs fuse-sshfs java-1.6.0-openjdk xen-runtime ruby

	## update services


	/usr/sbin/chroot "/mnt/$BASE_NAME" /sbin/chkconfig --level 1345 sshd --add

	/usr/sbin/chroot "/mnt/$BASE_NAME" /sbin/chkconfig --level 1345 sshd on

	/usr/sbin/chroot "/mnt/$BASE_NAME" /sbin/chkconfig --level 345 ip6tables off

	## add scripts to startup

	cp -R /$SCRIPT_DIR/* "/mnt/$BASE_NAME/$SCRIPT_DIR"

	chmod +x /mnt/$BASE_NAME/$SCRIPT_DIR/*

	echo "source /$SCRIPT_DIR/awssetup" >> /mnt/$BASE_NAME/etc/rc.local

	## wrap everything up

	yum -c yum.all.repo --installroot="/mnt/$BASE_NAME" clean all

	sync

	umount "/mnt/$BASE_NAME/proc"

	umount "/mnt/$BASE_NAME"

view raw addmorepckgs.sh hosted with ❤ by GitHub

You can add your own little flare to the above section to talk to chef, puppet, or equivalent to install the rest of the server, its up to you.

Now ship it off to amazon, you’ll have to register it once you are done.

	ec2-bundle-image --image "$IMAGE_DIR/$IMG_FILE" --prefix "$BASE_NAME" --cert "$EC2_CERT" --privatekey "$EC2_PRIVATE_KEY" --user "$AWS_USER" --destination "$IMAGE_DIR" --arch "$ARCH"

	ec2-upload-bundle --manifest "$IMAGE_DIR/$BASE_NAME.manifest.xml" --bucket "$BUCKET_NAME" --access-key "$AWS_ACCESS_KEY_ID" --secret-key "$AWS_SECRET_ACCESS_KEY"

view raw ec2push.sh hosted with ❤ by GitHub

Below is the amazon setup script (awssetup) that is referenced above and will be executed on startup, taken from an amazon fedora 8 image, mostly verbatim.

	#!/bin/sh
	# $Rev: 45452 $
	# This script will be executed after all the other init scripts.
	# You can put your own initialization stuff in here if you don't
	# want to do the full Sys V style init stuff.
	echo "-----VERSION : ""\$Rev: 45452 $""-----"
	# Stuff we want to do once at launch and never again:
	if [ -f "/root/firstrun" ]; then
	# Update AMI tools to the latest version:
	# [ -x "/usr/local/sbin/update-tools.sh" ] && /usr/local/sbin/update-tools.sh
	# Try to find kernel modules matching current kernel:
	# [ -x "/usr/local/sbin/update-modules.sh" ] && /usr/local/sbin/update-modules.sh
	# Some kernels use xvc0 as their serial console device:
	if [ -c /dev/xvc0 ]; then
	if ! grep -q 'co:2345:respawn:/sbin/agetty xvc0 9600' /etc/inittab; then
	echo 'co:2345:respawn:/sbin/agetty xvc0 9600 vt100' >> /etc/inittab
	echo 'xvc0' >> /etc/securetty
	kill -1 1
	fi
	fi
	# Ensure devpts is mounted to prevent ssh hang-ups
	mount \| grep devpts > /dev/null 2>&1
	if [ $? -ne 0 ] ; then
	devpts="none /dev/pts devpts gid=5,mode=620 0 0"
	( grep -v "\#" /etc/fstab \| grep devpts > /dev/null 2>&1 ) \|\| echo $devpts >> /etc/fstab
	mount -a >/dev/null 2>&1
	fi
	# Randomise the root password as the last operation
	# We ideally have some more entropy at this stage
	echo "-----RANDOMISING ROOT PASSWORD-----" \|logger -s -t "ec2"
	dd if=/dev/urandom count=128 2>/dev/null\|md5sum\|passwd --stdin root >/dev/null 2>&1
	rm -f /root/firstrun
	# Regenerate the host keys at this stage
	# Having more entropy to work with
	echo "-----TRIGGERING HOST KEYS REGENERATION-----"\|logger -s -t "ec2"
	echo "Removing existing keys"\|logger -s -t "ec2"
	rm -f /etc/ssh/ssh_host_key.pub /etc/ssh/ssh_host_rsa_key.pub /etc/ssh/ssh_host_dsa_key.pub \
	/etc/ssh/ssh_host_key /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_dsa_key
	echo "Bouncing sshd to force regeneration"\|logger -s -t "ec2"
	/sbin/service sshd restart
	echo "Setting sshd to start as a service"\|logger -s -t "ec2"
	/sbin/chkconfig --level 2345 sshd on
	fi
	touch /var/lock/subsys/local
	# Get your chosen keypair credentials
	/var/awsscripts/get-credentials.sh
	givenhostname=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`
	echo "HOSTNAME=$givenhostname" >> "/etc/sysconfig/network"
	# =Output ssh host keys to console=
	[ -f /etc/ssh/ssh_host_key ] \|\| (ssh-keygen -f /etc/ssh/ssh_host_key -t rsa1 -C 'host' -N '' \| logger -s -t "ec2")
	[ -f /etc/ssh/ssh_host_rsa_key ] \|\| (ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -C 'host' -N '' \| logger -s -t "ec2")
	[ -f /etc/ssh/ssh_host_dsa_key ] \|\| (ssh-keygen -f /etc/ssh/ssh_host_dsa_key -t dsa -C 'host' -N '' \| logger -s -t "ec2")
	echo "-----BEGIN SSH HOST KEY FINGERPRINTS-----" \|logger -s -t "ec2"
	ssh-keygen -l -f /etc/ssh/ssh_host_key.pub \|logger -s -t "ec2"
	ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub \|logger -s -t "ec2"
	ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key.pub \|logger -s -t "ec2"
	echo "-----END SSH HOST KEY FINGERPRINTS-----" \|logger -s -t "ec2"

view raw awssetup.sh hosted with ❤ by GitHub

Here is the script for get-credentials.sh that is called in the above script, which is also found in a base amazon image.

	#!/bin/bash
	# Version : $Rev: 45328 $
	# root's public keys
	PUB_KEY_URI=http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
	# PUB_KEY_URI=http://169.254.169.254/1.0/meta-data/public-keys/0/openssh-key
	PUB_KEY_FROM_HTTP=/tmp/openssh_id.pub
	PUB_KEY_FROM_EPHEMERAL=/mnt/openssh_id.pub
	ROOT_AUTHORIZED_KEYS=/root/.ssh/authorized_keys



	# We need somewhere to put the keys.
	if [ ! -d /root/.ssh ] ; then
	mkdir -p /root/.ssh
	chmod 700 /root/.ssh
	fi

	# Fetch credentials...
	echo "-----CREDENTIALS RETRIEVAL-----"\|logger -s -t "ec2"
	# First try http
	echo "Attempting to retrieve public key from [$PUB_KEY_URI]"\|logger -t "ec2" -s
	curl --retry 3 --retry-delay 5 --silent --fail -o $PUB_KEY_FROM_HTTP $PUB_KEY_URI
	if [ $? -eq 0 -a -e $PUB_KEY_FROM_HTTP ] ; then
	if ! grep -q -f $PUB_KEY_FROM_HTTP $ROOT_AUTHORIZED_KEYS
	then
	cat $PUB_KEY_FROM_HTTP >> $ROOT_AUTHORIZED_KEYS
	echo "New key added to authrozied keys file from parameters"\|logger -t "ec2" -s
	else
	echo "Already have your key"\|logger -t "ec2" -s
	fi
	rm -f $PUB_KEY_FROM_HTTP

	elif [ -e $PUB_KEY_FROM_EPHEMERAL ] ; then
	echo "Meta-data fetch failed, trying from legacy ephemeral store location [$PUB_KEY_FROM_EPHEMERAL]"\|logger -t "ec2" -s
	# Try back to ephemeral store if http failed.
	# NOTE: This usage is deprecated and will be removed in the future
	if ! grep -q -f $PUB_KEY_FROM_EPHEMERAL $ROOT_AUTHORIZED_KEYS
	then
	cat $PUB_KEY_FROM_EPHEMERAL >> $ROOT_AUTHORIZED_KEYS
	echo "New key added to authrozied keys file from ephemeral store"\|logger -t "ec2" -s
	fi
	chmod 600 $PUB_KEY_FROM_EPHEMERAL

	fi


	if [ ! -f $ROOT_AUTHORIZED_KEYS ]
	then
	echo "!!! FATAL ERROR !!! No able to find authorized_keys file [$ROOT_AUTHORIZED_KEYS]"\|logger -t "ec2" -s
	else
	echo "Setting permissions on $ROOT_AUTHORIZED_KEYS to 0600"\|logger -t "ec2" -s
	chmod 600 $ROOT_AUTHORIZED_KEYS
	fi

view raw get-credentials.sh hosted with ❤ by GitHub

Works with the following:

Kernel - aki-a71cf9ce
Ramdisk - ari-a51cf9cc