Create a EC2 Amazon Machine Image - AMI

Customize an existing AMI image

It's much easier to customize an existing AMI image to your own taste than to start from scratch so let's do that. First we need to find an image that is close to what we want. Read EC2 quickstart on how to setup the EC2 API tools.

$ ec2-describe-images -a 
…
IMAGE	ami-f0f61599	alestic-64/debian-5.0-lenny-base-64-20091011.manifest.xml	063491364108	available	public		x86_64	machine	aki-a3d737ca	ari-7cb95a15		instance-store
…

This debian lenny image provided by alestic.com will serve as our base for running application servers and MySQL Cluster SQL and data nodes on. 

We are going to launch the image and install some software:

• Glassfish 3 (min req java 1.6.0.17)
• MySQL Cluster 7.0.x
• Munin
• Monit

Then re-bundle and upload our new image as our own to S3 for storage. We leave everything as "default" for now, i.e., the image/server will not be configured as a MySQL server or an application server at launch. 

Launch the debian image

Since the debian image is bundled as 64-bit we need to specify the instance type because as default m1.small is used which is for 32-bit platforms. The most recent supported instances types can be found here.

Standard instances

• m1.small
  1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of local instance storage, 32-bit platform
• m1.large
  7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platfor
• m1.xlarge
  15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

High-Memory Instances

• m2.2xlarge
  34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform
• m2.4xlarge
  68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

High-CPU Instances

• c1.medium
  1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of local instance storage, 32-bit platform
• c1.xlarge
  7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

m1.large is fine for our needs.

$ ec2-run-instances ami-f0f61599 --instance-type=m1.large -key alex-keypair
RESERVATION     r-6b962203      242055002701    default
INSTANCE        i-0f460c67      ami-f0f61599                    pending alex-keypair    0               m1.large       2009-12-15T13:16:29+0000 us-east-1b      aki-a3d737ca    ari-7cb95a15            monitoring-disabled                    instance-store

Logon to the instance.

$ ssh -i ~/.ec2/pk-alex-keypair root@ec2-174-129-172-248.compute-1.amazonaws.com 
Linux domU-12-31-39-05-3C-21 2.6.21.7-2.fc8xen-ec2-v1.0 #1 SMP Tue Sep 1 10:25:30 EDT 2009 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Amazon EC2 Debian 5.0.3 lenny AMI built by Eric Hammond
http://alestic.com  http://ec2debian-group.notlong.com

domU-12-31-39-05-3C-21:~#

Install your software, tune the OS etc

$ vi /etc/motd
$ dpkg -l
$ aptitude update
$ aptitude install sun-java6-jdk
$ wget http://download.java.net/glassfish/v3/release/glassfish-v3.zip
$ aptitude install unzip
$ cd /opt
$ unzip /root/glassfish-v3.zip
$ ln -s glassfishv3 glassfish
$ /opt/glassfish/bin/asadmin start-domain
Waiting for DAS to start .......
Started domain: domain1
Domain location: /opt/glassfishv3/glassfish/domains/domain1
Log file: /opt/glassfishv3/glassfish/domains/domain1/logs/server.log
Admin port for the domain: 4848
Command start-domain executed successfully.
$ # Enable glassfish admin port
$ ec2-authorize default -p 4848
GROUP           default
PERMISSION              default ALLOWS  tcp     4848    4848    FROM    CIDR    0.0.0.0/0
$ # launch browser at http://ec2-174-129-172-248.compute-1.amazonaws.com:4848
$ /opt/glassfish/bin/asadmin stop-domain
Waiting for the domain to stop ....
Command stop-domain executed successfully.
$ aptitude install munin munin-node munin-plugins-extra
$ aptitude install monit
$ wget http://dev.mysql.com/get/Downloads/MySQL-Cluster-7.0/mysql-cluster-gpl-7.0.9-linux-x86_64-glibc23.tar.gz/from/http://ftp.jaist.ac.jp/pub/mysql/
$ tar zxvf mysql-cluster-gpl-7.0.9-linux-x86_64-glibc23.tar.gz
$ ln -s mysql-cluster-gpl-7.0.9-linux-x86_64-glibc23 /usr/local/mysql
$ rm mysql-cluster-gpl-7.0.9-linux-x86_64-glibc23.tar.gz

In a future post I'll show you a script to launch multiple AMI instances and starting MySQL Cluster nodes on them based on how the MySQL Cluster configuration file is defined.

Re-bundle image and upload to S3

Remember that you cannot modify an AMI image after it has been created so if you make this image public for everyone to access then remove any history/logs etc that you don't want anyone to see. For  the next steps you need your AWS certificate, private key and the access key which can be found here. 

First we need to copy the AWS certifcate and private key to /tmp on our instance. We don't want our credentials included in the image so make sure the files are in /tmp or anyother directory that will not be included in the image.

$ scp -i ~/.ec2/pk-alex-keypair ~/.ec2/*.pem root@ec2-174-129-172-248.compute-1.amazonaws.com:/tmp

Let's create the actual bundle. Your access user ID is your account number, 12 digits and optionally hyphenated. The manifest prefix is the only description of the image so it's important to be as detailed as possible.

$ ec2-bundle-vol \
 --cert /tmp/cert-*.pem \
 --privatekey /tmp/pk-*.pem \
 --user 1234-5678-9012 \
 --destination /mnt \
 --arch `uname -m` \
 --size 10240 \
 --prefix 'debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216'
Copying / into the image file /mnt/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216...
Excluding: 
         /selinux
         /sys
         /proc/bus/usb
         /proc
         /dev/pts
         /dev
         /media
         /mnt
         /proc
         /sys
         /etc/udev/rules.d/70-persistent-net.rules
         /etc/udev/rules.d/z25_persistent-net.rules
         /root/.ssh
         /tmp
         /mnt/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216
         /mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.001628 s, 644 MB/s
mke2fs 1.41.3 (12-Oct-2008)
Bundling image file...
Splitting /mnt/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.tar.gz.enc...
Created debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.00
Created debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.01
Created debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.02
Created debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.03
Created debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.04
Created debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.05
...
Generating digests for each part...
Digests generated.
Unable to read instance meta-data for product-codes
Creating bundle manifest...
ec2-bundle-vol complete.

You need your AWS access key and secrect access key to upload the image to S3. The S3 bucket name is like a namespace for your files/objects.

$ ec2-upload-bundle \
 --bucket 'mysql-cluster-7-images' \
 --access-key <AWS access key> \
 --secret-key <AWS secret key> \
 --manifest /mnt/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.manifest.
xml \
 --location US
Uploading bundled image parts to the S3 bucket mysql-cluster-7-images ...
Uploaded debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.00
Uploaded debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.01
...
Uploaded debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.64
Uploaded debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.65
Uploading manifest ...
Uploaded manifest.
Bundle upload completed.

Register/deregister new AMI image

Finally it's time to register your new image and it's done on your localhost.

$ ec2-register --description 'Debian 5.0 base 64 image with MySQL Cluster 7.0.9' --name debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216 mysql-cluster-7-images/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.manifest.xml
IMAGE   ami-a045a7c9
$ ec2-describe-images
IMAGE   ami-a045a7c9    242055002701/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216 242055002701    available       private         x86_64  machine aki-a3d737ca    ari-7cb95a15            instance-store
IMAGE   ami-9130d1f8    mysql-cluster-7-images/ubuntu-9.04-glassfishv2-base-64-20090717.manifest.xml    242055002701   available        private         x86_64  machine aki-b51cf9dc    ari-b31cf9da            instance-store

Public and private images

The newly created image is private by default and can only be used by you however you can modify the permissions to allow another user or all, i.e., the public to use the image as well.

$ ec2-modify-image-attribute --launch-permission --add all ami-a045a7c9
launchPermission        ami-a045a7c9    ADD     group   all
$ ec2-describe-images 
IMAGE   ami-a045a7c9    242055002701/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216 242055002701    available       public          x86_64  machine aki-a3d737ca    ari-7cb95a15            instance-store
IMAGE   ami-9130d1f8    mysql-cluster-7-images/ubuntu-9.04-glassfishv2-base-64-20090717.manifest.xml    242055002701   available        private         x86_64  machine aki-b51cf9dc    ari-b31cf9da            instance-store
$ ec2-modify-image-attribute --launch-permission --remove all ami-a045a7c9
launchPermission        ami-a045a7c9    REMOVE  group   all
$ ec2-describe-images 
IMAGE   ami-a045a7c9    242055002701/debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216 242055002701    available       private         x86_64  machine aki-a3d737ca    ari-7cb95a15            instance-store
IMAGE   ami-9130d1f8    mysql-cluster-7-images/ubuntu-9.04-glassfishv2-base-64-20090717.manifest.xml    242055002701   available        private         x86_64  machine aki-b51cf9dc    ari-b31cf9da            instance-store

Deregister and remove image. You need the EC2 AMI tools to be able to remove the S3 bundle from localhost.

$ ec2-deregister ami-a045a7c9
IMAGE   ami-a045a7c9
$ ec2-delete-bundle --bucket mysql-cluster-7-images --access-key AKIAJBGBDP5JJTZANHOQ  --secr
et-key o1EIll8+ktJcpPn8xz3RtoXo2VF596YOFU8y+WI4 --prefix debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216
Deleting files:
   - debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.manifest.xml
   - debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.00
   - debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.01
   - debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.02
...
Deleted debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.64
Deleted debian-5.0-mysql-cluster-7.0.9-glassfisv3-base-64-20091216.part.65
ec2-delete-bundle complete.