CloudFormation 101 – Part 03

This is part 03 in this series. Previously in

Part 01 – I created a VPC, and internet gateway and attached the internet gateway to the VPC

Part 02 – I added a public and private subnet in one availability zone

In this part, I am adding
– an EIP for the NAT gateway,
– a NAT gateway
– public and private route tables with default routes, and
– associating the route tables with their respective subnets.

The complete (includes all parts 01-03) code is below:

AWSTemplateFormatVersion: '2010-09-09'
#
## The Description section (optional) enables you to include comments about your template.
#
Description:  
  Create VPC, and related components
#
## Parameters section to customize your templates
#
Parameters:
  VPCName:
    Description: Name of the VPC
    Type: String
    Default: "MyVPC"    
    MinLength: '1'
    MaxLength: '30'
    AllowedPattern: '^[a-zA-Z]+[0-9a-zA-Z\-]*$'
    ConstraintDescription: Must contain alphabets and/or numbers.

  VpcCIDR:
    Description: Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 10.0.0.0/16    
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

  PublicSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
    Type: String
    Default: 10.0.1.0/24
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

  PrivateSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
    Type: String
    Default: 10.0.3.0/24
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

#
## Resources created by the stack
#
Resources:
  #
  ## Create the VPC
  ##
  ## Uses the intrinsic function Ref to get the value of the VPC Name
  ## from parameters above
  #
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref VPCName
  #
  ## Create the IGW
  #
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Ref VPCName
  #
  ## Connect the IGW to the VPC
  #
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC
  #
  ## Create a public subnet
  ##
  ## The VpcId is obtained by referring back to the VPC created above
  ##
  ## The CIDR block is from the parameters
  ##
  ## The Availability Zone is obtained by querying the available availability
  ## zones in this region and returning the first (offset 0) entry
  ##
  ## The MapPublicIpOnLaunch is set to true indicating that instances launched 
  ## in this subnet receive a public IPv4 address
  #
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PublicSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Subnet (AZ1)
  #
  ## Create a private subnet
  ##
  ## The MapPublicIpOnLaunch is set to false indicating that instances launched 
  ## in this subnet will not receive a public IPv4 address
  #
  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs  '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Subnet (AZ1)

  #
  ## Create an Elastic IP (EIP) address
  ##
  ## The below section also uses the DependsOn attribute. With the DependsOn 
  ## attribute you can specify that the creation of a specific resource follows 
  ## another. When you add a DependsOn attribute to a resource, that resource is 
  ## created only after the creation of the resource specified in the DependsOn 
  ## attribute. In this case the NAT Gateway is dependent on the prior creation 
  ## of the InternetGatewayAttachment and the attachment to the VPC 
  #
  NatGateway1EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} NatGateway1 EIP

  #
  ## Create a NAT GW
  ##  
  ## Connect the EIP created above to the NAT GW
  ##
  ## Default to public connectivity
  ##
  ## Connecting to the subnet by using the parameter
  #
  NatGateway1:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway1EIP.AllocationId
      SubnetId: !Ref PublicSubnet1
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} NatGateway 1

  #
  ## Create a public route table
  #
  PublicRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Routes
  #
  ## Add a public route in the above route table to allow the 
  ## subnets to access the internet through the IGW
  #
  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway
  #
  ## Associate the public route tables with the public subnets
  #
  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable1
      SubnetId: !Ref PublicSubnet1

  #
  ## Create a private route table
  #
  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Routes (AZ1)
  #
  ## Add a route in the above route table to allow the 
  ## subnets to access the internet through the IGW
  #
  DefaultPrivateRoute1:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway1

  #
  ## Associate the private route tables with the private subnets
  #
  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      SubnetId: !Ref PrivateSubnet1

#
## Resources created by the stack
##
## Uses the intrinsic function Sub to get the stack name 
## from parameters above and substitute it into the name of
## the internet gateway
#          
Outputs:
  VPC:
    Description: Name of the VPC
    Value: !Ref VPC
    Export:
      Name: !Sub '${AWS::StackName}'

  InternetGateway:
    Description: Internet Gateway 
    Value: !Ref InternetGateway
    Export:
      Name: !Sub '${AWS::StackName}-InternetGateway'

  PublicSubnet1:
    Description: AZ1 - public subnet
    Value: !Ref PublicSubnet1
    Export:
      Name: !Sub '${AWS::StackName}-PublicSubnet1'
      
  PrivateSubnet1:
    Description: AZ1 - private subnet 01
    Value: !Ref PrivateSubnet1
    Export:
      Name: !Sub '${AWS::StackName}-PrivateSubnet1'
      
  NatGateway1EIP:
    Description: NAT Gateway EIP
    Value: !Ref NatGateway1EIP
    Export:
      Name: !Sub '${AWS::StackName}-NatGateway1EIP'
      
  NatGateway1:
    Description: NAT Gateway 1
    Value: !Ref NatGateway1
    Export:
      Name: !Sub '${AWS::StackName}-NatGateway1'

  PublicRouteTable1:
    Description: Public route table
    Value: !Ref PublicRouteTable1
    Export:
      Name: !Sub '${AWS::StackName}-PublicRouteTable1'

  PrivateRouteTable1:
    Description: Private route table - AZ1 - private subnet 01
    Value: !Ref PrivateRouteTable1
    Export:
      Name: !Sub '${AWS::StackName}-PrivateRouteTable1'

CloudFormation – Create and customize a windows server using user data

This blog demonstrates how to

  • create a windows server via CloudFormation
  • create a user directory
  • install the aws cli
  • add the aws cli to the windows path
  • Create a log file to track the output of the commands executed in the user data section
  • Download a file from an s3 bucket

By using these commands, you can download and install software after the Windows instance is created.

The IAM profile assigned to the EC2 instance should have access to the s3 bucket used.

The code is below:

AWSTemplateFormatVersion: 2010-09-09
Description: Innovate - Create App and Web Servers
Resources:
  rMyBastionInstance:
    Type: 'AWS::EC2::Instance'
    Properties:
      ImageId: 'ami-I-want-to-use'
      KeyName: 'my_windows_key'
      IamInstanceProfile: 'My-IAM-profile'
      InstanceType: 'm4.2xlarge'
      SecurityGroupIds: 
        - sg-security-group-id
      SubnetId: 'my-subnet-id'
      BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
                VolumeType: gp2
                DeleteOnTermination: true
                VolumeSize: 100
                Encrypted: true
      UserData:
        'Fn::Base64': !Sub |
          <script>
              cd \
              mkdir tempdc
              msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi /qn
              cd \tempdc
              echo Current date and time > c:\tempdc\s3_output.txt
              echo %DATE% %TIME% >> c:\tempdc\s3_output.txt
              SET PATH=%PATH%;C:\Program Files\Amazon\AWSCLIV2
              echo %PATH% >> c:\tempdc\s3_output.txt
              aws s3 cp s3://mybucket/my-file.exe c:\tempdc\my-file.exe >> c:\tempdc\s3_output.txt
          </script>
      Tags:
        - Key: Name
          Value: MyWindowsInstance

CloudFormation 101 – Part 02

In the previous blog, I had created a VPC and internet gateway and attached the internet gateway to the VPC. I will add a public and private subnet in one availability zone in this post.

Some definitions:
Subnets
A subnet is a range of IP addresses in your VPC. You can launch AWS resources into a specified subnet.

Public subnet
A public subnet contains resources that will be connected from and to the internet—for example, load balancers, web servers, etc.

Private subnet
A private subnet contains resources that will not be accessed directly from the internet—for example, application servers, databases, etc.

It is critical to place your resources in the appropriate subnet. Placing resources with sensitive code or information in a public subnet would open them up to compromise by bad actors, of which there are many on the internet 😊.

The code is below:

AWSTemplateFormatVersion: '2010-09-09'
#
## The Description section (optional) enables you to include comments about your template.
#
Description:  
  Create VPC, an internet gateway, and attach the internet gateway to the VPC
#
## Parameters section to customize your templates
#
Parameters:
  VPCName:
    Description: Name of the VPC
    Type: String
    Default: "MyVPC"    
    MinLength: '1'
    MaxLength: '30'
    AllowedPattern: '^[a-zA-Z]+[0-9a-zA-Z\-]*$'
    ConstraintDescription: Must contain alphabets and/or numbers.

  VpcCIDR:
    Description: Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 10.0.0.0/16    
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

  PublicSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
    Type: String
    Default: 10.0.1.0/24
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

  PrivateSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
    Type: String
    Default: 10.0.3.0/24
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

#
## Resources created by the stack
#
Resources:
  #
  ## Create the VPC
  ##
  ## Uses the intrinsic function Ref to get the value of the VPC Name
  ## from parameters above
  #
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref VPCName
  #
  ## Create the IGW
  #
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Ref VPCName
  #
  ## Connect the IGW to the VPC
  #
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC
  #
  ## Create a public subnet
  ##
  ## The VpcId is obtained by referring back to the VPC created above
  ##
  ## The CIDR block is from the parameters
  ##
  ## The Availability Zone is obtained by querying the available availability
  ## zones in this region and returning the first (offset 0) entry
  ##
  ## The MapPublicIpOnLaunch is set to true indicating that instances launched 
  ## in this subnet receive a public IPv4 address
  #
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PublicSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Subnet (AZ1)
  #
  ## Create a private subnet
  ##
  ## The MapPublicIpOnLaunch is set to false, indicating that instances launched 
  ## in this subnet will not receive a public IPv4 address
  #
  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs  '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Subnet (AZ1)

#
## Resources created by the stack
##
## Uses the intrinsic function Sub to get the stack name 
## from parameters above and substitute it into the name of
## the internet gateway
#          
Outputs:
  VPC:
    Description: Name of the VPC
    Value: !Ref VPC
    Export:
      Name: !Sub '${AWS::StackName}'

  InternetGateway:
    Description: Internet Gateway 
    Value: !Ref InternetGateway
    Export:
      Name: !Sub '${AWS::StackName}-InternetGateway'

  PublicSubnet1:
    Description: AZ1 - public subnet
    Value: !Ref PublicSubnet1
    Export:
      Name: !Sub '${AWS::StackName}-PublicSubnet1'
      
  PrivateSubnet1:
    Description: AZ1 - private subnet 01
    Value: !Ref PrivateSubnet1
    Export:
      Name: !Sub '${AWS::StackName}-PrivateSubnet1'

CloudFormation 101 – Part 01

Some quick blogs on using CloudFormation to provision AWS infrastructure using code. I have attempted to put descriptive comments in line to explain what each sub-unit of code is doing.

This YAML code creates a VPC, an internet gateway, and attaches the gateway to the VPC.

Some definitions:
VPC
Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you’ve defined. This virtual network closely resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

Internet Gateway
An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between your VPC and the internet.

VPC Gateway Attachment
Attaches an internet gateway, or a virtual private gateway to a VPC, enabling connectivity between the internet and the VPC.

The code is below:

AWSTemplateFormatVersion: '2010-09-09'
#
## The Description section (optional) enables you to include comments about your template.
#
Description:  
  Create VPC, an internet gateway, and attach the internet gateway to the VPC
#
## Parameters section to customize your templates
#
Parameters:
  VPCName:
    Description: Name of the VPC
    Type: String
    Default: "MyVPC"    
    MinLength: '1'
    MaxLength: '30'
    AllowedPattern: '^[a-zA-Z]+[0-9a-zA-Z\-]*$'
    ConstraintDescription: Must contain alphabets and/or numbers.

  VpcCIDR:
    Description: Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 10.0.0.0/16    
    MinLength: '10'
    MaxLength: '18'
    AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
    ConstraintDescription: Must be a valid CIDR range of the form x.x.x.x/x.

#
## Resources created by the stack
#
Resources:
  #
  ## Create the VPC
  ##
  ## Uses the intrinsic function Ref to get the value of the VPC Name
  ## from parameters above
  #
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref VPCName
  #
  ## Create the IGW
  #
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Ref VPCName
  #
  ## Connect the IGW to the VPC
  #
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

#
## Resources created by the stack
##
## Uses the intrinsic function Sub to get the stack name 
## from parameters above and substitute it into the name of
## the internet gateway
#          
Outputs:
  VPC:
    Description: Name of the VPC
    Value: !Ref VPC
    Export:
      Name: !Sub '${AWS::StackName}'

  InternetGateway:
    Description: Internet Gateway 
    Value: !Ref InternetGateway
    Export:
      Name: !Sub '${AWS::StackName}-InternetGateway'
      

Creating an EC2 in AWS via Cloud Formation

This blog post explains how to create (“spin up” in the cool lingo 🙂 ) an EC2 in AWS using Cloud Formation. I will attempt to explain each line or section of code before the code itself. The complete cloud formation template is available at the end of this post. This template is written in JSON. It could also have been done in YAML.

This is the opening curly brace. Just make sure you have one at the end to close it out. Strongly suggest using a JSON formatter and syntax checker.

{

These lines identify the AWS template format version and a description tag for documentation purposes

    "AWSTemplateFormatVersion":"2010-09-09",
    "Description":"Linux EC2 to test with",

The parameters section contains the parameters that will be passed from my Jenkins job to this cloud formation template. Each of these can have a type, default value, and description.

    "Parameters":{
        "CFstackname":{
            "Type":"String"
        },
        "AppID":{
            "Type":"String",
            "Default":"ABC",
            "Description":"Name of the application. Default=ABC"
        },
        "Role":{
            "Type":"String",
            "Default":"App",
            "Description":"Enter the Role name. Default=App"
        },
        "UserID":{
            "Type":"String",
            "Description":"Enter the userid of the owner(s)"
        }
    },

This next section specifies the resources that are to be created by this cloud formation template.

    "Resources":{

The type of the resource to be created is an EC2 instance

        "EC2Instance":{
            "Type":"AWS::EC2::Instance",

This section contains the properties to describe the resource.

            "Properties":{

This parameter specifies the instance type which in this case is a t3 small

                "InstanceType":"t3.small",

The DisableApiTermination attribute controls whether the instance can be terminated using the console, CLI, or API. While writing this blog post I did learn that the DisableApiTermination attribute does not prevent Amazon EC2 Auto Scaling from terminating an instance. Just a useful piece of information.

                "DisableApiTermination":"false",

This specifies the network interfaces. The AssociatePublicIpAddress is set to false in this case. The subnet ID is your subnet i.e. logical subdivision of an IP network. The group set contains the security groups that you want to be attached to this EC2 instance. These need to be predefined.

                "NetworkInterfaces":[
                    {
                        "AssociatePublicIpAddress":"false",
                        "DeviceIndex":"0",
                        "SubnetId":"my-subnet-01",
                        "GroupSet":[
                            "sg-for-me-01",
                            "sg-for-me-02"
                        ]
                    }
                ],

This next parameter specifies the Amazon machine image (AMI) that you wish to use to create this instance. In my case, I had a pre-created AMI that I was using as it was already encrypted with my KMS key

                "ImageId":"ami-dean-01",

Here I’m specifying the block devices (disks) to be associated with this machine. The volume size is specified in gigabytes. Storage is deleted when the EC2 is terminated. The disk is to be encrypted with the KMS key specified

                "BlockDeviceMappings":[
                    {
                        "DeviceName":"/dev/sda1",
                        "Ebs":{
                            "VolumeSize":"100",
                            "DeleteOnTermination":true,
                            "VolumeType":"gp2",
                            "Encrypted":"true",
                            "KmsKeyId":"my magical KMS key"
                        }
                    }
                ],

These are the tags that are to be associated with this EC2. Some of them are from the parameters of the top of the template. This is a free form to the extent that you can specify any tag string as the key and any value as the value.

                "Tags":[
                    {
                        "Key":"UserID",
                        "Value":"xyz01"
                    },
                    {
                        "Key":"Name",
                        "Value":"My Linux Machine"
                    },
                    {
                        "Key":"Role",
                        "Value":"App"
                    }

Close all the curly braces or else bad things will happen 🙂

 
                ]
            }
        }
    },

This section is the outputs that I am expecting. The first is the instance id, followed by the Availability Zone (AZ) and the private IP

    "Outputs":{
        "InstanceId":{
            "Description":"InstanceId of the newly created EC2 instance",
            "Value":{
                "Ref":"EC2Instance"
            }
        },
        "AZ":{
            "Description":"Availability Zone of the newly created EC2 instance",
            "Value":{
                "Fn::GetAtt":[
                    "EC2Instance",
                    "AvailabilityZone"
                ]
            }
        },
        "PrivateIP":{
            "Description":"PrivateIP of the newly created EC2 instance",
            "Value":{
                "Fn::GetAtt":[
                    "EC2Instance",
                    "PrivateIp"
                ]
            }
        }
    }
}

Complete cloud formation template

{
    "AWSTemplateFormatVersion":"2010-09-09",
    "Description":"Linux EC2 to test with",
    "Parameters":{
        "CFstackname":{
            "Type":"String"
        },
        "AppID":{
            "Type":"String",
            "Default":"ABC",
            "Description":"Name of the application. Default=ABC"
        },
        "Role":{
            "Type":"String",
            "Default":"App",
            "Description":"Enter the Role name. Default=App"
        },
        "UserID":{
            "Type":"String",
            "Description":"Enter the userid of the owner(s)"
        }
    },
    "Resources":{
        "EC2Instance":{
            "Type":"AWS::EC2::Instance",
            "Properties":{
                "InstanceType":"t3.small",
                "DisableApiTermination":"false",
                "NetworkInterfaces":[
                    {
                        "AssociatePublicIpAddress":"false",
                        "DeviceIndex":"0",
                        "SubnetId":"my-subnet-01",
                        "GroupSet":[
                            "sg-for-me-01",
                            "sg-for-me-02"
                        ]
                    }
                ],
                "ImageId":"ami-dean-01",
                "BlockDeviceMappings":[
                    {
                        "DeviceName":"/dev/sda1",
                        "Ebs":{
                            "VolumeSize":"100",
                            "DeleteOnTermination":true,
                            "VolumeType":"gp2",
                            "Encrypted":"true",
                            "KmsKeyId":"my magical KMS key"
                        }
                    }
                ],
                "Tags":[
                    {
                        "Key":"UserID",
                        "Value":"xyz01"
                    },
                    {
                        "Key":"Name",
                        "Value":"My Linux Machine"
                    },
                    {
                        "Key":"Role",
                        "Value":"App"
                    }
 
                ]
            }
        }
    },
    "Outputs":{
        "InstanceId":{
            "Description":"InstanceId of the newly created EC2 instance",
            "Value":{
                "Ref":"EC2Instance"
            }
        },
        "AZ":{
            "Description":"Availability Zone of the newly created EC2 instance",
            "Value":{
                "Fn::GetAtt":[
                    "EC2Instance",
                    "AvailabilityZone"
                ]
            }
        },
        "PrivateIP":{
            "Description":"PrivateIP of the newly created EC2 instance",
            "Value":{
                "Fn::GetAtt":[
                    "EC2Instance",
                    "PrivateIp"
                ]
            }
        }
    }
}

Copying a file from a Linux on-prem server to an AWS RDS Oracle instance

Today we had a requirement to copy a file from a Linux on-prem server to an AWS RDS Oracle instance. We did this using a DB link and utl_file utility. For the rest of this post, “ONP” will refer to the on-prem instance, and “ADS” will refer to the AWS RDS Oracle instance.

On the AWS RDS Oracle instance, create a DB link to the on-prem instance

create database link AWS_TO_ON_PREM CONNECT TO ONPUSER
IDENTIFIED BY SECRET-PASSWORD
USING '(DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ONPREM.HOST.COM)(PORT = 1776))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = ONPDB)
    )
  )'
   ;

On the AWS RDS Oracle instance, create a directory to store the file

exec rdsadmin.rdsadmin_util.create_directory('ADS_DIRECTORY');

Check that the directory got created

Select * from dba_directories;

As AWS manages the path internally, the above SQL will return the actual directory path.

On the on-prem instance, create a directory pointing to the location of the file that you want to copy to AWS

CREATE OR REPLACE DIRECTORY
FOR_AWS AS '/onprem/host/file/location/file_for_aws.bin';

GRANT READ, WRITE ON DIRECTORY FOR_AWS TO ONPUSER;

Now to copy the file from the on-prem instance to the AWS instance. This is accomplished via the below PL/SQL which is to be executed while connected to the AWS RDS Oracle instance. In our case, the data was binary (raw) data

declare
  ONP    utl_file.file_type@AWS_TO_ON_PREM;
  ADS    utl_file.file_type;
  ldata  raw(32767);
begin
  ONP := utl_file.fopen@AWS_TO_ON_PREM(location=>'FOR_ADS', filename=>'file_for_aws.bin', open_mode=>'rb');
  ADS := utl_file.fopen(location=>'ADS_DIRECTORY', filename=>'file_for_aws.bin', open_mode=>'wb');
  begin
    loop
      begin
        utl_file.get_raw@AWS_TO_ON_PREM(ONP, ldata, 32767);
        utl_file.put_raw(ADS, ldata, true);
      exception
        when no_data_found then
          exit;
      end;
    end loop;
  end;
  utl_file.fclose@AWS_TO_ON_PREM (ONP);
  utl_file.fclose(ADS);  
exception
  when others then
    utl_file.fclose@AWS_TO_ON_PREM(ONP);
    utl_file.fclose(ADS);
    raise;
end;
/

After the above is complete, check that the file size in AWS matches the file size from on-prem with this sql

select    filesize, mtime 
from      table(rdsadmin.rds_file_util.listdir('ADS_DIRECTORY')) 
where     filename='file_for_aws.bin'
;

If you need to delete a file in AWS, you can use this command

exec utl_file.fremove('directory-name','file-name');

ORA-15032, ORA-15017 and ORA-15040 issues with an ASM disk group

While creating a new Oracle database with ASM on an AWS EC2 platform I encountered the below errors while attempting to mount one of the ASM disk groups:

ALTER DISKGROUP VOTE_DISK mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "VOTE_DISK" cannot be mounted
ORA-15040: diskgroup is incomplete

When I checked the v$asm_diskgroup, I saw

SQL> select NAME,TOTAL_MB,FREE_MB from v$asm_diskgroup;  

NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
ASMDATA1                            716796     716726
DBVOT                                    0          0
ASMDATA2                            716796     716726
ASMARCH                             716796     716726

The goal was to create the VOTE_DISK diskgroup on the DBVOT volume that was in turn mapped to the /dev/bqdx disk. The AWS Console indicated that the /dev/bqdx was attached to the EC2 but was not visible in the OS for some reason. I issued the command:

/usr/bin/echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | /usr/sbin/fdisk /dev/bqdx

Which resulted in:

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): Building a new DOS disklabel with disk identifier 0x29ba6b8d.

Command (m for help): Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): Partition number (1-4, default 1): First sector (2048-104857599, default 2048): Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599): Using default value 104857599
Partition 1 of type Linux and of size 50 GiB is set

Command (m for help): Selected partition 1
Hex code (type L to list all codes): Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

However, when I attempted to create the disk I got the error:

root@10-118-134-71:/root # /usr/sbin/oracleasm createdisk DBVOT /dev/bqdx1
Device "/dev/bqdx1" is already labeled for ASM disk "DBVOT"

I then attempted to drop the disk group

SQL> drop diskgroup DBVOT;
drop diskgroup DBVOT
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup "DBVOT" does not exist or is not mounted

I then tried with the FORCE option and succeeded

SQL> drop diskgroup DBVOT force including contents;

Diskgroup dropped.

I then deleted the disk

root@10-118-134-71:/root # /usr/sbin/oracleasm deletedisk DBVOT 
Clearing disk header: done
Dropping disk: done

and checked to make sure it was no longer visible

[oracle@10-118-134-71 ~]$ /usr/sbin/oracleasm listdisks
DB1
DG10
DG11
DG12
DG2
DG3
DG4
DG5
DG6
DG7
DG8
DG9

Then I recreated the disk

root@10-118-134-71:/root # /usr/sbin/oracleasm createdisk DBVOT /dev/bqdx1
Writing disk header: done
Instantiating disk: done

and listed the disks to confirm

root@10-118-134-71:/root # /usr/sbin/oracleasm listdisks
DG1
DG10
DG11
DG12
DG2
DG3
DG4
DG5
DG6
DG7
DG8
DG9
DBVOT

Then I created the disk in ASM

SQL> CREATE DISKGROUP DBVOT EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/DBVOT';

Diskgroup created.

Installing the Oracle Client 12c on an AWS Linux EC2 – Silent Mode

After the EC2 is built, run the below commands as root. Depending on the network security, you may have to either set or unset proxy variables.

/usr/bin/yum install -y binutils.x86_64 compat-libcap1.x86_64 compat-libstdc++-33.i686 \
compat-libstdc++-33.x86_64 gcc.x86_64 gcc-c++.x86_64 glibc-devel.i686 glibc-devel.x86_64 glibc.i686 \
glibc.x86_64 libXtst.x86_64 libXtst.i686 ksh.x86_64 libaio.x86_64 libaio.i686 libaio-devel.i686 \
libaio-devel.x86_64 libgcc.i686 libgcc.x86_64 libstdc++.x86_64 libstdc++.i686 libstdc++-devel.i686 \
libstdc++-devel.x86_64 libXi.i686 libXi.x86_64 make.x86_64 sysstat.x86_6

The above command will install all of the prerequisites required for the Oracle Client 12c .

Create the install group

/usr/sbin/groupadd oinstall

Create the dba group

/usr/sbin/groupadd dba

Create the oracle user and add to the groups

/usr/sbin/useradd -g oinstall -G dba oracle

Set the oracle password

passwd oracle

Respond to the above prompt with the oracle password that you want to use.

Make a directory for the software

mkdir /apps/opt/oracle

Change ownership of the directory to the oracle user and the oinstall group

chown oracle:oinstall /apps/opt/oracle

Change to oracle user

su - oracle

Copy the software into the above directory via ftp, sftp, scp etc. The directory will contain file

linuxamd64_12102_client.zip

Unzip the foftware

unzip linuxamd64_12102_client.zip

Change directory to the client directory that was unzipped.

cd /apps/opt/oracle/client

Run the installer in silent mode

./runInstaller -silent \
 -responseFile /apps/opt/oracle/client/response/client_install.rsp    \
   oracle.install.client.installType=Runtime           \
   UNIX_GROUP_NAME=dba                                 \
   INVENTORY_LOCATION=/apps/opt/oracle/oraInventory            \
   SELECTED_LANGUAGES=en                               \
   ORACLE_HOME=/apps/opt/oracle/base/rtclient64            \
   ORACLE_BASE=/apps/opt/oracle/base                           \
   waitForCompletion=true

At the end of the installation, you should see the message

The installation of Oracle Client 12c was successful.
Please check '/apps/opt/oracle/oraInventory/logs/silentInstall2017-08-30_06-00-25PM.log' for more details

Exit from oracl back to root

exit

As a root user, execute the following script

/apps/opt/oracle/oraInventory/orainstRoot.sh

The above script will display messages similar to

Changing permissions of /apps/opt/oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /apps/opt/oracle/oraInventory to dba.
The execution of the script is complete.

Set up environment variables for the oracle user. Go to the oracle user’s home

su - oracle
cd

Edit the oracle users

vi .bash_profile

Setup the below variable to make it easier for the users to access sqlplus etc.

# Oracle client variables
export ORACLE_HOME=/apps/opt/oracle/base/rtclient64
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
PATH=$ORACLE_HOME/bin:$PATH
which sqlplus

Test sqlplus

sqlplus user_id

SQL*Plus: Release 12.1.0.2.0 Production on Wed Aug 30 18:23:27 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Enter password: 

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Data Mining and Real Application Testing options

AWS – EC2 with Oracle for a I/O intensive application – Part 02

This week one of our applications is attempting to benchmark the performance of an Oracle database running on an Exadata machine versus running in the AWS cloud on an x1.16xlarge EC2. Our initial thought was to define the storage as two ASM disk groups, each consisting of four 400 gb disks at 6,000 IOPS each per ASM disk group. Assuming that a given data file was spread across all four disks in a disk group, we could have speeds approaching 20,000 IOPS.

During testing, Amazon recommended that the ASM data be placed on single disks running at 20,000 IOPS. The current maximum IOPS that can be provisioned is 20,000 IOPS. Additionally, AWS requires that the IOPS:gb ratio be maintained at a maximum of 50:1. In order to meet these requirements, we decided to create two mount points, each of 1000 gb at 20,000 IOPS. An additional 1000 gb at 20,000 IOPS was also added for the archive logs.

We created a json template to spin up the three disks and mounted them to the existing EC2. Before mounting, make sure that the mount points are not already being used by currently attached disks. After the disks are mounted, we attached them to ASM and waited for the rebalance to complete. After the rebalance was completed, we dropped the old disks that were provisioned at 6000 IOPS. The application saw a 67% decrease in elapsed time on some of the queries.