AWS Auto Scaling using Python Boto3

Auto Scaling Group in AWS configure using Python Boto3 

What is Auto Scaling means? 

This is key capability or power of Cloud Computing Engineers believe in their skills on scaling abilities. 

Amazon EC2 Auto Scaling helps to maintain application availability and lets automatically add or remove EC2 instances using scaling policies that we define.  There are 2 types of scaling policies : Dynamic or predictive. These scaling policies let us add or remove EC2 instances capacity to service established or real-time demand patterns.

It contains various steps involved in Auto Scaling process using Python Boto3 we will explore every step that accumulate to form a complete automation solution for a DevOps project.

ASG Groups associated with ELB and EC2 instances

 

Understanding AWS Auto scaling configuration steps

Check any running instances Create launch configuration Configure ASG for Auto scaling Verify the configuration Disable Auto Scaling
In order to setup Auto Scaling, we need Launch configuration to be created first followed by Auto Scaling group

1. Check any running instances

Get the running EC2 instance list

import boto3
ec2_resource=boto3.resource('ec2')

instances = ec2_resource.instances.filter(
	Filters=[{'Name':'insance-state-name','Values':['running']}
	])
for i in instances:
    print(instance.id, instance.state)

Check the output

 

2. Create launch configuration 

 object client referring to 'autoscaling' method: create_launch_configuration() method is used to create launch configuration

as_client = boto3.client('autoscaling')	

ami_id = input("Please enter AMI id to use in Auto Scaling:")
keyname = input("Please enter Key Name to use for instances:")

response = as_client.create_launch_configuration(
	LaunchConfigurationName = 'vybhava_lc',
	ImageId = ami_id,
	KeyName = kayname,
	SecurityGroup = ['vybhava_sg']
	InstanceType = 't2.micro'
)
print(response)

When you observe the printed output have 'HTTPStatusCode as 200 then it is successful. 3. Creating Auto Scaling Group The create_auto_scaling_group() method is used to create Auto Scaling group.

asg_resp = as_client.create_auto_scaling_group(
	AutoScalingGroupName='vybhava_asg',
	LaunchConfigurationName='vybhava_lc',
	MinSize=1, MaxSize=2, DesiredCapacity=1,
	LoadBalancerNames=['vybhava_lb']
	AvailabilityZones=['ap-south-1b','ap-south-1c']
)
print (asg_resp)
print (asg_resp['ResponseMetadata']['HTTPStatusCode']) #Try this 

Check the output and compare it withe AWS console

 

4. Updating ASG 

You can try scale up by increase MinSize, MaxSize, DesiredCapacity values up similarly down. When you do update remember that is should not pass LoadBalancerNames.

update_resp = as_client.update_auto_scaling_group(
	AutoScalingGroupName='vybhava_asg',
	LaunchConfigurationName='vybhava_lc',
	MinSize=1, MaxSize=2, DesiredCapacity=1,

	AvailabilityZones=['ap-south-1b','ap-south-1c']
)

print(update_resp)

When you run down to 0 value then terminated ASG instances.

In Auto Scaling group configuration, Number of instances always remain up and running is the meaning of parameter 'DesiredCapacity'. Auto Scaling Group defining we need to define the Load Balancer in Auto Scaling setup.

Manage ELB using Python Boto3

 Managing AWS ELB automations using Boto3 Python scripts. 

What is ELB?

ELB automatically distributes incoming network traffic (may be HTTP/HTTPS requests) across multiple EC2 instances running in different Availability Zones.

ELB to EC2 instances

 In this post I will experimenting on the following: 

• Create ELB 
• Apply security group
• Create Health check
• Register instances with ELB  
• Delete ELB services

1. How to Create ELB using Python Boto3 automations?

The create_load_blancer method is used to create a Load Balancer.

object: client refers to elb 

method: create_load_blancer() which requires LoadBalancerName, Listeners that is again a dictionary of Protocol,LoadBalancerPort,InstanceProtocol, InstancePort, And another parameter 'AvailabilityZone' is important that refer to region on which you ar working on.

import boto3
elb_client=boto3.client('elb')

response = elb_client.create_load_blancer(
	LoadBalancerName='vybhava_lb',
	Listeners=[{'Protocol': 'HTTP', 'LoadBalancerPort':80, 'InstanceProtocol':'HTTP', 'InstancePort': 80},
	AvailabilityZone=['ap-south-1a']
	)
print(response)
print(response['DNSName'])

2. Apply security group on ELB using Boto3

object: client refers to elb method: apply_security_groups_to_load_balancer() here we can use existing security group(that is created during EC2 instance creation) can be associated with the ELB as well.

import boto3
elb_client=boto3.client('elb')
response=elb_client.apply_security_groups_to_load_balancer(
	LoadBalancerName='vybhava_lb',
	SecurityGroups='sg-vybhava_lb']
	)
print(response)

3. How to create Health check?

object: client refers to elb method: configure_health_check() without healthcheck there is not strong bonding between ELB and EC2 instances. This will validate to check weather the incoming request should be send to the instance or not based on the parameters here we provide.

import boto3
elb_client=boto3.client('elb')
health_check_resp=elb_client.configure_health_check(
	LoadBalancerName =  'vybhava_lb',
	HealthCheck={
		'Target':'TCP:22',
		'Interval':10,
		'Timeout':5,
		'UnhealthyThreshold':5,
		'HealthyThreshould': 5
	}
	
print(health_check_resp)	

The health_check_resp object output c should have 200. Http code for success.

4. How to register instance with ELB services?

EC2 Instance(s) will be attached to ELB object: client method: register_instances_with_load_balancer

import boto3
elb_client=boto3.client('elb')

instanceid=input('Please enter instance id to attach to ELB:')
attachinstance=elb_client.register_instances_with_load_balancer(
	LoadBalancerName='vybhava_lb',
	Instances=[	{
		'InstanceId': instanceid,
	}	]
)
print(attachinstance)

Observe that 'Instance Count' value changes according to attached number of instances. In the bottom tabs look for 'Instances'.

5. How to deregister instance from ELB using Boto3?

De-register EC2 instances from ELB, its reverse process what we did for registration.The object: client method: register_instances_with_load_balancer

import boto3
elb_client=boto3.client('elb')

deattachinstance=elb_client.deregister_instances_with_load_balancer(
LoadBalancerName='vybhava_lb',
	Instances=[	{
		'InstanceId': instanceid,
	}	]
)
print(deattachinstance)

6. How to delete ELB service using Boto3?

The removal of elb can be performed on elb_client, to delete ELB we need to use param for delete_loadbalancer is existing elb services name.

object: client refer to elb service

method: delete_loadbalancer()

import boto3
elb_client=boto3.client('elb')

response=elb_client.delete_loadbalancer(LoadBalancerName='vybhava_lb')
print(response)

Goto the aws console and confirm that no ELB in that region where we did all the above. 

 Refrences: https://unbiased-coder.com/boto3-load-balancer-guide/

AWS Manage S3 Buckets using Python Boto3

 In this post, I will show you 

how to create S3 bucket, 

how to put objects into the bucket, 

how to upload multiple objects in s3, 

how to download multiple objects, 

how to control access policy, and 

how to host a static website in S3. 

1. How to create S3 Bucket using Python3 Boto3?

Object : resource method: create_bucket 

important method  parameters: 

 ACL : private or public 

 Bucket - name of the bucket name this should be unique for each bucket 

 CreateBucketConfiguration - have the LocationConstrating that is region on which you want to host your s3 bucket.

import boto3
s3_resource = boto3.resource('s3')
bucket = s3_resource.create_bucket(ACL='private',
			Bucket='vybhava2023demo.com',
			CreateBucketConfiguration={
			'LocationConstrating': 'us-west-2'
			}) 
			
print("Successfully create bucket:", bucket)

2. How to put the objects into the S3 Bucket using Boto3

When you want to add file from local system we can use Python file methods and have a object reference here f is the reference to the file opened for read operation.

f=open('greet.txt').read()
s3_client =boto3.client('s3')
respose = s3_client.put_object(
	ACL='private',
	Body=f,
	Bucket='vybhava2023demo.com',
	Key='greet.txt'
	)

3. Delete object from S3 Bucket

Delete object from S3 bucket 

Object : client 

method : delete_object()

s3_client = boto3.client('s3')
response = s3_client.delete_object(
	Bucket='vybhava2023demo.com',
	Key='greet.txt'	
)

4. List all content objects in a Bucket

List all the contents of a Bucket this may be file objects those are put into the Bucket earlier.

Object: client

method: list_objects

s3_client = boto3.client('s3')
response = s3_client.list_objects(
	Bucket='vybhava2023demo.com'
	)

for content in response['Contents']:
	print(content['Key'])

5. List all S3 Buckets

Get the s3_client object 

method: list_buckets : 

On the aws-cli run the command: aws s3 ls

# File: list-s3.py 
# Description: This script will list all s3 buckets using client interface

s3_client = boto3.client('s3')
list_buckets=s3_client.list_buckets()
print(list_buckets)

# run 2
for b in list_buckets['Buckets']:
	print(b['Name'])

Hope you enjoyed this post!! Please write back your errors and exceptions when you run the Boto3 programs for AWS services and resources.

Manage AWS EC2 Instances using Python Boto3 script

Hey Welcome! back to Automations with Python for AWS!! 

Now IT market says AWS is the top number one Cloud Computing platform. That is why I've selected this AWS automations using Python Boto3.

In this post we will be exploring the AWS EC2 Instance related operations, and manage them in a reusable form.

• Create EC2 instance using Python Boto3
• Launch AWS EC2 instance using Python Boto3 script
• Stop AWS EC2 instance using Python Boto3 script
• Start AWS EC2 Instance using Python Boto3 script
• Terminate AWS EC2 instance using Python Boto3 script
• Fetching Public IP of given instance-id

How do you Create EC2 instance using Python3 Boto3? 

 Creating EC2 instance using Boto3 Python code

#=============================================
# File : create_ec2.py
# Description: Create EC2 instance by Boto3

import boto3
ec2 = boto3.resource('ec2')

instances = ec2.create_instances(
        ImageId="ami-0dafa01c8100180f8",
        MinCount=1,
        MaxCount=1,
        InstanceType="t2.micro",
        KeyName="KeyPair1"
    )
    

Launch instance

import boto3
ec2_client = boto3.client('ec2')

# This function will requires 
# image_id [Required] based on the region this will be changing
# instance_type [optional] default t2.micro type otherwise you can provide
# max [optional] default as 1, you can provide maximum number of instances
def launch_instance(image_id, instance_type='t2.micro', max=1):
	resource = ec2_client.run_instances(ImageId=image_id,
					InstanceType=instance_type,
					MinCount=1, MaxCount=max)
									
	for instance in resource['Instances']:
		print(instance['InstanceId'])

# main program
launch_instance('yourami-id')

To control the ec2 instances individual AWS cli commsnd  test Start, Stop

ec2.start_instances 

ec2.stop_instances

and terminate the ec2 instance
List AWS EC2 Instances using python boto3 script

import boto3
ec2_client = boto3.client('ec2')
resp = ec2_client.describe_instances()
for reservation in resp['Reservations']:
	for instance in reservation['Instances']:
		print("Running Instance Image ID: {} Running instance Instance Type: {} Running Instance Keyname {}".format(instance['InstanceId'],instance['InstanceType'],instance['KeyName']))

Python code with Menu driven program to manage EC2 instances

import boto3
import time

ec2 = boto3.resource('ec2')
ec2_client = boto3.client('ec2')

## Display all instances
def display_instances():
    for instance in ec2.instances.all():
        print (instance.id , instance.state)

## Stop instance by given instance id
def stopinstance():    
    instanceid=input("Please enter instanceid:")
    response = ec2_client.stop_instances(InstanceIds=[instanceid])
    time.sleep(90)
    display_instances()

## Terminate instance by given instance id
def terminateinstance():    
    instanceid=input("Please enter instanceid:")
    response = ec2_client.terminate_instances(InstanceIds=[instanceid])
    time.sleep(90)
    display_instances()
    
## Start instance by given instance id
def startinstance():    
    instanceid=input("Please enter instanceid:")
    response = ec2_client.start_instances(InstanceIds=[instanceid])
    time.sleep(90)
    display_instances()

## Main Program
def main():
    while True:
        menu_list=["Display All Instance","Stop Instance","Start Instance","Terminate Instance","Exit"]
        i=1
        for item in menu_list:
            print (i,item)
            i=i+1
        choice=int(input("Please enter Menu Choice:"))
        if choice==1:
            display_instances()
        elif choice==2:
            stopinstances()
        elif choice==3:
            startinstances()
        elif choice==4:
            terminateinstance()            
        else:
            exit()

if __name__ == '__main__':
    main()        
 

Enjoy the automations with Python Boto3 for AWS...

Python Automations using Boto3 for AWS

The objective of this post is for experimenting with AWS Boto3 automations and AWS Lambda, most of the realtime usecase on AWS cloud controlled and acceleration with them. We should know as SRE or DevOps Engineer how to refer to the Boto3 documentation where AWS team provided lot of details and examples of each Boto3 method, I'm pretty sure you could make great automations with this Boto3 module explore ideas.

Python for AWS Using Boto3

Let's jump on it...

Prerequisites

You must have AWS account [this can be your company provided or free-tire account].

Getting Started Python Automations using Boto3 

Step 1: Add User in IAM 

Let's get into the IAM adding user you can provide username as devops-admin

Select AWS Access TYPE {tic} programmatic access this allows to access key ID and secret access key for the AWS API, CLI SDK (Boto3 uses this).

Set permissions for 'devopsuser' Select policy type as 'AdministrationAccess' to access AWS resources and services [For this testing purpose only]. When you proceed for 'Create' user you will get the 'Add user' page you can download .csv file that contains - Access key and Secret access key[show/hide] store them in a text file as well.

Step 2: Download and install aws cli

[optional for to run from Laptop] Download 'aws cli' AWS command line on your laptop.

aws-cli is a unified tool to manage your AWS services. This can be used for automate multiple AWS services. From Mac/Linux you can install using following command:

pip install awscli

Select the Options as per your Laptop - Windows 32bit or 64bit installer, And install awscli on your laptop.

Note that Amazon Linux will have awscli already installed.

Step 3: Configuring aws cli 

This will work only when you have awscli installed, Run the following command:

aws configure 

Setting up aws-cli environment using 'aws configure' command

use the step1 outcome here Acess Key, Secret access key and then choose region as per your location generally in Projects we have to use company provided region. Choice of Output always JSON format[default] leave it to none for this experiment. 

This above set aws configuration will be used in Boto3 Python programs automatically once it set otherwise we can provide as parameters in the program.

Step 4: Python 3 installation

To install on RedHat family Linux(Rocky, CentOS, Suse) you can use the following commands

Check for latest Python installed

python -V

yum info python

Python package manager Pip is required, Let's do Pip installation

wget https://bootstrap.pypa.io/get-pip.py
  python get-pip.py

Boto3 on Ubuntu :

apt-get install Python3-pip3
pip3 install boto3

Verify pip version details

pip show

Finally, all set to go!! Let's do Boto3 installation 

pip install boto3

You could see in the installion summary it installs botocore s3transfer and boto3

You can verify on the same with the following :

pip3 show boto3