python Archives - IT - Engineering - Cloud

January 11, 2019February 12, 2019

AWS Lambda to auto start stop Ec2 instance on schedule using python and boto3

Use this lambda function to auto start stop all Ec2 instances based on schedule from tags.

#Auto Shutodown - Start EC2 instances based on tags import boto3 import os import json import croniter import datetime # Enter the region your instances are in. Include only the region without specifying Availability Zone; e.g., 'us-east-1' region = 'us-west-2'


EC2_STATUS_CODE_RUNNING = 16

EC2_STATUS_CODE_STOPPED = 80
def lambda_handler(event, context):

    ec2 = boto3.client('ec2', region_name=region)
    #auto_start_stop_tag = 'tc:uptime_schedule_gmt'

    auto_start_tag = 'tc:start_time_schedule_gmt_24h_format'

    auto_shutdown_tag = 'tc:shutdown_time_schedule_gmt_24h_format'
    instances_to_shutdown = []

    instances_to_start = []

    # Query ec2 machines for auto_start_stop_tag,

    #instances_with_schedules = get_instance_schedules(auto_start_stop_tag)

    instances_with_start_schedules = get_instance_schedules(auto_start_tag)

    instances_with_shutdown_schedules = get_instance_schedules(auto_shutdown_tag)

    print("instances_with_start_schedules: %s" % instances_with_start_schedules)

    print("instances_with_shutdown_schedules: %s" % instances_with_shutdown_schedules)
    for instance_id, values in instances_with_start_schedules.items():

        now = datetime.datetime.now()

        print("now: %s" % now)

        iterator = croniter.croniter(values['Schedule'], now)

        next_run_time = iterator.get_next(datetime.datetime)

        print("next_run_time: %s" % next_run_time)

        duration_until_next_run_time = next_run_time - now

        print("duration_until_next_start_time: %s" % duration_until_next_run_time)

        duration_of_one_hour = datetime.timedelta(hours=1)
        if duration_until_next_run_time <= duration_of_one_hour and values['State']['Code'] == EC2_STATUS_CODE_STOPPED:
            print("true")
            print("instance_to_stop.append(%s)" % instance_id)
            instances_to_start.append(instance_id)

      
    for instance_id, values in instances_with_shutdown_schedules.items():
        now = datetime.datetime.now()
        print("now: %s" % now)
        iterator = croniter.croniter(values['Schedule'], now)
        next_run_time = iterator.get_next(datetime.datetime)
        print("next_run_time: %s" % next_run_time)
        duration_until_next_run_time = next_run_time - now
        print("duration_until_next_shutdown_time: %s" % duration_until_next_run_time)
        duration_of_one_hour = datetime.timedelta(hours=1)
        if duration_until_next_run_time <= duration_of_one_hour and values['State']['Code'] == EC2_STATUS_CODE_RUNNING:
            print("instance_to_shutdown.append(%s)" % instance_id)
            instances_to_shutdown.append(instance_id)

    if len(instances_to_shutdown) > 0:

        ec2.stop_instances(InstanceIds=instances_to_shutdown)

        print('stopped your instances: ' + str(instances_to_shutdown))

        send_shutdown_notification(instances_to_shutdown, "STOPPED")
    if len(instances_to_start) > 0:

        ec2.start_instances(InstanceIds=instances_to_start)

        print('started your instances: ' + str(instances_to_start))

        send_start_notification(instances_to_start, "STARTED")
def send_shutdown_notification(instances, event):

    instances_json_object = {"instances":instances, "event":event}

    instances_json_string = json.dumps(instances_json_object)

    instances_json_bytes = instances_json_string.encode('utf-8')
    lambda_arn = os.environ['LAMBDA_NOTIFICATION_SHUTDOWN_ARN']

    lambda_client = boto3.client("lambda")

    lambda_client.invoke(

        FunctionName=lambda_arn,

        InvocationType='Event',

        LogType='None',

        Payload=instances_json_bytes

    )
def send_start_notification(instances, event):

    instances_json_object = {"instances":instances, "event":event}

    instances_json_string = json.dumps(instances_json_object)

    instances_json_bytes = instances_json_string.encode('utf-8')
    lambda_arn = os.environ['LAMBDA_NOTIFICATION_START_ARN']

    lambda_client = boto3.client("lambda")

    lambda_client.invoke(

        FunctionName=lambda_arn,

        InvocationType='Event',

        LogType='None',

        Payload=instances_json_bytes

    )
def get_instance_schedules(tag_name):

    # When passed a tag key, tag value this will return a list of InstanceIds that were found.
    ec2client = boto3.client('ec2')
    response = ec2client.describe_instances(

        Filters=[

            {

                'Name': 'tag-key',

                'Values': [tag_name]

            }

        ]

    )

    instancelist = {}

    for reservation in (response["Reservations"]):

        for instance in reservation["Instances"]:

            tag_value = ''

            for tag in instance['Tags']:

                if tag['Key'] == tag_name:

                    tag_value = tag['Value']

                    break

            instancelist[instance["InstanceId"]] = {'Schedule':tag_value,'State':instance['State']}

return instancelist

January 11, 2019June 9, 2020

Get all All AWS Ec2 snapshop reports with python and boto3

Use this python script to get all EC2 snapshot report in your AWS account.

import boto3 def lambda_handler(event, context): session = boto3.Session(profile_name='saml') ec2client = session.client('ec2') instances_with_volumes = get_instance_ids(ec2client, "tc:OpsAutomatorTaskList")

for instance_id, volume_ids in instances_with_volumes.items(): response = ec2client.describe_snapshots( Filters = [ {'Name':'volume-id', 'Values': volume_ids} #{'Name':'start-time', 'Values': ['2019-01-01']} ] #MaxResults = 10 ) #print(response['ResponseMetadata']) #print("instance_id:%s, number_of_snapshots:%s" % (instance_id, len(response['Snapshots']))) if len(response['Snapshots']) > 0: #print(response['Tags']) #Print Header print('InstanceId' + ' , ' + 'Description' + ' , ' + 'Encrypted' + ' , ' + 'OwnerId' + ' , ' + 'Progress' + ' , ' + 'SnapshotId' + ' , ' + 'StartTime' + ' , ' + 'State' + ' , ' + 'VolumeId' + ' , ' + 'VolumeSize') for snapshot in response['Snapshots']: print(instance_id + ' , ' + snapshot['Description'] + ' , ' + str(snapshot['Encrypted']) + ' , ' + str(snapshot['OwnerId']) + ' , ' + snapshot['Progress'] + ' , ' + snapshot['SnapshotId'] + ' , ' + str(snapshot['StartTime']) + ' , ' + snapshot['State'] + ' , ' + snapshot['VolumeId'] + ' , ' + str(snapshot['VolumeSize']) )


def get_instance_ids(ec2client, tag_name):

# When passed a tag key, tag value this will return a list of InstanceIds that were found.
response = ec2client.describe_instances(

Filters=[

{

'Name': 'tag-key',

'Values': [tag_name]

}

]

)

instancelist = {}

for reservation in (response["Reservations"]):

for instance in reservation["Instances"]:
ebs_volume_ids = []

for ebs in instance['BlockDeviceMappings']:

ebs_volume_ids.append(ebs['Ebs']['VolumeId'])
instancelist[instance["InstanceId"]] = ebs_volume_ids
return instancelist
lambda_handler('','')

January 9, 2019January 9, 2019

How to call a shell script from python code?

This is how to How to call a shell script from python code

import subprocess subprocess.call(['sh shell_script.sh arguments'])

or

import sys, os os.system("sh shell_script.sh arguments")

Output Example

January 8, 2019September 3, 2020

Quicksort Algorithm Implementation with Python

QuickSort is an O(nlogn) efficient sorting algorithm, serving as systematic method for placing elements of an array in order. Quicksort is a comparison sort, meaning that it can sort items of any type for which a “less-than” relation (formally, a total order) is defined. In efficient implementations it is not a stable sort, meaning that the relative order of equal sort items is not preserved. Quicksort can operate in-place on an array, requiring small additional amounts of memory to perform the sorting. It is very similar to selection sort, except that it does not always choose worst-case partition.
Source: https://en.wikipedia.org/wiki/Quicksort
Below are 2 versions of the Quicksort Algorithm Implementation with Python. The first version is easy, but use more memory. The second version use the memory very efficiently.

I- QuickSort Algorithm Implementation with Python (Memory intensive version)

QuickSort Algorithm Implementation with Python method1

Output:

II- Quicksort Algorithm Implementation with Python (with Memory Optimisation)

III- QuickSort Algorithm Implementation with Python for Both Methods with duration captured

Output

III- Most Efficient Quicksort implementation in Python on array of integer represented as string. Example: Array=[“1″,”237373737″,”3″,”1971771717171717″,”0”]

def QuickSort(array):

return sorted(array, key=lambda x: (len(x),x))

input:

6
31415926535897932384626433832795
1
3
10
3
5

Output:

1
3
3
5
10
31415926535897932384626433832795

Just to clarify that lambda part, in case someone else doesn't understand how exactly string comparison works: '2' > '1' is True, but '2' > '10' is also True, as well as '2' > '1000'. That's why the strings are sorted by length first, because len('2') < len('10').

IV- Build up a sorted array, one element at a time. Print the array after each iteration of the insertion sort, i.e., whenever the next element has been inserted at its correct position

python build up sorted array input output

January 7, 2019January 9, 2019

Binary Search Algorithm Implementation with Python

Binary Search is an algorithm that finds the position of a target in a sorted array. Binary search compares the target value to the middle element of the array. If they are not equal, the half in which the target cannot lie is eliminated and the search continues on the remaining half, again taking the middle element to compare to the target value, and repeating this until the target value is found. If the search ends with the remaining half being empty, the target is not in the array. Even though the idea is simple, implementing binary search correctly requires attention to some subtleties about its exit conditions and midpoint calculation, particularly if the values in the array are not all of the whole numbers in the range.

Source: https://en.wikipedia.org/wiki/Binary_search_algorithm

Below the binary search algorithm implementation with python:

#Test_Binary_Search.py Array=[-2,1,0,4,7,8,10,13,16,17, 21,30,45,100,150,160,191,200] Target=201 n=len(Array) def binarySearch(A,T): L=0 R=n-1 while (L <= R): m=int((L + R)/2) if ( A[m] < T ): L = m +1 elif (A[m] > T): R = m-1 else: print("Target is at %s:" % m) return m print("Target not found") return "unsuccessful"

binarySearch(Array, Target)

Binary Search Algorithm implementation with python

July 21, 2016July 22, 2016

Script with hash tables on windows and Linux

How to declare and write a script with hash tables on windows and linux

Hash tables with powershell on windows

Declaration:
$states=@{“Alberta” = “Calgary”; “British Columbia” = “Vancouver”; “Ontario” = “Toronto” ; “Quebec” = “Montreal”}

Name
_____ Value
_______

Alberta Calgary

British Columbia Vancouver

Ontario Toronto

Quebec Montreal

Add new key-value in hashtable:
$states.Add(“Manitoba”,”Winnipeg”)

Remove key-value in hashtable:
$states.Remove(“Manitoba”,”Winnipeg”)
Change value in hashtable:
$states.Set_Item(“Ontario”,”Ottawa”)
Retrieve value in hashtable:
$states.Get_Item(“Alberta”)
Find key in hashtable:
$states.ContainsKey(“Alberta”)
Find Value in hashtable:
$states.ContainsValue(“Calgary”)
Count items in hashtable:
$states.Count
Sort items by Name in hashtable:
$states.GetEnumerator() | Sort-Object Name -descending
Sort items by Value in hashtable:
$states.GetEnumerator() | Sort-Object Value -descending
Hash tables with perl on linux or windows

Declaration:
my %hash = (); #Initialize a hash
my $hash_ref = {}; # Initialize a hash reference. ref will return HASH
Clear (or empty) a hash
for (keys %hash)
{
delete $hash{$_};
}
Clear (or empty) a hash reference
for (keys %$href)
{
delete $href->{$_};
}
Add a key/value pair to a hash
$hash{ ‘key’ } = ‘value’; # hash
$hash{ $key } = $value; # hash, using variables
Using Hash Reference
$href->{ ‘key’ } = ‘value’; # hash ref
$href->{ $key } = $value; # hash ref, using variables
Add several key/value pairs to a hash
%hash = ( ‘key1’, ‘value1’, ‘key2’, ‘value2’, ‘key3’, ‘value3’ );
%hash = (
key1 => ‘value1’,
key2 => ‘value2’,
key3 => ‘value3’,
);

Copy a hash
my %hash_copy = %hash; # copy a hash
my $href_copy = $href; # copy a hash ref
Delete a single key/value pair
delete $hash{$key};
delete $hash_ref->{$key};

Hash tables with python on linux or windows

Hash tables are called dictionary in python.
Declaration:
dict = {‘Name’: ‘Zara’, ‘Age’: 7, ‘Class’: ‘First’}
Accessing Values
print “dict[‘Name’]: “, dict[‘Name’]
print “dict[‘Age’]: “, dict[‘Age’]
Output:
dict[‘Name’]: Zara
dict[‘Age’]: 7
Updating Dictionary
dict = {‘Name’: ‘Zara’, ‘Age’: 7, ‘Class’: ‘First’}

dict[‘Age’] = 8; # update existing entry
dict[‘School’] = “DPS School”; # Add new entry
Delete Dictionary Elements
#!/usr/bin/python

dict = {‘Name’: ‘Zara’, ‘Age’: 7, ‘Class’: ‘First’}

del dict[‘Name’]; # remove entry with key ‘Name’
dict.clear(); # remove all entries in dict
del dict ; # delete entire dictionary

Source:

https://technet.microsoft.com/en-us/library/ee692803.aspx
http://www.cs.mcgill.ca/~abatko/computers/programming/perl/howto/hash/
http://www.tutorialspoint.com/python/python_dictionary.htm

September 8, 2015January 22, 2019

reverse a string on Linux and Windows

How to reverse a string on Linux and Windows

Via shell script on Linux

sh-3.2# vi reverse.sh
#### Start Script #####
#!/bin/bash
input_string=”$1″
reverse_string=””

input_string_length=${#input_string}
for (( i=$input_string_length-1; i>=0; i– ))
do
reverse_string=”$reverse_string${input_string:$i:1}”
done

echo “$reverse_string”
##### End Script #####

Let’s run it:

sh-3.2# chmod 775 reverse.sh
sh-3.2# ./reverse.sh Etienne
enneitE

Via powershell script on Windows

#Let’s use the script reverse.ps1 below.
######
$string=”Etienne”
$string_array=$string -split “”
[array]::Reverse($string_array)
$string_array -join ”

#####Output#####
PS C:\Users\etienne_noumen\Documents\Etienne\Scripting> .\reverse.ps1

E t i e n n e

enneitE

Via powershell script on Windows in one line

([regex]::Matches($String,’.’,’RightToLeft’) | ForEach {$_.value}) -join ”

Via batch script on Windows

::Note: ReverseStr also calls StrLen
::and string length is not greater than 80 chars
:: but can be changed.

@echo off
SetLocal EnableDelayedExpansion
cls
set Str=Etienne
call :StrLen %Str%
echo Length=%Len%
call :ReverseStr %Str%
echo String=%Str%
echo Reverse Str=%Reverse%
exit /b

::—————-
:: Calc Var Length
::—————-
:: %*=Str to Check
:: Returns %Len%
:: —————
:StrLen %*
set Data=%*
for /L %%a in (0,1,80) do (
set Char=!Data:~%%a,1!
if not “!Char!”==”” (
set /a Len=%%a+1
) else (exit /b)
)
exit /b

::—————
:: Reverse String
::—————
:: %* Str to Reverse
:: Returns %Reverse%
::——————
:ReverseStr %*
set Data=%*
call :StrLen %Data%
for /L %%a in (!Len!,-1,0) do (
set Char=!Data:~%%a,1!
set Reverse=!Reverse!!Char!
)
exit /b

Via perl script on Windows or Linux

Via python script on Windows or Linux

def reverse_string(a_string)
return a_string[::-1]
reverse_string(“etienne”) returns “enneite”
Source:

http://www.computing.net/answers/programming/reverse-a-string-in-dos/26004.html

July 28, 2015July 19, 2016

prompt and read input variables from keyboard

Let’s find how to prompt and read input variables from keyboard while executing a script using shell, perl, python, batch and powershell (windows and Linux)

On Linux via shell

read -p “Enter your name: ” name
echo “Hi, $name. Let’s be friend!”

On Windows via powershell

$name=read-host “Enter your name:”
write-host “Hi $name, Let’s be friend!”
On Windows via batch

Set /p Name=”Enter your name:”
echo “Hi %name%, Let’s be friend!”
On Windows or Linux via perl

print “Enter your name “;
my $name = ;
chomp $name; # Get rid of newline character at the end
print “Hello $name, let’s be friend”;
On Windows or Linux via python

name=input(“Enter your name: “)
print (“Hello ” + name + ” let’s be friend”)

Name _____	Value _______
Alberta	Calgary
British Columbia	Vancouver
Ontario	Toronto
Quebec	Montreal

This is how to How to call a shell script from python code

How to declare and write a script with hash tables on windows and linux

Hash tables with powershell on windows

Hash tables with perl on linux or windows

Hash tables with python on linux or windows

Source:

How to reverse a string on Linux and Windows

Via shell script on Linux

Via powershell script on Windows

Via powershell script on Windows in one line

Via batch script on Windows

Via perl script on Windows or Linux

Via python script on Windows or Linux

Let’s find how to prompt and read input variables from keyboard while executing a script using shell, perl, python, batch and powershell (windows and Linux)

On Linux via shell

On Windows via powershell

On Windows via batch

On Windows or Linux via perl

On Windows or Linux via python