Init Script Alternative

Install upstart package from repo, avaiable in both debian and centos.

To create a job to be started automatically when Ubuntu starts. As written example, suppose create the following file /etc/init/testservice.conf with sudo:

# testservice - test service job file

description "my service description"
author "Me <myself@i.com>"

# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas

# When to start the service
start on runlevel [2345]

# When to stop the service
stop on runlevel [016]

# Automatically restart process if crashed
respawn

# Essentially lets upstart know the process will detach itself to the background
# This option does not seem to be of great importance, so it does not need to be set.
#expect fork

# Specify working directory
chdir /home/user/testcode

# Specify the process/command to start, e.g.
exec python mycommand.py arg1 arg2

To 'manually' start or stop the process use

sudo start testservice
sudo stop testservice

The above example works great for everything , but however, there is short cut too :

The solution is to use a shutdown-hook for the system. In Ubuntu it is very easy to add a shutdown-hook by using upstart.

The upstart job config file (/etc/init/shutdown-hook.conf) should look something like this:

description "run at shutdown"

start on starting rc
task
exec /bin/bash /etc/my_service/upload_log_to_s3.sh log

Finally I just needed the script (/etc/my_service/upload_log_to_s3.sh) for uploading a log file to S3:

#!/bin/bash

log_file_ext=$1

gzip -c /var/log/my_service/*.$log_file_ext > /tmp/log.gz

aws s3 cp /tmp/log.gz s3://my_service_bucket/logs/`date +%Y-%m-%dT%H:%M:%SZ`.log.gz

The upload script will just gzip the log file (needed as I’m using delaycompress), rename the log file to the current timestamp, and upload the file using aws-cli. The argument sets the file extension of the log file, which is necessary to be able to upload both the current (.log) as well as the previous log file (.log.1).

Post HTTP API grafana Query

The details are as follows:

URL : http://www.nasheikh.com:3000/api/datasources?=

Option : Post

Headers :  Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

Payload : 

{

        "name":"test3",

        "type":"influxdb_08",

        "url":"http://localhost:8086",

        "access":"proxy",

        "basicAuth":false,

        "database":"server1",

        "user":"root",

        "password":"root"

}


Do this in one liner in curl :

curl -H "Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @datainject.json http://www.nasheikh.com:3000/api/datasources?=

Here datainject.json is the file which is pasted above just saved as a json file and passed in curl.

Watch a file for its content, and perform an action accordingly:

Parent Script

#!/bin/bash

while true; do
  change=$(inotifywait -e close_write,moved_to,create .)
  change=${change#./ * }
  if [ "$change" = "changingfile.txt" ]; then ./child.sh; fi
done

Child Script

#!/bin/bash

rm -fr middle*

count=0

eraser=0

while read server

do

let count=$count+1

sed 's/server1/'${server}'/g;s/Tracker/'${server}'/g' center > middle.$count
sed 's/test1/'${server}'/g' datainject.json > newdata.$count.json
curl -H "Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @newdata.$count.json http://www.nasheikh.com:3000/api/datasources?=
sleep 2

done < network.txt

eraser=$(echo middle.$count)

sed -i '$ s/.$//' $eraser

sleep 2

cat prepend middle* final > merged.json

curl -H "Authorization: Bearer eyJrIjoiNzNjc2s5amxkZW9aUE81UXV3dVVOcmRHNTJhRkVOY1oiLCJuIjoiYWRtaW4iLCJpZCI6MX0=" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @merged.json http://nasheikh.com:3000/api/dashboards/db

rm -fr middle* merged.json newdata*

========================================================================

Update Server List from Amazon VPC

#!/bin/bash

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicIpAddress]' --output text --profile nash >> mainusa.txt

while true

do

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicIpAddress]' --output text --profile nash >> freshusa.txt

diff -Naur mainusa.txt freshusa.txt > my.patch

check=$(cat my.patch | wc -l)

echo $check

if [ "$check" -gt 0 ]

then

patch mainusa.txt < my.patch

else

echo "no chnage"

fi

rm -fr freshusa.txt my.patch

sleep 30

done

Random Stuff 2

Calculate Response time of web page

#!/bin/sh

while true
do
script -c "time -p wget http://52.20.204.148/ -q --output-document=/dev/null 2>/tmp/tmp.log" -f /tmp/tmp.log > /dev/null
sleep 5
NUM=$(cat /tmp/tmp.log | grep "real" | cut -d " " -f2 | tr -cd '[:digit:].')
curl -X POST -d '[{"name":"foo","columns":["val"],"points":[['$NUM']]}]' "http://52.20.204.148:8086/db/server1/series?u=root&p=root"
done

========================================================================

Type of web server response per minute

#!/bin/bash

while true
do
#set -x
mkdir -p /home/ec2-user/tmp/percfile
rm -f /home/ec2-user/tmp/percfile/*
dd=`date +%d`
mon=`date +%m`
yea=`date +%Y`
for (( i = 1; i >0; i-- ))
do
grep $(date +%d/%b/%Y:%H:%M -d "-$i  min") /var/log/httpd/access_log >> /home/ec2-user/tmp/percfile/log1mins
GOOD=$(grep "200" /home/ec2-user/tmp/percfile/log1mins | wc -l)
BAD=$(grep -v "200" /home/ec2-user/tmp/percfile/log1mins | wc -l)
curl -X POST -d '[{"name":"response","columns":["good","bad"],"points":[['$GOOD','$BAD']]}]' "http://localhost:8086/db/server1/series?u=root&p=root"
done
sleep 1m
done

========================================================================

No of hits to a web server per second

#!/bin/sh

frequency=1

lastCount=`wc -l /var/log/httpd/access_log | sed 's/\([0-9]*\).*/\1/'`

while true

do

newCount=`wc -l /var/log/httpd/access_log | sed 's/\([0-9]*\).*/\1/'`

diff=$(( newCount - lastCount ))

rate=$(echo "$diff / $frequency" |bc -l)

#echo $rate

curl -X POST -d '[{"name":"hps","columns":["val"],"points":[['$rate']]}]' "http://localhost:8086/db/server1/series?u=root&p=root"

lastCount=$newCount

sleep $frequency

done

========================================================================

Gets your public IP

#!/bin/bash

while true

do

sleep 1

myip=$(curl wgetip.com 2> /dev/null) > /dev/null

sleep 1

curl -X POST -d '[{"name":"response","columns":["val1"],"points":[["'$myip'"]]}]' "http://$myip:8086/db/server1/series?u=root&p=root"

sleep 1

done

========================================================================

Service status in boolean

#!/bin/bash

while true

do

sleep 1

pidof  httpd >/dev/null

if [[ $? -eq 0 ]] ; then

curl -X POST -d '[{"name":"response","columns":["val2"],"points":[[0]]}]' "http://localhost:8086/db/server1/series?u=root&p=root"

else

curl -X POST -d '[{"name":"response","columns":["val2"],"points":[[1]]}]' "http://localhost:8086/db/server1/series?u=root&p=root"

sleep 1

fi

done

========================================================================

Type of web server response per minute

#!/bin/sh

#while true

#do

NOW=$( tail -n1 /var/log/httpd/access_log | head -1 | awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' )

REF=$(( $NOW - 60 ))

NUM=$REF

LINE=1

COUNT=0

BAD=0

while [ $NUM -ge $REF ]

do

NUM=$( tail -n$LINE /var/log/httpd/access_log | head -1 | awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' )

tail -n$LINE /var/log/httpd/access_log | head -1 | grep -i "200" >/dev/null && let COUNT=$COUNT+1

tail -n$LINE /var/log/httpd/access_log | head -1 | grep -v "200" >/dev/null && let BAD=$BAD+1

let LINE=$LINE+1

done

let COUNT=$COUNT-1

let BAD=BAD-1

curl -X POST -d '[{"name":"response","columns":["good","bad"],"points":[['$COUNT','$BAD']]}]' "http://localhost:8086/db/server1/series?u=root&p=root"

#sleep 15m

#done

========================================================================

Sql access through php

<?php

$servername = "localhost";

$username = "root";

$password = "qwe";

$dbname = "grafana";

$homepage = file_get_contents("/home/comb.txt");

// Create connection

$conn = mysqli_connect($servername, $username, $password, $dbname);

// Check connection

if (!$conn) {

    die("Connection failed: " . mysqli_connect_error());

}

$sql = "UPDATE dashboard SET data='$homepage' WHERE slug='dash2'";

if (mysqli_query($conn, $sql)) {

    echo "Record updated successfully";

} else {

    echo "Error updating record: " . mysqli_error($conn);

}

mysqli_close($conn);

?>

========================================================================

Run local script on remote server ( with root priviliges )

Parent Script :

#!/bin/bash
SOURCEFILE=$1
SOURCEIP=$2
TARGETIP=$3
FILE=$(basename $1)
echo "Enter the path"
read path
scp -3 -r -i /root/asw/Admaxim-23-06-2015.pem ec2-user@$SOURCEIP:$SOURCEFILE ec2-user@$TARGETIP:/tmp    
ssh -t -t -i /root/asw/Admaxim-23-06-2015.pem ec2-user@$TARGETIP sudo -i "bash -s" -- < ./remold.sh "$FILE" "$path"
exit

Child Script :

#!/bin/bash

cd /tmp

yes | cp -r $1 $2

rm -fr $1

exit

exit

========================================================================

SSH into a remote server with non root account with login key and go root automatically :

#!/bin/bash
ssh -t -t -i /root/asw/keyfile.pem user@$1 sudo -i "bash -s"

========================================================================

Pass defined column value to loop  :

#!/bin/bash

while read server
do
echo ${server}
done < final | awk '{print $1}'

========================================================================

DB creation :

curl -X POST "http://localhost:8086/db?u=root&p=root" -d '{"name": "'${server}'"}'

========================================================================

Random Stuff

This is the bash script

#!/bin/bash

NOW=$( tail -n1 log1 | awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' ) ## get last line timestamp
REF=$(( $NOW - 15*60 ))  ##previous 15 minute
NUM=$REF
LINE=1
COUNT=0
while [ $NUM -ge $REF ] ## Run until the condition is not met
do
NUM=$( tail -n$LINE log1 | head -1 | awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' ) ## pass value of each time stamp and compare with the while condition
tail -n$LINE log1 | head -1 | grep -i "ready" && let COUNT=$COUNT+1  ## count the number of ready occurrence
let LINE=$LINE+1 ## for backward iteration
done
echo "Total Error = "$COUNT" " ## display count

The log file used is "log1"

10.183.253.51 - - [18/Mar/2013:22:25:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:26:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:27:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:28:00 -0700] GET /adserver/ready HTTP/1.1 200 1 0
10.183.253.51 - - [18/Mar/2013:22:29:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:30:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:31:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:32:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:33:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:34:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:35:00 -0700] GET /adserver/ready HTTP/1.1 200 1 0
10.183.253.51 - - [18/Mar/2013:22:36:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:37:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:38:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:39:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:40:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:41:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:42:00 -0700] GET /adserver/ready HTTP/1.1 200 1 0
10.183.253.51 - - [18/Mar/2013:22:43:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:44:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:45:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:46:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:47:00 -0700] GET /adserver/ready HTTP/1.1 200 1 0
10.183.253.51 - - [18/Mar/2013:22:48:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:49:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:50:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:51:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:52:00 -0700] GET /adserver/ready HTTP/1.1 200 1 0
10.183.253.51 - - [18/Mar/2013:22:53:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:54:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1
10.183.253.51 - - [18/Mar/2013:22:55:00 -0700] POST /adserver/getOpenRTBSmaatoInfo HTTP/1.1 204 - 1

x-----------------------------------------------------------------------------------------------------------------------x

This is the bash script

#!/bin/bash

NOW=$( tail -n1 log2 | awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' ) ## take first time instance ( user specified can also be parsed )
REF=$(( $NOW - 15*60 ))  ##previous 15 minute PN. second time instance ( user specified can also be parsed )
NUM=$REF
LINE=1
while [ $NUM -ge $REF ]
do
NUM=$( tail -n$LINE log2 | head -1 | awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' )
tail -n$LINE log2 | head -1 | grep -v "click" | cut -d "=" -f4 | cut -d " " -f1 >> number.txt    ## Values retrieved from the log are temporarily stored here
SUM=$(awk '{ SUM += $1} END { printf "%.2f",  SUM }' number.txt) ## values are retireved and added up gradually step by step
let LINE=$LINE+1 ## backward iteration for faster processing
done
rm -fr number.txt ## external temporary file created has been removed
echo "The Sum of Bit is "$SUM"" ##  print the sum

The log file used is "log2"

10.183.252.22 - - [16/Mar/2013:00:00:06 -0700] GET /adtracker/track/track/imp?id=11062&cpnsite_id=23709&bid=0.3 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:01:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:02:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:03:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:04:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:05:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:06:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:07:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:08:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:09:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:10:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:11:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:12:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:13:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:14:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:15:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:16:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:17:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:18:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:19:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:20:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:21:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:22:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:23:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:24:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:26:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:27:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:28:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:29:06 -0700] GET /adtracker/track/track/imp?id=11390&cpnsite_id=23609&bid=0.2 HTTP/1.1 3
10.183.252.22 - - [16/Mar/2013:00:30:06 -0700] GET /adtracker/track/track/click?id=11062 HTTP/1.1 3

x-----------------------------------------------------------------------------------------------------------------------x

Just enter the below command at the shell

# cat log2 | grep -v "click" | cut -d "?" -f2 | cut -d "&" -f1

The log file used is "log2"

All ready given above.

x-----------------------------------------------------------------------------------------------------------------------x

To increase the file descriptor limit, edit the kernel parameter file /etc/sysctl.conf.

# nano /etc/sysctl.conf

fs.file-max = 600000

Apply the changes without a reboot:

# sysctl -p

To check the new limit :

# more /proc/sys/fs/file-max

x-----------------------------------------------------------------------------------------------------------------------x

Sync Time between servers

Option 1

# date --set="$(ssh user@server date)"

Or

# ssh user@host "date --set \"$(date)\""

manually on all three servers

Option 2

‘ntpdate 192.168.1.1’ to be done on each server

or add

server 192.168.1.1 prefer to /etc/ntp.conf on each server

although we will require a self hosted NTP server.

Option 3

Set all 3 server timezone to same timezone and use ntpdate pool.ntp.org to get them adjusted to same time.
Offset of the servers with each other can be checked by running command ntpq -p and by adding each servers ip's between them.
The offset is usually in milli-seconds and that too because of latency.

x-----------------------------------------------------------------------------------------------------------------------x

Change Timezone of a machine

# cp /etc/localtime /root/old.timezone
# rm /etc/localtime
# ln -s /usr/share/zoneinfo/America/New_York /etc/localtime
# nano /etc/sysconfig/clock  /* to make the change permanent */

ZONE=”America/New_York”  /* change value according to your need */

reboot the machine and then :

# date

or

# ntpdate pool.ntp.org     /* NTP service should be installed and running */

x-----------------------------------------------------------------------------------------------------------------------x

We are assuming here that all 5 servers are already added in known host and id_rsa.pib file has been transferred from sender to each receiving server and added in ~/.ssh/authorized_keys. Also root user is allowed login for all servers.

This is the bash script

#/usr/bin/bash

# There are three variables accepted via commandline
# $1 = first parameter (/source_path/source_filename)
# $2 = second parameter (/target_directory/)
# $3 = third paramter (file that contains list of hosts)

SOURCEFILE=$1
TARGETDIR=$2
HOSTFILE=$3

if [ -f $SOURCEFILE ] ## check for file existence
then
   printf "File found, preparing to transfer\n"
   while read server  ## initialize array
   do
      scp -vp $SOURCEFILE ${server}:$TARGETDIR  ## do transfer accordingly
   done < $HOSTFILE ## list of ip's which are stored in an external file
else
   printf "File \"$SOURCEFILE\" not found\n"
   exit 0
fi
exit 0

Command at prompt # ./scp.sh /file/to/be/transferred /destination/of/file/to/be/ /directory/of/ip/list/network.txt

The file used is "network.txt" for IP list

192.168.119.135
192.168.119.136
192.168.119.137
192.168.119.138
192.168.119.139

x-----------------------------------------------------------------------------------------------------------------------x

AWS Spot Pricing

This is done in 2 scripts :

./spothead.sh

#!/bin/bash

n=$(date "+%H:%M:%S")
while read server
do
#var=$(. ./spotprice.sh "${server}")
#echo "AFTER: $var"
./spotprice.sh "${server}"
done < types.txt
echo ""
printf "PN: -  TOP most value shows the SMOOTHEST graph  -\n    -   Rest of the values follows accordingly   -\n    -        LAST value represent WORST!         -"
echo ""
echo ""
echo "-------------------------------------------------------"
printf "|   Gausian Curve    |  Current Price  |    Server    | \n"
printf "| (top value = best) |  "%s\ at" "$n"   |    (Type)    | \n"
echo "-------------------------------------------------------"
cat merged.txt | sort -n
echo ""
rm -fr merged.txt

========================================================================

./spotprice.sh

#!/bin/bash

NOW=$(date "+%Y-%m-%dT%H:%M:%S")
REF=$(date -d "-1 days" +"%Y-%m-%dT%H:%M:%S")
aws ec2 describe-spot-price-history --instance-types $1 --availability-zone us-east-1e --product-description "Linux/UNIX (Amazon VPC)" --start-time $REF --end-time $NOW --query 'SpotPriceHistory[*].[SpotPrice]' --output text >> number.txt
SUM=$(awk '{delta = $1 - avg; avg += delta / NR; mean2 += delta * ($1 - avg); } END { printf "%0.7f \n" , sqrt(mean2 / NR); }' number.txt)
CP=$(tail -1 number.txt)
echo "      $SUM          $CP        $1" >> merged.txt
rm -fr number.txt

That will get the job done!!

Access Log Manipulation

#!/bin/bash

tail -n1 /var/log/httpd/access_log | awk '{print $4}' | while read line
do
in=${line}
#echo "Input: $in"
rfc_form="${in:1:2}-${in:4:3}-${in:8:4} UTC ${in:13:2}:${in:16:2}:${in:19:2}"
#echo "Converted: $rfc_form"
epoch_time=$(date -d "$rfc_form" +%s)
echo "Only Seconds: $epoch_time"
#date -d "$rfc_form"
date -d @$epoch_time
done

One more technique:

echo "18-06:25:34 " | awk -F: '{ print ($1*24*3600) + ($2*3600) + ($3*60) + $4 }'

========================================================================

Take this log file :

[
    [
        "52.4.30.109"
    ], 
    [
        "54.210.143.237"
    ], 
    [], 
    [], 
    [
        "54.210.164.230"
    ], 
    [], 
    [], 
    [
        "54.210.247.55"
    ], 
  
    [
        "54.210.162.121"
    ], 
    [
        "54.210.145.45"
    ], 
    [], 
    [
        "107.23.2.112"
    ], 
    [], 
    [], 
    [
        "52.7.146.177"
    ], 
    [
        "54.210.222.151"
    ], 
    [
        "54.210.170.90"
    ], 
    [], 
    [
        "54.210.163.107"
    ], 
    [
        "54.210.171.244"
    ], 
    [
        "52.7.33.54"
    ], 
    [
        "54.210.192.121"
    ], 
    []
]

Apply this :

cat new.txt | awk '{$1=$1}{ print }' | sed 's/[][]//g;s/,//g;s/"//g' | sed '/^$/d'

Expect Magic !!

Full Script:

#!/bin/bash

rm -fr new*
rm -fr fresh*

aws ec2 --region eu-central-1 describe-instances --query 'Reservations[*].Instances[*].PublicIpAddress' >> neweu.txt
aws ec2 --region us-east-1 describe-instances --query 'Reservations[*].Instances[*].PublicIpAddress' >> newusa.txt

cat neweu.txt | awk '{$1=$1}{ print }' | sed 's/[][]//g;s/,//g;s/"//g' | sed '/^$/d' >> fresheu.txt
cat newusa.txt | awk '{$1=$1}{ print }' | sed 's/[][]//g;s/,//g;s/"//g' | sed '/^$/d' >> freshusa.txt

counteu=$(cat fresheu.txt | wc -l)
countusa=$(cat freshusa.txt | wc -l)

echo "EU count is $counteu"
echo "USA count is $countusa"

rm -fr new*
rm -fr fresh*

OR do this simply:

aws ec2 --region eu-central-1 describe-instances --query 'Reservations[*].Instances[*].[PublicIpAddress, Tags[0].Value]' --output text | sort -k 2

aws ec2 --region us-west-1 describe-instances --query 'Reservations[*].Instances[*].[PublicIpAddress, Tags[0].Value]' --output text | sort -k 2