Pooling JSON data into MySql Table sequentially!

We have JSON data :

[{
"timeMillis": 1452502968583,
"latitude": -33.9041,
"longitude": 170.7297,
"current": 15.7,
"direction": "GROUND"
}, {
"timeMillis": 1452506410228,
"latitude": -33.6113,
"longitude": 172.1532,
"current": 8.1,
"direction": "GROUND"
}, {
"timeMillis": 1452513792306,
"latitude": -48.9861,
"longitude": 169.7887,
"current": -38.5,
"direction": "GROUND"
}]

save it as file.json

Now use this :

<?php
    $con = mysqli_connect("localhost","root","","lightning");

    // Check connection
    if (mysqli_connect_errno())
      {
      echo "Failed to connect to MySQL: " . mysqli_connect_error();
      }

    $jdata = file_get_contents("/home/file.json");
    
    $data = json_decode($jdata, true);
    foreach($data as $mydata)
    {
        $sql = "INSERT INTO data(timeMillis, latitude, longitude, current) VALUES (".$mydata['timeMillis'].",".$mydata['latitude'].",".$mydata['longitude'].",".$mydata['current'].")";   
        if(!mysqli_query($con, $sql))
        {
            die('Error : ' . mysql_error());
        }
    }

    mysqli_close($con);
    //echo "Le bhai, ho gaya tera data store!"         
?>

Job Done!

KickStarting and Linux WSUS Techinque

Make a Webserver:

# yum -y install httpd

Create a folder 

# mkdir  -p /var/www/html/install

Then mount the cd

# mount /dev/cdrom /media

Copy the data

# cp -R /media/* /var/www/html/install

Verify it 

# diff -q -r /media /var/www/html/install

There are may ways to create a kickstart file for example use a gui tool :

# yum install system-config-kickstart

Or use one like this:

#Kickstart file automatically generated by anaconda.

#version=DEVEL

install                                 # instead of "upgrade"

text            # you're not going to be standing there watching it, are you?

url --url http://10.10.10.1/pub/        # use this instead of CD-ROM Option

bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

zerombr yes

clearpart --all --initlabel

part / --fstype ext4 --size 4096 --grow

part swap --recommended

selinux --disabled

lang en_US.UTF-8

keyboard us

network --onboot yes --device eth0 --bootproto dhcp --noipv6

rootpw  --iscrypted $6$Y3aFxc5JNbESX3hb$

firewall --service=ssh

authconfig --enableshadow --passalgo=sha512

timezone --utc Pacific/Auckland

repo --name="CentOS"  --baseurl=http://10.10.10.1/pub/ --cost=100    #replace the base url from #cdrom or any other to suit yours

reboot

%packages --nobase

@core

%post 

cd /tmp 

wget http://10.10.10.1/post-install/myrepo.sh 

chmod +x local-repo.sh 

./repo.sh

%end

Save this file as ks.cfg into /var/www/html/install

Now we create our own repo:

# mkdir -p /var/www/html/reposit

# yum install createrepo

# createrepo   /var/www/html/reposit

# rsync -avz rsync://centos.ar.host-engine.com/6.6/os/x86_64/ /var/www/html/reposit

# createrepo --update /var/www/html/reposit

Once this is all done, we make a script 

# mkdir -p /var/www/html/post-install

# cd /var/www/html/post-install

# nano myrepo.sh

cd /etc/yum.repos.d

rm *.repo  

wget http://10.10.10.1/post-install/myrepo.repo

yum clean all

yum -y update

Save it and create a local repo, also in the same folder like this:

# nano myrepo.repo

[myrepo]

name=myrepo

baseurl=http://10.10.10.1/reposit/

gpgcheck=1

gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

Save it.

Now boot up a client and when the installer screen comes in press tab and enter this all in one line:

vmlinuz initrd=initrd.img ks=http://10.10.10.1/install/ks.cfg append ip=10.10.10.100 netmask=255.255.255.0 ksdevice=eth0

Job Done!

Getting a DESKTOP on centos!

Do a minimal install of centos 6.5 and start doing this:

# yum -y groupinstall "Desktop" "Desktop Platform" "X Window System" "Fonts" "General Purpose Desktop"

# init 5

To have CentOS boot into runlevel 5 “X11” instead of runlevel 3 “Full multiuser mode”, modify the /etc/inittab file to change start up level like this :

# nano /etc/inittab

id:3:initdefault:

to

id:5:initdefault:

Job Done!

AWS Auto Scaling - Complete CLI Solution

#! /bin/bash

function ASG {
read -e -p "Enter Min no. of instances(Integer Value)...: " min
echo ""
read -e -p "Enter Max no. of instances(Integer Value)...: " max
echo ""
read -e -p "Enter Desired no. of instances(Integer Value)...: " desired
echo ""
aws autoscaling create-auto-scaling-group --auto-scaling-group-name $Cluster-$COMPONENT-ASG --launch-configuration-name RTB-$COMPONENT-ASG --min-size $min --max-size $max --desired-capacity $desired --availability-zones us-east-1e --health-check-type ELB --health-check-grace-period 240 --default-cooldown 300 --load-balancer-names "RTB" --vpc-zone-identifier subnet-fea1f0d5 ; check;
}

function config {
echo ""
aws autoscaling create-launch-configuration --launch-configuration-name $Cluster-$COMPONENT-ASG --image-id ami-b7f68bd2 --instance-type $COMPONENT --spot-price "0.27" --no-associate-public-ip-address  --security-groups sg-665df302 --user-data $File --key-name rtbservers --instance-monitoring Enabled=false --no-ebs-optimized --block-device-mappings "[{\"DeviceName\":\"/dev/xvda\",\"Ebs\":{\"SnapshotId\":\"snap-68dc2c09\",\"VolumeSize\":10,\"VolumeType\": \"gp2\",\"DeleteOnTermination\": true}},{\"DeviceName\": \"/dev/sdb\",\"VirtualName\":\"ephemeral0\"}]" ; check && ASG;
}

function check {
if [ $? -eq 0 ]
then
echo "Command Executed Successfully"
echo ""
else
echo "Error, Please try again!"
echo ""
exit 1
fi
}

function type {
echo ""
echo "Please select the desired Instance type"
echo ""

select COMPONENT in c3.2xlarge c3.4xlarge m3.2xlarge r3.2xlarge r3.4xlarge g2.2xlarge
do
case $COMPONENT in

c3.2xlarge) config;
            break;;
c3.4xlarge) config;
            break;;
m3.2xlarge) config;
            break;;
r3.2xlarge) config;
            break;;
r3.4xlarge) config;
            break;;
g2.2xlarge) config;
            break;;
esac
done
}

echo ""
echo "Please select the desired Cluster type"
echo ""

select Cluster in RTB RTB3 RTB4
do
case $Cluster in
RTB) File="file:///operations/users/naveed/bootscriptrtb.txt"
     type;
     break;;
RTB3) File="file:///operations/users/naveed/bootscriptrtb3.txt"
      type;
      break;;
RTB4) File="file:///operations/users/naveed/bootscriptrtb4.txt"
      type;
      break;;
esac
done

echo "Please configure AS Policy"
echo ""
read -e -p "Enter No. of Instance to be affected (Integer Value)...: " aff
echo ""

aws autoscaling put-scaling-policy --policy-name increase-policy --auto-scaling-group-name $Cluster-$COMPONENT-ASG --scaling-adjustment $aff --adjustment-type ChangeInCapacity --output text > increase.txt
inc=$(cat increase.txt)
aws autoscaling put-scaling-policy --policy-name decrease-policy --auto-scaling-group-name $Cluster-$COMPONENT-ASG --scaling-adjustment -$aff --adjustment-type ChangeInCapacity --output text > decrease.txt
dec=$(cat decrease.txt)

echo "Please configure AS Alarm"
echo ""
read -e -p "Enter High CPU threshold (Integer Value)...: " high
echo ""
read -e -p "Enter Lower CPU threshold (Integer Value)...: " low
echo ""

aws cloudwatch put-metric-alarm --alarm-name $Cluster-$COMPONENT-AddCapacity --metric-name $Cluster-$COMPONENT-CPU-HIGH --namespace AWS/EC2 --statistic Average --period 300 --threshold $high --comparison-operator GreaterThanOrEqualToThreshold --dimensions "Name=AutoScalingGroupName,Value=$Cluster-$COMPONENT-ASG" --evaluation-periods 2 --alarm-actions $inc
aws cloudwatch put-metric-alarm --alarm-name $Cluster-$COMPONENT-RemoveCapacity --metric-name $Cluster-$COMPONENT-CPU-LOW --namespace AWS/EC2 --statistic Average --period 300 --threshold $low --comparison-operator LessThanOrEqualToThreshold --dimensions "Name=AutoScalingGroupName,Value=$Cluster-$COMPONENT-ASG" --evaluation-periods 2 --alarm-actions $dec

rm -fr increase.txt decrease.txt

echo "Job Finished!"
echo ""

Fixing the GPG missing keys error

Sometimes we face the missing gpg key issue when we add diffrent sources to sources.list files, well this can be fixed easily:
Im gonna show an example where im using ubuntu 14 and try to first add sources of debian jessie 8.

first go to http://debgen.simplylinux.ch/

get your source list, for eg:

deb http://ftp.us.debian.org/debian testing main contrib non-free
deb-src http://ftp.us.debian.org/debian testing main contrib non-free

deb http://security.debian.org/ jessie/updates main contrib non-free
deb-src http://security.debian.org/ jessie/updates main contrib non-free

paste this in

nano /etc/apt/sources.list

then run this script:

#!/bin/bash

apt-get update 2> /tmp/keymissing; for key in $(grep "NO_PUBKEY" /tmp/keymissing |sed "s/.*NO_PUBKEY //"); do echo -e "\nProcessing key: $key"; gpg --keyserver pgpkeys.mit.edu --recv $key && gpg --export --armor $key | apt-key add -; done
apt-get update 2> /tmp/keymissing; for key in $(grep "NO_PUBKEY" /tmp/keymissing |sed "s/.*NO_PUBKEY //"); do echo -e "\nProcessing key: $key"; gpg --keyserver subkeys.pgp.net --recv $key && gpg --export --armor $key | apt-key add -; done

# it works if the command finished and says things like:
#  gpg: requesting key 46925553 from hkp server pgpkeys.mit.edu
#  gpg: key 46925553: "Debian Archive Automatic Signing Key (7.0/wheezy)
#  <ftpmaster@debian.org>" not changed
#  gpg: Total number processed: 1
#  gpg:              unchanged: 1

# If doesnt work and stuck on
# gpg: requesting key 46925553 from hkp server pgpkeys.mit.edu

# Then copy key from this line
# "Processing key: 8B48AD6246925553"
# and google search "8B48AD6246925553" this will return keyserver, like below, put those in for --keyserver line

# And look for keyserver

# Works for debian: pgpkeys.mit.edu
# Works for ubuntu: subkeys.pgp.net

This will work for both, ubuntu and debian - vice versa.

Special thanks to :  http://ram.kossboss.com/fix-missing-gpg-key-apt-get/

Init Script Alternative

Install upstart package from repo, avaiable in both debian and centos.

To create a job to be started automatically when Ubuntu starts. As written example, suppose create the following file /etc/init/testservice.conf with sudo:

# testservice - test service job file

description "my service description"
author "Me <myself@i.com>"

# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas

# When to start the service
start on runlevel [2345]

# When to stop the service
stop on runlevel [016]

# Automatically restart process if crashed
respawn

# Essentially lets upstart know the process will detach itself to the background
# This option does not seem to be of great importance, so it does not need to be set.
#expect fork

# Specify working directory
chdir /home/user/testcode

# Specify the process/command to start, e.g.
exec python mycommand.py arg1 arg2

To 'manually' start or stop the process use

sudo start testservice
sudo stop testservice

The above example works great for everything , but however, there is short cut too :

The solution is to use a shutdown-hook for the system. In Ubuntu it is very easy to add a shutdown-hook by using upstart.

The upstart job config file (/etc/init/shutdown-hook.conf) should look something like this:

description "run at shutdown"

start on starting rc
task
exec /bin/bash /etc/my_service/upload_log_to_s3.sh log

Finally I just needed the script (/etc/my_service/upload_log_to_s3.sh) for uploading a log file to S3:

#!/bin/bash

log_file_ext=$1

gzip -c /var/log/my_service/*.$log_file_ext > /tmp/log.gz

aws s3 cp /tmp/log.gz s3://my_service_bucket/logs/`date +%Y-%m-%dT%H:%M:%SZ`.log.gz

The upload script will just gzip the log file (needed as I’m using delaycompress), rename the log file to the current timestamp, and upload the file using aws-cli. The argument sets the file extension of the log file, which is necessary to be able to upload both the current (.log) as well as the previous log file (.log.1).

Post HTTP API grafana Query

The details are as follows:

URL : http://www.nasheikh.com:3000/api/datasources?=

Option : Post

Headers :  Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

Payload : 

{

        "name":"test3",

        "type":"influxdb_08",

        "url":"http://localhost:8086",

        "access":"proxy",

        "basicAuth":false,

        "database":"server1",

        "user":"root",

        "password":"root"

}


Do this in one liner in curl :

curl -H "Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @datainject.json http://www.nasheikh.com:3000/api/datasources?=

Here datainject.json is the file which is pasted above just saved as a json file and passed in curl.

Watch a file for its content, and perform an action accordingly:

Parent Script

#!/bin/bash

while true; do
  change=$(inotifywait -e close_write,moved_to,create .)
  change=${change#./ * }
  if [ "$change" = "changingfile.txt" ]; then ./child.sh; fi
done

Child Script

#!/bin/bash

rm -fr middle*

count=0

eraser=0

while read server

do

let count=$count+1

sed 's/server1/'${server}'/g;s/Tracker/'${server}'/g' center > middle.$count
sed 's/test1/'${server}'/g' datainject.json > newdata.$count.json
curl -H "Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @newdata.$count.json http://www.nasheikh.com:3000/api/datasources?=
sleep 2

done < network.txt

eraser=$(echo middle.$count)

sed -i '$ s/.$//' $eraser

sleep 2

cat prepend middle* final > merged.json

curl -H "Authorization: Bearer eyJrIjoiNzNjc2s5amxkZW9aUE81UXV3dVVOcmRHNTJhRkVOY1oiLCJuIjoiYWRtaW4iLCJpZCI6MX0=" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @merged.json http://nasheikh.com:3000/api/dashboards/db

rm -fr middle* merged.json newdata*

========================================================================