LearnAdmin

What is Software delivery ?

2017-10-31T20:15:00.002+05:30

Implementing CI/CD is part of Devops . Here CD is continouse Delivery .

It include steps we need to do to make code available to customer.

Which means running that code on production servers , making that code resilient .

Which means no bugs , no security issues , No outages and no traffic spikes .

Python script to check if file is locked also display last modified time

2017-08-04T11:46:00.000+05:30

To lock file for testing use this command

The below command will lock /home/ubuntu/fileloc

flock -x -w 5 /home/ubuntu/fileloc echo "4" > /home/ubuntu/fileloc && sleep 5

------------------------

test.py

------------------------

import os, time

import datetime as dt

def is_locked(filepath):

locked = None

file_object = None

if os.path.exists(filepath):

try:

print "Trying to open %s." % filepath

buffer_size = 8

# Opening file in append mode and read the first 8 characters.

file_object = open(filepath, 'a', buffer_size)

if file_object:

print "%s is not locked." % filepath

locked = False

except IOError, message:

print "File is locked (unable to open in append mode). %s." % \

message

locked = True

finally:

if file_object:

file_object.close()

print "%s closed." % filepath

else:

print "%s not found." % filepath

return locked

def wait_for_files(filepaths):

for filepath in filepaths:

if is_locked(filepath):

print "%s is currently in use." % \

(filepath)

st = os.stat(filepath)

mtime = dt.datetime.fromtimestamp(st.st_mtime)

print('%s modified %s'%(filepath, mtime))

# Test

if __name__ == '__main__':

files = [r"/home/ubuntu/filelock"]

print wait_for_files(files)

-------------------------

python test.py

Install and configure postges-xl

2017-08-02T14:37:00.000+05:30

Other than official documentation i did not find much help . It took one hour for me to setup . As i thought its straight forward setup . Download source , compile it and install it . But there is more to it

Below are steps i followed :

Download a stable branch

git clone --depth=1 -b XL9_5_STABLE git://git.postgresql.org/git/postgres-xl.git

cd postgres-xl/

./configure

make

cd contrib/pgxc_ctl

make

cd postgres-xl/ ; make install

cd contrib/pgxc_ctl ; make install

Now installation is done

Add below in /etc/profile

export LD_LIBRARY_PATH=/usr/local/pgsql/lib
export PGDATA=/usr/local/pgsql/data
export PATH=$PATH:/usr/local/pgsql/bin

Then add postgres user
and set /user/local/pgsql as its home directory

chown postgres:postgres -R /user/local/pgsql

Now we need to setup postgres-XL . Since it is distributed database we need atleast two nodes . In general every component sits on its own node and also have stand by nodes .

For this tutorial purpose i am using two nodes node1 and node2

Major components of Postgres-XL

Global Transaction Monitor (GTM) : The Global Transaction Monitor ensures cluster-wide transaction consistency. GTM is responsible for issuing transaction ids and snapshots as part of its Multi-version Concurrency Control.

Coordinator : The Coordinator manages the user sessions and interacts with GTM and the data nodes.

Data Node : The Data Node is where the actual data is stored.
I have tried installing everything on node1 for demo purpose but default port of coordinator and data node are same also the share same /tmp directory structure.

So we might need to change port number then temp directories etc

Below is the error for reference purpose

postgres@Node1:~$ postgres --datanode -D /usr/local/pgsql/dbdata -h 0.0.0.0 -p 5432 -c gtm_host=node1 -c gtm_port=6666 &
[3] 35190
postgres@Node1:~$ LOG: could not bind IPv4 socket: Address already in use
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
WARNING: could not create listen socket for "0.0.0.0"
FATAL: could not create any TCP/IP sockets
^C
[3]+ Exit 1 postgres --datanode -D /usr/local/pgsql/dbdata -h 0.0.0.0 -p 5432 -c gtm_host=node1 -c gtm_port=6666
postgres@Node1:~$ postgres --datanode -D /usr/local/pgsql/dbdata -h 0.0.0.0 -p 5433 -c gtm_host=node1 -c gtm_port=6666 &
[3] 35196
postgres@Node1:~$ LOG: database system was shut down at 2017-08-01 16:29:12 UTC
FATAL: lock file "/tmp/.s.PGPOOL.6667.lock" already exists
HINT: Is another postmaster (PID 35113) using socket file "/tmp/.s.PGPOOL.6667"?
LOG: pool manager process (PID 35199) exited with exit code 1
LOG: terminating any other active server processes
LOG: startup process (PID 35198) exited with exit code 2
LOG: aborting startup due to startup process failure

[3]+ Exit 1 postgres --datanode -D /usr/local/pgsql/dbdata -h 0.0.0.0 -p 5433 -c gtm_host=node1 -c gtm_port=6666

Now this two node setup
Node1 will have Global Transaction monitor + Coordinator
Node2 will be the Data node .
We need to download and install from source on both the servers then
One Node1
Init GTM
Create a directory in /usr/local/pgsql called gtm

postgres@Node1:~/data$ initgtm -D /usr/local/pgsql/gtm -Z gtm
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.

fixing permissions on existing directory /usr/local/pgsql/gtm ... ok
creating configuration files ... ok
creating control file ... ok

Success.
You can now start the GTM server using:

gtm -D /usr/local/pgsql/gtm
or
gtm_ctl -Z gtm -D /usr/local/pgsql/gtm -l logfile start
Then start GTM service
gtm -D /usr/local/pgsql/gtm -h 0.0.0.0 -n node1 -p 6666 &

Now coordinator service
Create a directory pgdata or coordinator and run below

postgres@Node1:~$ initdb -D /usr/local/pgsql/pgdata --nodename=node1
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /usr/local/pgsql/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/pgdata/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
creating cluster information ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
You can now start the database server of the Postgres-XL coordinator using:

postgres --coordinator -D /usr/local/pgsql/pgdata
or
pg_ctl start -D /usr/local/pgsql/pgdata -Z coordinator -l logfile

You can now start the database server of the Postgres-XL datanode using:

postgres --datanode -D /usr/local/pgsql/pgdata
or
pg_ctl start -D /usr/local/pgsql/pgdata -Z datanode -l logfile

Now start the service

postgres --coordinator -D /usr/local/pgsql/pgdata -h 0.0.0.0 -p 5432 -c gtm_host=node1 -c gtm_port=6666 &
Now on node2 after you compile from source and followed steps till adding content in /etc/profile and adding of postgres user is done
Create a data folder and initialize as above
initdb -D /usr/local/pgsql/data --nodename=node2
Then start service
postgres --datanode -D /usr/local/pgsql/data -h 0.0.0.0 -p 5432 -c gtm_host=node1 -c gtm_port=6666 &

Now you can run psql and connect

postgres@Node2:~$ psql
psql (PGXL 9.5r1.5, based on PG 9.5.6 (Postgres-XL 9.5r1.5))
Type "help" for help.

postgres=#

Creating a dummy process in linux with a process name for testing and other purpose

2017-05-12T13:39:00.002+05:30

bash -c 'exec -a ProcessName sleep 1000000' &

nagios https proxy monitoring plugin or Bash website monitoring script

2017-03-23T16:31:00.002+05:30

#!/bin/bash

if [ $# -eq 0 ]
then
echo "No arguments supplied"
fi

if [ $# -eq 1 ]
then

output=`curl -x 10.11.10.10:3128 -L $1 -m 10 -o /dev/null || echo "Website_down"`

if [ "$output" == "Website_down" ] ; then

echo "Website is down CRITICAL "
exit 2

else

echo "Website_up $1 OK"
exit 0
fi

fi

Adding client certificate file using chef data bags

2017-03-23T16:30:00.000+05:30

Below is just an example

logstash_cert_secret = Chef::EncryptedDataBagItem.load_secret("/etc/chef/secrets/logstash_client_key")

logstash_certs = Chef::EncryptedDataBagItem.load("certs", "logstash_client", logstash_cert_secret)

directory "/etc/pki/tls/certs/" do

action :create

mode "0755"

owner "root"

group "root"

end

file "/etc/pki/tls/certs/logstash_client.crt" do

action :create

mode "0644"

owner "root"

group "root"

content logstash_certs["crt"].join("")

end

Search and replace in all files in a current directory

2017-03-03T13:22:00.000+05:30

grep -rl 'pc961' ./|xargs sed -i 's/pc961/pc/g'

Ansible define a basic windows hosts

2017-03-03T13:18:00.003+05:30

[windows]

10.10.10.120

[windows:vars]

ansible_connection=winrm

ansible_ssh_user=Administrator

ansible_ssh_pass=AdminPa$$

Three basic Git commands

2017-01-09T19:02:00.001+05:30

1) git add -A

git add -A is equivalent to git add .; git add -u.

git add . ( This will add all files but does not remove files )
git add -u ( This will add only changes but will not add any new files created )

So we use

git add -A ( this will take care of both the commands )

2) git commit -m “Commit changes text ex: I have added new function xyz to the file”

3) git push origin

example of branch name : develop

Another 3 frequently used commands include :

git pull
git status
git merge origin/Branchname

Installing configuring Ansible and deploying haproxy playbook

2016-09-14T16:24:00.000+05:30

Basic installation

sudo yum install ansible (or) sudo pip install ansible

Configuring your hosts file with inventory


cat /etc/ansible/hosts 

[haproxy]

10.0.184.12   



[all_servers:vars]

ansible_connection=ssh 

ansible_user=root

ansible_ssh_private_key_file=/Users/home/yoursshkey.pem


I have only one server in my inventory where i will be setting up an haproxy 
For easy management we will be writing common variables under [all_servers:vars]


When we run ansible play books hosts will be picked from the above inventory file

Example : hosts: haproxy  ( this will run on all servers under haproxy )

Now test servers doing a ping ( ping in ansible will connect to the server using ssh )


 ansible haproxy -m ping


10.0.184.12 | SUCCESS => {


    "changed": false, 


    "ping": "pong"


}


Then Download ready made packages from ansible galaxy 
https://galaxy.ansible.com/devops/haproxy/
Also download dependencies that will configure EPEL repositories
https://galaxy.ansible.com/sfromm/epel/
Then create a common playbook for both the roles main.yml




- hosts: haproxy

  roles:

     - { role: devops.epel }





- hosts: haproxy

  roles:

     - role: devops.haproxy

       haproxy_stats:

         name: 'global_monitor'

         ip: "{{ ansible_default_ipv4.address }}"

         port: '38888'

         stats:

           enabled: True

           hide_version: true

           uri: /slb_stats_url

           realm: Welcome\ to\ slb\ monitor

           auth: admin:admin

           refresh: 2s

       haproxy_frontends:

       - name: 'fe-testsite'

         ip: '{{ ansible_default_ipv4.address }}'

         port: '80'

         maxconn: '1000'

         default_backend: 'be-testsite'

       haproxy_backends:

       - name: 'be-testsite'

         description: 'testsite'

         servers:

           - name: 'be-testsite-01'

             ip: '192.168.1.100'

sh-3.2# ansible-playbook main.yml

PLAY [haproxy] *****************************************************************

TASK [setup] *******************************************************************
ok: [10.0.184.12]

TASK [devops.epel : Installs python dependencies] ******************************
changed: [10.0.184.12] => (item=[u'libselinux-python'])

TASK [devops.epel : Disable SElinux] *******************************************
changed: [10.0.184.12]

TASK [devops.epel : create EPEL yum repository] ********************************
changed: [10.0.184.12]

TASK [devops.epel : import EPEL GPG key] ***************************************
changed: [10.0.184.12]

PLAY [haproxy] *****************************************************************

TASK [setup] *******************************************************************
ok: [10.0.184.12]

TASK [devops.haproxy : Include OS-specific variables.] *************************
ok: [10.0.184.12]

TASK [devops.haproxy : Installs haproxy as well as socat for socket api.] ******
changed: [10.0.184.12] => (item=[u'haproxy', u'socat'])

TASK [devops.haproxy : Ensure HAProxy is started and enabled on boot.] *********
changed: [10.0.184.12]

TASK [devops.haproxy : Ensure chroot directory exists.] ************************
ok: [10.0.184.12]

TASK [devops.haproxy : Create directory for the frontend] **********************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the frontends] *********************************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{haproxy_frontends}}').
This feature will be
removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [10.0.184.12] => (item={u'ip': u'10.0.184.12', u'maxconn': u'1000', u'default_backend': u'be-testsite', u'port': u'80', u'name': u'fe-testsite'})

TASK [devops.haproxy : Create directory for the backends] **********************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the backends] **********************************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{haproxy_backends}}').
This feature will be
removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [10.0.184.12] => (item={u'servers': [{u'ip': u'192.168.1.100', u'name': u'be-testsite-01'}], u'description': u'testsite', u'name': u'be-testsite'})

TASK [devops.haproxy : Create directory for the listen sections] ***************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the listen sections] ***************************
skipping: [10.0.184.12] => (item=haproxy_listen)

TASK [devops.haproxy : Create directory for the userlists] *********************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the userlist sections] *************************
skipping: [10.0.184.12] => (item=haproxy_userlists)

TASK [devops.haproxy : Create the compiled folder] ****************************
changed: [10.0.184.12]

TASK [devops.haproxy : Empty the folder if not already empty] ******************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the global config] *****************************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the default config] ****************************
changed: [10.0.184.12]

TASK [devops.haproxy : Build up the stats config] ******************************
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the frontends configuration file] **************
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the backends configuration file] ***************
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the listen sections configuration file] ********
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the userlists sections configuration file] *****
changed: [10.0.184.12]

TASK [devops.haproxy : Assemble the final configuration file] ******************
changed: [10.0.184.12]

RUNNING HANDLER [devops.haproxy : restart haproxy] *****************************
changed: [10.0.184.12]

PLAY RECAP *********************************************************************
10.0.184.12 : ok=31 changed=27 unreachable=0 failed=0

This is just a test deployment that will forward request to background test server 192.168.1.100

Kibana4 apache configuration with authentication

2015-09-04T14:35:00.000+05:30

We need to configure httpd to listen on port 80 which does a reverse proxy to localhost on 5601.

Make sure kibana is listening only on internal port before this settings are done .

ServerAdmin devops@learnadmin.com

ServerName kibana.learnadmin.com

DocumentRoot /var/www/auth

AuthType Basic

AuthName "Authentication Required"

AuthUserFile "/var/www/auth/htpasswd"

Require valid-user

ProxyPass / http://localhost:5601/

ProxyPassReverse / http://localhost:5601/

Use below command to create a htpasswd file and enable authentication

htpasswd -c /var/www/auth/htpasswd devops

Monitoring of kibana and logstash services and start them if process not found

2015-09-04T14:28:00.000+05:30

Kibana and logstash might run out of memory and application might get closed or get terminated. In that case we will be running the below script to monitor services and start them if not running .

There are other parameters that need to be verified before going for this option like java heap size etc.

#!/bin/bash

LOGSTASH=logstash

KIBANA=kibana

var1=`ps -ef|grep -i $LOGSTASH|sed -e '/grep/d'`

if [ -z "$var1" ]; then

nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf &

echo "Process logstash not found" | mail -s "Process logstash not found creating a new one" devops@learnadmin.com

else

echo "process logstash found."

var2=`ps -ef|grep -i $KIBANA|sed -e '/grep/d'`

if [ -z "$var2" ]; then

nohup /opt/kibana/bin/kibana status &

echo "Process kibana not found" | mail -s "Process kibana not found creating a new one" devops@learnadmin.com

else

echo "process kibana found."

Chef provision a esxi virtual machine or a virtual server

2015-08-24T12:13:00.000+05:30

We need to install esxi knife plugin on our workstation

gem install knife-esx

Add below params in knife.rb

knife[:esx_host] = "esx"
knife[:esx_username] = "root"
knife[:esx_password] = "password of your esxi server"

Need to create a ubuntu template ( or custom OS template as per requirement )
Below example shows how to provision a ubuntu template .

ssh root@esxi

mkdir -p /vmfs/volumes/datastore1/esx-gem/templates

vmkfstools -i /vmfs/volumes/datastore1/ubuntu-12.10-x64_template/*.vmdk –diskformat thin /vmfs/volumes/datastore1/esx-gem/templates/ubuntu-12.10-x64_template.vmdk

esx template list ( should display the template we installed )

knife esx vm create --vm-name server1 --use-template ubuntu-12.10-x64_template --verbose true --distro ubuntu12.04-19-gems --vm-memory 2048 -x provision -i ~/.ssh/id_rsa

How to use chef encrypted databags

2015-01-12T19:16:00.002+05:30

Note: Please comment the below line in knife.rb file else it will automatically decrypt values
and show when we use "knife data bag show" command
I initially thought knife was not encrypting my values and had to debug ..which took lot of time.

[root@ec2-test .chef]# cat knife.rb | grep secret

#knife[:secret_file] ='/root/encrypted_data_bag_secret'

knife data bag create --secret-file /root/.chef/encrypted_data_bag_secret testbag password

{

"id": "password",

"password": "this is test password key"

}

[root@ec2-test .chef]# knife data bag show testbag password

id: password

password:

cipher: aes-256-cbc

encrypted_data: KMHzeFQwfm0wWeHFymxfJsMo425CP+wlwoZ6xN7waVlgUNOmRrr/+jOtDLIN

s7Xl

iv: TYPYnSqYTcmU8ZWE2sIt4A==

version: 1

Once encrypted if you try to edit the encrypted databag it shows like this

{

"name": "data_bag_item_testbag_password",

"json_class": "Chef::DataBagItem",

"chef_type": "data_bag_item",

"data_bag": "testbag",

"raw_data": {

"id": "password",

"password": {

"encrypted_data": "KMHzeFQwfm0wWeHFymxfJsMo425CP+wlwoZ6xN7waVlgUNOmRrr/+jOtDLIN\ns7Xl\n",

"iv": "TYPYnSqYTcmU8ZWE2sIt4A==\n",

"version": 1,

"cipher": "aes-256-cbc"

}

To show the decrypted values we can use

[root@ec2-test .chef]# knife data bag show testbag password --secret-file /root/.chef/encrypted_data_bag_secret

id: password

password: this is test password key

---------------------------
Some important commands related to data bags

Generate random secret key

openssl rand -base64 512 >encrypted_data_bag_secret

Generate random password

date | md5 in mac

date | md5sum in any linux machine

Setting a Secure Flag to cookie in httpd – Apache

2014-07-30T15:57:00.003+05:30

First check if mod_headers is loaded. Else enable that module

LoadModule headers_module modules/mod_headers.so

Once the module is loaded .

Header edit Set-Cookie ^(.*)$ $1;Secure

Add above line to your virtual host in httpd.conf/ssl.conf as per your config

What this does is it adds a secure flag to your cookie. This flag is set by application server when sending
a new cookie to client/user when sending http response.This also mean that server will not send cookie over http. It sends only via https .This prevents unauthorized access to cookie data

This can be done via code in java or can be done @ apache/httpd config level.

Here is more info on code level changes click here..!!

Get SSL certificate expire date / Openssl get certificate expire date

2014-07-09T15:29:00.003+05:30

openssl x509 -in /home/certificates/certname.crt -noout -enddate

rssh chroot jailroot sftp creation - By Vinodh tiruttani

2014-06-27T14:04:00.000+05:30

Steps for chroot jail folder creation:

The chroot command changes its current and root directories to the provided directory and then run command, if supplied, or an interactive copy of the user's login shell. Please note that not every application can be chrooted.

Creating User
[root@learnadmin ~]# /usr/sbin/useradd testrssh
[root@learnadmin ~]# su testrssh
[testrssh@learnadmin root]$ cd /home/testrssh/
[testrssh@learnadmin ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/testrssh/.ssh/id_dsa):
Created directory '/home/testrssh/.ssh'.
Enter passphrase (empty for no passphrase): < Type passphrase >
Enter same passphrase again: < Re-type passphrase >
Your identification has been saved in /home/testrssh/.ssh/id_dsa.
Your public key has been saved in /home/testrssh/.ssh/id_dsa.pub.
The key fingerprint is:
92:c9:fe:9f:47:29:ba:15:29:9c:51:84:0f:c6:d8:0b testrssh@learnadmin.com
[testrssh@learnadmin ~]$
[testrssh@learnadmin ~]$ cd .ssh/
[testrssh@learnadmin .ssh]$ pwd
/home/testrssh/.ssh
[testrssh@learnadmin .ssh]$ cp id_dsa.pub authorized_keys2
[testrssh@learnadmin .ssh]$ ls -l
total 24
-rw-r--r-- 1 testrssh testrssh 618 Jun 12 18:12 authorized_keys2
-rw------- 1 testrssh testrssh 736 Jun 12 18:11 id_dsa ( 600 permission)
-rw-r--r-- 1 testrssh testrssh 618 Jun 12 18:11 id_dsa.pub
[testrssh@learnadmin .ssh]$
[testrssh@learnadmin .ssh]$ exit
[root@learnadmin ~]# /usr/sbin/usermod -s /usr/bin/rssh testrssh
Download from net rssh-2.3.3.tar.gz from http://www.pizzashack.org/rssh/downloads.shtml
[root@learnadmin ~]# tar xvf rssh-2.3.3.tar.gz
[root@learnadmin ~]# cd rssh-2.3.3
[root@learnadmin rssh-2.3.3]# ./configure
[root@learnadmin rssh-2.3.3]# make
[root@learnadmin rssh-2.3.3]# make install
[root@learnadmin rssh-2.3.3]# cd ..
Download from net rssh-2.3.3-1.x86_64.rpm
[root@learnadmin ~]# rpm -ivh rssh-2.3.3-1.x86_64.rpm
Preparing... ########################################### [100%]
1:rssh ########################################### [100%]
[root@learnadmin ~]# pwd
/root

· Converting rssh chroot Jail folder – Execute the below script from the below location

[root@learnadmin ~]# cp /usr/share/doc/rssh-2.3.3/mkchroot.sh .
[root@learnadmin ~]# chmod 775 mkchroot.sh
[root@learnadmin ~]# ./mkchroot.sh /home/testrssh/
NOT changing owner of root jail.
NOT changing perms of root jail.
setting up /home/testrssh//usr/bin
setting up /home/testrssh//usr/libexec/openssh
setting up /home/testrssh//usr/libexec
Copying libraries for /usr/bin/scp.
        /lib64/libcrypto.so.6
        /lib64/libutil.so.1
        /usr/lib64/libz.so.1
        /lib64/libnsl.so.1
        /lib64/libcrypt.so.1
        /lib64/libresolv.so.2
        /usr/lib64/libgssapi_krb5.so.2
        /usr/lib64/libkrb5.so.3
        /usr/lib64/libk5crypto.so.3
        /lib64/libcom_err.so.2
        /usr/lib64/libnss3.so
        /lib64/libc.so.6
        /lib64/libdl.so.2
        /usr/lib64/libkrb5support.so.0
        /lib64/libkeyutils.so.1
        /usr/lib64/libnssutil3.so
        /usr/lib64/libplc4.so
        /usr/lib64/libplds4.so
        /usr/lib64/libnspr4.so
        /lib64/libpthread.so.0
        /lib64/libselinux.so.1
        /lib64/libsepol.so.1
Copying libraries for /usr/libexec/openssh/sftp-server.
        /lib64/libcrypto.so.6
        /lib64/libutil.so.1
        /usr/lib64/libz.so.1
        /lib64/libnsl.so.1
        /lib64/libcrypt.so.1
        /lib64/libresolv.so.2
        /usr/lib64/libgssapi_krb5.so.2
        /usr/lib64/libkrb5.so.3
        /usr/lib64/libk5crypto.so.3
        /lib64/libcom_err.so.2
        /usr/lib64/libnss3.so
        /lib64/libc.so.6
        /lib64/libdl.so.2
        /usr/lib64/libkrb5support.so.0
        /lib64/libkeyutils.so.1
        /usr/lib64/libnssutil3.so
        /usr/lib64/libplc4.so
        /usr/lib64/libplds4.so
        /usr/lib64/libnspr4.so
        /lib64/libpthread.so.0
        /lib64/libselinux.so.1
        /lib64/libsepol.so.1
Copying libraries for /usr/bin/rssh.
        /lib64/libc.so.6
Copying libraries for /usr/libexec/rssh_chroot_helper.
        /lib64/libc.so.6
copying name service resolution libraries...
tar: Removing leading `/' from member names
        lib/libnss_files-2.5.so
tar: /lib/libnss1_files*: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors
        lib/libnss_files.so.2
Setting up /etc in the chroot jail
cp: omitting directory `/etc/ld.so.conf.d'
Chroot jail configuration completed.
NOTE: if you are not using the passwd file for authentication,
you may need to copy some of the /lib/libnss_* files into the jail.

NOTE: you must MANUALLY edit your syslog rc script to start syslogd
with appropriate options to log to /home/testrssh//dev/log. In most cases,
you will need to start syslog as:
    /sbin/syslogd -a /home/testrssh//dev/log
NOTE: we make no guarantee that ANY of this will work for you... if it
doesn't, you're on your own. Sorry!
Note: while executing above command some files are not copied, we need to copy manually.
[root@learnadmin ~]#
[root@learnadmin ~]# cd /home/testrssh/
[root@learnadmin testrssh]# cd dev/
[root@learnadmin dev]# mknod -m 666 zero c 1 5
[root@learnadmin dev]# mknod -m 666 null c 1 3
[root@learnadmin dev]# cd ../lib
[root@learnadmin lib]# cp /lib/ld-linux.so.2 .
[root@learnadmin lib]# cp /lib/libc.so.6 .
[root@learnadmin lib]# cd ../lib64/
[root@learnadmin lib64]# cp /lib64/ld-linux-x86-64.so.2 .
[root@learnadmin lib64]# cp /lib64/libnss_* .
[root@learnadmin lib64]# cd
[root@learnadmin]# vi /etc/rssh.conf ( add the user and uncomment 2 lines)
allowscp
allowsftp
user=testrssh:011:00011:"/home/testrssh"
:wq!
[root@learnadmin ~]#
Done rssh chroot jail folder is created
-------------------------------------------------
Testing rssh login from remote system
Login to server from remote server
Copy id_dsa key from Server ( user: testrssh )
[root@clientmachine test-rssh-keys]# vi id_dsa_testrssh

-----BEGIN DSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,42ED97DC52451DE9

UAh2bmigMZ5DN0KVadfR9bRrFL7l0mUMF7iCjgbqHA3wgNfPttytDmt3jJR5Mmb+
uxl0s1D1JwDvNstUgTL9fDC+TnrQXj+0YVQ7bvHPLdcyTEUm3MXNAbqv3PUoFAVa
qfLPtybOHCADOA6wH+vcJ4Mt2NM/M+fbMbcmBsrMw3fYzB2na5Zs2OoVxKLbVPEo
mKP3Eu5Eo1nC6wApQkBzO7ZJDq3x7YjxPPQJZ82j/vVVuZsGj0IFBRm1Esj8gABp
MlQfTzK0wnbbCJ2L3CqRPS7gyyKCpdQtuNYliDh8E+uXAsCjIx+rksfRJkTmKTDj
Huob0no/69che/tnBzhzQV3B6C1BrJ4CPElRVOoq2niQIy4ruku5wK9ONm3J50It
wc3uADpOgBUStavVpuUJTU4Oe8rVraUhEyBB0iOpYySGrAAfEtabwCwB0OvK+bht
1mLleTDGadms07+psexV4L66bmDY6ypu7dtcGWuFXQP5UQdOHqS7ehdEtCpuboer
BVkcrdivDHoc88cScuDOytPsFyMiHJ5fkrMLtKWQBTF3I7SiDqVBq/lKFcubZK3Q
K/PpJyasnxlcaLJLFkFibw==
-----END DSA PRIVATE KEY-----
:wq!
[root@ clientmachine test-rssh-keys]# chmod 600 id_dsa_testrssh
[root@ clientmachine test-rssh-keys]# sftp -o IdentityFile=id_dsa_testrssh testrssh@sftp server
Connecting to 10.x.x.x...
Enter passphrase for key 'id_dsa_testrssh':
sftp> ls
dev    etc    lib    lib64 usr
sftp> cd /root
Couldn't canonicalise: No such file or directory
sftp> cd /tmp
Couldn't canonicalise: No such file or directory
sftp>
rssh chroot jail folder is working.

Linux grep a process and kill all process id's

2014-04-25T11:39:00.001+05:30

kill -9 `ps -ef | grep ProcessName | grep -v grep | awk '{print $2}'`

How to parse xml tags in bash or retrieve xml tag values in linux

2014-04-10T14:43:00.000+05:30

Consider the following example file test.xml

</header>

grep -E -m 1 -o "(.*)" test.xml

This will return "test123"

if you want only the value we need to pipe it further

grep -E -m 1 -o "(.*)" test.xml | sed -e 's,.*$[^<]*$.*,\1,g'

This will return "test123"

fix failed: NT_STATUS_BAD_NETWORK_NAME"

2014-03-26T17:18:00.001+05:30

smbclient -L 127.0.0.1 -U testuser ( will list directories )

smbclient //127.0.0.1/Pubshare -U testuser ( will connect )

Simple gpg encryption setup

2014-03-10T11:56:00.000+05:30

Generation of encryption keys
-----------------------------------

gpg --gen-key

gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) "

Real name: learnadmin
Email address: admin@learnadmin.com
Comment: this is comment section
You selected this USER-ID:
"learnadmin (this is comment section) "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.
----------------
Here give password which is used for decryption of files. ( my testpassphrase is test@123 )
----------------
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.++++++++++++++++++++++++++++++.+++++++++++++++.+++++++++++++++...++++++++++++++++++++++++++++++++++++++++..+++++..+++++..++++++++++++++++++++.+++++..........+++++

Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 162 more bytes)
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++.+++++++++++++++..+++++.+++++.+++++++++++++++.+++++.++++++++++++++++++++.+++++++++++++++.++++++++++...+++++++++++++++....+++++++++++++++..++++++++++++++++++++>++++++++++>+++++.>.+++++........<+++++.......................................................................>+++++....................<+++++......>+++++................................................................................................................+++++^^^^
gpg: /home/username/.gnupg/trustdb.gpg: trustdb created
gpg: key 4F13D8CF marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 1024D/4F13D8CF 2014-03-10
Key fingerprint = BB11 7A85 CC87 55D2 6F35 D34C CD32 5F04 4F13 D8CF
uid learnadmin (this is comment section)
sub 2048g/5A897101 2014-03-10

------------------------
Tip for creating random key for cert generation :
Depending on your disk type
dd if=/dev/sda of=/dev/zero
(or)
dd if=/dev/hda2 of=/dev/zero
2911235+0 records in
2911234+0 records out
1490551808 bytes (1.5 GB) copied, 8.24918 seconds, 181 MB/s

The above command with take input form your disk and writes it randomly to /dev/zero

---------------------------

gpg --import Yourkey.gpg ( not required . This key need to be imported if you are using this encryption on other server or other client )

Create a file test.txt and try to encrypt and check if it is working ...!!

gpg --always-trust --encrypt --recipient "learnadmin (this is comment section) " test.txt

gpg --batch --passphrase test@123 --decrypt test.txt.gpg>test.txt

Graphite & statsd installation and Configuration

2013-12-26T15:50:00.000+05:30

How to install Graphite and statsd
-----------------------------------

yum install python-devel.noarch
yum install pycairo.x86_64 Django.noarch django-tagging.noarch python-twisted.noarch python-zope-interface.x86_64 python-zope-interface httpd memcached python-memcached
yum install fontconfig.x86_64 fontconfig-devel.x86_64
yum install mod_wsgi.x86_64
yum install python-pip.noarch

pip-python install whisper
pip-python install carbon
pip-python install graphite-web

cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite.conf
cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py

python /opt/graphite/webapp/graphite/manage.py syncdb
chown -R apache:apache /opt/graphite/storage/

wget -O /etc/init.d/carbon https://gist.github.com/raw/3652720/6978bda604c794e21165a81c6b879528d19e8bb4/carbon.init.sh
chmod 0755 /etc/init.d/carbon
chkconfig --add carbon

/opt/graphite/bin/carbon-cache.py start
/etc/init.d/httpd start

yum install nodejs npm

cd /opt/graphite
git clone https://github.com/etsy/statsd.git
cd statsd
cp exampleConfig.js local.js

/etc/init.d/carbon start

/etc/init.d/memcached start
/etc/init.d/statsd start

Configurations
--------------------
Graphite Configuration: /opt/graphite
Statsd configugration: /opt/graphite/statsd

Graphite/Statsd Logs
---------------------------

Graphite: /opt/graphite/storage/log/webapp
Statsd: /var/log/statsd/
statsderr.log statsd.log

Bash script to mask credit card number or any data in logs.

2013-12-20T17:32:00.001+05:30

create a temp test file for testing your data

cat test.txt

hi this is my credit card number 1234567891234567
i have 22 this 22 is just for testing 1234567890123456 and this one

Here is the format which varies according to your requirement .

In my case there are 16 digits .

I want to mask first 6 number here is the regular expression or bash command

sed -e :a -e "s/[0-9]\{6\}$[0-9]\{10\}$/\*\*\*\*\*\*\1/;ta" test.txt

here 6 and 10 are number of masks and number of visible characters respectively.

you can alter them according to your needs.

And the stars can be replaced by # or the value you want to mask with .

The number of starts should be equal to number of masks in my case it is 6

Bash script get yesterdays or one day ago's date or date-1

2013-10-30T16:17:00.000+05:30

date -d '1 day ago' +"%Y-%m-%d"

Linux list all processes based on memory consumption .

2013-08-22T12:23:00.000+05:30

list top 5 processes
ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5

list all processes in assending order .
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS