ExploreIT

TensorFlow

noreply@blogger.com (sridhar) — Mon, 26 Nov 2018 01:26:00 -0800

TensorFlow !!

one more cool yet powerful framework/library to develop ML programs.

The underlying basis for this is a directed graph. The data otherwise called edges flows through nodes which are otherwise called operations.

TensorFlow normally works in a lazy way i.e., it builds the graphs and execution happens later during the programming although, there is an 'eager' way to execute in runtime mode. And, the TensorFlow programming happens at "estimator" api level. Or there is another high level API called 'keras' which can be used to define models and execute at ease.

This is how it can be installed..

conda - package manager can be used for this. update conda and then use conda to upgrade all the packages if you are not sure which ones are dependent packages like numpy etc.,
And, then install python packages using pip3 to install/upgrade tensorflow and keras.

ML how to..

noreply@blogger.com (sridhar) — Sun, 11 Nov 2018 01:15:00 -0800

How to approach an ML problem.. here is what I think

1. Take a look at the data and decide what do you want to do like, what to predict - Explore the data

2. Decide what type of ML problem is it - unsupervised / supervised (classification / regression)

3. Clean up the data .. remove unwanted data, noise, decide on features and create features, normalize feature values - data cleanup and feature engineering

4. Create data sets - training/validation/test data sets

5. Train the model - supply train and validation sets, choose the model parameters, loss function, number of iterations etc., - Create and train a model

6. evaluate the error on training set and validation set. Tune the parameters and re-train until the error is close on both training and validation sets - model validation and tuning

7. Check the model prediction on the test data set which is the data set that the model was not aware of the output label value in the above phases - final check

8. all good ? release to live data. If not, repeat above by revisiting input features and model and model parameters to get a better fit. - going live

Machine Learning

noreply@blogger.com (sridhar) — Sat, 10 Nov 2018 23:09:00 -0800

Machine Learning - as the name suggest, it is about teaching a machine to learn about something. The crust is about how to teach a machine.. the better you teach the better it learns.

How we teach a machine ? well, remember the word 'maths' ? it is every where.. most of the logical problems are solved through maths and that is how machines work. so, we teach the same way.. to do that, it is required to collect 'enough' data sets and feed it to a machine asking to fit a mathematical equation. By doing so, the machine learns about the data and its dynamics .. once it is done, the machine would be able to predict the outcome for a future data point.

the first step in doing so, is identify what kind of ML problem is it. Generally, an ml problem falls into either of the two categories or sometimes both ! depends on what you want..

ML Problem
/ \
/ \
supervised unsupervised
/ \ |
/ \ |
classification regression grouping/clustering

supervised problems are the cases where the data sets contains both input (features) and output (label). The teaching process will be carried out in guidance with the output so that, ml algorithm will know how is it doing how to evaluate the model so that it better fits to the data.

This is further categorized into two types. 'classification' is type of problem where the output is basically classified into two or a few unique values and the goal is to fit a model so that, the future data point will be classified into the given sets. like what is the number that a given picture contains..

Regression is the second type of supervised problems where the output variable is a continuous label instead of limited/discrete value. like whether your favorite team wins a match against a given opponent.

Unsupervised problems are the cases where the outcome is not really known. So, the goal is to fit a model in order to identify the clusters in the data sets.

There are lot of ML Algorithms and libraries and frameworks available as opensource and as commercial products in the market already..

Web Application's Navigation Paths and Page Performance

noreply@blogger.com (sridhar) — Fri, 12 Jan 2018 23:03:00 -0800

I have been trying to find out a better way to visualize and better understand how a web application is being accessed by the end users and how the end user journey looks like and which pages are indirectly became the close groups based on how the end users are distributed and their interests in the application features. And, finally how smooth is the path between pages..

And, stumbled on a wonderful visualization tool called Gephi. And, below is how a sample website looks like depicting the user navigation paths, the page grouping and page performance which is characterized by the weights of the path between the pages. In the graph terminology, the pages are nodes and the paths between pages are called edges which are attributed by response times as weight.

A Sample Splunk Dashboard

noreply@blogger.com (sridhar) — Wed, 4 Oct 2017 06:39:00 -0700

A sample dashboard built using Splunk.. showing some meaningful stats from some pseudo data.
Have to say wow ! it was so quick to build and the responsiveness of the features are so fast .. and it's fun learning this stuff..

Applying machine learning to study and do capacity planning

noreply@blogger.com (sridhar) — Thu, 25 May 2017 07:01:00 -0700

ML algorithms and libraries have evolved and the scope of applying the techniques has also grown a lot. I am trying here to apply one of such techniques to study the performance metrics and then estimate the required capacity at a future load.

Lets take a simple set like pageviews and heap usage
Example:
pv, heapusage(mb)
105,637
110,638
115,640
120,642
125,644
130,646
135,648
140,650
145,652

Now, since this is not a classification type dataset, the regression models can be used. There are a few regression models from one of the coolest ML libraries - scikit (sklearn).

let's take a linearRegression model and try to fit it on the set.

The logic to implement the model is:

- declare the data set into arrays (have not changed the parameters like squaring them etc.,)

- create sets for training and validation as 20% is used for validation

- choose the model as linearRegression

- fit the model on the train set to make it learn the trends

- get the score of learning to verify how well it has fit on the dataset

- get the coefs and intecepts used to fit the model

- predict the validation set and compare it with actual values in validation set

- if it is all good, then try to predict the load for future increased load

Once, this is done, I have build wrappers and a simple UI.

UI Page 1: To simply upload the feature file i.e. the metrics file

UI Page 2: This will show the result. The correlation between the metrics, validation sets and how close the predicted values and finally the predicted output i.e. required Heap Memory for the given page views.

Although this is a simple demonstration, more features can be added..

some useful linux commands

noreply@blogger.com (sridhar) — Sun, 23 Apr 2017 07:13:00 -0700

Below are some of the useful linux commands to diagnose issues..

To know about processes, parent and call heirarchy and to know the process's resource usage etc., we can use pstree, ps, top -H etc.,

Example:
pstree

init─┬─Xvnc
├─crond
├─firefox───9*[{firefox}]
├─gnome-terminal─┬─bash
│ ├─gnome-pty-helpe
│ └─{gnome-terminal}
├─gnome-terminal─┬─bash───su───bash───startWebLogic.s───java───103*[{java}]

ps -e f

132 ? Sl 0:01 gnome-terminal
137 ? S 0:00 \_ gnome-pty-helper
138 pts/1 Ss 0:00 \_ bash
353 pts/1 S 0:00 | \_ su
357 pts/1 S+ 0:00 | \_ bash
460 pts/1 S 0:00 | \_ /bin/sh ./startWebLogic.sh
510 pts/1 Sl 26:44 | \_ /...../jdk/bin/java -server -Xms256m -Xmx1024m -Dwe
993 pts/0 Ss+ 0:00 \_ bash

while ps -ef can give you full command of the process
And, it will also show the parent process id and process cpu, rss size etc.,

To know all the list of files that the process has accessed, lsof is the command..

/usr/sbin/lsof -p 510

So the weblogic server which is a JVM has accessed 2K+ files. And example is below..

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 510 root cwd DIR 202,2 4096 5657417 .../DefaultDomain

FD -is the file descriptor like below
cwd current working directory;
mem memory-mapped file;
mmap memory-mapped device;
pd parent directory;
rtd root directory;
tr kernel trace file

TYPE is the file type like a REGulare or DIRectory..
NODE is the inode number
DEVICE indicates the device type and partition numbers

There are couple of other useful commands to debug issues like top -H -p will give the thread level processor utilization which is useful when debugging high CPU consumption
issues in a JVM..

strace is another useful command to know what a process is doing at OS level like socket connections, reads etc.,

netstat is another command with options like -nap gives the information on the connections and rec/send Q which can indicate any network or program slow or being blocked and which
connections in which state etc.,

Disk space check commands like df -h, du -sh . etc., are useful to verify sizes and space on disk and network shares

free -g is another useful command to know how much memory has been consumed in physical, swap and cache/buffer areas.

SAR is another great collection of metrics rangning from processes, memory, swap activity, CPU, load average, disk, network etc.,

To flush out the cache and buffers drop_caches is what that needs to be cleared out. Example as shown below

free -m
total used free shared buffers cached
Mem: 15500 15330 169 0 249 11778
-/+ buffers/cache: 3302 12197
Swap: 10047 1704 8342

echo 1 > /proc/sys/vm/drop_caches --- this will flush out the cache and buffers.

free -m
total used free shared buffers cached
Mem: 15500 3367 12132 0 0 635
-/+ buffers/cache: 2731 12768
Swap: 10047 1704 8342

The above is useful if you want something to be loaded into memory again with some runtime changes and also to test out something like a disk speed

To check disk speed: DD - a linux command is a very useful one.

Example Write test:
date; time dd if=ip/test.txt of=op/test.txt bs=1024k count=1000
Sun Apr 23 06:46:17 PDT 2017
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.15715 seconds, 332 MB/s

real 0m3.164s
user 0m0.000s
sys 0m1.104s

-- date is to print out which date and time is to time the command and it is writing a file from ip dir to op dir with block size as 1K and a 1000 times i.e. a total of 1GB. This has spent 3.16s total i.e. 'real' means the elapsed time and system mode CPU time is 1.1 s and nothing really in user mode. So its basically the speed i.e. 332 MB/s took that much time..

But to test it in isolation i.e. read and write speeds, one can use /dev/zero which is a special null char file gives as many as read..

date; time dd of=/dev/zero if=test.txt bs=1024k count=1000 --- read speed
date; time dd if=/dev/zero of=test.txt bs=1024k count=1000 --- write speed

some of other commands like ping, tracert can be used to test the network speed and network paths..

Test JMS Connections

noreply@blogger.com (sridhar) — Fri, 3 Mar 2017 23:22:00 -0800

Below a simple code to test the jms connection. This will time the initialcontext and the queue connection steps. You can get a queue and factory details from the weblogic console. Services - Messaging - JMSModules. Pre-reqs are like, you need to create a JMS server, JMSModule, ConnectionFactory, Queue.

import java.io.*;
import java.util.Hashtable;
import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

public class TestJMS
{

public final static String JNDI_FACTORY="weblogic.jndi.WLInitialContextFactory";
public final static String JMS_FACTORY="testFactory";
public final static String QUEUE="testQ";
private QueueConnectionFactory qconFactory;
private QueueConnection qcon;
private Queue queue;

long t1,t2;

public void initialize(Context ctx, String queueName)throws NamingException, JMSException
{

t1=System.currentTimeMillis();
qconFactory = (QueueConnectionFactory) ctx.lookup(JMS_FACTORY);
qcon = qconFactory.createQueueConnection();
t2=System.currentTimeMillis();
System.out.println(t2-t1);// this is to measure how long does the connection takes
}

public void close() throws JMSException {
qcon.close();
}

public static void main(String[] args) throws Exception {
long t1,t2;

Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_FACTORY);
env.put(Context.PROVIDER_URL, args[0]);
t1=System.currentTimeMillis();
InitialContext ic = new InitialContext(env);
t2=System.currentTimeMillis();
System.out.println("inittime:"+(t2-t1));// this is to measure how long does the initial context takes

TestJMS qs = new TestJMS();
qs.initialize(ic, QUEUE);
qs.close();
}

}

Before executing this program, the environment variables needs to be set. This can be done by running setDomainEnv.sh under weblogic home-userprojs-domains..bin and run the java code from here by passing the t3 url as argument.

Java EE 7 APIs

noreply@blogger.com (sridhar) — Wed, 1 Feb 2017 07:26:00 -0800

EJB
session (stateful and stateless)
Message driven beans
Async local session in ejb lite
nonpersistent timers in ejb lite

Java Servlets
Nonblocking IO - for scalablity
http protocol upgrade

JSF
the UI framework for the web apps supports many features like
rendering, input validation, event handling, data conversion, page navigation,
EL and new ones like HTML 5, Faces flows, resource libraries..

JSP
JSTL (tag libraries)
Java Persistent API
JAX-RS - Java API for RESTful services
Managed Beans
Dependancy injection for Java and Java EE
Bean Validation
JMS
Java EE Connector Architecture
JavaMail API
JAAC - Java Authorization Contract for Container
JASPIC - Java Aithentication Service Provider for Continers
Java API for WebSocket
JSON-P - Java API for JSON processing
Concurrency Utilities for Java EE
Batch Applications for Java Platform
JDBC
JNDI
JavaBeanc Activation Framework
JAXP
JAXB (Java Architecture for XML binding)
JAX-WS
SAAJ - SOAP with attachments API for Java
JAAS - Java authentication and authorization service
Annotations

Weblogic Monitoring Sessions

noreply@blogger.com (sridhar) — Wed, 30 Nov 2016 00:47:00 -0800

Here is the sample script to monitor weblogic server app level session counts. This helps in understanding which apps the sessions are being created the most and how frequent etc., and this can be extended to get session time details as well.

These stats can help in estimating the session memory footprints like sizing a JVM heap.

def getSessionStats():
connect(adminname,adminPd, url);
servers = domainRuntimeService.getServerRuntimes();
timestmp = datetime.datetime.utcnow();
for server in servers:
apps = server.getApplicationRuntimes();
for app in apps:
crs = app.getComponentRuntimes();
for cr in crs:
if (cr.getType() == 'WebAppComponentRuntime'):
snm = cr.getName();
opens = repr(cr.getOpenSessionsCurrentCount());
totals = repr(cr.getSessionsOpenedTotalCount());
highs = repr(cr.getOpenSessionsHighCount());
fo.write(str(timestmp)+','+snm+','+opens+','+totals+','+highs+'\n')

import datetime
adminPd=''
url='t3://...:'
adminname='faadmin'
fo=open("filepathtoopen","a+")
fo.write('Time, Name , OpenSessionCount, TotalSessions, SessionCountHigh \n')
AdminPorts=[ '' ]
for port in AdminPorts:
urltoconnect=url+port
getSessionStats()

fo.close()

JDK 8 - HOTSPOT JVM

noreply@blogger.com (sridhar) — Tue, 18 Oct 2016 05:32:00 -0700

With JDK 8 getting adopted vastly, I just want to give a high level view of hotspot memory management and the garbage collection techniques

Quickly, about Garbage Collection

Allocates objects to young generation

Promotes aged objects to old generation

Marking old generation objects

Recovering space by removing unreachable or compacting live objects

Available collectors in Hotspot JVM

1. Serial collector – for single processor and low concurrent apps. -XX:+UseSerialGC.

2. Parallel collector (or throughput collector) – by default one on the server machines and medium to large scale apps. User -XX:+UseParallelGC. Its multithreaded GC. Select this when high throughput is needed.

3. Mostly concurrent collector – performs most of its work concurrently i.e. app is still running. Select this when faster response times are needed. Since the app is running most of the time without any pauses, means RTs are not impacted often by the pause times. There are 2 types of mostly concurrent collectors – Concurrent Mark & sweep, G1 (garbage first gc – a generational algo preferred for heaps more than 4GB).

*throughput is termed as time spent in GC vs time spent in app. The more timespent is app means it has high amount of resources to deliver.

* for generational GCs i.e. GCs with young,eden,tenured generations, both young and major gcs do pause the app in all types of GC algos. Minor/young gc happens very fast but a major gc happens when there is no space for tenured objects, so it involves entire heap collections i.e. identifying the reachable objects, marking and sweeping unreachable objs and compacting the space to avoid fragmentation. So it pauses longer time. CMS, G1 tries to reduce this by doing them always in the background while app is running and pausing app only in certain times and for short duration

Tuning first steps:

Go with the defaults

- serial for single core machine,

- parallel for high throughput requirement i.e. where App’s RTs can be acceptable at a bit higher sec that means giving longer pause times i.e. less frequent pauses – overall, the app will be paused at times but longer (means high throughput but breaches RT as it causes spike) and the parallel GC does its work with multiple threads in parallel to each and consumes CPU.

- Mostly concurrent – on large apps, multi processor machines, when RT is important. Since RT is important, the pause times are less but may be frequent however, some part of the gc work like marking can happen without any pause. Use CMS when heap when heap less than 4gb else use G1
- Once the default is selected, check whether the requirement is met else, try playing around increasing the XMX. Then try tuning the heap areas.

Parallel Collector:

- -XX:+UseParallelGC

- both minor and major collections are executed in parallel

- number of threads is calculated as 5/8 * N (number of HW threads) or can be set manually as -XX:ParallelGCThreads=

- since higher number of threads causes fragmentation when promoting objects from young to tenured spaces, try reducing the number or increase tenured space to cope up with freagmentation.

- parallel collector tunes automatically based on behavior. So no need to specify generation sizes or granular level tunings. Behavior is mentioned are max gc pause time, throughput, foot print of heap size.

- Max pausetime target: -XX:MaxGCPauseMillis=

- Throughput target: -XX:GCTimeRatio= i.e. it tries to spend 1/(1+N) percentage in GC. By default is 99 i.e. i/(1+99) = 1% in GC , 99% leaving it to app

- Footprint : is basically the –Xmx

- parallel collector tries to meet max pausetime first then throughput then footprint targets. It plays around growth/shrink percentages of generations to achieve these targets.

Mostly concurrent collectors:

CMS: if the importance is faster RT i.e. lesser pause times and can afford CPU

- -XX:+UseConcMarkSweepGC.

- reduces pause time by always keeping few threads to concurrently (i.e. without pausing the threads) to mark the reachable & unreachable objects and sweeping of unreachable objects but pauses during a major gc during movement of references. It always tries to keep the tenured space clean to avoid longer pauses or accumulation of objects. When it fails keep up the tenured space free then a full collection happens with all the threads paused (a failure case)

- In CMS, the tenured and young can happen independently as they have different threads running in concurrent to the application.

- CMS output is a bit different to other GC outputs. -verbose:gc and -XX:+PrintGCDetails (-XX:+PrintAllGCDetails to print more details). It has below output

CMS-initial-mark indicates the start of the concurrent collection cycle

CMS-concurrent-mark indicates the end of the concurrent marking phase

CMS-concurrent-preclean

CMS-remark

CMS-concurrent-sweep marks the end of the conc sweeping phase

CMS-concurrent-reset (getting ready for next collection)

- CMS and Parallel does compactions for the whole heap when there is no consecutive space available

G1: for large heaps and pause times targets can be met at higher probability and also achieving high throughput (i.e. more time given to app)

- heap will be partitioned into a set of equally sized heap regions, each a contiguous range of virtual memory. the algo performs a concurrent global marking phase to determine the live objects of the heap. After the marking phase completes, it collects the mostly empty regions first, to yield a large amount of free space. So it is called Garbage-First.

- This always works to reduce fragmentation by compacting during collections.

- G1 is beneficial when there is large amount of live data i.e. 50% of heap, allocation rate changes, when there is long collection time or if there is long compaction times

Default configuration on server class machines

On server-class machines, the following are selected by default if not specified otherwise.

Throughput garbage collector

Initial heap size of 1/64 of physical memory up to 1 GB

Maximum heap size of 1/4 of physical memory up to 1 GB

Server runtime compiler

And, to view the default configurations use java -XX:+PrintFlagsFinal -version

JMeter Solr Banana

noreply@blogger.com (sridhar) — Sun, 2 Oct 2016 12:08:00 -0700

Want to give some colors to JMeter ? We all know JMeter is a great tool and helps load testing the apps.. how about bringing two more awesome tools on to the table - Solr and Banana

With Solr, we can store/index/search large amount of data and Banana is a pretty tool with lot of latest HTML and javascript capabilities to draft some cool graphs to show the trends.

So, how we can use these tools ..

we can do a lot in fact, but to start with,

- Load test results to display the runtime stats - Response times/Transaction throughput/Bytes and the list goes on..
- Monitoring system resources like Memory/CPU/Disk/Load, JVM garbage collection activity etc.,

So, its all sounds interesting ? and do you think we can really build a good monitoring tool ? Well, below are couple of dashboards

Load Test Report:

System Resources

These are few sample dashboards. The actual setup does a lot more.. the remote agents collect the data and pushes them to solr and the dashboard refreshes the stats.

Sizing JVMs and VM Memory

noreply@blogger.com (sridhar) — Sat, 24 Sep 2016 05:50:00 -0700

How to estimate the amount of JVM heap and host memory requirement in complex cases where there are more JVM instances per VM and more apps per JVM and have different usage requirement per app ?

let's see some of the metrics to collect to construct an equation on estimating the sizes.

list the apps per instance. Ideally there will be multiple apps deployed on one instance of JVM and all these apps are not equally used and each app will have its own characteristics thus have uneven memory requirements i.e. based on req. classes, code logic and constructs etc.,

And identify the 'typical' transactions.. ideally, in any app, 20% of features are executed 80% of times

Take a first measurement - server start up heap. Once the server is up completely, get the first point after a full GC (hotspot) or first OC (jrockit)

Now, ideally the first user access is demanding one as it loads lot of stuff. So, do just logins into each of the app and take a note of OCs/FGCs for each user. So, per app, the numbers gives the amount of heap required for first login

So, XMS - to make it good enough, it can be sum of (server startup heap + sum of all above deltas )

Now, exercise the typical 20% transaction as single user per app and do not logout - so these are typical active users on the system - call these deltas as h1, h2 etc . This can be on a warmed up server but best is to have no other logged in users in the system. Each delta can be noted or take the final delta after all different app users are in and executed the typical flows but not logged out.

XMX i.e. the max amount of memory can be XMS + ( total delta * (%app1 conc. users) + ... )

for hotspot jvms or other jvms where there are more heap spaces like young/survivor, perm etc., they can be calculated as proportionate ratio of XMX ..

And, typical host memory usage requirement can be calculated as approx 1.8 times of XMX

If someone can simulate load and want to bare the cost, time and complexity of all setup to do that then it is probably best to get the numbers based on test results.., the benefits in doing so can be not just limited to JVM heap but gives the other resource pools across the layers..

Mobiles - The way they transformed made things reachable to some but created complexity to others

noreply@blogger.com (sridhar) — Wed, 14 Sep 2016 06:57:00 -0700

I am talking about the quality checks for the applications built for mobile devices. Mobile devices transformation has to be considered as the fastest change in technology so as the complexity in delivering applications on this new platform which is becoming a must !

Besides making sure the features work i.e. testing the functionality, it is the most important task to ensure the app actually perform as expected. someone can say what is there to think about as it is a light weight code .. there is infact a lot that could disrupt the performance - examples - have to support different vendors like android/APPLE, version of operating systems, underlying hardware, resolutions, GPU, CPUs, memory, network carriers and subscribed bandwidths like 2G/3G/4G, inter-apps interruptions, backend thread support for async calls .. what not !

Well there is a lot, I can think of how one can approach to validate performance of the mobile applications and the upstream servers by using some of the great tools in the market.

Below are some of the tasks to evaluate performance and what these tools can do..

Test and monitor the app's performance on different emulators. If it has to be done independently then each IDE is needed like android studio, IOS IDE and respective skill set as well. Perfecto addresses this with its support to multiple devices

Test the app's performance on different real devices- can't do it manually.. can not have a farm of devices.. perfecto can do it using its mobile device cloud.. or its emulator is good enough as well

Simulate apps performance under different network speeds on devices/emulators - can do by using various softwares like act but again perfecto along with shunra integration can manage devices by emulating various network bandwidths

Capture the traffic and simulate 1000's of devices load to app servers - this can be done using emulators and tcp capture.. but perfecto-shunra can do and the captured traffic can be fed to Loadrunner. And, Shunra can virtualize networks to emulate load from different locations/carriers/bandwidths during load tests

At the same time one would have to recheck the app's performance on the device while server under load - again by perfecto

Its a different world now with mobiles/tablets/wearables that move .. Apps need to support all these !

Gone are those days where apps are accessed from a standalone PC ..

End to end request processing time

noreply@blogger.com (sridhar) — Wed, 24 Aug 2016 05:18:00 -0700

In a simplistic view, in a typical online system, this is where one needs to check to know any slowness

Browser rendering time - if the page size too big or too complex with java scripts and style sheets
then the rendering time could take time..
# of static content being used on the web page has an impact on the overall load time. And, if they are not cacheable, the round trips even from a CDN could have an impact on the load time
Total download time for a page depends on how big is the response content and how many susbrequests are triggered part of the page and the network time and entire server side processing time
Page load time is the time by when user sees the page.. so above all have impact on it.

coming to network, it is important to know what is the path the request is traversing from client to the server. Is it taking longest path via CDN or how the addresses being resolved over public internet and any proxy being used etc., the network delay and packet loss are important factors to keep an eye on.

coming to server side, the request processing time depends on many factors.. it depends on underlying infrastructure, resources, architecture, code logic implementations etc.,
but there are few check points to look at to break it down..
checking at http server layer tells the variation between end page load time and total server time. This helps identify any network delays.
checking the difference between http server to app server times tells if there is any delay in the middle layers like authentication routing..
on the app server side, it could spend time in many places.. the routing can happen to many servers or even to external systems. using runtime instrumentation tools, it is possible to break down the time spent in pure code, wait times due to synchronous code blocking, gc pause times, time spent in reading from sockets/wirte to while interacting with DB or in the calls to other servers while making remote calls like rjvm to EJBs or service calls etc., by breaking this way, each underlying activity and the delays can be identified

Each of the above ones are not more than just an index to an ocean of tuneable metrics that each underlying technology modules contains. However, where to look at and what to tune is the key.

Application performance - odd bits

noreply@blogger.com (sridhar) — Wed, 24 Aug 2016 03:51:00 -0700

we do normally warm up the systems and concerned about the how it performs under load but will it meet customer expectation or improve customer experience or avoid frustrating scenarios ?
how about the 'first time' and idle case performances ?

did you ever check what is the single user response times and request processing times on server side ? if it is not performing any good for single user then it will not do for multiple. The base performance for single user is what needs to tuned first

what about first time cases ? we can not ask users of the system to hang on until system warms up ! then who will use it first :) so, analysing cold case system performance is important. it is important to know what happens on first access across the layers in cold cases i.e. after a system restart, app servers or entire VMs or DB etc., and what needs to be tweaked to get better first time performance.

how about sleeping systems ! not all systems work round the clock atleast not all days in a week.. so did you ever check what is the resource usage of idle systems ? there could be some unexpected code path executions might happen even under no load case. So it is important to know the resources usage under no load as they might also run all the time.

how about apps performance under a very slow network access or networks with high packet loss. how reliable are the systems..

what happens when infrastructure fails ? how many users can still continue doing what ever they are doing with out any interruption and feel the same performance of the app while a DR process kicks in..

how about the systems with long running sessions.. users may not logout for long time and it is required to keep the sessions so long and what is the impact !

how the applications handle when majority of end users do not logout but just closes the browser ! how to handle the memory usage in those cases..

Java EE - EJB

noreply@blogger.com (sridhar) — Sat, 20 Aug 2016 03:39:00 -0700

Enterprise Java Beans - EJB

EJBs are the Java EE server side components which implement app's business logic. They normally be deployed in EJB containers provided by the app servers. Implementing EJBs can provide scalability to the application and better handling of security and transactions. EJBs can also implement webservices.

Types of EJB:
Session - To implement user actions
- stateful: maintains a state for the client and this bean can not be shared.
- stateless: does not maintain state so can be allocated to any clients. They are reusable. So this
pool will be less compared to stateful.
- singletone session beans: have only one instance for the whole time. They can get initiated when the app starts. Although they act as stateless but there is no pool because of single existence

Message Driven Beans (MDB): for listening to the messages either from queues or JMS
(if someone read about entity beans, they are now part of persistence API..(referring to Java EE7)

Implementation is simple.. infact, the annotations made it so..lets says to write a simple stateless EJB, just annotate the class with @Stateless and implement the business logic.. and to call this EJB, a servlet can do ..ofcourse, you need to annotate with @WebServlet(urlPatterns="/") the url pattern says the context root. And, as usual extend the class with HttpServlet. And, to access the EJB, just annotate with @EJB and then the declare the EJB instance.

However, in a typical EJB implementation, there could be all type of session beans - stateless, stateless bean implementing a service, stateful bean accessed remotely etc.,

A remote interface is required for the beans which allow remote access. This remote business interface defines all the business methods of a bean and annotated with Remote from javax.ejb pacakge and it gets implemented by the session bean. A session bean can be an end point for a web service.

A stateful session bean can have methods annotated with Remove which can be invoked by client to remove the instance.

In the case of singleton bean, the concurrent access from the clients can be controlled in two ways - container managed or bean managed by annotating accordingly. And, the methods must be annotated with the locktype i.e. read or write so that concurrent access can allowed or provided with a synchronous mechanism respectively..

For stateless session beans to implement service end points, they must be annotated with @Webservice and the business methods that are exposed must be annotated with @Webmethods. They can also implement async methods so that clients no need to wait for response from long running methods.

Coming to EJB pools - in weblogic, there is an element called max-beans-in-free-pool in weblogic-ejb-jar.xml. This determines how many EJBs must be made available in free pool. max-beans-in-pool will put a cap on the pool limit. For MDBs, the container will create as many instances required based on the size limited to max-beans-in-free-pool. Default MDB threads are 16 but this can be changed by having custom queue or workmanagers

Oracle VM

noreply@blogger.com (sridhar) — Sun, 14 Aug 2016 04:41:00 -0700

Virtualization is a technique to share the hardware resources among multiple systems or users to achieve optimal usage of resources and reducing costs.

Although virtualization is a generic one conceptually, lets talk on server virtualization. This means a bunch of HW resources like CPUs, Memory, Disks, ports etc., are shared among multiple OSs either of same type or multiple type. So, to achieve this we need someone or something to manage the underlying HW and above running guests (OSs). This 'manager' is what is called a 'Hypervisor'.

There are couple of types of Hypervisors
Native or bare metal hypervisor - this is the software which directly runs on host's hardware to control the hardware and monitor the guests OS. so imagine this something that mediates between guest OS and underlying hardware. Example of
such implementation are Oracle VM, VMware EXXi Xen, Microsoft Hyper-V

The other hypervisor is made to run within a traditional operating system and then guest OSs can run on top of it. Example Oracle VirtualBox (which can be installed on an PC where windows is the base OS but virtual box can then host another guest OS like linux..

Oracle VM Server:

This can be installed on X86 instruction set based platforms with Xen hypervisor (GPU licensed) or on SPARC platforms (which will have its own hypervisor).
In general, the above implementation has their own firmware/hardware, a hypervisor and then a super domain/vm which controls the resource allocation to other guest VMs (also called domains or simply guests)

So, simply Oracle VM server is a collection of hardware (CPU, Mem, Network, IO etc.,), hypervisor (for managing underlying baremetal i.e. the hardware), domains (the VMs with thier own set of OS except Dom0 which a complete linux kernel and manages all the other Domains).

Lets explore some interesting things related to Oracle VM

CPU capacity:
how to determine the cpu capacity on a vm server
xm info is the command to use. for example, as shown below, the number of cpus are 72 which are ideally the threads. There are 2 nodes, 18 cores per socket and 2 threads per core
i.e. 2 * 2 * 18 = 72 threads (0 to 71 total, 0-35 on sock1, 36-71 on sock2)

nr_cpus : 72
nr_nodes : 2
cores_per_socket : 18
threads_per_core : 2

The cpu topology can be viewed by using the commnad xenpm get-cpu-topology
CPU core socket node
CPU0 0 0 0
CPU1 0 0 0
CPU2 1 0 0
CPU3 1 0 0
..
xm info also gives the high level vm details like what bit it supports, what instruction set (like intel x86), number of real cpus, number of nodes, number of sockets, number of threads per core, cpu frequency,
memory, pagesize etc.,
In an hyperthreaded model, each core will run 2 threads instead of one. and this would have counter effects but could improve efficiency..

vCPUs
virtual cpus are the cpus that are assigned to a guest/domu i.e. a virtual machine which runs on a domu can be assigned 10 CPUs which are considered as virtual cpus and the actual bindings to real cpu depends on how they are configured. For example below, vm1 is a virtual machine with the id=1, has 3 vCPUs which are in bind state and mapped to CPUs 3,6 7. This vm1 is configured to have the cpu affinity as 2-35 which is first socket on a 2 socket 72 core machine. so, since there is no absolute binding, the mapping can change in runtime and depends on the availability,
the vcpus can be mapped to any of the real cpus in the range 2-35.

xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
vm1 1 0 3 -b- 5354.1 2-35
vm1 1 1 6 -b- 2312.4 2-35
vm1 1 2 7 -b- 2337.8 2-35

you can pin the CPUs for guest vms runtime but to change any affinity to dom0 requires a reboot.. and dom0 always takes the top priority.
And, it is always good to monitor the real cpu usage from the vm server to check how in an oversubscribed case, the busy vms on the same socket could impact each other..

JNDI - Java Naming and Directory Interface

noreply@blogger.com (sridhar) — Sun, 14 Aug 2016 03:13:00 -0700

JNDI - Java Naming and Directory Interface

what is JNDI - its a naming service
why is it needed - in a distributed enterprise application, there are multiple resources like DB pools or business components like EJBs deployed on the Java EE containers and they need a way to locate. JNDI serves that purpose.

Applications can use annotations to locate the resource. Like datasources which are nothing but database resources, provides connection to database. when the application code refers to a
datasource and invokes JDBC API to getconnection, it gets a physical connection. In case connection pooling is implemented then it gets a handle to pooled connection object instead of direct physical connection.
These connections need to be closed and when closed will go back to the pool. The pool of database connections will give better performance and better connection handling mechanism.

Similarly JNDI mapping can be done to other services like JMS, LDAP etc.,

Below shown are some of the resources on Glassfish server and the JNDI mapping.

Oracle database performance

noreply@blogger.com (sridhar) — Mon, 8 Aug 2016 06:54:00 -0700

Lets explore Oracle database performance aspects on a high level..

some of the key terms to know about:

SGA: Shared Global Area - is basically a collection of memory units or structures used by all the processes on a db instance.
PGA: Program Global Area- is a memory region specific to a process (server process or bg process).
Buffer Cache: is basically a buffer to keep data blocks read from data files. The buffer cache is shared across the users.
Shared pool: basically contains the program data like parsed SQLs, PL/SQL code, data dictionaries etc., and this is accessed almost in every DB operation.

And lets see some of the interesting views
gv$process contains details on the currently active processes which are either on CPU or on latchwait or in spinning on a latch.. also contains PGA details for this process
gv$sgastat contains details on the system global area (SGA). For each pool - shared/large/java/stream pools
gv$session contains details for each current session. data from this view is sampled every sec and put into V$ACTIVE_SESSION_HISTORY. From 11G Rel 2 onwards, each individual req can be traced with the help of ECID.
gv$pgastat contains details on PGA usage

we can take a periodic snapshots on the above tables to analyze further..
example: CREATE SNAPSHOT snapshotonprocess AS SELECT * FROM gv$process
and then the processes pga can be calculated.. like select inst_id, count(*) cnt, Round(sum(PGA_USED_MEM) / 1024 / 1024 / 1024, 2) from snapshotonprocess group by inst_id order by inst_id
- adding a criteria like 'background is not null' will give pga stats for background processes

similarly, we can query sga stats, shared pools, shared pool usage, buffer cache usage and active sessions etc.,

Another good place to look at for session performance and to point out the slow SQLs or event waits is by querying V$ACTIVE_SESSION_HISTORY.
V$ACTIVE_SESSION_HISTORY contains sampled session activity in the database. samples are taken every sec. So, it is possible to calculate howmuch time spent on DB and on which queries for a end user request by tracing the ECIDs in this view.

Lets look at awesome reports by Oracle database on performance of the database.

AWRs- A great performance report on DB workloads. It contains information about DB time, system resource usage, waits or other events that could impact performance, SQL statistics like long running SQLs or resource intensive sqls or
SQLs with high buffer gets.
we can check how the buffer cache and shared pool size changed from the beginning to the end of the snapshots.
logical reads(the more the better), physical reads(the less the better), hardparses(the more over a warmedup system indicates the plans are not good perhaps and might need to gather stats or something attached to the queries in runtime), rollbacks etc.,
latches(a short lived serialization methods that oracle uses on shared structures) efficiency metrics, top 5 foreground events, CPU and memory stats are good place to start with if there is a overall db performance hit.
If the performance is specific to SQLs then other areas to look at in AWRs are related SQL performance - # of execs, time taken per exec, cpu used, buffer gets, hard parses etc., And, up on identifying the SQLs (best way is to match the ECIDs for the frontend initiated SQLs),
SQLHC is next step to analyze further on the historical performance and the to analyze the SQL execution plan, indexes, bind variables/conditions etc.,

To have a better insight, keep a good baseline AWR in the system so that any future snapshot can be compared against it..

To create a awr snap:
EXEC DBMS_WORKLOAD_REPOSITORY.create_snapshot;
select snap_id from dba_hist_snapshot order by snap_id asc;

To generate AWR: select output from table(dbms_workload_repository.awr_report_html(,,,));

To declare the basline:
BEGIN DBMS_WORKLOAD_REPOSITORY.create_baseline ( start_snap_id => <###>, end_snap_id => <###>, baseline_name => 'baseline'); END;
/

To compare AWRs:
@$ORACLE_HOME/rdbms/admin/awrddrpi.sql
or
select * from TABLE(DBMS_WORKLOAD_REPOSITORY.awr_diff_report_html(,,,,,,,));

Other good reports to look at:
ADDM report - an Automatic Db Disgnostic Monitoring report.. helps identifying the issues in an Active Workload Repository

Weblogic MBean Monitoring

noreply@blogger.com (sridhar) — Sat, 16 Jul 2016 11:43:00 -0700

Weblogic provides a nice feature to monitor the server resources i.e. through MBeans.

weblogic mbeans types - configuration MBeans, Runtime MBeans and Application Defined MBeans

These can be browsed from EM console - go to admin server EM url for the domain which you are interested. Navigate to the server under the domain from EM console and then menu - System MBean Browser

you can see the above 3 types of MBeans tree structure. Normally for runtime stats, we check Runtime MBeans. say com.bea - select a server - ServerRuntime - and select a type of MBean to browse and select the server

Example MBean for JDBC Data Sources

MBean Name com.bea:ServerRuntime=Server_1,Name=Server_1,Location=Server_1,Type=JDBCServiceRuntime
There are many attributes for this MBean and interesting one to monitor the resource usage is 'JDBCDataSourceRuntimeMBeans'
This attribute will list all the data sources configured for the server - click on a datasource to see its current usage

This is same as what we monitor from Admin console - datasources - monitoring ..

The same can also be done via a script using python/wlst

------------------------------------------------------------------------------------
def getConnectionPoolStat():
connect('adminUserName', 'adminPassword', 'adminURL')
name=cmo.getName()
domainRuntime()
cd('ServerRuntimes')
srvrs=domainRuntimeService.getServerRuntimes()
crntdr=pwd()
for srv in srvrs:
srvName=srv.getName()
cd(crntdr)
print srvName
cd(srvName +'/JDBCServiceRuntime/' + srvName)
allDS=cmo.getJDBCDataSourceRuntimeMBeans()
cd('JDBCDataSourceRuntimeMBeans')
for ds in allDS:
dsName = ds.getName()
cd(dsName)
lcc=cmo.getLeakedConnectionCount()
ccy=cmo.getCurrCapacity()
accc=cmo.getActiveConnectionsCurrentCount()
frrc=cmo.getFailedReserveRequestCount()
ctc=cmo.getConnectionsTotalCount()
acavg=cmo.getActiveConnectionsAverageCount()
cchc=cmo.getCurrCapacityHighCount()
achc=cmo.getActiveConnectionsHighCount()
ftrc=cmo.getFailuresToReconnectCount()
st=name + ',' + dsName + ',' + srvName + ',' + str(acavg) + ',' + str(accc) + ',' + str(achc) + ',' + str(ctc) + ',' + str(ccy)+','+str(cchc)+ ',' + str(lcc) +',' + str(frrc)+','+ str(ftrc) + '\n'
fo.write(st)
cd('..')

fo=open("outputcsvfile","w+")
fo.write('DomainName , DataSourceName, ServerName, Active Connections Avg Count, Active connections current count, Active Connections high count, Connections Total Count, Current Capacity , Current High Capacity, Leaked Connection Count, Failed ReserveRequest Count, FailuresToReconnectCount \n')
getConnectionPoolStat()
fo.close()

This needs to be executed from wlst in oracle common home
sessions per app can be collected through weblogic_j2eeserver:app_session..

Java Web Services

noreply@blogger.com (sridhar) — Fri, 11 Dec 2015 04:41:00 -0800

Web services - in my simplistic view, these are the software components exposed over web (HTTP) as a service so various other programs or software modules can be able to interact. For easier understanding purpose, it is similar to how a web application deployed on an application server exposes its functions through a UI that a client program like internet browsers can access and display it. Web service even though communicate over HTTP, the exchange of information happens in an XML document with certain rules - called as SOAP (Simple Object Access Protocol)

Java provides an API to develop these components. This is called JAX-WS (Java API for XML Web Services).
And these can be implemented in 2 ways - RPC (remote procedure call) and in a message oriented i.e. in a document style

As this API makes it so simple, we just need below components to develop a simple webservice using JAX-WS

Lets see an RPC style implementation

1. Need a remote i.e. endpoint interface class
2. A remote implementation class
3. A remote publisher class
4. And a client

Below is the self explainable simple code logic

----------------
package com.myWSPackage;
import javax.jws.WebMethod;
import javax.jws.WebService;
import javax.jws.soap.SOAPBinding;
import javax.jws.soap.SOAPBinding.Style;

@WebService
@SOAPBinding(style = Style.RPC)
public interface IamJustAnInterface{
@WebMethod String implementMe(String name);
}
----------------
package com.myWSPackage;
import javax.jws.WebService;

@WebService(endpointInterface = "com.myWSPackage.IamJustAnInterface")
public class IamAnImplementer implements IamJustAnInterface{
@Override
public String implementMe(String name) {
try{Thread.sleep(10000);}catch(Exception e){}
return "Hi There - at your service : " + name;
}
}
-------------------
package com.myWSPackage;
import javax.xml.ws.Endpoint;

public class IamAPublisher{

public static void main(String[] args) {
Endpoint.publish("http://localhost:7777/ws/hi", new IamAnImplementer());
}
}
---------------
package com.myWSPackage;
import java.net.URL;
import javax.xml.namespace.QName;
import javax.xml.ws.Service;

public class WSClient{
public static void main(String[] args) throws Exception {
URL url = new URL("http://localhost:7777/ws/hi?wsdl");
QName qname = new QName("http://myWSPackage.com/", "IamAnImplementerService");
Service service = Service.create(url, qname);
IamJustAnInterface ifi = service.getPort(IamJustAnInterface.class);
System.out.println(ifi.implementMe("Buddy"));
}
}
---------------------------------------------------

what happens on the client:
I have intentionally put a sleep in the service interface implementation class to capture the stacks

SocketNativeIO.readBytesPinned(FileDescriptor, byte[], int, int, int)
SocketNativeIO.socketRead(FileDescriptor, byte[], int, int, int) line: 32
SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)
SocketInputStream.read(byte[], int, int) line: 129
BufferedInputStream.fill() line: 218
BufferedInputStream.read1(byte[], int, int) line: 258
BufferedInputStream.read(byte[], int, int) line: 317
HttpClient.parseHTTPHeader(MessageHeader, ProgressSource, HttpURLConnection) line: 690
HttpClient.parseHTTP(MessageHeader, ProgressSource, HttpURLConnection) line: 633
HttpURLConnection.getInputStream() line: 1195
HttpURLConnection.getResponseCode() line: 379
HttpClientTransport.readResponseCodeAndMessage() line: 198
HttpTransportPipe.process(Packet) line: 151
HttpTransportPipe.processRequest(Packet) line: 83
DeferredTransportPipe.processRequest(Packet) line: 78
Fiber.__doRun(Tube) line: 587
Fiber._doRun(Tube) line: 546
Fiber.doRun(Tube) line: 531
Fiber.runSync(Tube, Packet) line: 428
Stub.process(Packet, RequestContext, ResponseContextReceiver) line: 211
SEIStub.doProcess(Packet, RequestContext, ResponseContextReceiver) line: 124
SyncMethodHandler.invoke(Object, Object[], RequestContext, ResponseContextReceiver) line: 98
SyncMethodHandler.invoke(Object, Object[]) line: 78
SEIStub.invoke(Object, Method, Object[]) line: 107 ----> proxy and stub creation has been taken care by JDK
$Proxy19.implementMe(String) ---------> is the remote call to the webservice's method
WSClient.main(String[]) line: 15 ---------> is our client main method

what happens on the server side

Thread.sleep(long) ------> just sleeps for 10 sec before it sends out the response
IamAnImplementer.implementMe(String) line: 11 ---> call to the interface method which was exposed as a service
NativeMethodAccessorImpl.invoke0(Method, Object, Object[])
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object[]) line: 597
InstanceResolver$1.invoke(Packet, Method, Object[]) line: 235
InvokerTube$2.invoke(Packet, Method, Object[]) line: 135
EndpointMethodHandler.invoke(Packet) line: 246
SEIInvokerTube.processRequest(Packet) line: 82
Fiber.__doRun(Tube) line: 587
Fiber._doRun(Tube) line: 546
Fiber.doRun(Tube) line: 531
Fiber.runSync(Tube, Packet) line: 428
WSEndpointImpl$2.process(Packet, WebServiceContextDelegate, TransportBackChannel) line: 232
HttpAdapter$HttpToolkit.handle(WSHTTPConnection) line: 460
HttpAdapter.handle(WSHTTPConnection) line: 233
WSHttpHandler.handleExchange(HttpExchange) line: 95
WSHttpHandler.handle(HttpExchange) line: 80
Filter$Chain.doFilter(HttpExchange) line: 65
AuthFilter.doFilter(HttpExchange, Filter$Chain) line: 65
Filter$Chain.doFilter(HttpExchange) line: 68
ServerImpl$Exchange$LinkHandler.handle(HttpExchange) line: 557
Filter$Chain.doFilter(HttpExchange) line: 65
ServerImpl$Exchange.run() line: 529
ThreadPoolExecutor$Worker.runTask(Runnable) line: 886
ThreadPoolExecutor$Worker.run() line: 908
Thread.run() line: 662 ----> the publisher start a thread starts

The document style implementation is all the same way as above except the style declared in the remote
interface class will change to DOCUMENT.

The WSDL will be different for each of these implementations although there seems to be no change in the execution path i.e. stack traces shown above.

In Java EE7, there is support to implement RESTful webservices through the API called JAX-RS.
These are best suited when there is no need for state maintenance, can rely on underlying server caching mechanism and the content will not change dynamically and for light weight message exchanges like handsets, portable communication devices etc,. The RESTful services do not require XML messages or WSDL based API implementations.. They depend/implement the REST verbs like PUT/GET/POST/DELETE.. the implementation is very simple - a root reosource class annotated with the @PATH tells where the implemented java class is located and its methods annotated with the verbs like @GET, @Producer etc., tells which type of request handling will be done, what kind of text will be produced and the output etc., the servlet mapping defined the application's web.xml will tell the resource path and the mapped classes ro invoke.

Java String, StringBuilder and StringBuffer

noreply@blogger.com (sridhar) — Wed, 25 Nov 2015 02:51:00 -0800

What is so different between String, StringBuffer, StringBuilder ?

Strings are simple, constant size and represents a series of characters.
Since they are constants, strings can not be changed and you can simply exchange them.
StringBuilders make them a bit flexible and changeable. While StringBuilder and StringBuffer provides some similar features, StringBuffer is better when it is multi-threaded usage as the methods are synchronized.

A simple example:

public class StringAndStringBuilder {
public static void changeString(StringBuilder sb){
sb.delete(0, sb.length());
sb.append("changed");
}
public static void changeString(String s){
s="changed";

}

public static void main(String[] args) {
StringBuilder sb=new StringBuilder("nothing");
changeString(sb);
System.out.println("value after change via stringbuilder "+ sb);
String s = new String("nothing");
changeString(s);
System.out.println("value after change via string "+ s);

}
}
output:
value after change via stringbuilder changed
value after change via string nothing

Java Synchronization

noreply@blogger.com (sridhar) — Wed, 25 Nov 2015 02:13:00 -0800

Lets see how synchronization works in Java. Below is a sample program which has three parts
- TestSynchronization is a the main class
- TestThread is a private class extends Thread.
- IamSynchronized is one more private class which has synchronized code logic. one part is just a synchronous block and the other is a synchronous method.

public class TestSynchronization {
public static void main(String args[]) {

IamSynchronized IS = new IamSynchronized();
TestThread T1 = new TestThread( "Thread - 1 ", IS );
TestThread T2 = new TestThread( "Thread - 2 ", IS );
try{Thread.sleep(60000);}catch(Exception e){}
T1.start();
T2.start();
}
}

class TestThread extends Thread {
private Thread t;
private String threadName;
IamSynchronized IS ;
TestThread(String name, IamSynchronized is){
threadName = name;
IS = is;
}
public void run() {
IS.justIterate();
System.out.println("calling sync method - Thread: "+threadName);
IS.syncMethod();
System.out.println("Thread " +threadName + " done");
}

public void start ()
{
System.out.println("Starting " + threadName );
if (t == null)
{ t = new Thread (this, threadName);
t.start ();
}
}

}

class IamSynchronized {
public void justIterate(){
synchronized(this){
try {
for(int i = 5; i > 0; i--) {
System.out.println("Iteration -- " + i );
Thread.sleep(1000);
}} catch (Exception e) { System.out.println("Interrupted"+e); }
}
}
public synchronized void syncMethod(){
{
try {
for(int i = 5; i > 0; i--) {
System.out.println("Iteration -- " + i );
Thread.sleep(1000);
}
} catch (Exception e) {
System.out.println("Interrupted"+e);
}
}
}
}

Lets see how the locking works and how the control flow happens when 2 parallel threads trying to execute the same logic.

Starting Thread - 1
Starting Thread - 2
I am in the non synchronous part of justIterate method - Thread Thread - 1
I am Thread Thread - 1 in sync block Iteration -- 5
I am in the non synchronous part of justIterate method - Thread Thread - 2 -- THREAD 2 IS BLOCKED BY THREAD1
I am Thread Thread - 1 in sync block Iteration -- 4
I am Thread Thread - 1 in sync block Iteration -- 3
I am Thread Thread - 1 in sync block Iteration -- 2
I am Thread Thread - 1 in sync block Iteration -- 1
I am Thread Thread - 2 in sync block Iteration -- 5
calling sync method - Thread: Thread - 1 -- THREAD 1 IS BLOCKED BY THREAD2
I am Thread Thread - 2 in sync block Iteration -- 4
I am Thread Thread - 2 in sync block Iteration -- 3
I am Thread Thread - 2 in sync block Iteration -- 2
I am Thread Thread - 2 in sync block Iteration -- 1
calling sync method - Thread: Thread - 2 -- THREAD 2 IS AGAIN BLOCKED BY THREAD1 BUT IN SYNC MTHOD
I am Thread Thread - 1 in sync method Iteration -- 10
I am Thread Thread - 1 in sync method Iteration -- 9
I am Thread Thread - 1 in sync method Iteration -- 8
I am Thread Thread - 1 in sync method Iteration -- 7
I am Thread Thread - 1 in sync method Iteration -- 6
Thread Thread - 1 done
I am Thread Thread - 2 in sync method Iteration -- 10 -- SINCE THREAD1 RELEASES LOCK, THREAD2 CONTINUES
I am Thread Thread - 2 in sync method Iteration -- 9
I am Thread Thread - 2 in sync method Iteration -- 8
I am Thread Thread - 2 in sync method Iteration -- 7
I am Thread Thread - 2 in sync method Iteration -- 6
Thread Thread - 2 done

Below stack trace gives an idea on how the locks being handled by the JVM

"Thread - 2 " prio=6 tid=0x000000000b550800 nid=0x2478 waiting for monitor entry [0x000000000d48f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at IamSynchronized.justIterate(TestSynchronization.java:47)
- waiting to lock <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:24)
at java.lang.Thread.run(Thread.java:722)

"Thread - 1 " prio=6 tid=0x000000000b550000 nid=0x18c4 waiting on condition [0x000000000d17f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at IamSynchronized.justIterate(TestSynchronization.java:49)
- locked <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:24)
at java.lang.Thread.run(Thread.java:722)

--------------------------------------------------

"Thread - 2 " prio=6 tid=0x000000000b550800 nid=0x2478 waiting on condition [0x000000000d48f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at IamSynchronized.justIterate(TestSynchronization.java:49)
- locked <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:24)
at java.lang.Thread.run(Thread.java:722)

"Thread - 1 " prio=6 tid=0x000000000b550000 nid=0x18c4 waiting for monitor entry [0x000000000d17f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at IamSynchronized.syncMethod(TestSynchronization.java:56)
- waiting to lock <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:26)
at java.lang.Thread.run(Thread.java:722)

--------------------------------------------------

"Thread - 2 " prio=6 tid=0x000000000b550800 nid=0x2478 waiting for monitor entry [0x000000000d48f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at IamSynchronized.syncMethod(TestSynchronization.java:56)
- waiting to lock <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:26)
at java.lang.Thread.run(Thread.java:722)

"Thread - 1 " prio=6 tid=0x000000000b550000 nid=0x18c4 waiting on condition [0x000000000d17f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at IamSynchronized.syncMethod(TestSynchronization.java:58)
- locked <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:26)
at java.lang.Thread.run(Thread.java:722)

--------------------------------------------------

"Thread - 2 " prio=6 tid=0x000000000b550800 nid=0x2478 waiting on condition [0x000000000d48f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at IamSynchronized.syncMethod(TestSynchronization.java:58)
- locked <0x00000007d5fbab18> (a IamSynchronized)
at TestThread.run(TestSynchronization.java:26)
at java.lang.Thread.run(Thread.java:722)

Eclipse MAT - Analyzing heap dumps

noreply@blogger.com (sridhar) — Tue, 13 Oct 2015 08:52:00 -0700

Eclipse Memory Analyzer Tool is one of the best tools to analyze JVM heap dumps and to understand the memory allocations, memory leaks etc., Below listed are kind of pointers to what MAT gives and what they actually mean in analyzing the heap dump

The Overview report is the best one to start. It gives
size of your heap dump - which could be very less than the XMX or in most of the cases a bit larger than the XMX. The first case examples are - OOM thrown not in the heap but other areas like perm space, TLA or may be it just could not allocate a large char array say gt. 2GB while there is still lot of free space or sometimes not able to dump large collection of objects which are pending with finalizer thread.

Classes: number of classes loaded

Objects: number of objects loaded

Class Loaders: number of class loaders

Unreachable objects histogram: these are normally removed during analysis but can be set not to by using preference 'keep_unreachable_objects'

The overview report also gives pointers on where to look..

Histogram - gives the number of instances per class.
you can obtain the shallow & retained sizes (shallow heap is the mem. occupied by one object of a class. retained heap is the sum of shallow sizes of all the objects of a class)
you can calculate the approx. or precise retained size of a class or package. group the histogram view at package level or a class loader..
you can also explore the top packages or class loaders by going through outgoing references. outgoing references are basically the tree nodes or leaves that a specific object refers to. Incoming references of an object are basically all the links i.e. references that kept the object alive

Dominator tree - is an object graph showing all the dominators list. it is basically showing all the 'dominating' links of an object tree.

top consumers - it is similar to grouping the histogram view by packages. But nice report to know which packages are the top consumers of the memory. So that we can further explore which classes of the top consuming packages are being instantiated or retaining more space and what are their immediate dominators or gc roots

duplicate classes - gives the list of duplicate classes loaded by different class loaders. If your perm gen space is filling up then probably this view will give good indication on what classes are being loaded and duplicate list.

In addition to above, there are 2 good reports given by MAT

Leak Suspects:It points out what are the biggest objects occupying the memory. They may or may not be the leaks but worth to begin the investigation with. It tells how many instances and memory consumed by each instance. you can also get the gc root information like which gc root like a thread, class loader etc., have instantiated the objects. (This can be obtained from leak hunter report.) It also gives system & thread overview, top consumers, class histograms.

Top Components: use this if there are no heavy objects found by leak suspects. This report will give detailed analysis for all the objects occupying gt. 1% of heap.

Finalizers: this is one more important feature to see what the finalizers are doing. Finalize methods will be executed to free up the object. sometimes if there a long running code in finalize method or a lock in the finalize method could delay in cleaning up the heap. This could result in large accumulation of objects pending for finalization. in MAT, go to Java Basics -> Finalizer Overivew to view these stats. If Finalizer Overview does not show any data then check the thread which is doing finalization like 'Finalizer thread' in weblogic and check how much space it has retained. If that does not show any significant memory held and the heap dump is not the same size as reported in verbose GC logs just before the dump then live monitoring of class histogram is the option. It could be that there are huge number of objects loaded and not live and waiting to get finalized.

you can also query the heap using OQL, analyze threads to see thread stacks, URLs, thread states etc., java collection usage,

Knowing all the above, A quick way to analyze the heap dump is take a look at the leak suspects after getting a high level idea from overview report. see if there are any large collection of an objects is being loaded and by which class loader. now get the path to gc roots (or merge shortest path option for top instances by retained size) from histogram if not given in the suspects report, to find out who is/are holding these objects in the memory. once we know who is holding, now see what the object contains i.e. the content. sometimes there will be a single object holding large amount of memory like some big xml report generation or there could be large collection of small objects occupying the memory. In the second case see why so many instances rather than the size of each instance.