Performancing Metrics

Vamsi Tokala's blog

Tuesday, June 18, 2013

Synthetic monitoring

I have recently published a document on synthetic monitoring. Abstract is given below

Abstract:

Many large organizations have started implementing web content management (WCM) platform to maximize their online presence and improve operational efficiency across web based initiatives.  These platforms have to support a large number of users accessing the application from multiple countries/regions with optimum performance. Platforms also have to achieve high availability for the published websites and its content management platform.

IT operations team responsible for managing the digital platform  has to ensure the performance and availability requirements are met and take corrective actions if required.   It must have the process which can alert the team with a performance issue as and when it is detected, understand the severity and complexity of the issue and diagnosing the issue with the available information.

Synthetic monitoring can help IT operations for managing and optimizing web application availability, performance of applications from the end-user perspective. This paper highlights the approach used to implement synthetic monitoring for a WCM platform and lessons learned from its implementation.

https://docs.google.com/file/d/0BzOftuUbGkRvOUYxWU1iMmVGMmM/edit?usp=sharing

@2013, copyright Vamsidhar Tokala

Friday, September 14, 2012

Akamai Configuration

I have done my first akamai CDN implimentation  for  CQ5 publish in my project.   I have listed the prerequisites and the configuration I have implimented.


DNS Setup

DNS Domain has to be configured to set-up with Akamai and for the domain to work its need to be registered and mapped to below name servers. This will allow Domain configurations to work

Following URL is configured for Akamai testing:

http://XXX-performance-testing.utils-XXX.com:4503/


Note: There is another workaround i.e. to register a temp domain via http://dnsdynamic.org
We can register a doming e.g. XXX-performance-testing.dnsdynamic.com and map it to IP: XXX.XXX.XXX.XXX

Akamai Configuration


Akamai is configured in the following fashion for optimal performance



ORIGIN SERVER CONFIGURATION

DIGITAL PROPERTIES CONFIGURATION SUMMARY

Default Configuration

(All Digital Properties) Origin: origin-nexus-performance-testing.utils-XXX.com

CP Code: Site Accelarator2 (125924)

Forward HOST header: Digital Property

Cache Key: Origin Server (origin-nexus-performance-testing.utils-XXX.com)

Compression: Yes



CACHE KEY: IGNORE CASE

OPTION VALUE

Ignore case in the cache key: Enabled



LAST MILE ACCELERATION

OPTION VALUE

Apply to content with Content-type: text/html*, application/x-javascript*, text/css*, image/png image/jpeg image/gif

Disable for Old Browsers: Yes



HTTP POST

OPTION VALUE

HTTP POST Requests: Enabled

Allow POST Without Content Length: Disabled



ORIGIN DNS OPTIMIZATION

OPTION VALUE

DNS Refresh: Enabled

DNS Refresh Timeout: 2 hours



TIME TO LIVE RULES

# MATCH CRITERIA ACTION

Default Rule TTL: 1 day

Default Error Response Rule 10 seconds

Default Cache Redirect Response Rule Cache Redirect Responses

1 Host: nexus-performance-testing.utils-XXX.com

File Extension: aif aiff au avi bin bmp cab carb cct cdf class css

dcr doc dtd exe flv gcf gff gif grv hdml hqx ico ini

jpeg jpg js mov mp3 nc pct pdf png ppc pws swa swf

txt vbs w32 wav wbmp wml wmlc wmls wmlsc xsd zip

TTL: 1 day



REPORTING

OPTION VALUE

HTTP Headers to Include in Logs: User-Agent

Cookie Values: - None -



SUREROUTE / TIERED DISTRIBUTION OPTIONS

OPTION VALUE

Tiered Distribution: Enabled

Tiered Distribution Map Name: ch2.akamai.net

SureRoute: On

SureRoute Test Object URI: /akamai/sureroute-test-object.html

Race Statistic TTL: 86400 seconds

Force Test Race Protocol to HTTPS: No



SITE ACCELERATOR

OPTION VALUE

Site Accelerator Enabled



GENERATE REDIRECT RULES

# MATCH CRITERIA GENERATE REDIRECT

1 Path: /index.html (Ignore Case On)

Redirect Name: Redirect_geometrixx

Redirect Option: Relative - Change URI

Response Code: 301 - Permanent Redirect

URL: /content/geometrixx/en.html



BROWSER CACHE CONTROL HEADERS

# MATCH CRITERIA CACHE CONTROL HEADERS

Default Rule Pass through the origin's Cache-Control

headers to the browser.

@2011, copyright Vamsidhar Tokala

Wednesday, March 7, 2012

Performance testing IPAD/Mobile applicatins




Recently customers are requesting for performance testing IPAD/Mobile applications and I came across such a requirement recently. The challenge is to simulate the traffic and the typical approach to capture traffic using tools like LoadRunner may not work in this case due to the reasons below.

·         Tools like LoadRunner can’t be installed in IPAD/smart phones to record the flow and capture the communication.

·         Network capturing tools like Wireshark and Fiddler can’t be installed in IPAD.
The approach is to route the traffic from IPAD to a hub/router and capturing the communication using network monitoring tools like wireshark.

1)      Hub takes packets sent from one port and transmits (repeats) them to every other port on the device. For example, if a computer on port 1 of a 4-port hub needs to send data to a computer on port 2, the hub sends those packets to ports 1, 2, 3, and 4.  When computer sends data, all computers connected to the hub receive it.  Connect a computer to the hub which has Wireshark or fiddler installed to capture the communication.

2)      Second approach is to connect to a router and allow traffic to go through the proxy.
The below link has the approach followed by my team to capture communication using wireless router and fiddler
Finally download the BPC toolkit available in my blog to convert fiddler output to LoadRunner script.
 
Note: HP developed some Mobile apps protocol to test Mobile applications which takes Wireshark network trace as an input to create a script. Purchasing license for this protocol is not required if the above approach is followed

@2011, copyright Vamsidhar Tokala

Wednesday, February 1, 2012

Identifying Memory leaks in .Net using winDBG

Identifying Memory Leaks due to object references in Finalize queue


Load the SOS debugger extension for a CLR 4.0 application

.loadby sos clr

1) Identify the objects in finalizeQueue which survived Garbage collection

!fq

SyncBlocks to be cleaned up: 0

MTA Interfaces to be released: 0

STA Interfaces to be released: 0

----------------------------------

generation 0 has 1 finalizable objects (000000008ba63058->000000008ba63060)

generation 1 has 15 finalizable objects (000000008ba62fe0->000000008ba63058)

generation 2 has 14884 finalizable objects (000000008ba45ec0->000000008ba62fe0)

Ready for finalization 0 objects (000000008ba63060->000000008ba63060)



000007ff01d58610 2 1024 System.Data.DataTable

000007ff0164e298 4 1120 System.Diagnostics.Process

000007ff01d82738 8 1728 System.Data.DataColumn

000007ff017235a0 16 1920 System.Threading.OverlappedData

000007ff001f1780 28 2464 System.Threading.Thread

000007ff006764b8 52 3744 System.Reflection.Emit.DynamicResolver

000007ff0026fba8 66 4224 System.Threading.ReaderWriterLock

000007ff01e7bb00 314 10048 System.Data.SqlClient.SNIPacket

000007ff01e717e0 546 21840 System.Data.SqlClient.SNIHandle

000007ff01e49f30 314 32656 System.Data.SqlClient.SqlConnection

000007ff02933bf0 302 45904 System.Data.SqlClient.SqlDataAdapter

000007ff0166d240 398 70048 System.Diagnostics.PerformanceCounter

000007ff01e0d0e8 1510 338240 System.Data.SqlClient.SqlCommand

000007ff0057d8f8 11209 358688 System.WeakReference



2) You can also print the details of the finalizable objects for Gen2 using the above details

dd 000000008ba45ec0 000000008ba62fe0


3) Identify the suspected object having leaks

!dumpheap -type System.Data.SqlClient.SqlCommand

Address MT Size

0000000010c43f40 000007ff01e0d0e8 224

0000000010c44020 000007ff01e0d0e8 224

0000000010c44198 000007ff01e0d0e8 224

0000000010c44278 000007ff01e0d0e8 224

0000000010c444e8 000007ff01e0d0e8 224


4) Identify the GC roots for the objects. It contains the call stack

!gcroot 0000000010c43f40

Scan Thread 15 OSTHread 1bc0

Scan Thread 16 OSTHread 1d74

Scan Thread 19 OSTHread 3200

Scan Thread 17 OSTHread 21f0

Scan Thread 18 OSTHread 3564

Scan Thread 20 OSTHread 322c

Scan Thread 21 OSTHread 2b80

Scan Thread 28 OSTHread 2e5c

Scan Thread 29 OSTHread 35d0

Scan Thread 30 OSTHread 3bc

Scan Thread 31 OSTHread 2770

Scan Thread 33 OSTHread 19b4

Scan Thread 34 OSTHread 534

Scan Thread 35 OSTHread 3280

Scan Thread 36 OSTHread 1908

Scan Thread 37 OSTHread 344c

Scan Thread 38 OSTHread 13d4

Scan Thread 39 OSTHread 21d4

Scan Thread 40 OSTHread 31a4

DOMAIN(0000000001AB88B0):HANDLE(Pinned):1217c0:Root: 000000002070f040(System.Object[])->

00000000108c9ac0(System.Collections.Hashtable+SyncHashtable)->

00000000108c91e8(System.Collections.Hashtable)->

000000001171efc0(System.Collections.Hashtable+bucket[])->

00000000120a3768(System.Collections.Hashtable)->

00000000120a37c0(System.Collections.Hashtable+bucket[])->

00000000120a3820(System.Collections.Hashtable)->

00000000120fa180(System.Collections.Hashtable+bucket[])->

00000000120b7200(System.Collections.Generic.Dictionary`2[[ATOM.AS.CobolBase.CobolProgramName, ATOM.AS.CobolBase],[ATOM.AS.CobolBase.CobolProgram, ATOM.AS.CobolBase]])->

00000000120b7360(System.Collections.Generic.Dictionary`2+Entry[[ATOM.AS.CobolBase.CobolProgramName, ATOM.AS.CobolBase],[ATOM.AS.CobolBase.CobolProgram, ATOM.AS.CobolBase]][])->

00000000120b7258(ATOM.AS.CobolBase.CobolProgram)->

00000000120b7310(System.Collections.Generic.List`1[[ATOM.AS.CobolBase.IProgramEvents, ATOM.AS.CobolBase]])->

0000000019b0f918(System.Object[])->

0000000019b0f590(ATOM.AS.DataAccess.DataAccess)->

0000000019b13c40(ATOM.AS.DataAccess.ProviderSQL)->

0000000019b13d70(System.Data.SqlClient.SqlCommand)->

0000000019b15978(System.Data.SqlClient.SqlCommand+CachedAsyncState)




@2011, copyright Vamsidhar Tokala

Monday, August 29, 2011

Derive Concurent Users using Little's law

The below real time example will help in deriving the concurrent users using Little's law. In the below example

  • Adjusted Hits, Median Think Time derived from production logs for one hour period.
  • Avg RT/Step  is the Target Response SLA required to met under load
  • No of Steps is no of transactions per business flow
  • Concurrent Users is derived using little's law Throughput* (Avg RT+ Think Time)





 


@2011, copyright Vamsidhar Tokala

Friday, June 17, 2011

Performance Tuning Tips using Oracle Indexes


1)   Consider indexing keys that are used frequently in WHERE clauses and join tables and with high selectivity. The selectivity of an index is the percentage of rows in a table with the same value for the indexed key. 
a.   Good Selectivity
A table having 1000 records and one of its indexed column has 800 distinct values, then the selectivity of this index is 800 / 1000 = 0.8
b.   Bad Selectivity
lf an index on a table of 1000 records had only 5 distinct values, then the index's selectivity is 5 / 1000 = 0.005 and in this case a query which uses the limitation of such an index will return 1000 / 5 = 200 records for each distinct value. Full table scan is better than an indexed scan in this scenario

2)   Oracle database does not create an index for foreign key constraint automatically. If foreign key column often used in join conditions then create an index on them to enhance the join process

3)   Consider indexing foreign keys for referential integrity constraints in cases where large number of concurrent inserts updates and delete statements access the parent and client tables. Indexing foreign key columns helps avoid full table scans when searching for the matching rows in the child table when DML is performed on parent table. Without an index in the child table a table level lock may occur

4)   When choosing to index a key, consider whether the performance gain for queries is worth the performance loss of inserts, updates and deletes and use of space required storing the index.

5)   Deleting a row from a table in the oracle database results in the deletion of the index entry.  Updates to the key columns result in a logical deletion and insertion of index.

6)   It is recommended that after periods of high DML activity, you verify index statistics and reorganize indexes
Select used, monitoring,
From v$object_usage where index_name =name of the index

Saturday, April 23, 2011

White Paper - Java RMI Communication with LoadRunner

Java RMI

Java RMI Communication with LoadRunner

       Vamsidhar Tokala


This paper provides tips and tricks for HP LoadRunner JAVA RMI configuration, scripting, and execution. This guide is intended to help testers to understand the concepts of Java RMI and to use HP LoadRunner for Java RMI Scripting using Java Record and Replay protocol.

Introduction


LoadRunner is one of the very few tools available in the market which supports Java RMI protocol.  Information and the expertise available in the public domain is very limited for this protocol and developing test scripts for applications that use RMI involves Java programming, and requires understanding of the Java classes involved in client/server communications.  This article will enable testers with a solid background to address the challenges ahead.

Java RMI


JAVA RMI is distributed object protocol for communication between distributed objects.  It provides infrastructure services such as error, exceptional handling, parameter passing and security context. Client stubs and server skeletons are generated automatically and act as an interface between the application and the rest of the RMI system.

When a client invokes a call to a business object, it needs to get a remote object reference (Stub) to the remote object (skeleton), which intercepts the call and invokes the corresponding method. Remote objects works with container to execute resource management strategies such as instance pooling and provide support for transaction persistence and security

The way client gets a remote reference to an object is by first looking up home interface of the object via JNDI(Java Naming And Directory Interface) and then invoke create method defined in the home interface. If the object client wants to call already exists, then client can invoke one of the find methods defined in the home interface after the JNDI lookup. Server typically provides a naming server for the clients to look up objects deployed on it.

When a client invokes a method call of an object of an application deployed in the server, client actually invokes a corresponding method call on a local proxy (Stub) of the object deployed in server. Local proxy serializes the parameters of the call and sends them to server side proxy (Skeleton). Skeleton collaborates with the container to provide middle ware services such as concurrency control, connection pooling. Once the call is processed, results will be returned to Skeleton, stub and then to client eventually.

RMI Communication with LoadRunner


LoadRunner will act as a client stub during RMI communication with the server. It acts like a proxy of the object on the server side. It sends the method call to the Skeleton, which is present on the server side.

When remote calls from separate clients are placed (executing in different JVMs) each call will run in a separate thread. However, if you make concurrent calls from the same client then calls will execute on the same server thread.

Configure LoadRunner’s Java RMI recorder


In order to allow LoadRunner to capture RMI communication with the application ensure the following is done

1.        Install JDK. JRE alone is not sufficient for LoadRunner to record the RMI communication
2.        Identify the jar files required for the application to run.  Do the following steps to determine the same
·         Manually access the application through browser/batch file.
·         Go to Control Panel – java – Temporary internet files – Settings - Select View applications - Show JNLP Descriptor

3.        Download all the jar files listed in the jnlp file to a local folder from the application server (Refer JNLP section to look at a sample file). Alternatively look in the Java console for all JARs downloaded to the cache, and copy these to a local directory.(When an application lunches from the browser a copy of all the jar will get downloaded to a temporary directory in local machine.)

4.        Get all the application properties and parameters in JNLP file (see below for a sample format)

5.        Create the Java security policy file - d:\all.policy - with contents:

grant {
permission java.security.AllPermission;
};

The java.security.AllPermission is a permission that implies all other permissions. Granting AllPermission should be done with extreme care. Thus, it grants code the ability to run with security disabled. Extreme caution should be taken before granting such permission to code. This permission should be used only during testing.

6.        Configure class path   in the environment variables for all the jar files in the local folder plus
\bin
classes
\classes\srv
\jre\lib\rt.jar

b) Set %PATH% to point to the JDK and verify this with java -version

7.        Create a batch file with the information available from JNLP file as shown below

-Djava.security.policy=d:\all.policy
-DUSE_JAAS="false" -Dclient_master.properties="/retek/client_master.properties"
-Xms256M -Xmx256M
-Djava.naming.provider.url="opmn:ormi://XXX:6007:XXX-rpm/rpm12"
-DNAMING_URL="opmn:ormi://XXX:6007:XXX-rpm/rpm12"
-Djava.naming.factory.initial="oracle.j2ee.rmi.RMIInitialContextFactory"
-DNAMING_FACTORY="oracle.j2ee.rmi.RMIInitialContextFactory"
com.retek.rpm.gui.security.RpmUIClient

8.        Run the batch file in the local machine. It should invoke the application with out any issues.  In case of any issues while lunching the application using batch file, relook at all the steps from 1 to 8.

9.        LoadRunner has predefined hooks for Java RMI called rmi.hooks which is present Dat folder of LoadRunner installation directory.  The LoadRunner recorder uses these hooks to capture all Java classes as they are loaded into the Java Virtual Machine (JVM). When ever a hooks method is called, the same will be recorded into LoadRunner Script.  
To enable hooks, just remove ‘;’ character present is rmi.hooks file located at \dat & \classes directories.

Customized hooks for any classes or methods specific to the application can also be created with a filename called user.hooks and place it under LoadRunner/classes directory. 

10.     Under LoadRunner recording options: Configure the class path as in step 6 and add each of the jar files in the same order as in JNLP file (which is very important) in LoadRunner recording options.

11.     Compile and run the empty Java script with out errors.

12.     Record the application either with the java application by specifying the application parameters defined in the batch file or by selection the application type as batch file. LoadRunner launches the application and captures the methods and classes defined in the hooks file.

Script Parameterization

1.        Typical LoadRunner functions like vuser_init(); vuser_end() will not have any impact in the script. So after the script is recorded move the login part of the application code to init () function and logout to end () and the rest in action () function of the script.

·         init() method will be called once in the beginning of the run
·         action() method will be called once per each iteration
·         end() method will be called once at the end of the run
2.        Parameters defined in the script should assign to  java objects  as shown below for different data types
·         String promotionEventDescription = "";
·         com.retek.platform.util.type.RDate promotionEventStartDate =     new com.retek.platform.util.type.RDate ("", "MM/dd/yyyy"); //application specific date object
·         long departmentId = Long.parseLong("");//long
·         String departmentDisplayId = ""; //string

Serialization and De serialization

During the recording process LoadRunner automatically serializes the objects. Serialized objects are flattened into bytes and subsequently inflated in the future.  LoadRunner saves the objects into binary files with sequentially numbered and saved under LoadRunner Vuser directory. First file will be named as 1 and the second file as 2 and so on...

These classes captured in the script are serialized and deserialized during replay.  The below example code is generated by VuGen

 _object_array83 = new java.lang.Object[] {_clientcontext1, _tslcascadesearchcriteriaimpl1};  // IDLSequence
_string22 = "com.retek.rpm.app.item.service.TslCascadeSearchCriteriaImpl __CURRENT_OBJECT = {" +
  "}";
 _class10 = (java.lang.Class)lr.deserialize(_string22,21);  // RMIComponent
 _class_array27 = new java.lang.Class[] {_class1, _class10};  // IDLSequence
_object27 = _rpmcommandexecutionserviceremote27.executeCommand("com.retek.rpm.app.item.service.TslCascadeAppService", "findItems", (java.lang.Object[])_object_array83, (java.lang.Class[])_class_array27);

)

Forms of deserialization

There are two forms of deserialization used in LoadRunner.
lr.deserialize (int, boolean)
First form uses the int argument and it takes the Sequence number of the serialized object of the file saved under LoadRunner Vuser directory. If the boolean flag for the second argument is true, it uses LoadRunner's serialization method. If it is false, it uses Java's standard serialization method.
LoadRunner only unfolds objects when you are using LoadRunner's mechanism, (lr.deserialize(int, true)) and when the size of the object is smaller than the limit specified in the recording options. In cases where, for complication purposes, the object cannot be made sense out of or is larger than the size limit, LoadRunner automatically folds the object into serialbytexx.dat; sometimes it uses Java's standard mechanism, and therefore, it does not unfold the objects inside the script.
  lr.deserialize(String, int)
The second form records the serialized object inside the script source as a string and uses that as the object to be deserialized. In the below example _string15 will be used as an object to be deserialized
_rpmcommandexecutionserviceremote23 = _rpmcommandexecutionserviceremotehome45.create();
       
 _string15 = "com.retek.platform.bo.IdentifiableReference __CURRENT_OBJECT = {" +
              "com.retek.platform.bo.ObjectId objectId = {" +
                    "java.lang.Long value = {" +
                          "super = {" +
                          "}" +
                          "long value = #1#" +
                    "}" +
              "}" +
              "java.lang.String referencedClassName = #com.retek.rpm.domain.zonestructure.bo.ZoneGroupImpl#" +
              "java.lang.Long version = {" +
                    "super = {" +
                    "}" +
                    "long value = #0#" +
              "}" +
        "}";
 _identifiablereference1 = (com.retek.platform.bo.IdentifiableReference)lr.deserialize(_string15,14);  // RMIComponent

Having the larger serialized object inside the script source as a string is not recommended unless correlation is needed to the object. The bigger the Java serialized object, the more memory the JVM will need for VuGen to record it as unfolded. Thus, the recording may simply crash if "Object Size limit" is set to be too high in the recording options.

To increase/decrease the object size for serialization can be set through Recording options -Serialization tab - Limit Object Size

Correlation

Correlation for the serialized objects inside the script is straight forward. In the below example, the below value 14548 is replaced by a variable

_string32_StringBuffer.append("java.util.ArrayList __CURRENT_OBJECT = {" +
"}" +
                                            "com.retek.platform.bo.IdentifiableReference promotionReference = {" +
                                                  "com.retek.platform.bo.ObjectId objectId = {" +
                                                        "java.lang.Long value = {" +
                                                              "super = {" +
                                                              "}" +
                                                              //"long value = #14548#" +
                                                              "long value = ##" +
                                                        "}" +
                                                  "}" +

Object Inspection


In case you need to inspect an object from the script then import pf-joi-full.jar in LoadRunner and add following lines:

_clientcontext1 = com.retek.platform.service.ClientContext.getInstance();
        org.pf.joi.Inspector.inspect(_clientcontext1);

This will popup a window of JOI (Java Object Inspector) and you will see all members and data of the object

References

Enterprise Java Bean Response Time analysis – Santoshan Kumaran, Te-kai Liu
http://www.oracle.com/technetwork/java/javase/tech/index-jsp-136424.html
http://download.oracle.com/javase/1.4.2/docs/guide/security/permissions.html
http://grids.ucs.indiana.edu/ptliupages/projects/HPJava/theses/slim/dissertation/dissertation/node11.html