Application Performance Testing Best Practices

Application Scalability:

1. Define Application Scalability in two ways:    a. Vertical         b. Horizontal
2. Vertical scalability is defined as the scalability of an application as additional CPUs are added to the same server
3. Horizontal scalability is defined as the scalability of an application as additional servers are added to the environment
4. Scalability of an application is important for large enterprise Portal deployments where an individual server cannot support the anticipated load. In such cases, the Portal provides the capability to cluster multiple instances of the application. This can be achieved by either installing multiple instances of the Portal on a single server with a large number of CPUs or having each instance run in a separate Java Virtual Machine (JVM), a combination of both
5. Performance improvements are depends an operating system, application server and JVM handle the scheduling of threads across a larger number of CPUs. This is also subject to change as subsequent versions of JVMs are released to market. In addition, careful tuning of the application server is required to ensure that the load on a single instance can be most effectively supported (e.g., Web Sphere Web Container Thread Pool parameters)
6. For horizontal scalability, the only limitation on the scalability of the Portal will most likely be from the network or database perspective (the database is a shared resource between all instances of the Portal in a particular cluster). Any other system component that is shared by all Portal instances (e.g., LDAP Directory server) can potentially create a bottleneck and prevent further scaling of the application.
7. Portal instances running in a clustered environment it is required Portal network throughput is below 70% of the maximum network bandwidth available at peak periods (i.e. maximum of 8 MB/sec on a 100 Mb/sec network). Network throughput levels above this will impact the average response times for end users during these periods.
8. Database CPU utilization of above 75% at peak periods can greatly impact the Portal performance depending on the types of queries being executed at that time

Web Server:

1. If a web server (e.g., Apache, Sun ONE, IIS) is using proxy to accept request from the application server in a production environment, it is recommended to host the web server on a separate physical server. Such a configuration may be encountered in an Internet deployment with a DMZ (Demilitarized Zone)
2. The net effect of this is that more resources, in the form of memory and CPU, will be available to the application server to process requests. This will have a small impact on the response times due to an additional network hop but will increase the overall throughput of the application due to the additional CPU resources that will be available

Application Server:
Tuning parameters for Web Sphere & Apache Tomcat application servers
WebSphere:

1. The optimal number for the Web Container Thread Pool Maximum Size is 75 (default is 50). Both lower and higher values result in slightly lower throughput.
2. Increasing Thread Pool Minimum Size has adverse effect on performance.
3. Similarly, increasing Thread Inactivity Timeout has a slightly negative effect on performance.
4. Running Performance Monitoring Service introduces about 30% overhead

Apache Tomcat:
The development time issues relate to how the Java code for the Web application was designed and implemented. Again, there is a whole set of implementation best practices surrounding this area, such as:

1. Do not create sessions for JSPs if they are not required
2. Do not store large objects in your session.
3. Time out sessions quickly, and invalidate your sessions when you are done with them.
4. Use the right scope for objects.
5. Use connection pooling for improving performance.
6. Cache static data.
7. Use transfer objects to minimize calls to remote services.
8. Minimize logging from Web applications, or use simple logging formats

Database Server:
For large enterprise deployments of the Portal, it is recommended that a dedicated database server be used to host the Portal schema. In addition, certain database subsystems (e.g., MetaStore, Users) can be further distributed across several database servers. Deploying the Portal schema in this manner will help to ensure that the database will not become a bottleneck as the Portal is scaled to support a continually growing user base
Oracle 11g:
To assist in the rollout, build a list of tasks that increase the chance of optimal performance in production and enable rapid debugging of the application. Do the following:

1. When you create the control file for the production database, allow for growth by setting MAXINSTANCES, MAXDATAFILES, MAXLOGFILES, MAXLOGMEMBERS, and MAXLOGHISTORY to values higher than what you anticipate for the rollout. This technique results in more disk space usage and larger control files, but saves time later should these need extension in an emergency
2. Set block size to the value used to develop the application. Export the schema statistics from the development or test environment to the production database if the testing was done on representative data volumes and the current SQL execution plans are correct
3. Set the minimal number of initialization parameters. Ideally, most other parameters should be left at default. If there is more tuning to perform, then this appears when the system is under load
4. Be prepared to manage block contention by setting storage options of database objects. Tables and indexes that experience high INSERT/UPDATE/DELETE rates should be created with automatic segment space management. To avoid contention of rollback segments, use automatic undo management
5. All SQL statements should be verified to be optimal and their resource usage understood
6. Validate that middleware and programs that connect to the database are efficient in their connection management and do not logon or logoff repeatedly
7. Validate that the SQL statements use cursors efficiently. The database should parse each SQL statement once and then execute it multiple times. The most common reason this does not happen is because bind variables are not used properly and WHERE clause predicates are sent as string literals. If you use precompiles to develop the application, then make sure to reset the parameters MAXOPENCURSORS, HOLD_CURSOR, and RELEASE_CURSOR from the default values before precompiling the application
8. Validate that all schema objects have been correctly migrated from the development environment to the production database. This includes tables, indexes, sequences, triggers, packages, procedures, functions, Java objects, synonyms, grants, and views. Ensure that any modifications made in testing are made to the production system
9. As soon as the system is rolled out, establish a baseline set of statistics from the database and operating system. This first set of statistics validates or corrects any assumptions made in the design and rollout process