<$BlogRSDUrl$>

Monday, October 04, 2010

Top 10 Java Issues in Production

10 Heap/network issues

Tools: dtrace, hprof, introscope, jconsole, visualvm, yourkit, azul zvision
Invasive tools : bci, jvmti, jvmdi

OS Tools : dtrace, osprofile, vtune

Network/DISK tools: ganglia, iostat, lsof, nagios, netstat, snoop

Gotcha:
  • instrumentation is not cheap
  • avoid expensive heap walks
  • thread dumps
  • asynchronous logging
  • finish task and then increment performance counter
  • jconsole is cheap

9 Leaks

symptoms :
  • app consumes all memory
  • heap trend is ramping saw tooth
  • finally Out of Memory
Tools : yourkit, hprof, eclipse mat, jconsole, jhat, jps, visualvm, azul zvision

Theory:
  • Allotted Vs Live Objects
  • Finalizers, Classloaders
8 I/O

I/O Serialization
Symptoms :
  • Multi-node scale-out does not scale linearly
  • Spending time CPU & IO
Tools :
  • cpu profiling
  • io profiling
Solution
  • pick a high performance serialization library
  • avro, krgo
I/O Limits/Tuning

Symptoms:
  • Too many file descriptors, cursors etc
  • inconsistent response times
Tools:
  • Nagios
  • pkg
  • rpm
  • info
  • ulimit
  • yum
Solution
  • check OS patches
  • check user and process limits
I/O Sockets, Files, DB

Symptoms:
  • socket open/close takes a long time
  • JRMP timeouts, long JDBC calls
  • running out of files, cursors
Tools:
  • dbms tools, du, iostat, gmon, lsof, netstat
Workaround
  • ping/telnet tests
7 Locks and Synchronization

Symptoms:
  • Adding users/threads/CPUs causes slow down
  • High lock acquire times and contention
  • race conditions, dead locks
  • I/O under load
Tools:
  • dtrace, lockstat, azulzvision
  • thread dumps
  • IBM visual analyzer ( j.u.c in eclipse )
Solution:
  • Use non-blocking collections

6 Endless Compilation

symptoms:
  • Time in compiling
  • Time in interpretor
Tools:
  • -XX:+PrintCompilation
  • CPU profiler

5 Endless Exceptions

symptoms:
  • Application spending time filling stack trace
tools:
  • CPU profiler, zvision
  • thread dumps
  • Track caller/callee
  • repeated kill -3
Solution:
  • don't throw , return
  • JVM's don't optimize exception paths
4 Fragmentation

Symptoms:
  • performance degrades over time
  • "Full GC" makes problem go away
  • Lot of free memory, but in tiny fragments
Tools;
  • GC logging flags, for CMS -XX:PrintFLSStatistics=2 -XX:+PrintCMSInitializationStatistics
  • Fragger
Solution:
  • Upgrade to latest CMS
  • azul zing and gen pausless GC
  • pooling similar sized and similar aged objects together

3 GC Tuning

Symptoms:
  • Entropy(GC) = number of GC flags
  • Too many free parameters
  • 64bit/ large heap no solution
  • Constant 40%-60% CPU utilization by GC
  • Scheduled reboot before full GC
  • Full-time engineer working on GC flags
Workarounds:
  • Ask JVM vendor for one flag solution
  • G1 GC, Zing GPGC
2 Spikes in Load

Symptoms:
  • Rush hour traffic, tax day, black friday
  • outage under spikes, power law of networks
Solution:
  • Measure
  • Test with realistic load and realistic multi-node setup
  • build redundancy
1 Versionitis

Symptoms:
  • different nodes have different configuration, stack components, versions
  • classpath dist/* , -verbose
  • hard to reproduce
Solutions:
  • Method
  • Version control
  • rigor
0 Collapsing under load

juc profiling


Tuesday, August 26, 2008

Useful commands in Linux and Solaris

  1. truss in solaris , strace in linux


Tuesday, May 06, 2008

Useful Maven commands

checkout a project from subversion

mvn scm:checkout -DconnectionUrl=scm:https://svn.url.project.name -DcheckoutDirectory=name

Labels:


Java Classloaders

Classloaders are responsible for finding and loading a class.

Class itself is a java object.

Following statements are true,
  1. All classloaders except bootstrap classloaders have a parent classloader
  2. All class loaders are of type java.lang.ClassLoader

Custom class loaders

A custom Class Loader can be created by extending java.lang.ClassLoader.
Two things are important in creating a custom classloader.
  1. Setting parent classloader
  2. implementing findClass() method

Labels:


Friday, August 24, 2007

Auction site for security vulnerabilities
Wabi Sabi Labi

Labels:


Monday, July 23, 2007

List of User groups in Bay area.

ACCU  http://www.accu-usa.org/
Bay Area Debian http://bad.debian.net/
BAFUG S.F. http://www.bafug.org/
BALUG http://www.balug.org/
[BAWUG http://www.bawug.org/]
BayPIGgies http://www.baypiggies.net/
BayLISA http://www.baylisa.org/
BayLISA Monitoring SIG
http://www.groundworkopensource.com/community/monitoring-sig.html
BUUG http://www.buug.org/
CABAL http://linuxmafia.com/cabal/
CalLUG http://www.ocf.berkeley.edu/~linux/
[CCSF LUG mailing list only. Dead site:
http://www.ccsf.cc.ca.us/Organizations/Linux_Users_Group/]
Cocoaheads http://cocoaheads.org/us/SiliconValleyCalifornia/
CSUEB LUG http://acc.csueastbay.edu/~linuxug/kwiki/
EBLUG http://www.eblug.org/
Linux Consultants' Lunch inquired about missing mailing list, 2006-02-21.
Linux User Group Hayward (Linuxdojo) http://www.linuxdojo.net/
Linuxstammtisch http://electriclichen.com/linuxbierwanderung/local/#sanfrancisco
NBLUG http://www.nblug.org/
Oakland Perl Mongers http://oakland.pm.org/
PC Clubhouse Linux SIG http://www.pcc.org/sigs/#Linux
PenLUG http://www.penlug.org/
SFBA NetBSD UG http://www.imrryr.org/NetBSD/sfbanug.html
SF LUG http://www.sf-lug.org/
SF OpenBSD UG http://www.sfobug.org/
SF Perl UG http://sf.pm.org/
SF PostgreSQL UG http://pugs.postgresql.org/sfpug/
SVBUG http://www.svbug.com/
SVLUG http://www.svlug.org/
Silicon Valley Perl UG http://www.rf.net/~sv-pm-org/
Silicon Valley Wireless Users and Experimenters http://www.svwux.org/
SJSU LUG http://sjsulug.engr.sjsu.edu/
Smaug http://www.scruz.org/
Sonoma County Sysadmins (SoCoSA) http://socosa.org/
SULUG http://sulug.stanford.edu/
Starship Augusta Ada http://trek.starshine.org/

Linux User Group of Davis (LUGOD) http://www.lugod.org/
Linux User Group of Sacramento (SacLUG) http://www.saclug.org/


Sites of sporadic interest:
Elec. Flea Mkt. 2nd Sat. at De Anza, Mar-Oct: http://www.asvaro.org/fleamarket
Intrusion Detection UG http://idug.cryptojail.net/
Mindsource BOFs http://www.mindsrc.com/
Linux.org events http://www.linux.org/event/
Mozilla Conference
LWCE http://www.linuxworldexpo.com/
Holidays, time changes! see: http://linuxmafia.com/bale/holidays
O'Reilly Open Source Convention, http://conferences.oreillynet.com/os2006/
EFF Bay Chapter http://www.eff.org/bayff/
Bay Area Cypherpunks http://www.cryptorights.org/lists/meetingpunks/
South Bay Community Network sbay.org
Computer History Museum http://www.computerhistory.org/events/
Annual Linux Picnix http://www.linuxpicnic.org/
Stacey's Book Store Events http://www.staceys.com/sanfranciscoevents.html
USENIX Association http://www.usenix.org/events/
CodeCon http://www.codecon.org/
Computers, Freedom, and Privacy http://www.cfp.org/
IEEE Security Symposiums http://www.ieee-security.org/
May 21-24, 2006: http://www.ieee-security.org/TC/SP2006/oakland06.html
Computer Professionals for Social Responsibility http://www.cpsr.org/
Vintage Computer Festival http://www.vintage.org/events.php (November?)
Howard's Events list: http://lg-events.blogspot.com/

Dead groups:
BANG http://www.bang.org/
Internet Developer/Technology Group http://www.inetdevgrp.org/
LBNL LUG http://lug.lbl.gov/
Linux Events http://linevents.com/ (gone)
Linux Movies Group http://www.linuxmovies.org/
SCLUG http://linus.got.net/
sig-beer-west http://www.extasia.org/sig-beer-west/
SlugLUG http://sluglug.ucsc.edu/
WebGuild http://www.webguild.org/

Monday, June 04, 2007

What would happen when tried to compile and run this program?

public class StaticVar
{
void execute(String b){
static String str;
str = b;
}

public static void main( String[] args ) {
StaticVar var = new StaticVar();
var.execute( "blah");
}
}
1. Compilation Error
2. Runtime Error.
3. Compiles and Runs with no errors.

Answer : 1

Labels:


Wednesday, May 02, 2007

Nice posting for converting bytes to string and vice versa.

http://forum.java.sun.com/thread.jspa?threadID=659432&messageID=3873051

This page is powered by Blogger. Isn't yours?