Headless Java Monster

You know you are up against the same fellow if you start seeing the

java.awt.HeadlessException, typically running off a virtual server, or in the rare  case of a dedicated server without a monitor (aka head).

The solution is simple. First shut down the application, tomcat, etc. that got the exception.

1. Install the X display manager. 

On a Ubuntu on the other hand, you could install the Xvfb package (via apt, synaptic, etc.)

2. Start X:

3. Export display:
With those done, now you should have entered the simpler "No X11 DISPLAY variable" zone. Simply export the display variable to fix this.

(In the Ubuntu case above you have to export DISPLAY=:1)

4. Allow all users to connect/ use this Display variable:

Now restart the application, tomcat, etc. that you were trying to run initially & it should work. Hope nothing headless ever troubles no man!

C# - A home away from home

For someone with years of experience in Java, the stint to code in C# seemed like a cake walk. The large scale port of popular Java frameworks into dot net such as NHibernate, Spring.net, NUnit, etc., make life all that much easier for anyone starting to bridge the gap.

There are however some bits that trouble us Java natives no end. Particularly anything & everything to with the web.config file. This one file has enough traps in it to confuse the hell out of any sane minded developer. The file has hints for the IDE (Visual studio), the framework (Asp.net), the web server (IIS), & everyone else connected to the runtime.

The other bit that seems bothersome is how dependencies, packages & Dll's get referenced. Particularly with frequently changing Dll's, & the tools optimized for caching, it gets difficult to know what version is really loaded & running.

Anyway, once you get past these issues, the ride ahead is through familiar grounds.

Remotely Debug Solr Cloud in Eclipse Using JPDA, JDWP, JVMTI & JDI

The acronym's first:
JPDA - Java Platform Debug Architecture
JDWP - Java Debug Wire Protocol
JVMTI - JVM Tool Interface
JDI - Java Debug Interface

To debug any of the open source Java projects such as Solr using Eclipse, rely on the JDWP feature available within any standard JVM. You can get a lot more info about the terms and architecture here.

At a high level the concept is that there is a JVM to be debugged (Solr) & a client side JVM debuggee (Eclipse). The two communicate over the JDWP. Thanks to a standardized wire protocol the client may even be a non JVM application which subscribes to the protocol.

One of the two JVMs acts as the debugging server (the one that waits for the client to connect). The other JVM acts as the debugging client which connects to the debugger server, to start the debugging process.

In our case, to keep things simple let Solr be the debugger server, while Eclipse can be the debugger client. The configurations then are as follows.

On Solr side (assuming Solr Cloud):

java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Djetty.port=7200 -Dhost=myhost -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -Djava.util.logging.config.file=etc/logging.properties -DnumShards=3 -DzkHost=zk1:2171 -jar start.jar

Note: Since we have set suspend = y, Solr side will stay suspended until the Eclipse debugger client has connected

On Eclipse side:
Go to Run > Debug Configurations > Remote Java Application
Then choose Standard Socket Attach. Host: localhost (or IP). Port: 8000 (the same as set above)

Also in Eclipse you should have checked out the Solr source code from Solr trunk as a project. This will allow you to put break points at appropriate location to help with the debugging. So go on give this a shot and happy debugging!

Java Classloaders

Key things about Java classloaders:
1. There is a hierarchy among classloaders:
Bootstrap <---|
                      Extension <--|
                                          System <---|
Custom

- Child classloaders (typically) delegate class loading to parent classloaders. child class loader’s findClass() method is not called if the parent classloader can load the class.
- Custom classloaders can override the default delegation chain to a certain extent.
- Due to the delegation chain of classloaders, ensure classes to be loaded by custom classloaders are not present on the system class path, boot class path, or extension class path.

2. Bootstrap classloader is a special classloader included with the JVM, written in native code. Bootstrap classloader is tasked with loading all core classes part of the jre.
None of the other classloaders can override the Bootstrap classloader's behvaiour.

3. All other classloaders are written in Java.

4. Extension classloader loads classes from the extension directories: JAVA_HOME/jre/lib/ext/

5. A more popular alternative to using the Extension classloader, is to use the System classloader which loads classes from the CLASSPATH environment variable location.

6. Finally, Custom classloaders can be written to override certain defaults like delegating classloading to parents, etc. Custom classloaders is commonly used by Application servers (such as Tomcat).

7. Separate Namespaces per Classloader:
Same class loaded by two different classloaders, are considered different. Trying to cast an object of one class (loaded by classloader 1) to a reference of the other (loaded by classloader 2, though identical in terms of its fully qualified class name) will result in a ClassCastException.

8. Lazy loading and Caching of Classes:
Classloaders load classes lazily. Once loaded classloaders cache all previously loaded classes for the duration of the JVM.

Key Methods of Classloaders:
To be detailed..

Dynamic Reloading of Classes:
Due to the non-overridable behaviour of caching of classes by classloaders, reloading of classes within a running JVM poses problems. To reload a class dynamically (a common use case for app. servers), a new instance of the classloader itself needs to be created.
Once the earlier classsloader is orphaned/ garbage, classes loaded & cached by it (and reachable only via the now GC'd classloader) also become garbage, which can then be collected by the GC.

Phantom References in Java

Depending upon the tenacity of the grasp of the reference object, Java language has defined four different kinds of reference types - Strong, Soft, Weak & Phantom.

A simple search leads you to good explanations on the topic. Particularly the article by Ethan, this SO discussion, the official Java docs from Oracle, and this article on reachability are useful.

The focus of this article is to explain the working of the example in KDGregory's post on Phantom References.

You must understand the call sequence shown above and the purpose of the participating classes. The key points are:

(i) PooledConnection: class extends Connection class.

(ii) Client Applications: only interact with instances of  the wrapper PooledConnection class. Applications get hold of new connections via the getConnection() for the ConnectionPool.

(iii) ConnectionPool: has a fixed number of Connections (base).

(iv) IdentityHashMap: Strong references to currently assigned/ in use Connections are maintained in two IdentityHashMaps - ref2Cxt & cxt2Ref. The references in the two HashMaps is the PhantomReference to the newly created PooledConnection.

The code is able to handle the following cases with regard to releasing of connections:

Case 1: A well behaving client, which calls PooledConnection.close():
This in turn calls the ConnectionPool.releaseConnection(Connection), passing in the associated the Connection object, leading to returning of the associated Connection object to the Connection Pool.

Case 2: Misbehaving client, Crashed client, etc. which does NOT call PooledConnection.close():
Once the client application stops using the PooledConnection object, the PooledConnection object goes out of scope. At this stage the PhantomReference associated with the PooledConnection object (see wrappedConnection()), gets enqueued in the associated ReferenceQueue.

In the ConnectionPool.getConnection() method, the call to ReferenceQueue.remove(), returns the PhantomReference to the particular  PooledConnection. Passing in this PhantomReference to ConnectionPool.releaseConnection(Reference), locates the associated Connection object from the _ref2Cxt, IdentityHashMap, & returns the Connection back to the Connection Pool.

This automatic reclaiming of the precious (yet orphaned) Connection is the crux of KD Gregory's Phantom Reference example.

Java Generics PECS & Get-Put Principle, Non-Reifiable Types & Erasure, Covariance & Contravariance

Joshua Bloch's in the book Effective Java has introduced the PECS (Producer Extends Consumer Super) mnemonic for managing type hierarchies via Java Generics, i.e. Covariance & Contravariance for Generics.

Covariance & Contravariance

Covariance is a subtyping principle where a child/ sub-type can be used in place of parent/ super-type. In Java land this is seen with Arrays, Method Overriding (Java 5 onwards) & Generics Extends.

Contravariance is the reverse, where a parent/ super-type can be used in place of a sub-type. In Java, contravariance is seen with Generics Super.

PECS

Now coming back to PECS, also referred to as the Get-Put, principle of Generics. The key thing about PECS is that it is defined from the persepective of the Collection object in focus, and not the caller/ client using the Collection object.

Case 1: When the Collection is being used to retrieve existing (previously added) data, from the Collection's perspective it is a Producer of data. As per PECS it needs to Extend.
The caller in this case can be sure that results have objects who's parent is MyParent. So the caller can safely iterate over results as an immutable collection of MyParent objects.

However, any kind of addition into the Collection <? extends MyParent> is unsafe. MyParent can have any number of subtypes (MyChild1, MyChild2, etc.) and there is no way for the caller application to know this specific subtype. So at this stage any addition is not allowed.

Case 2: When the Collection is being used to store new data, from the Collection's perspective it is a Consumer of data. As per PECS it needs to define Super.

Type Erasure & Non-Reifiable Nature of Generics

Java Generics is implemented through erasure entirely at the compiler level. The type info. is discarded from within the bytecodes (i.e. List<String> & List have identical bytecodes). This also causes the non-reifiable behaviour of Generic types, i.e. the inability to determine the type info. from the bytecode at runtime. (Note that Arrays are reifiable as shown in E.g. 2 above).

The PECS rule is therefore to make the compiler operate defensively.  Once type info has been associated with a given Collection via Generics, the PECS restriction is there to enable compile time detection of potential unsafe usage.