Leaking streams

I recently fixed a mistake of my own. I’d assumed the JDBC driver, or more specifically the PreparedStatement, would close the InputStream I handed it when the transaction finished. Unfortunately it does no such thing. Our custom InputStream kept some of the data in memory and spilled the rest into a file in the tmp directory. The file was supposed to be deleted when the associated stream closed, and as a precaution we also set deleteOnExit.

Because close was never called on the stream, new files sprang up in tmp like mushrooms after rain. After the server had been running for a while, tmp filled up and we ran out of disk space or file descriptors. The catch was that cluster nodes aren’t restarted all that often, so deleteOnExit was just for show. Hibernate also created the stream, since it was used as a user type in the mapping. An entity selected from the database could have an associated stream without you realizing it. Fortunately, most of that code was isolated in the persistence layer.

Finding and fixing all the unclosed streams was tricky because of the indirection through Hibernate and the user type. A leak remained somewhere in the persistence code and I couldn’t spot it by reading the code. Eventually I wrote debugging code that printed a stack trace in the stream’s constructor, marked it with the identity hash code, and put the same code into the file created in tmp. By going through the log and matching files left behind to the marked traces, I finally found and fixed the remaining leaks.

That ad hoc fix wasn’t enough for me. Anyone who uses that user type, which we use for binary data in the database generally, can forget about the indirection and create another leak. I also wanted to make sure no higher layer forgot to close the stream and release the file. The technique I’d used to find the remaining leaks had worked well, so it made sense to take it further.

I moved all the tracing code into an aspect. Whenever we need it for integration or manual testing, we can enable the aspect and have it print the unclosed streams, or rather the code that opened them, when the JVM exits. The output looks roughly like this.

-------------------------------------------------------

Unclosed stream allocated from:

java.lang.Thread.getStackTrace(Thread.java:1436)

cz.sweb.pichlik.leakdetector.LeakDetector.ajc$afterReturning$cz_sweb_pichlik_leakdetector_LeakDetector$1$5f03d6db(LeakDetector.aj:29)

cz.sweb.pichlik.leakdetector.FooInputStream.(FooInputStream.java:12)

cz.sweb.pichlik.leakdetector.LeakDetectorTest.testSmoke(LeakDetectorTest.java:11)

…rest omitted

I left out the full stack trace above for the article. The code that caused the leak follows.

public class LeakDetectorTest {

@Test

public void testSmoke() throws Exception{

new FooInputStream(getClass().getResourceAsStream("LeakDetectorTest.class")).close();

for(int i = 0; i < 5; i++) {

new FooInputStream(null);

}

}

}

Here’s the aspect.

public aspect LeakDetector {

private final Map<Integer, StackTraceElement[]> openStreams;

pointcut streamConstructor(): target(FooInputStream) && (execution(FooInputStream.new(..)));

pointcut streamClose(): target(FooInputStream) && (execution(void close()));

public LeakDetector(){

openStreams = new ConcurrentHashMap<Integer, StackTraceElement[]>();

Runtime.getRuntime().addShutdownHook(new LeakPrinter(openStreams));

}

/**

* Puts a stream into watched area.

*/

after() returning: streamConstructor() {

StackTraceElement stacktrace[] = Thread.currentThread().getStackTrace();

Integer streamId = getObjectIdentifier(thisJoinPoint);

openStreams.put(streamId, stacktrace);

}

/**

* Removes a stream from watched area.

*/

after() returning: streamClose() {

Integer streamId = getObjectIdentifier(thisJoinPoint);

openStreams.remove(streamId);

}

private Integer getObjectIdentifier(JoinPoint jp) {

return System.identityHashCode(jp.getTarget());

}

}

The aspect contains two pointcut expressions, one to catch stream creation and another to catch closing. Each pointcut has its own advice. The first takes a snapshot of the current stack trace when the stream is constructed and stores it in a map along with the identity hash code. The second removes the corresponding entry from the map by identity hash code when the stream’s close method is called. The aspect also creates a thread in its constructor and registers it as a shutdown hook. The thread just prints the contents of the map. At the end of the program, the only traces left in the map show where unclosed streams were created.

The most suitable way to use the aspect seems to be load-time weaving. You don’t need a special compilation step because the aspect is woven in as classes load. If you want to play with it, there’s a Maven project with the source code and the AspectJ compiler configured.

Related articles

• Transparent caching with aspect-oriented programming (This article also explains the basics of AOP.)