Locks and a Two-in-the-Morning Wake-Up Call

Whenever your phone rings at two in the morning, it's trouble. It's either a jilted lover or something has blown up in production. It's even worse when it directly affects customers, especially those overseas who are dealing with it during prime time. It happened to me recently. The problem was specific, but there are a few general lessons to take from it. I realized that while reading Synchronized Drowning, which describes something similar.

I'd written code to load a public key for signature verification. The key rotated every few hours and was stored on the GlusterFS distributed file system, so it made sense to avoid reading it on every request. The code looked roughly like this.

private Key key;
        
        Key getKey(){
            Lock readLock = readWriteLock.readLock();
            try {
                readLock.lock();
                return key;
            } finally {
                readLock.unlock();
            }
        }

void loadKey() {
            Lock writeLock = readWriteLock.writeLock();
            try {
                writeLock.lock();
                //load
                BufferedReader br = new BufferedReader(new FileReader(new File(keyFile))));
                ....
            } finally {
                writeLock.unlock();
            }
        }

Using a read/write lock meant threads didn't block one another when reading the key. When a new key was detected, we called loadKey. That method took the write lock and blocked all but one thread until the key had loaded.

Everything worked until the thread reading the file froze somewhere deep in the operating system, waiting forever on the disk. The only thing that helped was unmounting the disk and restarting the JVM. With the write lock held effectively forever, all the other threads were left waiting for it to be released. Here are a few general lessons from my mistakes.

• Whenever you call an external system, especially over the network, expect it to slow down unexpectedly. Rule number one is to set sensible timeouts at every level, from the OS and JVM to the application.
• When you use a Lock and the implementation supports it, prefer the tryLock method, which attempts to acquire the lock within the given timeout.
• The one thing you can count on is that the technology you use will fail. It pays to ask yourself, “What happens if?” If I'd asked that here and thought through the consequences, I would have used timeouts and kept the scope of the lock much narrower. It would have been enough to protect access to the instance variable holding the key.
• Keep the threads that serve HTTP traffic out of blocking I/O. Use an API with nonblocking support for all I/O operations.