Java: Character Encoding

One of the most common questions in Java discussion groups

is about character encoding. It usually comes up when reading or writing through

servlets or JSP, databases and files. More generally, it applies to operations

at the boundary between the JVM and the outside world, where a byte representation

is converted to characters or the other way around. For me the clearest explanation

of encoding in Java came from Makub in a discussion group

on java.cz. I’ll try to pass it on.

I’m fine because everything is Unicode

You can find that headline in any number of Java brochures. It’s true

that Java represents all characters internally as Unicode. Unicode assigns each character

a unique number and defines three forms for representing it,

in bytes, words or double words (8, 16 or 32 bits).

The fact that Java uses Unicode internally, in UTF-16, is repeated so often

that many developers come to believe they don’t have to think about encoding.

The opposite is true. Any operation at the boundary between the JVM and the outside

world, such as reading a text file or writing to a database, converts

between the internal Unicode representation of a character and its byte representation

in either direction.

The external byte representation follows the encoding you choose. If you don’t

specify one, the JVM uses its default encoding, which depends on the environment

it’s running in. Problems arise when converting a character to its external byte representation

or back again. If the JVM default differs from the encoding we want for input or output

and we don’t tell Java explicitly which one to use, the conversion

to bytes or back follows the default.

Handling an HTTP request in a servlet is a classic example. If you start reading

the request parameters before calling

setCharacterEncoding, the servlet container

uses the default encoding for all the parameters.

A simple rule follows from this: explicitly

set the encoding for every input and output operation outside the JVM.

The trouble with that rule is that developers often don’t realize what counts as outside the JVM.

That includes working with a database, accessing the file system or

handling an HTTP request, for example.

I’ve saved a few practical tips for last. They solve 90% of the problems.

General: keep text out of source code

Put all application text into a resource bundle and you’ll save yourself

a lot of trouble.

General: reading and writing text resources

Use java.io.InputStreamReader and java.io.OutputStreamWriter.

General: set the compiler’s encoding

If you have application text in source code, such as exception messages,

remember to tell the compiler what encoding to use. That’s what the

encoding option is for.

General: convert properties-based resource bundles before use

If your resource bundle is based on a properties file, run it 
through native2ascii as part of the build. The utility is

a standard part of the Java SDK.

Ant task:

<target name="native2ascii"> 
    <native2ascii 
 encoding="utf-8" 
 src="${src.dir}" 
 dest="${web.dir}/WEB-INF/classes" 
 includes="**/*.properties" 
 ext=".properties"/> 
</target>
 

Server: set the encoding for HTTP requests and responses

Before you read the first parameter or write the first character to the output,

set the desired encoding for both. For HTTP requests, a servlet filter

is a clean way to do it. It avoids problems that can arise

when someone else gets to the parameters before you do,

an input validator, for example.

public class RequestEncodindFilter implements Filter{    
  public void doFilter(
  ServletRequest rq, 
  ServletResponse re, 
  FilterChain chain) 
    throws IOException, ServletException {
        rq.setCharacterEncoding("utf-8");//kodovani          
        chain.doFilter(rq,re);  
}
public void init(FilterConfig fc) throws ServletException {}
public void destroy() {}
}

web.xml

<filter>

<filter-name>RequestEncodindFilter</filter-name>

<filter-class>filterpackage.RequestEncodindFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>RequestEncodindFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>

For JSP pages, use the page directive and the attributes

pageEncoding and contentType.

<%@ page contentType=“text/html; charset=utf-8” pageEncoding=“utf-8” %>

Database: set the JDBC driver’s encoding

The jTDS driver has a parameter for setting the encoding used with

CHAR, VARCHAR and TEXT types.

Connection string MySQL JDBC ovladače:
jdbc:mysql://localhost/devdb1?useUnicode=true&characterEncoding=UTF-8