UTF-8 — The little-endian and big-endian problem

The Czech title’s “little and big Indian” is really a play on big-endian and little-endian. It has nothing to do with
Winnetou and his companions. On the
IVTLog blog,
dj Padák found a problem with UTF-8 character encoding.
A little square appears in the resulting page. See the
picture.
Here’s how dj Padák describes it.

If you assemble the resulting page in PHP with include or require, an odd character appears at the beginning of each included file. I’d like to know what the problem is.

The problem is caused by text editors that put
the control characters \u010F\u00BB\u017E at the start of the file. They’re called ByteOrderMagic (BOM),
and their job is to say whether characters, or more precisely their representation as bytes,
are stored in big-endian or little-endian form.

What the control characters mean

To understand big-endian and little-endian, we need to realize that
in UTF-8, characters that don’t fit into one byte are encoded in two bytes. In that case
you need to tell whether the more significant byte comes first or
second. With big-endian it comes first. With little
-endian it’s the other way around.

A character with hexadecimal value 0x00FF would be stored in this order in big-endian form.

• The more significant byte, 00
• The less significant byte, FF

Byte order matters for processing, but the marker that tells you which order is used
should be suppressed in the output and used
only internally. Some editors and applications put these
control characters in the output. That’s what PHP does here, as you can see
in the picture. Java had the same problem in JDK 1.3, but it was fixed in later
versions.

One solution is to delete the control characters and save the file. I do that
in Eclipse, my development environment. Other editors
can probably show the characters as well. This little workaround can cause trouble, though, if you meet a program that
assumes the opposite byte order to the one you used.

September 11, 2003 — More information about the BOM in UTF-8