UTF-8 bit by bit (2001)

栏目: IT技术 · 发布时间: 4年前

内容简介:Richard Suchenwirth 2001-02-28 - From a delightful debugging chat at theTcl chatroom, I was brought to write down what I think on UTF-8 analysis (cf.Unicode and UTF-8, see also).I imagine a UTF-8 string as a railroad. It operates single-unitThe xxxxx bits

Richard Suchenwirth 2001-02-28 - From a delightful debugging chat at theTcl chatroom, I was brought to write down what I think on UTF-8 analysis (cf.Unicode and UTF-8, see also).

I imagine a UTF-8 string as a railroad. It operates single-unit railcars (one-byte ASCII characters, to be known from the fact that the highest bit is 0), and trains (sequences of two or more bytes that together form a character). Each train consists of exactly one locomotive (you see I'm European) and one or more trailers . The locomotive indicates the length of the train, including itself, in the highest bits that form a consecutive row of 1's, and one 0 bit. Examples:

0xxxxxxx : I'm a railcar, just a single unit
 110xxxxx : I'm leading a train of length 2

The xxxxx bits are used for other purposes (locomotives carry some freight too ;-)

Trailers indicate that they are trailers by the initial bit sequence 10. This way, they can't be mistaken for railcars or locomotives. E.g.

10yyyyyy: I'm a trailer

The freight of the train is in the x's and y's. In that concrete case, a C program reported to have received the bytes C3 and A4. Written as binary, that's

Now for clarity we delimit the indicators with parens:

(110)00011 (10)100100

and can just remove them:

00011 100100
 11100100 => E4, the iso8859-1 value for German ä ("a umlaut").

The generalized rule for the indicator of each byte is "those bits from highest(leftmost) down, up to and including the first zero bit".

Now going the other way. In orthodox UTF-8, a NUL byte (\x00) is represented by a NUL byte. Plain enough. But in Tcl we sometimes want NUL bytes inside "binary" strings (e.g. image data), without them terminating it as a real NUL byte does. To represent a NUL byte without any physical NUL bytes, we treat it like a character above ASCII, which must be a minimum two bytes long:

(110)00000 (10)000000 => C0 80

Whoops. Took us a while, but now we can read UTF-8, bit by bit.

andrewsh 2010-03-12 - Please note that 0xc0 0x80 sequence is illegal in the "Real" UTF-8: [], []


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

因计算机而强大

因计算机而强大

[美]西摩 佩珀特 Seymour Papert / 梁栋 / 新星出版社 / 2019-1 / 38

本书有两个中心主题—— 孩子可以轻松自如地学习使用计算机; 学习使用计算机能够改变他们学习其他知识的方式。 (前苹果公司总裁 约翰·斯卡利) 最有可能带来文化变革的就是计算机的不断普及。 计算机不仅是一个工具,它对我们的心智有着根本和深远的影响。 计算机不仅帮助我们学习 ,还帮助我们学习怎样学习。 计算机是一种调解人与人之间关系的移情对象。 一个数学的头脑......一起来看看 《因计算机而强大》 这本书的介绍吧!

HTML 压缩/解压工具
HTML 压缩/解压工具

在线压缩/解压 HTML 代码

JS 压缩/解压工具
JS 压缩/解压工具

在线压缩/解压 JS 代码

RGB转16进制工具
RGB转16进制工具

RGB HEX 互转工具