Lotus Multi-Byte Character Set

The Lotus Multi-Byte Character Set (LMBCS) is a proprietary multi-byte character encoding originally conceived in 1988 at Lotus Development Corporation with input from Bob Balaban and others.[1] Created around the same time and addressing some of the same problems, LMBCS could be viewed as parallel development and possible alternative to Unicode.[1] For maximum compatibility, later issues of LMBCS incorporate UTF-16 as a subset.[2][3]

Commercially, LMBCS was first introduced as the default character set of Lotus 1-2-3 Release 3 for DOS in March 1989[1][4] and Lotus 1-2-3/G Release 1 for OS/2[1] in 1990 replacing the 8-bit Lotus International Character Set (LICS) and ASCII used in earlier DOS-only versions of Lotus 1-2-3 and Symphony.[5] LMBCS is also used in IBM/Lotus SmartSuite, Notes and Domino,[1] as well as in a number of third-party products.

LMBCS encodes the characters required for languages using the Latin,[6] Arabic, Hebrew, Greek and Cyrillic[6] scripts, the Thai, Chinese, Japanese[6] and Korean writing systems, and technical symbols.

Encodings

Technically, LMBCS is a lead-byte encoding where code point 00hex as well as code points 20hex (32) to 7Fhex (127) are identical to ASCII[1] (as well as to LICS).[5]

Code point 00hex is always treated as NUL character to ensure maximum code compatibility with existing software libraries dealing with null-terminated strings[1] in many programming languages such as C.[lower-alpha 1] This applies even to the UTF-16be codes, where code words with the form xx00hex are mapped to private-use codes with the form F6xxhex during encoding in order to avoid the use of NUL bytes,[7] and to escaped control characters, where 20hex is added to the C0 (but not C1) control characters following the 0Fhex lead byte.[7]

Code points 01hex to 1Fhex, which serve as control codes in ASCII, are used as lead bytes to switch the definition of code points above 7Fhex between several code groups (similar to code pages) and at the same time determine either a single- or multi-byte nature for the corresponding code group.[1]

For example, code group 1 (with group byte 01hex)[1] is almost identical to the SBCS code page 850, whereas code group 16 (with group byte 10hex)[1] is similar to the Japanese MBCS code page 932. Multi-byte characters can thus occupy two or three bytes.[7][6]

In canonical LMBCS, each character starts with its group byte.[1] To reduce the length, in optimized or compressed LMBCS a default code group or optimization group code can be defined on a per application or process basis (ideally chosen according to the highest likelihood of occurrence)[1] and must be communicated to the interpreting code in some way (f.e. by specifying the corresponding "LMBCS-n" name).[8] Thereby, the group byte can be omitted for these characters.[1] Lotus 1-2-3 retrieves the optimization group code from the file header of the corresponding source file,[7] whereas for Lotus Notes the optimization group code is fixed to be always 01hex.[2][7]

Default Group Bytes Description
N/A 00hex 1[7] NUL
LMBCS-1 01hex 2[7] Code page 850 (DOS Latin-1)[2][7]
LMBCS-2 02hex 2[7] Code page 851 (DOS Greek)[2][7]
LMBCS-3 03hex 2[7] Code page 1255 (Windows Hebrew)[2][7]
LMBCS-4 04hex 2[7] Code page 1256 (Windows Arabic)[2][7]
LMBCS-5 05hex 2[7] Code page 1251 (Windows Cyrillic)[2][7]
LMBCS-6 06hex 2[7] Code page 852 (DOS Latin-2)[2][7]
N/A 07hex 1[7] BEL[2]
LMBCS-8 08hex 2[7] Code page 1254 (Windows Turkish)[2][9][7]
N/A 09hex 1[7] TAB[2][9][7]
N/A 0Ahex 1[7] LF[2][9][7]
LMBCS-11 0Bhex 2[7] Code page 874 (Thai)[9][7]
(LMBCS-12) 0Chex 2[7] Reserved[2]
N/A 0Dhex 1[7] CR[2][9][7]
(LMBCS-14) 0Ehex 2[7] Reserved[2]
(LMBCS-15) 0Fhex 2[7] Remapped C0/C1 control codes[7]
LMBCS-16 10hex 3[7] Code page 932/[2]943[7] (Japanese / Shift-JIS)[2][9]
LMBCS-17 11hex 3[7] Code page 949/[2]1261[7] (Korean)[2][9]
LMBCS-18 12hex 3[7] Code page 950[2][7] (Traditional Chinese / Taiwan / Big5)[2][9]
LMBCS-19 13hex 3[7] Code page 936/[2]1386[7] (Simplified Chinese)[2][9]
(LMBCS-20) 14hex 3[7] UTF-16 (Unicode)[2][3][7]
N/A 15hex 3 Reserved[2]
N/A 16hex 3 Reserved[2]
N/A 17hex 3 Reserved[2]
N/A 18hex 3 Reserved[2]
N/A 19hex 1[7] Lotus 1-2-3 system range[9][7]
N/A 1Ahex 3 Reserved[2]
N/A 1Bhex 3 Reserved[2]
N/A 1Chex 3 Reserved[2]
N/A 1Dhex 3 Reserved[2]
N/A 1Ehex 3 Reserved[2]
N/A 1Fhex 3 Reserved[2]

Character set

Without prefix byte the code points 32 (20hex) to 127 (7Fhex) are interpreted as follows (corresponding to LMBCS codes 32 to 127):

Single byte codes (ASCII/ISO-646-US[10])
_0 _1 _2 _3 _4 _5 _6 _7 _8 _9 _A _B _C _D _E _F
2_
32
SP
0020
!
0021
"
0022
#
0023
$
0024
%
0025
&
0026
'
0027
(
0028
)
0029
*
002A
+
002B
,
002C
-
002D
.
002E
/
002F
3_
48
0
0030
1
0031
2
0032
3
0033
4
0034
5
0035
6
0036
7
0037
8
0038
9
0039
:
003A
;
003B
<
003C
=
003D
>
003E
?
003F
4_
64
@
0040
A
0041
B
0042
C
0043
D
0044
E
0045
F
0046
G
0047
H
0048
I
0049
J
004A
K
004B
L
004C
M
004D
N
004E
O
004F
5_
80
P
0050
Q
0051
R
0052
S
0053
T
0054
U
0055
V
0056
W
0057
X
0058
Y
0059
Z
005A
[
005B
\
005C
]
005D
^
005E
_
005F
6_
96
`
0060
a
0061
b
0062
c
0063
d
0064
e
0065
f
0066
g
0067
h
0068
i
0069
j
006A
k
006B
l
006C
m
006D
n
006E
o
006F
7_
112
p
0070
q
0071
r
0072
s
0073
t
0074
u
0075
v
0076
w
0077
x
0078
y
0079
z
007A
{
007B
|
007C
}
007D
~
007E
DEL/
007F/2302

  Letter  Number  Punctuation  Symbol  Other  Undefined

Group 1

LMBCS group 1 code points 128 (80hex) to 255 (FFhex) are identical to the corresponding code points in code page 850 (DOS Latin-1), whereas code points 1 (01hex) to 127 (7Fhex) are defined according to the following exception list (corresponding to LMBCS codes 256 to 383):

LMBCS Group 1, lower half[11][10]
_0 _1 _2 _3 _4 _5 _6 _7 _8 _9 _A _B _C _D _E _F
0_
0
NUL
0000

263A

263B

2665

2666

2663

2660

2022

25D8

25CB

25D9

2642

2640

266A

266B

263C
1_
16

25BA

25C4

2195

203C

00B6
§
00A7

25AC

21A8

2191

2193

2192

2190

221F

2194

25B2

25BC
2_
32
¨
00A8
~
007E
˚
02DA
^
005E
`
0060
´
00B4

201C
'
0027

2026
-[lower-alpha 2]
2013

2014
[lower-alpha 3]
2018
[lower-alpha 3]
2019

2039

203A
3_
48
¨[lower-alpha 4]
00A8
~[lower-alpha 4]
007E
˚[lower-alpha 4]
02DA
^[lower-alpha 4]
005E
`[lower-alpha 4]
0060
´[lower-alpha 4]
00B4

201E

201A

201D

2017
nbsp[lower-alpha 3]
00A0
[lower-alpha 3]
FFFD
4_
64
Œ
0152
œ
0153
Ÿ
0178
˙[lower-alpha 3]
02D9
˚[lower-alpha 3][lower-alpha 4]
02DA

255E

255F

258C

2590
[lower-alpha 3]
25CA
[lower-alpha 3]
2318
[lower-alpha 3]
F8FF
[lower-alpha 3]
F8FE
[lower-alpha 3]
2126
5_
80

2568

2564

2565

2559

2558

2552

2553

256B

256A

2561

2562

2556

2555

255C

255B

2567
6_
96
ij
0133
IJ
0132

FB01

FB02
ʼn
0149
ŀ
0140
Ŀ
013F
¯[lower-alpha 3]
00AF
˘[lower-alpha 3]
02D8
˝[lower-alpha 3]
02DD
˛[lower-alpha 3]
02DB
ˇ[lower-alpha 3]
02C7
~[lower-alpha 3][lower-alpha 4]
007E
^[lower-alpha 3][lower-alpha 4]
005E
7_
112

2020

2021
Ħ[lower-alpha 3]
0126
ħ[lower-alpha 3]
0127
Ŧ[lower-alpha 3]
0166
ŧ[lower-alpha 3]
0167

2122

2113
Ŋ[lower-alpha 3]
014A
ŋ[lower-alpha 3]
014B
ĸ[lower-alpha 3]
0138
Kr[lower-alpha 5]
 

2310

20A4

20A7

  Letter  Number  Punctuation  Symbol  Other  Undefined  Mapped to a Unicode private use character

Group 2

LMBCS group 2 code points 128 (80hex) to 255 (FFhex) are identical to the corresponding code points in code page 851 (DOS Greek), whereas code points 1 (01hex) to 127 (7Fhex) are defined according to the following exception list:[lower-alpha 6]

LMBCS Group 2, lower half[11]
_0 _1 _2 _3 _4 _5 _6 _7 _8 _9 _A _B _C _D _E _F
0_
0
NUL
0000
ͺ
037A
΅
0385
Ϊ
03AA
Ϋ
03AB

2015
΄
0384
ʼ
02BC
ʽ
02BD

203E

F862

F863
1_
16

F864

F865

21D5

215E

215D

215C

215B

F867

21D1

21D3

21D2

21D0

F868

21D4

F869

F89F
2_
32

F89E

F89D

F89C

F89B

F89A

F899

F898

F897

F896

F895

F894

F893

F892

F891

F890

F88F
3_
48

F88E

F88D

F88C

F88B

F88A

F889

F888

F887

F886

F885

F884

F883

F882

F881

F880

F866
4_
64

2220

2207

F87F

F87E

F87D

F87C

F87B

F87A

F879

F878

F877

F876

F875

F874

F873

F872
5_
80

2202

2135

2111

211C

F871

F870

F86F

F86E

F86D

F86C

F86B

220B

2208

2209

2286

2287
6_
96

2297

2295

2713

22C0

2201

222B

2200

2203

F86A

2032

2033

221E

221D
φ
03C6

222A

2229
7_
112

2261

2245

2265

2264

2320

2321

2260

2248

2044

2219

2030

221A

207F

2205

2282

2283

  Letter  Number  Punctuation  Symbol  Other  Undefined  Mapped to a Unicode private use character

Group 6

LMBCS group 6 code points 128 (80hex) to 255 (FFhex) are identical to the corresponding code points in code page 852 (DOS Latin-2), whereas code points 1 (01hex) to 127 (7Fhex) are defined according to the following exception list:[lower-alpha 6]

LMBCS Group 6, lower half[11]
_0 _1 _2 _3 _4 _5 _6 _7 _8 _9 _A _B _C _D _E _F
0_
0
NUL
0000
ā
0101
Ĉ
0108
ĉ
0109
Ċ
010A
ċ
010B
Ē
0112
ē
0113
Ė
0116
ė
0117
Ĝ
011C
ĝ
011D
Ġ
0120
ġ
0121
Ģ
0122
ģ
0123
1_
16
Ĥ
0124
ĥ
0125
Ĩ
0128
ĩ
0129
Ī
012A
ī
012B
Į
012E
į
012F
Ĵ
0134
ĵ
0135
Ķ
0136
ķ
0137
Ļ
013B
ļ
013C
Ņ
0145
ņ
0146
2_
32
Ō
014C
ō
014D
Ŗ
0156
ŗ
0157
Ŝ
015C
ŝ
015D
Ũ
0168
ũ
0169
Ū
016A
ū
016B
Ŭ
016C
ŭ
016D
Ų
0172
ų
0173
Ā
0100
3_
48
4_
64
5_
80
6_
96
7_
112

  Letter  Number  Punctuation  Symbol  Other  Undefined

See also

Notes

  1. Lotus 1-2-3 Release 3.0 for DOS and newer versions are written in C.
  2. (U+2010), (U+2011), (U+2012), (U+2013)
  3. According to the documentation this code point is not supported by Lotus 1-2-3 Release 3.1+ for DOS and OS/2 and earlier.
  4. For compatibility with Lotus 1-2-3 Release 5.0.
  5. Unicode does not define a glyph for the crown currency symbol (Krone aka "Kr"), therefore this points to F8FBhex in the Unicode Private Use Area (PUA).
  6. According to the documentation code points 1 to 127 in this group are not supported by Lotus 1-2-3 Release 3.1+ for DOS and OS/2 and earlier. These versions only supported LMBCS code points 0 to 511, covering group 0 and 1 only.

References

  1. Balaban, Bob (2001). "Multi-Language Character Sets – What They Are, How To Use Them" (PDF). Looseleaf Software, Inc. Archived (PDF) from the original on 2016-11-25. Retrieved 2016-11-25.
  2. "Appendix A. Encoding Schemes". IBM Character Data Representation Architecture. IBM (CDRA). Lotus Multi-byte Character Set (LMBCS). Archived from the original on 2016-11-26. Retrieved 2016-11-26. For optimization purposes, the group byte is omitted in Notes for single-byte values between X'20' and X'FF'. For example, LMBCS is always optimized to group 0x01, which means that any character where the first byte is greater than 0x1F, has an implicit group byte of 0x01.
  3. Scherer, Markus; Murray, Brendan (2000-06-02). "Re: MS Excel, Lotus 123 & Unicode". Archived from the original on 2016-12-06. Retrieved 2016-12-06.
  4. "Kapitel 4. Kompatibilität mit anderen 1-2-3 Versionen – Zeichensätze" [Chapter 4. Compatibility with other 1-2-3 Versions – Character Sets]. Lotus 1-2-3 Version 3.1 Upgrader's Handbuch [Upgrader's handbook] (in German) (1 ed.). Cambridge, MA, USA: Lotus Development Corporation. 1989. pp. 4-10–4-11. 302173.
  5. Kamenz, Alfred; Vonhoegen, Helmut (1992). Das große Buch zu Lotus 1-2-3 für DOS (in German) (1 ed.). Data Becker. pp. 131–132, 357–358. ISBN 3-89011-375-3.
  6. Lotus – Inside Notes – The Architecture of Notes and the Domino Server (PDF). Lotus Development Corporation. 2000. Archived (PDF) from the original on 2016-12-12. Retrieved 2016-12-12. […] Notes uses a single character set, the Lotus Multibyte Character Set (LMBCS), to encode all text data used internally by its programs. Whenever Notes first inputs text encoded in a character set other than LMBCS, it translates the text into a LMBCS string, and whenever it must output text in a character set other than LMBCS, it translates the internal LMBCS string into the appropriate character set. Because all text is internally formatted by LMBCS, all text-processing operations […] are done in only one way. LMBCS uses up to three bytes in memory to represent a single text character […]
  7. Murray, Brendan; Snyder-Grant, Jim, eds. (2016) [2000-02-09]. "ucnv_lmb.c". International Components for Unicode. International Business Machines (IBM).
  8. Batutis, Edward J. (2001-11-03). "Re: converter types". International Components for Unicode (ICU). Archived from the original on 2016-12-06. Retrieved 2016-12-06.
  9. "LMBCS" (in Japanese). 2009-02-03. Archived from the original on 2016-11-26. Retrieved 2016-11-26.
  10. "Anhang 2. Der Lotus Multibyte Zeichensatz (LMBCS)" [Appendix 2. The Lotus Multibyte Character Set (LMBCS)]. Lotus 1-2-3 Version 3.1 Referenzhandbuch [Lotus 1-2-3 Version 3.1 Reference Manual] (in German) (1 ed.). Cambridge, MA, USA: Lotus Development Corporation. 1989. pp. A2-1–A2-13. 302168.
  11. "lmb-excp.ucm". 2000-02-10.

Further reading

  • Born, Günter (December 2000) [1990]. "Kapitel 2. LOTUS 1-2-3-Format (WK3)" [Chapter 2. Lotus 1-2-3 WK3 format]. Dateiformate – Eine Referenz – Tabellenkalkulation, Text, Grafik, Multimedia, Sound und Internet [File formats – a reference – spreadsheets, text, graphics, multimedia, sound and internet] (PDF) (in German). Bonn, Germany: Galileo Computing. ISBN 3-934358-83-7. Archived (PDF) from the original on 2016-11-29. Retrieved 2016-11-28. (Includes some information about LMBCS and Lotus system ranges.)
  • "LMBCS tables". User's Guide – 123 Release 4 for Windows (Fax). Lotus Development. 1995 [1994-01-01]. CHAPTER: Appendix A Using the Lotus Multibyte Character Set. Fax 10955. Retrieved 2016-12-06.
  • "Entering LMBCS characters". User's Guide – 123 Release 4 for Windows (Fax). Lotus Development. 1995 [1994-01-01]. CHAPTER: Appendix A Using the Lotus Multibyte Character Set. Fax 10954. Retrieved 2016-12-06.
  • Lotus Support. "Notes from Support: SMTP MTA International Character Sets". IBM developerWorks. IBM. Archived from the original on 2016-12-08. Retrieved 2016-12-08.
  • Character Translation Files (.CTF) by Notes 2.x and Country Language Service (.CLS) files by Notes 3.0 and higher contain information about LMBCS translation into other codepages
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.