Newline

Unicode

The Unicode standard defines a number of characters that conforming applications should recognize as line terminators:^[6]

 LF:     Line Feed, U+000A

 VT:     Vertical Tab, U+000B

 FF:     Form Feed, U+000C

 CR:     Carriage Return, U+000D

 CR+ LF: CR ( U+000D) followed by LF ( U+000A)

 NEL:    Next Line, U+0085

 LS:     Line Separator, U+2028

 PS:     Paragraph Separator, U+2029

This may seem overly complicated compared to an approach such as converting all line terminators to a single character, for example LF. However, Unicode was designed to preserve all information when converting a text file from any existing encoding to Unicode and back. Therefore, Unicode should contain characters included in existing encodings. NEL is included in EBCDIC with code (0x15). NEL is also a control character in the C1 control set.^[7] As such, it is defined by ECMA 48,^[8] and recognized by encodings compliant with ISO/IEC 2022 (which is equivalent to ECMA 35).^[9] C1 control set is also compatible with ISO-8859-1. The approach taken in the Unicode standard allows round-trip transformation to be information-preserving while still enabling applications to recognize all possible types of line terminators.

Recognizing and using the newline codes greater than 0x7F (NEL, LS and PS) is not often done. They are multiple bytes in UTF-8, and the code for NEL has been used as the ellipsis ('…') character in Windows-1252. For instance:

• ECMAScript^[10] accepts LS and PS as line breaks, but considers U+0085 (NEL) white space, not a line break.
• Windows 10 does not treat any of NEL, LS, or PS as line-break in the default text editor Notepad
• On Linux, a popular editor, gedit, treats LS and PS as newlines but does not for NEL.
• YAML^[11] no longer recognizes them as special, in order to be compatible with JSON.
• JSON treats LS and PS inside a String as letter while ECMAScript/Javascript treats them as new line and shows an error

The Unicode characters U+2424 (SYMBOL FOR NEWLINE, ␤), U+23CE (RETURN SYMBOL, ⏎), U+240D (SYMBOL FOR CARRIAGE RETURN, ␍) and U+240A (SYMBOL FOR LINE FEED, ␊) are intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline.

Escape sequences

An escape sequence is a combination of characters which represents no text; instead of being displayed (as text) it is supposed to be intercepted by the program and a special function is supposed to be performed. Escape sequences are also used to handle (set, search, replace, etc.) special characters.

In programming languages

To facilitate the creation of portable programs, programming languages provide some abstractions to deal with the different types of newline sequences used in different environments.

The C programming language provides the escape sequences '\n' (newline) and '\r' (carriage return). However, these are not required to be equivalent to the ASCII LF and CR control characters. The C standard only guarantees two things:

1. Each of these escape sequences maps to a unique implementation-defined number that can be stored in a single char value.
2. When writing to a file, device node, or socket/fifo in text mode, '\n' is transparently translated to the native newline sequence used by the system, which may be longer than one character. When reading in text mode, the native newline sequence is translated back to '\n'. In binary mode, no translation is performed, and the internal representation produced by '\n' is output directly.

On Unix platforms, where C originated, the native newline sequence is ASCII LF ( 0x0A), so '\n' was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode is a no-op, and Unix has no notion of text mode or binary mode. This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms.

The C library function fgets() is best avoided in binary mode because any file not written with the Unix newline convention will be misread. Also, in text mode, any file not written with the system's native newline sequence (such as a file created on a Unix system, then copied to a Windows system) will be misread as well.

Another common problem is the use of '\n' when communicating using an Internet protocol that mandates the use of ASCII CR+ LF for ending lines. Writing '\n' to a text mode stream works correctly on Windows systems, but produces only LF on Unix, and something completely different on more exotic systems. Using "\r\n" in binary mode is slightly better.

Many languages, such as C++, Perl,^[12] and Haskell provide the same interpretation of '\n' as C.

Java, PHP,^[13] and Python^[14] provide the '\r\n' sequence (for ASCII CR+ LF). In contrast to C, these are guaranteed to represent the values U+000D and U+000A, respectively.

The Java I/O libraries do not transparently translate these into platform-dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of CR, LF, or CR+ LF as a line terminator (see BufferedReader.readLine()). The System.lineSeparator() method can be used to retrieve the underlying line separator.

Example:

   String eol = System.lineSeparator();
   String lineColor = "Color: Red" + eol;

Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file.^[15]

Some languages have created special variables, constants, and subroutines to facilitate newlines during program execution. In some languages such as PHP and Perl, double quotes are required to perform escape substitution for all escape sequences, including '\n' and '\r'. In PHP, to avoid portability problems, newline sequences should be issued using the PHP_EOL constant.^[16]

Example in C#:

   string eol = Environment.NewLine;
   string lineColor = "Color: Red" + eol;
   
   string eol2 = "\n";
   string lineColor2 = "Color: Blue" + eol2;

Issues with different newline formats

A text file created with gedit and viewed with a hex editor. Besides the text objects, there are only EOL markers with the hexadecimal value 0A.

Even though the control characters are unambiguously defined in the corresponding character encoding table used by a text file, there still is an issue: there are different conventions to set and display a line break.

To denote a single line break, Unix programs use line feed, whose hexadecimal value in ASCII is 0a, while most programs common to MS-DOS and Microsoft Windows use carriage return+line feed, whose hexadecimal value in ASCII is 0d 0a. In ASCII, carriage return is a distinct control character.

The different newline conventions cause text files that have been transferred between systems of different types to be displayed incorrectly.

Text in files created with programs which are common on Unix-like or classic Mac OS, appear as a single long line on most programs common to MS-DOS and Microsoft Windows because these do not display a single line feed or a single carriage return as a line break.

Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extra CR may be displayed as a second line break, as ^M, or as <cr> at the end of each line.

Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file.

The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. For example, a compiler may fail with obscure syntax errors even though the source file looks correct when displayed on the console or in an editor. On a Unix-like system, the command cat -v myfile.txt will send the file to stdout (normally the terminal) and make the ^M visible, which can be useful for debugging. Modern text editors generally recognize all flavours of CR+ LF newlines and allow users to convert between the different standards. Web browsers are usually also capable of displaying text files and websites which use different types of newlines.

Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save—often inconsistently.

Most textual Internet protocols (including HTTP, SMTP, FTP, IRC, and many others) mandate the use of ASCII CR+ LF ( '\r\n', 0x0D 0x0A) on the protocol level, but recommend that tolerant applications recognize lone LF ( '\n', 0x0A) as well. Despite the dictated standard, many applications erroneously use the C newline escape sequence '\n' ( LF) instead of the correct combination of carriage return escape and newline escape sequences '\r\n' ( CR+ LF) (see section Newline in programming languages above). This accidental use of the wrong escape sequences leads to problems when trying to communicate with systems adhering to the stricter interpretation of the standards instead of the suggested tolerant interpretation. One such intolerant system is the qmail mail transfer agent that actively refuses to accept messages from systems that send bare LF instead of the required CR+ LF.^[17]

The standard Internet Message Format^[18] for eMail states: CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body.

The File Transfer Protocol can automatically convert newlines in files being transferred between systems with different newline representations when the transfer is done in "ASCII mode". However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively corrupting the file. FTP clients often employ some heuristics (for example, inspection of filename extensions) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.^[19]

Conversion between newline formats

Text editors are often used for converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCII CR/ LF conventions. The standard Windows editor Notepad is not one of them (although WordPad and the MS-DOS Editor are).

Editors are often unsuitable for converting larger files. For larger files (on Windows NT/2000/XP) the following command is often used:

D:\>TYPE unix_file | FIND /V "" > dos_file

On many Unix systems, the dos2unix (sometimes named fromdos or d2u) and unix2dos (sometimes named todos or u2d) utilities are used to translate between ASCII CR+ LF (DOS/Windows) and LF (Unix) newlines. Different versions of these commands vary slightly in their syntax. However, the tr command is available on virtually every Unix-like system and can be used to perform arbitrary replacement operations on single characters. A DOS/Windows text file can be converted to Unix format by simply removing all ASCII CR characters with

$ tr -d '\r' < inputfile > outputfile

or, if the text has only CR newlines, by converting all CR newlines to LF with

$ tr '\r' '\n' < inputfile > outputfile

The same tasks are sometimes performed with awk, sed, or in Perl if the platform has a Perl interpreter:

$ awk '{sub("$","\r\n"); printf("%s",$0);}' inputfile > outputfile  # UNIX to DOS  (adding CRs on Linux and BSD based OS that haven't GNU extensions)
$ awk '{gsub("\r",""); print;}' inputfile > outputfile              # DOS to UNIX  (removing CRs on Linux and BSD based OS that haven't GNU extensions)
$ sed -e 's/$/\r/' inputfile > outputfile              # UNIX to DOS  (adding CRs on Linux based OS that use GNU extensions)
$ sed -e 's/\r$//' inputfile > outputfile              # DOS  to UNIX (removing CRs on Linux based OS that use GNU extensions)
$ cat inputfile | tr -d "\r" > outputfile              # DOS  to UNIX (removing CRs using tr(1). Not Unicode compliant.)
$ perl -pe 's/\r?\n|\r/\r\n/g' inputfile > outputfile  # Convert to DOS
$ perl -pe 's/\r?\n|\r/\n/g'   inputfile > outputfile  # Convert to UNIX
$ perl -pe 's/\r?\n|\r/\r/g'   inputfile > outputfile  # Convert to old Mac

To identify what type of line breaks a text file contains, the file command can be used. Moreover, the editor Vim can be convenient to make a file compatible with the Windows notepad text editor. For example:

 $ file myfile.txt
 myfile.txt: ASCII English text
 $ vim myfile.txt

within vim

 :set fileformat=dos
 :wq

 $ file myfile.txt
 myfile.txt: ASCII English text, with CRLF line terminators

The following grep commands echo the filename (in this case myfile.txt) to the command line if the file is of the specified style:

$ grep -PL $'\r\n' myfile.txt # show UNIX style file (LF terminated)
$ grep -Pl $'\r\n' myfile.txt # show DOS style file (CRLF terminated)

For systems with egrep (extended grep) such as Debian (Linux) based systems and many other Unix systems, these commands can be used:

$ egrep -L $'\r\n' myfile.txt # show UNIX style file (LF terminated)
$ egrep -l $'\r\n' myfile.txt # show DOS style file (CRLF terminated)

The above grep commands work under Unix systems or in Cygwin under Windows. Note that these commands make some assumptions about the kinds of files that exist on the system (specifically it's assuming only Unix and DOS-style files—no Mac OS 9-style files).

This technique is often combined with find to list files recursively. For instance, the following command checks all "regular files" (e.g. it will exclude directories, symbolic links, etc.) to find all Unix-style files in a directory tree, starting from the current directory (.), and saves the results in file unix_files.txt, overwriting it if the file already exists:

$ find . -type f -exec grep -PL '\r\n' {} \; > unix_files.txt

This example will find C files and convert them to LF style line endings:

$ find -name '*.[ch]' -exec fromdos {} \;

The file command also detects the type of EOL used:

$ file myfile.txt
myfile.txt: ASCII text, with CRLF line terminators

Other tools permit the user to visualise the EOL characters:

$ od -a myfile.txt
$ cat -e myfile.txt
$ hexdump -c myfile.txt

dos2unix, unix2dos, mac2unix, unix2mac, mac2dos, dos2mac can perform conversions. The flip^[20] command is often used.