1. Introduction

When configuring software for compilation or simply running any script, we might sometimes encounter errors with a relatively cryptic text:

  • /bin/sh^M: bad interpreter
  • /bin/bash^M: bad interpreter
  • syntax error: unexpected end of file
  • -bash: $’sleep\r’: command not found

In this tutorial, we’ll tackle line endings and errors that we might see when they are incorrect. First, we discuss the general concept of how lines end. Next, we explore different problems that line endings can cause. After that, we’ll talk about specific types of scripts that can exhibit line-end issues. Finally, we delve into ways to make sure a file uses given characters as newlines.

For brevity and clarity, when the characters would otherwise be hidden, we often use <CR> to represent a carriage return and <LF> for the line feed character as per the standard. This is done in every context, including code snippets. Also, we use some terms interchangeably:

  • line ending
  • line break
  • newline

We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. It should work in most POSIX-compliant environments.

2. Line Endings

How a file line ends isn’t universal. Moreover, we can ignore the concept of lines altogether, instead treating files as binary, i.e., a sequence of bytes.

When it comes to text files with separate lines, Microsoft Windows, Apple macOS, and Linux differ in the default line endings that each operating system (OS) expects:

  • Microsoft Windows – <CR><LF>
  • Apple macOS – initially <CR>, currently <LF>
  • Linux – <LF>

Further, since some software products perform automatic conversions between these options, we can end up with different issues.

3. Problems With Line Endings

While many applications support all ways to end a line, some sensitive texts might not work without the correct ending.

3.1. Source Code File Newlines

Also called tokens, character sequences in any source code should adhere to the syntax of their language.

So, depending on the source file and how it’s used, we might encounter different problems stemming from its line endings.

On the one hand, some file types like the Portable Document Format (PDF) can work with any version of a newline, but we have to take the number of bytes into account for, e.g., the xref table:

$ cat lf.pdf
0 5
0000000000 65535 f
0000000018 00000 n
0000000077 00000 n
0000000178 00000 n
0000000457 00000 n
$ cat crlf.pdf
0 5
0000000000 65535 f
0000000021 00000 n
0000000086 00000 n
0000000195 00000 n
0000000490 00000 n

On the other hand, specific line breaks are sometimes a must as most interpreters and compilers need to know what to expect for a newline sequence in files.

3.2. Shell Newlines

Critically, attempting to mix and match line endings in a command, script, or a sensitive file can result in issues.

For instance, shells have strict rules when it comes to the type and order of character sequences:

$ test<LF>

Notably, the <LF> character is essential in this simple Bash test command example, as that’s what the shell expects.

To demonstrate, let’s try to prepend a <CR>, converting the line ending to another format:

$ test<CR><LF>
-bash: $'test\r': command not found

By using Ctrl-v Ctrl-m, we append <CR> to test. After that, we can either press Return or leverage Ctrl-m to add the actual terminator.

Of course, the command line is only accepted after <LF>, and that’s the only character omitted from the sequence, which the shell interprets as a command. Thus, Bash essentially tries and fails to execute the command test<CR>, which is similar to attempting to run testRANDOMCHARACTERS.

3.3. Shell Script Newlines

Of course, what applies to the shell is also valid for shell scripts. In addition, scripts have some unique constructs that can pose further issues.

One of these is the shebang, which hints at the interpreter of the source file:

$ cat script.sh
$ chmod +x script.sh
$ ./script.sh
-bash: ./script.sh: /bin/bash^M: bad interpreter: No such file or directory

Here, we see an error, which includes the special character notation ^M instead of <CR>.

The error itself states that /bin/bash<CR> is a bad interpreter, which it is, similar to test<CR>. In this case, the problem is in the #!/bin/bash shebang, which is only activated when running an executable script directly but is otherwise ignored:

$ cat script.sh
$ bash script.sh
script.sh: line 2: $'test\r': command not found

Notably, we don’t see a shebang problem here.

Let’s see a common scenario where errors like the above arise.

4. Autotools Configure Scripts

The Autotools configure script mechanism is mainly used to prepare code for compilation on a given platform:

  • detect available resources
  • set executable paths
  • identify library locations
  • configure linking

Often, we get configure scripts as part of a source package, which we’ve downloaded. In some cases, like when using Git or FTP, the downloads themselves can convert newlines.

Since it’s a shell script produced by Autoconf, configure is susceptible to all issues with newlines we already discussed:

$ cat configure
# Define the name and version of the package.<CR><LF>
$ ./configure
-bash: ./script.sh: /bin/sh^M: bad interpreter: No such file or directory

Notably, the shebang of configure causes an error due to unexpected line endings. However, the rest of the file might not raise any exceptions as it mainly, if not exclusively, consists of variable assignments.

Critically, this is a double-edged sword, as not erroring out doesn’t mean we won’t get issues down the line.

5. External Interpreter Scripts

Same as for internal scripts, the shell can understand the first line of external interpreter scripts as well. Let’s look at some common examples.

5.1. Perl

To begin with, Perl is a scripted language common to many Linux installations.

So, we can write a basic Perl script:

$ cat script.pl
#!/usr/bin/env perl<LF>
print "Perl script output."<LF>

In this short program, we just set the shebang to point to the perl interpreter and then print some text to the standard output:

$ chmod +x script.pl
$ ./script.pl
Perl script output.

Naturally, the code runs without issues.

However, let’s try to run the same script after replacing the <LF> with a <CR><LF>:

$ cat script.pl
#!/usr/bin/env perl<CR><LF>
print "Perl script output."<CR><LF>
$ ./script.pl
/usr/bin/env: ‘perl\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines

As expected, the shell detects the unexpected carriage return and considers it part of the line, leading to an [env]ironment search of perl\r. Naturally, this results in an error.

5.2. Python

Similar to Perl, unexpected line endings in Python scripts can lead to errors:

$ cat script.py
#!/usr/bin/env python<CR><LF>
print "Python script output."<CR><LF>
$ chmod +x script.py
$ ./script.py
/usr/bin/env: ‘python\r’: No such file or directory

Of course, the same error happens with python, python3, and other Python binaries.

However, because of the different executables for the Python interpreter, we might exhibit another very similar-looking error:

$ cat badscript.py
#!/usr/bin/env pythonr
print "Python script output."
$ chmod +x badscript.py
/usr/bin/env: ‘pythonr’: No such file or directory

In fact, the only difference between the two errors is the backslash. Although a very important character, the distinction might not be immediately obvious.

While the first error tells us that there is a <CR> at the end of the line that causes the interpreter name to be corrupted, the second one just notifies us that such an interpreter doesn’t exist, perhaps due to missing dependencies or because of a typo.

6. Fixing Line Endings

Generally, if a language or script expects given line endings, we should use them. Still, there are many options to detect and convert line breaks in files:

  • vi with the set ff=unix file format command, even automatically as vi +’:wq ++ff=unix’
  • sed with a simple substitution regular expression (regex) like ‘s/\r//’
  • tr with -d and the unwanted character, e.g., ‘\r’
  • awk with the gsub() function like ‘gsub(/\r/,””)’
  • perl with an -e one-liner like ‘s/\r//’
  • dos2unix for simple <CR><LF> to <LF> conversion

In addition, we can avoid the shebang issues if we run a script as an argument of its interpreter instead of directly. This fact is especially important for configure scripts:

$ ./configure
-bash: ./script.sh: /bin/sh^M: bad interpreter: No such file or directory
$ bash configure

Still, to avoid any trouble, using an editor like vi to manually edit or automatically convert line endings can be beneficial.

7. Summary

In this article, we talked about newlines, how they can affect file processing, and what we can do about it.

In conclusion, having the correct line endings is often critical for a file to properly display or run, especially when it comes to shell scripts.

Comments are open for 30 days after publishing a post. For any issues past this date, use the Contact form on the site.