Bash programmation (Cygwin): Illegal Character ^M [duplicate]

Question

I have a problem with a character. I think it's a conversion problem between dos and unix.

I have a variable that is a float value. When I print it with the echo command i get:

0.495959

But when I try to make an operation on that value with the bc command (I am not sure how to write the bc command).

echo $mean *1000 |bc

I get:

(standard_in) 1 : illegal character: ^M

I already use the dos2unix command on my .sh file. I think it's because my variable have the ^M character (not printed with the echo command)

How can i eliminate this error?

Hogue

How is $mean obtained? – Windup

Using ImageMagick "identify" command. But as I am on windows maybe it gives his result in a DOS format and not in Unix format. What causes a carriage character to be added.? – Hogue

Okay, but what's the command you are running so $mean is populated? Something like mean=`identify ...`... we need the complete line. – Windup

mean=identify -colorspace gray -format %[fx:mean] $jpg_frame1 – Hogue

Answer 1

I don't have Cygwin handy, but in regular Bash, you can use the tr -d command to strip out specified characters, and you can use the $'...' notation to specify weird characters in a command-line argument (it's like a normal single-quoted string, except that it supports C/Java/Perl/etc.-like escape sequences). So, this:

echo "$mean" * 1000 | tr -d $'\r' | bc

will strip out carriage-returns on the way from echo to bc.

You might actually want to run this:

mean=$(echo "$mean" | tr -d $'\r')

which will modify $mean to strip out any carriage-returns inside, and then you won't have to worry about it in later commands that use it.

(Though it's also worth taking a look at the code that sets $mean to begin with. How does $mean end up having a carriage-return in it, anyway? Maybe you can fix that.)

Quentinquercetin

It's actually best practice to convert ANSI-C style string into integer, I think. Only solution that worked for me! – Tutankhamen

Answer 2

This works:

${mean/^M/}

You can get ^M by typing Ctrl-V followed by Ctrl-M. Or, alternatively:

${mean/$(printf "\r")/}

The benefit of this method compared to @ruakh's is that here you are using bash built-ins only. The first will be faster as the second will run inside a subshell.

If you just want to "unixize" $mean:

mean="${mean/^M/}"

Edit: There's yet another way:

${mean/$'\r'/}

Hephaestus

Answer 3

Running Windows stuff in cygwin has one nasty side-effect as you found out - capturing the output of Windows programs in a cygwin bash variable will also capture the CR output by the program.

Judicious use of d2u avoids the issue - for example,

runtime="`mediainfo --Inform='Video;%Duration%' ${movie} | d2u`"

(Without the d2u, ${runtime} would have a CR tacked on the end, which causes the problem you saw when you feed it to 'bc' for example.)

Eiger

Answer 4

Maybe you should just save your script in UNIX format instead of DOS.

Deoxidize

I already use the dos2unix command to convert it. Is it what you mean? – Hogue

Yes, it is. I don't know why using dos2unix didn't solve your problem. The code looks good to me. I tried it and it works fine. But, if I save the script in dos mode, i get the same error: (standard_in) 1: illegal character: ^M. Usually i don't use dos2unix, just because I'm more comfortable with an editor. To replicate your code I used pspad. Its free, maybe you could give it a try. – Deoxidize

Answer 5

Try this:

echo `echo $mean` *1000 |bc

If echo really isn't printing it, it should work.

Sanfo

That's a good thought, but it won't help. If $mean contains ^M, and Bash recognizes ^M as a word separator (which, by default, it does not), then echo $mean * 1000 will already drop the ^M, because $mean isn't quoted. If it is echo that's adding the ^M, then it obviously won't work (as you've realized), and if the ^M is in the variable and Bash isn't recognizing it as a word separator, then the inner echo will still print the ^M and the outer echo will repeat it -- nothing gained. – Quentinquercetin

@Quentinquercetin You're right, I hadn't thought it very through! – Sanfo

Answer 6

^M is a carriage return character that is used in Windows along with newline (\n) character to indicate next line. However, it is not how it is done in UNIX world, and so bash doesn't treat at as a special character and it breaks the syntax. What you need to do is to remove that character using one of many methods. dos2unix tool can come handy, for example.

Agha

As I mentionned , I already used the dos2unix command on my .sh files. Is it what yu mean or do I have to use in another way? – Hogue

Answer 7

As others have pointed out, this is a Windows line ending issue. There are many ways to fix the problem, but the question is why did this happen in the first place.

I can see this happening in several places:

• This is a WINDOWS environment variable that was set when Cygwin started up. Sometimes these variables get a CRLF on the end of them. You mentioned this was a particular issue with this one variable, but you didn't specify where it was set.

• You edited this file using a Windows text editor like Notepad or Winpad.

Never use a text editor to edit a program. Use a program editor. If you like VI, download VIM which is available on Windows and comes on Cygwin (and all other Unix-based platforms). If VIM isn't for you, try the more graphically based Notepad++. Both of these editors handle end of line issues, and can create scripts with Unix line endings in Windows or files with Windows line endings in Cygwin.

---

• If you use VIM, you can do the following to change line endings and to set them:

  • To see the line ending in the current file, type :set ff? while in command mode.
  • To set the line ending for Unix, type :set ff=unix while in command mode.
  • To set the line ending for Windows, type :set ff=dos while in command mode.

• If you use Notepad++

  • You can go into the Edit-->EOL Conversion menu item and see what your current line ending setting (it's the one not highlighted) and change it.
  • To have Notepad++ use Unix line endings as the default, go into the Settings-->Preferences menu item. In the Dialog box, select the New Document/Default Directory tab. In the Format section which is part of the New Document section, select the line ending you want. WARNING: Do not select Mac as an option. That doesn't even work on Macs. If you have a Mac, select Unix.

Vaughan

Thanks, I'll give it a try. btw I was using Wordpad, and that probably causes the problem.. – Hogue