• Uncategorized

About bash : While-loop-stops-reading-after-the-first-line-in-Bash

Question Detail

I have the following shell script. The purpose is to loop thru each line of the target file (whose path is the input parameter to the script) and do work against each line. Now, it seems only work with the very first line in the target file and stops after that line got processed. Is there anything wrong with my script?

# SCRIPT: do.sh
# PURPOSE: loop thru the targets 


echo "proceed with $FILENAME"

while read LINE; do
   let count++
   echo "$count $LINE"
   sh ./do_work.sh $LINE
done < $FILENAME

echo "\ntotal $count targets"

In do_work.sh, I run a couple of ssh commands.

Question Answer

The problem is that do_work.sh runs ssh commands and by default ssh reads from stdin which is your input file. As a result, you only see the first line processed, because the command consumes the rest of the file and your while loop terminates.

This happens not just for ssh, but for any command that reads stdin, including mplayer, ffmpeg, HandBrakeCLI, httpie, brew install, and more.

To prevent this, pass the -n option to your ssh command to make it read from /dev/null instead of stdin. Other commands have similar flags, or you can universally use < /dev/null.

A very simple and robust workaround is to change the file descriptor from which the read command receives input.

This is accomplished by two modifications: the -u argument to read, and the redirection operator for < $FILENAME.

In BASH, the default file descriptor values (i.e. values for -u in read) are:

  • 0 = stdin
  • 1 = stdout
  • 2 = stderr

So just choose some other unused file descriptor, like 9 just for fun.

Thus, the following would be the workaround:

while read -u 9 LINE; do
   let count++
   echo "$count $LINE"
   sh ./do_work.sh $LINE
done 9< $FILENAME

Notice the two modifications:

  1. read becomes read -u 9
  2. < $FILENAME becomes 9< $FILENAME

As a best practice, I do this for all while loops I write in BASH.
If you have nested loops using read, use a different file descriptor for each one (9,8,7,…).

More generally, a workaround which isn’t specific to ssh is to redirect standard input for any command which might otherwise consume the while loop’s input.

while read -r LINE; do
   let count++
   echo "$count $LINE"
   sh ./do_work.sh "$LINE" </dev/null
done < "$FILENAME"

The addition of </dev/null is the crucial point here (though the corrected quoting is also somewhat important; see also When to wrap quotes around a shell variable?). You will want to use read -r unless you specifically require the legacy slightly odd behavior you get without -r.

Another workaround of sorts which is somewhat specific to ssh is to make sure any ssh command has its standard input tied up, e.g. by changing

ssh otherhost some commands here

to instead read the commands from a here document, which conveniently (for this particular scenario) ties up the standard input of ssh for the commands:

ssh otherhost <<'____HERE'
    some commands here

ssh -n option prevents checking the exit status of ssh when using HEREdoc while piping output to another program.
So use of /dev/null as stdin is preferred.

while read ONELINE ; do
   ssh [email protected]_xyz </dev/null <<EOF 2>&1 | filter_pgm 
   echo "Hi, $ONELINE. You come here often?"
   if [ ${PIPESTATUS[0]} -ne 0 ] ; then
      echo "aborting loop"
      exit ${PIPESTATUS[0]}
done << input_list.txt

This was happening to me because I had set -e and a grep in a loop was returning with no output (which gives a non-zero error code).

You may also like...

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.