• Uncategorized

About linux : Looping-through-the-content-of-a-file-in-Bash

Question Detail

How do I iterate through each line of a text file with Bash?

With this script:

echo “Start!”
for p in (peptides.txt)
echo “${p}”

I get this output on the screen:

./runPep.sh: line 3: syntax error near unexpected token `(‘
./runPep.sh: line 3: `for p in (peptides.txt)’

(Later I want to do something more complicated with $p than just output to the screen.)

The environment variable SHELL is (from env):


/bin/bash –version output:

GNU bash, version 3.1.17(1)-release (x86_64-suse-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

cat /proc/version output:

Linux version (geeko@buildhost) (gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)) #1 SMP Mon Nov 27 11:46:27 UTC 2006

The file peptides.txt contains:


Question Answer

One way to do it is:

while read p; do
echo “$p”
done outfile.txt

I’ve used these as written above because I have used text files where I’ve created them with one word per line. (See comments) If you have spaces that you don’t want splitting your words/lines, it gets a little uglier, but the same command still works as follows:

OLDIFS=$IFS; IFS=$’\n’; for line in $(cat peptides.txt); do cmd_a.sh $line; cmd_b.py $line; done > outfile.txt; IFS=$OLDIFS

This just tells the shell to split on newlines only, not spaces, then returns the environment back to what it was previously. At this point, you may want to consider putting it all into a shell script rather than squeezing it all into a single line, though.

Best of luck!
A few more things not covered by other answers:

Reading from a delimited file

# ‘:’ is the delimiter here, and there are three fields on each line in the file
# IFS set below is restricted to the context of `read`, it doesn’t affect any other code
while IFS=: read -r field1 field2 field3; do
# process the fields
# if the line has less than three fields, the missing fields will be set to an empty string
# if the line has more than three fields, `field3` will get all the values, including the third field plus the delimiter(s)
done < input.txt Reading from the output of another command, using process substitution while read -r line; do # process the line done < <(command ...) This approach is better than command ... | while read -r line; do ... because the while loop here runs in the current shell rather than a subshell as in the case of the latter. See the related post A variable modified inside a while loop is not remembered. Reading from a null delimited input, for example find ... -print0 while read -r -d '' line; do # logic # use a second 'read ... <<< "$line"' if we need to tokenize the line done < <(find /path/to/dir -print0) Related read: BashFAQ/020 - How can I find and safely handle file names containing newlines, spaces or both? Reading from more than one file at a time while read -u 3 -r line1 && read -u 4 -r line2; do # process the lines # note that the loop will end when we reach EOF on either of the files, because of the `&&` done 3< input1.txt 4< input2.txt Based on @chepner's answer here: -u is a bash extension. For POSIX compatibility, each call would look something like read -r X <&3. Reading a whole file into an array (Bash versions earlier to 4) while read -r line; do my_array+=("$line") done < my_file If the file ends with an incomplete line (newline missing at the end), then: while read -r line || [[ $line ]]; do my_array+=("$line") done < my_file Reading a whole file into an array (Bash versions 4x and later) readarray -t my_array < my_file or mapfile -t my_array < my_file And then for line in "${my_array[@]}"; do # process the lines done More about the shell builtins read and readarray commands - GNU More about IFS - Wikipedia BashFAQ/001 - How can I read a file (data stream, variable) line-by-line (and/or field-by-field)? Related posts: Creating an array from a text file in Bash What is the difference between thee approaches to reading a file that has just one line? Bash while read loop extremely slow compared to cat, why? ............................................................ Use a while loop, like this: while IFS= read -r line; do echo "$line" done

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.