Counting and modifying lines, words and characters in Linux text files

Linux includes some useful commands for counting when it comes to text files. This post examines some of the options for counting lines and words and making changes that might help you see what you want.

Counting lines

Counting lines in a file is very easy with the wc command. Use a command like that shown below, and you’ll get a quick response.

$ wc -l myfile
132 myfile

What the wc command is actually counting is the number of newline characters in a file. So, if you had a single-line file with no newline character at the end, it would tell you the file has 0 lines,

The wc -l command can also count the lines in any text that is piped to it. In the example below, wc -l is counting the number of files and directories in the current directory.

$ ls -l | wc -l
1184

If you pipe text to a wc command with a hyphen as its argument, wc will count the lines, words and characters.

$ echo hello to you | wc -
      1       3      13 -

The responses show the number of lines (1), words (3) and characters (13 counting the newline).

If you want to get the same information for a file, pipe the file to the wc command as shown below.

$ cat notes | wc -
     48     613    3705 -

Counting words

For just a word count, use the w option as shown in the examples below.

$ wc -w notes
613 TT2
$ date | wc -w
7

Counting characters

To count the characters in a file, use the -c option. Keep in mind that this will count newline characters as well as letters and punctuation marks.

$ wc -c TT2
3705 TT2

Counting instances of particular words

Counting how many times a particular word appears in a file is a lot more complex. To count how many lines contain a word is considerably easier.

$ cat notes | grep the | wc -l
32
$ cat notes | grep [Tt]he | wc -l
40

The second command above counts lines containing “the” whether or not the word is capitalized. It still doesn’t tell you how many times “the” appears overall, because any line containing the word more than once gets counted only once.

Ignoring punctuation and capitalization

Some words (e.g., “The” and “the”) will appear in your word lists more than once. You’re also going to see strings like “end” and “end.” since the commands described above don’t separate words from punctuation. To move past these problems, some additional commands are added in the examples that follow.

Removing punctuation

In the command below, a file containing a long string of punctuation characters is passed to a tr -d command that removes all of them from the output. Notice how everything except the “Characters ” string is removed from the output.

$ cat punct-chars
Characters .?,"!;:'{}[]():
$ cat punct-chars | tr -d '[:punct:]'
Characters

Changing text to all lowercase

A tr command can turn all character to lowercase to ensure that words that start with a capital letter (often because they start the sentence) or contain all capitals aren’t listed separately from those appearing in all lowercase.

$ echo "Hello to YOU" | tr '[A-Z]' '[a-z]'
hello to you

Using a script

The script below sets up three sets of commands for extracting the contents of a text file and extracting the words using increasingly more thorough strategies, so that you can see the output at each phase.

NOTE: The script passes the final collections of output to the column command to make the output a little easier to view.

#!/bin/bash

echo -n "file: "
read file

# separate file into wor-per-line format
tr -s '[:blank:]' '[n]' < $file > $file-2

# list words in columnar format
sort $file-2 | uniq -c | column

echo -n "try next command?> "
read ans

# removing punctuation
sort $file-2 | tr -d '[:punct:]' | uniq -c | column

echo -n "try next command?> "
read ans

# changing text to all lowercase
sort $file-2 | tr -d '[:punct:]' | tr '[A-Z]' '[a-z]' | uniq -c | column

The output below shows what you would see if you ran the script against the following Einstein quote:

"Two things are infinite: the universe and human stupidity; and I'm not sure about the universe."
― Albert Einstein

$ word-by-word
file: Einstein
      1 ―                     1 human                 2 the
      1 about                 1 I'm                   1 things
      1 Albert                1 infinite:             1 "Two
      2 and                   1 not                   1 universe
      1 are                   1 stupidity;            1 universe."
      1 Einstein              1 sure
try next command?> y
      1 ―                     1 human                 2 the
      1 about                 1 Im                    1 things
      1 Albert                1 infinite              1 Two
      2 and                   1 not                   2 universe
      1 are                   1 stupidity
      1 Einstein              1 sure
try next command?> y
      1 ―                     1 human                 2 the
      1 about                 1 im                    1 things
      1 albert                1 infinite              1 two
      2 and                   1 not                   2 universe
      1 are                   1 stupidity
      1 einstein              1 sure

Some of the effects of eliminating punctuation have a downside as they remove the apostrophes from contractions like “it’s”. The script also decapitalizes proper names.

Note that the hyphen is not removed from the Einstein quote by the punctuation elimination command. In addition, if your text includes left- and right-leaning double quotes, they also won’t be eliminated. This is because these characters are not included in the definition of ‘[:punct:]’.

Wrap-up

Linux includes a number of ways for counting lines, words and characters in text and for making modifications that help count the words. Some are just a bit more complex than others.