Lecture 10. Strings


Characters

Strings are not like integers, floats, and booleans. A string is a sequence, which means it is an ordered collection of other values, specifically a string is a sequence of characters. Of course, the real trouble comes when one asks what a character is. The characters that English speakers are familiar with are the letters A, B, C, etc., together with numerals and common punctuation symbols. These characters are standardized together with a mapping to integer values between 0 and 127 by the ASCII standard. There are, of course, many other characters used in non-English languages, including variants of the ASCII characters with accents and other modifications, related scripts such as Cyrillic and Greek, and scripts completely unrelated to ASCII and English, including Arabic, Chinese, Hebrew, Hindi, Japanese, and Korean. The Unicode standard tackles the complexities of what exactly a character is, and is generally accepted as the definitive standard addressing this problem.

A Char value represents a single character: it is just a 32-bit primitive type with a special literal representation and appropriate arithmetic behaviors, whose numeric value is interpreted as a Unicode code point. Here is how Char values are input and shown:


In [ ]:
'x'

In [ ]:
typeof('x')

You can convert a Char to its integer value, i.e. code point, easily:


In [ ]:
Int('x')

You can convert an integer value back to a Char just as easily:


In [ ]:
Char(120)

You can do comparisons and a limited amount of arithmetic with Char values:


In [ ]:
'A' < 'a'

In [ ]:
'A' <= 'a' <= 'Z'

In [ ]:
'A' <= 'X' <= 'Z'

In [ ]:
'x' - 'a'

In [ ]:
'A' + 1

A String is a Sequence

String literals are delimited by double quotes or triple double quotes:


In [ ]:
str = "Hello, World!\n"

In [ ]:
"""Contains "quote" characters"""

The backslash \ is used as an escape sequence.

If you want to extract a character from a string, you index into it:


In [ ]:
str[1]

In [ ]:
str[6]

In [ ]:
str[end]

All indexing in Julia is 1-based: the first element of any integer-indexed object is found at index 1.

In any indexing expression, the keyword end can be used as a shorthand for the last index. You can perform arithmetic and other operations with end, just like a normal value:


In [ ]:
str[end-1]

In [ ]:
str[end÷2]

The value of the index has to be an integer. Otherwise you get:


In [ ]:
str[1.5]

length

length is a built-in function that returns the number of characters in a string:


In [ ]:
fruit = "🍌 🍎 🍐"
len = length(fruit)

To get the last letter of a string, you might be tempted to try something like this:


In [ ]:
last = fruit[len]

String literals are encoded using the UTF-8 encoding. UTF-8 is a variable-width encoding, meaning that not all characters are encoded in the same number of bytes. This means that not every byte index into a UTF-8 string is necessarily a valid index for a character. If you index into a string at such an invalid byte index, an error is thrown:


In [ ]:
fruit[1]

In [ ]:
fruit[2]

In [ ]:
fruit[3]

In [ ]:
fruit[4]

In [ ]:
fruit[5]

In this case, the character 🍌 is a four-byte character, so the indices 2, 3 and 4 are invalid and the next character's index is 5; this next valid index can be computed by nextind(fruit, 1), and the next index after that by nextind(fruit, 4) and so on.


In [ ]:
i = nextind(fruit, 5)
fruit[i]

The function sizeof gives the number of bytes in a string:


In [ ]:
sizeof("∀ x ∈ X")

Traversal with a for loop

A lot of computations involve processing a string one character at a time. Often they start at the beginning, select each character in turn, do something to it, and continue until the end. This pattern of processing is called a traversal. One way to write a traversal is with a while loop:


In [ ]:
index = 1
while index <= sizeof(fruit)
    letter = fruit[index]
    println(letter)
    index = nextind(fruit, index)
end

This loop traverses the string and displays each letter on a line by itself. The loop condition is index <= sizeof(fruit), so when index is larger than the number of bytes in the string, the condition is false, and the body of the loop doesn’t run.

Fortunately, the above awkward idiom is unnecessary for iterating through the characters in a string, since you can just use the string as an iterable object:


In [ ]:
for letter in fruit
    println(letter)
end

Each time through the loop, the next character in the string is assigned to the variable letter. The loop continues until no characters are left.

The following example shows how to use concatenation (string multiplication) and a for loop to generate an abecedarian series (that is, in alphabetical order). In Robert McCloskey’s book "Make Way for Ducklings", the names of the ducklings are Jack, Kack, Lack, Mack, Nack, Ouack, Pack, and Quack. This loop outputs these names in order:


In [ ]:
prefixes = "JKLMNOPQ"
suffix = "ack"

for letter in prefixes
    println(letter * suffix)
end

Of course, that’s not quite right because “Ouack” and “Quack” are misspelled...

String Slices

A segment of a string is called a slice. Selecting a slice is similar to selecting a character:


In [ ]:
s = "Julius Caesar"
s[1:6]

The operator [n:m] returns the part of the string from the “n-eth” byte to the “m-eth” byte. So the same caution is needed as for simple indexing.

The end keyword can be used to indicate the last byte of the string:


In [ ]:
s[8:end]

If the first index is greater than the second the result is an empty string, represented by two double quotation marks:


In [ ]:
s[8:7]

An empty string contains no characters and has length 0, but other than that, it is the same as any other string.

String Interpolation

Constructing strings using concatenation can become a bit cumbersome, however. To reduce the need for these verbose calls to string() or repeated multiplications, Julia allows interpolation into string literals using $:


In [ ]:
greet = "Hello"
whom = "World"
"$greet, $(whom)!"

This is more readable and convenient and equivalent to the above string concatenation – the system rewrites this apparent single string literal into a concatenation of string literals with variables.

The shortest complete expression after the $ is taken as the expression whose value is to be interpolated into the string. Thus, you can interpolate any expression into a string using parentheses:


In [ ]:
"1 + 2 = $(1 + 2)"

Strings are immutable

It is tempting to use the [] operator on the left side of an assignment, with the intention of changing a character in a string. For example:


In [ ]:
greeting = "Hello, World!"
greeting[1] = 'J'

The “object” in this case is the string and the “item” is the character you tried to assign. For now, an object is the same thing as a value, but we will refine that definition later.

The reason for the error is that strings are immutable, which means you can’t change an existing string. The best you can do is create a new string that is a variation on the original:


In [ ]:
new_greeting = "J" * greeting[2:end]

This example concatenates a new first letter onto a slice of greeting. It has no effect on the original string.

Searching

What does the following function do?


In [ ]:
function find(word, letter)
    index = 1
    while index <= sizeof(word)
        if word[index] == letter
            return index
        end
        index = nextind(word, index)
    end
    -1
end

In a sense, find is the inverse of the [] operator. Instead of taking an index and extracting the corresponding character, it takes a character and finds the index where that character appears. If the character is not found, the function returns -1.

This is the first example we have seen of a return statement inside a loop. If word[index] == letter, the function breaks out of the loop and returns immediately.

If the character doesn’t appear in the string, the program exits the loop normally and returns -1.

This pattern of computation—traversing a sequence and returning when we find what we are looking for—is called a search.

Looping and Counting

The following program counts the number of times the letter a appears in a string:


In [ ]:
word = "🍌 🍎 🍐 🍌"
count = 0
for letter in word
    if letter == '🍌'
        count = count + 1
    end
end
println(count)

This program demonstrates another pattern of computation called a counter. The variable count is initialized to 0 and then incremented each time an '🍌' is found. When the loop exits, count contains the result—the total number of '🍌'’s.

String helper functions

As it turns out, there is a string helper function named search that is remarkably similar to the function find we wrote:


In [ ]:
search("Hello World!", 'o')

search can also find the first occurence of a substring:


In [ ]:
search("Julius Caesar", "Juli")

The function contains determines whether the second argument is a substring of the first:


In [ ]:
contains("Julius Caesar", "Caesar")

The in operator

The keyword in is a boolean operator that takes a character and a string and returns true if the first appears as in the second:


In [ ]:
'a' in "banana"

For example, the following function prints all the letters from word1 that also appear in word2:


In [ ]:
function in_both(word1, word2)
    for letter in word1
        if letter in word2
            println(letter)
        end
    end
end

With well-chosen variable names, Julia sometimes reads like English. You could read this loop, “for (each) letter in (the first) word, if (the) letter (appears) in (the second) word, print (the) letter.”

Here’s what you get if you compare "apples" and "oranges":


In [ ]:
in_both("apples", "oranges")

String Comparison

The relational operators work on strings. To see if two strings are equal:


In [ ]:
word = "Pear"
if word == "banana"
    println("All right, bananas.")
end

Other relational operations are useful for putting words in alphabetical order:


In [ ]:
if word < "banana"
    println("Your word, ", word, ", comes before banana.")
elseif word > "banana"
    println("Your word, ", word, ", comes after banana.")
else
    println("All right, bananas.")
end

Julia does not handle uppercase and lowercase letters the same way people do. All the uppercase letters come before all the lowercase letters.

A common way to address this problem is to convert strings to a standard format, such as all lowercase, before performing the comparison. This can be done with the function lowercase.

Debugging

When you use indices to traverse the values in a sequence, it is tricky to get the beginning and end of the traversal right. Here is a function that is supposed to compare two words and return true if one of the words is the reverse of the other, but it contains two errors:


In [ ]:
function is_reverse(word1, word2)
    if length(word1) != length(word2)
        return false
    end
    i = 1
    j = sizeof(word2)
    while j >= 0
        j = prevind(word2, j)
        if word1[i] != word2[j]
            return false
        end
        i = nextind(word1, i)
    end
    return true
end

The first if statement checks whether the words are the same length. If not, we can return false immediately. Otherwise, for the rest of the function, we can assume that the words are the same length. This is an example of the guardian pattern.

i and j are indices: i traverses word1 forward while j traverses word2 backward. If we find two letters that don’t match, we can return false immediately. If we get through the whole loop and all the letters match, we return true.

If we test this function with the words "pots" and "stop", we expect the return value true, but we get false:


In [ ]:
is_reverse("pots", "stop")

For debugging this kind of error, my first move is to print the values of the indices:


In [ ]:
function is_reverse(word1, word2)
    if length(word1) != length(word2)
        return false
    end
    i = 1
    j = sizeof(word2)
    while j > 0
        j = prevind(word2, j)
        println(i, " ", j)
        if word1[i] != word2[j]
            return false
        end
        i = nextind(word1, i)
    end
    return true
end

Now when I run the program again, I get more information:


In [ ]:
is_reverse("pots", "stop")

The first time through the loop, the value of j is 3, which is has to be 4.


In [ ]:
function is_reverse(word1, word2)
    if length(word1) != length(word2)
        return false
    end
    i = 1
    j = sizeof(word2)+1
    while j > 0
        j = prevind(word2, j)
        println(i, " ", j)
        if word1[i] != word2[j]
            return false
        end
        i = nextind(word1, i)
    end
    return true
end

If I fix that error and run the program again, I get:


In [ ]:
is_reverse("pots", "stop")

The iterations have to stop when i=4 and j=1.


In [ ]:
function is_reverse(word1, word2)
    if length(word1) != length(word2)
        return false
    end
    i = 1
    j = sizeof(word2)+1
    while j > 1
        j = prevind(word2, j)
        if word1[i] != word2[j]
            return false
        end
        i = nextind(word1, i)
    end
    return true
end

In [ ]:
is_reverse("pots", "stop")

In [ ]:
is_reverse("🍌 🍎 🍐 🍌", "🍌 🍐 🍎 🍌")

Case study: Word Play

This section presents the second case study, which involves solving word puzzles by searching for words that have certain properties. For example, we’ll find the longest palindromes in English and search for words whose letters appear in alphabetical order. And I will present another program development plan: reduction to a previously solved problem.

Reading word lists

For the exercises in this chapter we need a list of English words. There are lots of word lists available on the Web, but the one most suitable for our purpose is one of the word lists collected and contributed to the public domain by Grady Ward as part of the Moby lexicon project (see http://wikipedia.org/wiki/Moby_Project). It is a list of 113,809 official crosswords; that is, words that are considered valid in crossword puzzles and other word games. In the Moby collection, the filename is 113809of.fic. You can find this file in /ES123/data/words.txt.

This file is in plain text, so you can open it with a text editor, but you can also read it from Julia. The built-in function open takes the name of the file as a parameter and returns a file object you can use to read the file.


In [ ]:
fin = open("../data/words.txt")

fin is a common name for a file object used for input. The file object provides several functions for reading, including readline, which reads characters from the file until it gets to a newline and returns the result as a string with the newline removed:


In [ ]:
readline(fin)

The first word in this particular list is "aa", which is a kind of lava.

The file object keeps track of where it is in the file, so if you call readline again, you get the next word:


In [ ]:
readline(fin)

The next word is "aah", which is a perfectly legitimate word, so stop looking at me like that.

You can also use a file object as part of a for loop. This program reads words.txt and prints each word, one per line:


In [ ]:
for line in eachline("../data/words.txt")
    println(line)
end

The file is opened once at the beginning of iteration and closed at the end.

When you use open, you have to close the file manually with the function close.

Exercises

Exercise 1

Write a program that reads words.txt and prints only the words with more than 20 characters (not counting whitespace).

Exercise 2

In 1939 Ernest Vincent Wright published a 50,000 word novel called Gadsby that does not contain the letter 'e'. Since 'e' is the most common letter in English, that’s not easy to do.

In fact, it is difficult to construct a solitary thought without using that most common symbol. It is slow going at first, but with caution and hours of training you can gradually gain facility.

All right, I’ll stop now.

Write a function called has_no_e that returns true if the given word doesn’t have the letter 'e' in it.

Modify your program from the previous section to print only the words that have no 'e' and compute the percentage of the words in the list that have no 'e'.

Exercise 3

Write a function named avoids that takes a word and a string of forbidden letters, and that returns true if the word doesn’t use any of the forbidden letters.

Modify your program to prompt the user to enter a string of forbidden letters and then print the number of words that don’t contain any of them. Can you find a combination of 5 forbidden letters that excludes the smallest number of words?

Exercise 4

Write a function named uses_only that takes a word and a string of letters, and that returns true if the word contains only letters in the list. Can you make a sentence using only the letters acefhlo? Other than "Hoe alfalfa?"

Exercise 5

Write a function named uses_all that takes a word and a string of required letters, and that returns true if the word uses all the required letters at least once. How many words are there that use all the vowels aeiou? How about aeiouy?

Exercise 6

Write a function called is_abecedarian that returns true if the letters in a word appear in alphabetical order (double letters are ok). How many abecedarian words are there?

All of the exercises in the previous section have something in common; they can be solved with the search pattern. The simplest example is:


In [ ]:
function has_no_e(word)
    for letter in word
        if letter == 'e'
            return false
        end
    end
    true
end

The for loop traverses the characters in word. If we find the letter 'e', we can immediately return false; otherwise we have to go to the next letter. If we exit the loop normally, that means we didn’t find an 'e', so we return true.

You could write this function more concisely using the in operator, but I started with this version because it demonstrates the logic of the search pattern.

avoids is a more general version of has_no_e but it has the same structure:


In [ ]:
function avoids(word, forbidden)
    for letter in word
        if letter in forbidden
            return false
        end
    end
    true
end

We can return false as soon as we find a forbidden letter; if we get to the end of the loop, we return true.

uses_only is similar except that the sense of the condition is reversed:


In [ ]:
function uses_only(word, available)
 for letter in word
        if letter  available
            return false
        end
    end
    true
end

Instead of a list of forbidden letters, we have a list of available letters. If we find a letter in word that is not in available, we can return false.

uses_all is similar except that we reverse the role of the word and the string of letters:


In [ ]:
function uses_all(word, required)
    for letter in required
        if letter  word
            return false
        end
    end
    true
end

Instead of traversing the letters in word, the loop traverses the required letters. If any of the required letters do not appear in the word, we can return false.

If you were really thinking like a computer scientist, you would have recognized that uses_all was an instance of a previously solved problem, and you would have written:


In [ ]:
function uses_all(word, required)
    uses_only(required, word)
end

This is an example of a program development plan called reduction to a previously solved problem, which means that you recognize the problem you are working on as an instance of a solved problem and apply an existing solution.

Looping with indices

I wrote the functions in the previous section with for loops because I only needed the characters in the strings; I didn’t have to do anything with the indices.

For is_abecedarian we have to compare adjacent letters, which is a little tricky with a for loop:


In [ ]:
function is_abecedarian(word)
    previous = word[1]
    for c in word[2:end]
        if c < previous
            return false
        end
        previous = c
    end
    true
end

In [ ]:
is_abecedarian("flossy")

An alternative is to use recursion:


In [ ]:
function is_abecedarian(word)
    if length(word) <= 1
        return true
    end
    if word[1] > word[nextind(word, 1)]
        return false
    end
    is_abecedarian(word[2:end])
end

In [ ]:
is_abecedarian("flossy")

Another option is to use a while loop:


In [ ]:
function is_abecedarian(word)
    i = 1
    j = nextind(word, 1)
    while j <= sizeof(word)
        if word[j] < word[i]
            return false
        end
        i = j
        j = nextind(word, i)
    end
    true
end

In [ ]:
is_abecedarian("flossy")

The loop starts at i=1 and j=nextind(word, 1) and ends when j>sizeof(word). Each time through the loop, it compares the ith character (which you can think of as the current character) to the jth character (which you can think of as the next).

If the next character is less than (alphabetically before) the current one, then we have discovered a break in the abecedarian trend, and we return false.

Here is a version of is_palindrome that uses two indices; one starts at the beginning and goes up; the other starts at the end and goes down.


In [ ]:
function is_palindrome(word)
    i = 1
    j = prevind(word, sizeof(word)+1)
    while i<j
        if word[i] != word[j]
            return false
        end
        i = nextind(word, i)
        j = prevind(word, j)
    end
    true
end

Or we could reduce to a previously solved problem and write:


In [ ]:
function is_palindrome(word)
    is_reverse(word, word)
end

Debugging

Testing programs is hard. The functions in this chapter are relatively easy to test because you can check the results by hand. Even so, it is somewhere between difficult and impossible to choose a set of words that test for all possible errors.

Taking has_no_e as an example, there are two obvious cases to check: words that have an 'e' should return false, and words that don’t should return true. You should have no trouble coming up with one of each.

Within each case, there are some less obvious subcases. Among the words that have an 'e', you should test words with an 'e' at the beginning, the end, and somewhere in the middle. You should test long words, short words, and very short words, like the empty string. The empty string is an example of a special case, which is one of the non-obvious cases where errors often lurk.

In addition to the test cases you generate, you can also test your program with a word list like words.txt. By scanning the output, you might be able to catch errors, but be careful: you might catch one kind of error (words that should not be included, but are) and not another (words that should be included, but aren’t).

In general, testing can help you find bugs, but it is not easy to generate a good set of test cases, and even if you do, you can’t be sure your program is correct. According to a legendary computer scientist:

Program testing can be used to show the presence of bugs, but never to show their absence!— Edsger W. Dijkstra