Programming with Python

Vedran Šego, vsego.org

Contents:

1. Strings

2. Formats

3. Generator expressions

Strings

Strings are usually described as character sequences. However, in Python, they are better described as tuples of characters, in a sense that they function very much like immutable lists.

A string literal can be defined in the following ways:

Character sequences starting with a backslash (\) have a special meaning. Some of them can be seen here:

So, \\ inside a string means a single backslash (\), '\n' means new-line character, \" and \' mean quotation marks (instead of the end of the string), etc. This is called string escaping and the backslash is called an escape character.

More about string literals can be read in the official reference.

If we need to put many backslashes in a string (required by some applications, for example, regular expressions), we can also define so called raw strings, by prepending a string literal with r:

As we have already seen, print can be used to print a string:

In fact, we have been using this since we first encountered print:

We have also seen how to load a string:

and how to convert it to an integer or a "real" number:

Of course, if the string contains something other than an integer or a "real" number, an error occurs. For example:

Standard list/tuple operations work as we've seen with lists:

As we said before, strings are immutable. This means they cannot be changed:

in-place sorted or reversed, nor can they be extended or appended to:

However, these operations can be done by creating new strings. Something like this:

Notice that the sorting has created a list (of characters) and not a new string.

In order to fix this (so, to get a string), we have to merge the elements of a list into a string. This is done by join:

We can also do the opposite: split a string into a list of substrings, by some separator. For example,

To just convert a string to the list of its characters, we use the function list() which we have seen while working with iterables:

Python has a truly rich support for strings. Some of the available operations are:

See the full list of the supported string operations in the official documentation.

Regular expressions

For a truly powerfull (but sometimes complex) way to analyze strings, read about regular expressions in A.M. Kuchling's HOWTO. Those already familiar with regular expressions can read Python re reference.

There are numerous practical applications to regular expressions: data analysis, syntax highlighting, system maintenance and configuration (for example ModRewrite in Apache), etc. Most of the modern languages support the same regular expressions syntax (so called Perl compatible), which makes them quite portable. While often fairly simple, they can also get downright scary.

Regular expressions are beyond the scope of this course. You may use them if you want to, but it is not assumed that you are familiar with them.

String formats

Forming a string from several values can be tricky. For example, if we had x = 17 and y = 19, and wanted to print that information in the form

x = 17, y = 19

we have so far used the following syntax:

We had to convert the number x to a string using str(x) and then "glue" it with the string literals "x = " and ", y =" because otherwise print would add spaces.

Even with such conversions, we are limited with what we can do. For example,

• how would we print the value of pi up to only 3 decimal places?
• how to make it easy to customize messages in several languages, i.e., to tell Python to write something like

  Student <student's name> has enrolled in <name of the university> where {he/she} studied <what did they study>.

This is where formats come in handy.

There are three wide-spread ways to format strings in Python:

1. C-like formatting,
2. format function,
3. various templating engines.

The format function is very easy for a basic use, and it has a powerfull support for advanced formatting (like named arguments, embedded formats, etc).

The C-like formatting is somewhat shorter to invoke and it is common to many widely used languages (C/C++, PHP, Perl,...).

The third way, templating, is more powerful, usually with support for complex data structures. This is often used to produce whole documents, instead of just short strings.

format function

The basic use of the format function is like this:

"some string containing {}, possibly more than one".format(x)

In this case, {} will be replaced by the value in x and the new string will be returned.

The function accepts multiple arguments, usually one for each of the braces pairs. It then replaces the first pair of braces by the first argument, the second one with the second pair, and so on.

Using the unpacking operator * for sequences, we can also do it like this:

However, if we change the sentence to imply a different order of the parameters, we're in trouble:

There is an easy fix for this:

Named arguments can also come in handy:

or, using the unpacking operator ** for dictionaries:

The use that we have just seen makes it easy to make localised messages:

We can also align values on the screen:

Numbers can be easily formatted:

Any names or positional indices are given before ":". For example, the list of the tallest and the shortest people in the World:

There is much more that format can do. For the complete reference, read the official documentation.

C-like formatting

C-like formatting is somewhat less powerfull, but easier to invoke, which makes it ideal for simple formattings (for example, of numbers). The basic syntaxt is:

format_string % value

or, if there is more than one value to include,

format_string % a_tuple_of_values

For example:

More about C-like formatting can be read in the official reference.

Feel free to use it if you're already familiar with it. Otherwise, such a formatting is considered old and out of date, and it is advised to use the format function instead.

Generators and generator expressions

In this section we address some very useful, albeit Python-specific concepts:

• Iterator is an object representing a stream of data, i.e., capable of providing the first element, as well as traversing thorught all the elements from the beginning to the end.
  Note: Iterators also exist in C++, but it is a different concept and they have a different purpose there.
• Generator is a function that returns an iterator.
• Generator expression is an expression that returns an iterator.

Iterators are a powerful thing to use, but quite complex in all their generality, and they go well beyond the scope of this course.

We shall cover generator expressions, as they are a very compact, efficient, and often used way to deal with list-like data, and then we shall briefly see how generators work, as a bit more complex but quite powerful mechanism.

We have seen the function range before and we said that it

pretends to return a list of numbers (it returns something more complex, we can consider it a list for now)

Let us see what exactly does range return:

As we can see:

• A range and a list are not the same (they print differently).
• They have different types.
• However, they "contain" the same elements.

So, while acting the same way that lists do in a for loop, range actually behaves like a generator, which is a function that returns an iterator, which is what rng is.

Note: Strictly speaking, range a bit more than an ordinary generator, with some additional properties and functions that normal generators don't have, but for the sake of explaining how the generators work, we'll disregard those extras and use range as an example that we're quite familiar with.

The main difference between a list and an iterator is that a list actually contains all the data and allows us to access each of its elements.

An iterator, on the other hand, may contain the elements, but it almost never will. It can somehow get the first element, keep track of what the current element is, and "jump" to the next one. So, basically, it can provide everything a for loop needs, without any restrictions on where it gets its data.

For example, range(3, 7) will return an iterator that holds three values:

• starting value: 3,
• increment: 1 (default, since we didn't provide the third argument),
• end value: 7 (exclusive).

So, when we execute

for x in range(3, 7):
    print(x)

the following will happen:

1. for will "ask" range for its first value and obtain 3. It will then assign that 3 to x and execute its body (print(x)).
2. In the next step, for will "ask" range for its next value. Now, range remembers that its current value was 3, so it will compute the next one, and return 4, which will get assigned to x and the loop's body will once again be executed.
3. In the next step, for will "ask" range for its next value, and get 5.
4. In the next step, for will "ask" range for its next value, and get 6.
5. In the next step, for will "ask" range for its next value. However, range will see that the next value, 7, is too big, and it will gently explain for that it has no more values. for will then end, without further executions of its body.

The benefits

1. No extra memory is required, no matter how long the range is. For example, range(10**9) will not take memory for one billion integers.

2. There will be no delay before the for-loop starts, that would otherwise be needed to create such a long list of numbers.

3. It is not uncommon that the program stops before going through all the elements of a list or an iterator. Looping (either with loops or some other mechanisms) can be stopped prematurely. In these cases, an iterator will not even compute the unneeded values, potentially saving a lot of memory and processing time.

range here is just an example. Iterators can be used for all kinds of streaming data. For example, they are also used for reading files (which can easily be a few gigabytes big, which would put a real strain on the system if actually loaded at once).

Generators

So, how do we make our own generator (a function that returns an iterator)?

The key thing here is yield: a statement that acts like return, but it doesn't terminate a function (it merely "pauses" it).

Here is an example of a generator:

Note: This is a highly inefficient algorithm to find prime factors of a number. We shall see an improvement in several weeks. We use this one here because it closely follows the mathematical definition and, as such, it is easy to understand it.

So, how does the above work? Let us assume that $n = 84 = 2^2 \cdot 3 \cdot 7$.

When the for-loop is first encountered, prime(84) is called. It executes as any other function would, until it reaches yield p, which will first happen for p = 2. At this moment, the function returns p (i.e., 2) and stops executing, but it is not terminated, as it would have been if return was in place of yield.

So, the for-loop has now obtained its first value, 2, and it assigns it to prime. The body of the for-loop is now executed, i.e., cnt is increased from zero to one, and the message

Prime factor #1 of 84: 2

is printed.

After the body is executed, we return to the head of the for-loop. Recall that the function was not terminated, so it continues where it stopped when a yield statement was last invoked. The next prime factor that the for-loop in primes function finds is p = 3. Again yield p is invoked, the main for-loop gets its second value, prime = 3 is assigned, and the message

Prime factor #2 of 84: 3

is printed.

Back to the header of the for-loop, we reenter the still uninterrupted function primes, and continue after its last executed yield (generally, a generator can have many yield statements, not just one as in our example). The next p it will find is 7, which is again returned to the main for-loop and

Prime factor #3 of 84: 7

is printed.

We return to the header of the for-loop once again, reentering primes. However, this time, no new value of p will be found, and the function's own for-loop will end. Since there is no code after it, the primes function will finally come to an end (i.e., it will stop and not just "pause"). Now, the main for-loop is notified that there are no further values to be assigned to the prime variable, so it too quits.

The last command in our program is now executed, printing:

Number 84 has 3 distinct prime factors.

Iterators are easily converted to lists:

Generators are a fairly advanced concept and they will not be part of any examinations. However, the use of generators is encouraged. You can find more about them here.

Generator expressions

Generator expressions are expressions that return iterators. They behave in the same manner as generators, but their syntax is more compact and usually easier to understand as it is very similar to for loops.

Here is a simple example of a generator of the squares of the numbers from 0 to n-1:

Notice that the syntax of a generator is very much like what one would write in a mathematical document:

$gen$ contains values of $x^2$ for all integer $x$ in $[0, n \rangle$.

Even if we want to add some filtering condition, it is done in the same manner. For example,

$gen$ contains values of $x^2$ for all integer $x$ in $[0, n \rangle$ for which $x$ is a prime number.

is written like this:

As we can see, generator expressions look like neatly packed for-loops, which makes them rarely useful in for-loops themselves. For example,

or, without the auxiliary variable gen,

is better written as:

However, there are various other uses. The first of them is creating a list of easily computable values (complex operations don't fit in generator expressions):

A generator expression that creates a list is often called a list comprehension.

Note: The same construct can be used with sets (set comprehension) and dictionaries (dictionary comprehension). For example:

Additionally, there are various functions that take iterators as arguments. One of them is sum, which returns the sum of all the elements:

In these two examples, we merely wrote less code, but there was no gain in memory consumption nor the processing speed.

However, let us check if any of the above squares has the form p+2 for some prime number p. In other words, we want to check if x*x-2 is a prime for some x which is a prime itself:

In the above example, is_prime(number-2) gets computed until the first number for which it is true is encountered. This is 4, which happens for x = 2 when generating numbers:

Obviously, not all numbers satisfy this new condition (121 does not, because $121-2 = 119 = 7 \cdot 17$).

If we wanted to just check if all of the numbers have the form p+2 for some prime number p (i.e., if all x*x-2 are primes for some x which is a prime itself), we would have used the function all:

Notice that the following wouldn't work properly:

Generator expressions are meant for a single use. They know how to do "next", but there is no way to "reset" them (in the above example, the problem is that numbers is not reset). Depending on the exact generator that is used, there are various ways to deal with this, but this goes beyond the scope of this course.

More than one for clause

Generators and list comprehensions can have more than one for clause.

For example, here we create a list of all fully simplified fractions (memorized as strings) between zero and one (exclusively) with numerator and denominator between 1 and m.

Of course, there are more efficient ways to check if a fraction is fully simplified (hint: Euclidean algorithm), but we shall cover that later.

As always, all of these for statements could've looped over iterables other than iterators created by range. We could use lists, tuples, sets, etc.

For example, to limit the numerators to the set $\{1, 3, 5, 7, 11\}$ and the denominators to the set $\{17, 19, 23\}$ we can do this:

Examples

We are now ready to make some of the old pieces of code simpler.

Notes:

1. Generator expressions don't always speed things up.
   For example, the above code in d) is still very slow, as it tests all the numbers between 2 and n-1.

2. Repeating the same generator multiple times, like we did in b) and c), can actually make things slower (computing the same thing twice) and the code redundant (we copied the same code twice).
   To overcome these disadvantages, we can:

   • create a list of items (using a single generator) if speed is more of an issue than memory, or
   • clone the generator in some cases (read about itertools.tee), or a function that returns an iterator in other cases, or use a specific function for resetting that some iterators have (for example, those that read files).