Introduction to Python

Your completion of the notebook exercises will be graded based on your ability to do the following:

Understand: Do your pseudo-code and comments show evidence that you recall and understand technical concepts?

Apply: Are you able to execute code (using the supplied examples) that performs the required functionality on supplied or generated data sets?

Notebook objectives

By the end of this notebook you will be expected to:

• Identify basic concepts of programming;
• Understand Python syntax;
• Distinguish between the different native data types used in Python;
• Use flow control structures to implement conditional and repetitive logic;
• Understand functions as a block of organized, reusable code that is used to perform a single, related action, as being necessary for improving the modularity of an application, as well as allowing for code reuse within and across applications; and
• Understand a Python package as a file consisting of Python code that defines functions, classes, variables, and may also include runnable code. You should be able to import a Python package or specific functions from a package.

List of exercises

• Exercise 1: Printing strings.
• Exercise 2: Rolling dice.
• Exercise 3: Coin flip distribution.

Notebook introduction

In the Orientation Module, you were given a number of links that provided additional Python documentation and tutorials. This section will serve as a summarized version of useful commands that are needed to get non-technical users started, and equip them with the basics required to complete this course.

This course is not intended to be a Python training course, but rather to showcase how Python can be utilized in analysis projects and, more specifically, in analyzing social data. You will be provided with links that you can explore in your own time in order to further your expertise. You are welcome to offer additional suggestions to your peers in the online forums.

You should execute the cells with sample code and write your own code in the indicated cells in this notebook. When complete, ensure that you save and checkpoint the notebook, download a copy to your local workstation, and submit a copy to the Online Campus.

You can also visit the Python Language Reference for a more complete reference manual that describes the syntax and "core semantics" of the language.

Note to new Python users:

This notebook will introduce a significant amount of new content. You do not need to be able to master all of the components, but you are urged to work through the examples below, and to attempt following the logic. In this notebook, you will be introduced to Python syntax. The second notebook in this module will start to introduce various components of data analysis basics, and the third will guide you through a basic example from beginning to end. The focus will shift from programming to subject-related examples in subsequent modules.

1. Introduction to Python programming

Python is generally defined as a high-level, general-purpose, interpreted, dynamic computer programming language.

Let’s try to break this statement down, for those new to computer programming. High-level means that we can express instructions that computers are able to execute in a language that is closer to a human language, and not the machine language that a computer requires to run the instructions. The interpreter is a program that converts the high-level programming instructions into low-level programming instructions that a computer understands. Dynamic means that they do not enforce or check type-safety at “compile-time”, in favor of deferring such checks until “run-time”. This has many advantages, including lower development costs, rapid prototyping, and flexibility required by specific domains, such as data processing and analysis. This makes Python a perfect language for this course, as it is easy to understand (the expressions used are very similar to day-to-day conversations), it has a simple syntax, and provides objects that are appropriate for data processing and analysis.

Because it’s a programming language, there are some basic computer science concepts that are important to understand before you start coding.

1.1 Basic concepts

1.1.1 Variables

Variables are the backbone of any program and, therefore, the backbone of any programming language. A variable is used to store information to be referenced and manipulated in a set of instructions commonly referred to as a computer program. Variables also provide a way of labeling data with a descriptive name, in order for programs to be understood more clearly by the reader and ourselves (recall high-level from above). It is helpful to think of variables as containers that hold information. Their sole purpose is to label and store data in the computer's memory. This data can then be used throughout a program.

Variables can store various pieces of information such as the name of a person, that person's age or weight, or their date of birth. Each of these pieces of information contain different types of information. The data type of the variable containing the name of a person would be a string, while the age is stored as an integer, and the date of birth as a datetime object. Fortunately, in Python, unlike in Java or C, you do not need to tell the computer what type a variable is before you assign an object to it. This means that you can begin your program by referring to your name, say, as “X”.

>>> X = 'Mary'

In the middle of your program, you might change your mind, and reassign your “X” variable to your age.

>>> X = 21

This provides you with flexibility that you can exploit to quickly test ideas without worrying that your code will break when executed. As with anything, "with great flexibility, comes great responsibility!" It is best practice to use names that are intuitive for the data that the variable stores. The only exception is using names which are reserved by the programming language, called keywords. Keywords define the language's rules and structure, and they cannot be used as variable names. Inside Jupyter, a Python reserved name is highlighted differently to other names, as will shortly become clear.

A variable’s type also defines what you can do, and the behaviors you can expect, when you perform certain operations on it. This is illustrated later in this notebook.

1.1.2 Control structures

When we put together a set of instructions for the computer to execute, the computer reads the instructions sequentially. This is known as the "flow of the code". However, it is often the case that an instruction in the code requires a decision to be made. For example, say you have written a program to accept interns into your organization. To comply with the labor regulations governing your organization, you ask each candidate to enter their age amongst other details. Before accepting them as an intern, your program checks if the person's age is within the prescribed limits. If not, the application of a potential intern is rejected. Such decision points that affect the flow of the code occur frequently in computer programming and are known as flow control structures. Specifically:

A control structure is a block of programming that analyzes variables and chooses a direction in which to go based on given parameters. The term flow control details the direction the program takes (which way program control “flows”). Hence it is the basic decision-making process in computing; flow control determines how a computer will respond when given certain conditions and parameters.

Control structures make the programs function properly and, without control structures, your program’s code would only flow linearly from top to bottom. Changing what action a program takes, based on a variable's value, is what makes programming useful. Below, you will be introduced to the different control structures available in Python.

1.1.3 Data structures

A data structure is a particular way of storing and organizing data in a computer so that it can be used efficiently. Previously, you were introduced to the concept of variables using a person's name, age, and date of birth as examples. Imagine doing that for everyone who is enrolled in the MIT Big Data and Social Analytics certificate course. That would be a lot of variables that you would struggle to keep track of. A data structure is a way to get around having to create millions of variables. Python implements many data structures that you will be using, such as lists, dicts, tuples, and sequences. Moreover, other data structures can be built on top of these for efficiently working with data. In Python, the NumPy and Pandas modules provide data structures that are intuitive and expressive enough for many of our needs in data analysis.

1.1.4 Syntax

Each programming language is designed and implemented differently, and using it requires adherence to its syntax, which is the set of rules that define the combinations of symbols that are considered to be correctly-structured programs in that language. When these rules are followed, the programming language can understand your instructions and, therefore, you are able to create software that can be run. If you do not adhere to the rules of programming languages’ syntax, you will unfortunately run into errors and your programs will not be executed. As it turns out, the syntax of any programming language tends to be the biggest obstacle many people face when being introduced to programming for the first time, or when learning a new programming language. (Note that syntax errors are only one type of error among many that are possible.) Fortunately, Python will correctly identify syntax errors as it encounters them. This will be useful when trying to find errors in your code. Some key things to remember when using Python are highlighted below.

Case sensitivity

All variables are case-sensitive. Python treats “number” and “Number” as separate, unrelated entities.

Comments

Commented lines start with the hashtag symbol # and are ignored when interpreting the code at run-time.

Indentation

Code blocks are demarcated by their indentation. In particular, control structures end with a colon : at the decision point. Code blocks indicating action that must be taken when the decision evaluates to "True" are demarcated by indentation.

Note: Spaces and tabs do not mix
Because whitespace is significant, remember that spaces and tabs don't mix, so use only one or the other when indenting your programs. A common error is to conflate them. While they may look the same during editing, the interpreter will read them differently, and it will result in either an error or unexpected behavior. Most decent text editors can be configured to let the tab key emit spaces instead.

In summary, this section described what a variable is, how you can store information in it, and then retrieve that information at a later stage. The variable can have a name, and this name is usually informed by the kind of content you’ll be storing in the variable. So, if you’re storing your name in the variable, you would name the variable “yourName”. You would not have to give it that name, you could name the variable “WowImProgramming”, but that wouldn’t make much sense considering you are trying to store a person’s name. Finally, variables have types, and these types are used to help us organize what can and cannot be stored in the variable.

Hint: Having a type will help to open up what kinds of things we can do with the information inside the variable.

Example: If you have two integers (let’s say 50 and 32), you would be able to subtract one variable from the other (i.e., 50 – 32 = 18). But, if you had two variables that stored names (e.g., “Trevor” and “Geoff”) it wouldn’t make sense to subtract one from the other (i.e., “Trevor” – “Geoff”), because that does not mean anything.

Therefore, types are a powerful thing, and they help you make sense of what you can and cannot do with your variables.

2. Programming with Python

Now that the basic concepts have been introduced, let’s make these concrete using illustrative examples. This notebook uses Jupyter's code cells (in light gray background) to demonstrate these ideas. As you learned in the Orientation Module, you can execute instructions in a code cell and the output will be displayed below the cell, where applicable.

In most cases, a comment is included in the very first line of the cell to help in following the material. A comment is any line in the set of instructions that is preceded by a hashtag symbol. Inside Jupyter, once you have used the comment indicator, the line is highlighted in a different color and the font italicized, which is a very helpful visual aid for both readers and developers. See the example below:

Note:

The code cell below will not produce any output when executed.

2.1 Strings

Python works with "values" that can take different forms depending on the data type. For example, both a person's first name and age constitute different, valid values inside Python.

You can use the function “type()” and include the name of the value in between the parentheses to find out what type the value is. (Functions in Python are explained in more detail later in this notebook.)

From the above, you can see that our value 'Bob' has type str. The value of a string data type is a sequence of characters. Strings are typically used for text processing. You can perform concatenation and a number of other functions on strings.

2.2 Integers

Another example of a value is Bob’s age. Assuming Bob is 40 years old, the value can be specified as follows:

Note the difference in how the two values are specified: for string values the name is included between single or double quotes, whereas this is not needed for the age value. Had quotes been used in the latter, it would have expressed a different value of a different type.

2.3 Boolean

Let's introduce three new ideas with this example: how to compare two values, another valid data type in Python, and the concept of casting. First, let’s compare the two values (40 and '40') to see if they are of the same type.

What is happening here? We have used an operator (==) that accepts one value on either side, and compares these values to see if they are the same. The values being compared in this case are not 40 and '40', but their corresponding value types (i.e., integer, string, etc.). (If our intention is to compare the original values we would have used 40 == '40')

What type is the resulting False value?

The "False" value is of the “Boolean” type, which is another fundamental data type in programming. Boolean values can only take two values: "True" or "False".

It is often the case that, instead of working with the native data type, such as the Boolean data type that you have just seen, you want to use a different but equivalent data type. For example, if you accept that a "False" value represents the lack of something in a given context, you may want to represent that as an integer with the value 0. This is achieved using the concept of casting, which allows certain data types to be cast as a different type. In Python, you cast a value by calling the type that you want to cast it to, and including the value as an argument to that call. Thus, in this case, you achieve the following.

2.4 Floating point numbers

Floating point numbers are numbers that contain floating decimal points.

Remember that data types define what operations are allowed between values of the same type. However, certain data types, such as integers and floating numbers, can be used together in expressions. In this case, casting occurs without the need for the user to do it manually. The resulting value takes the type of the more encompassing data type of the involved values, as can be seen in the example below:

What happens when dividing two integer values?

2.5 Complex numbers

Complex numbers are important in many scientific and engineering disciplines. Python can also represent and perform calculations with complex numbers.

2.6 The assignment operator and variables

Our expressions, for example "type(40) == type('40')", can quickly become unwieldy and a source of errors that are difficult to debug. To help in manipulating values elegantly (and programmatically), it is common to use variables to store the value of interest. The variable is then used to reference the value of interest. To assign a value to a variable, we use the assignment operator "=" as in the following example:

Here, the value on the right-hand side of the assignment operator “=” has been assigned to the variable that has been cleverly named “a_boolean_value”. You can now use the variable in other parts of your code.

Let’s introduce another function, called “print” (another keyword), which prints the value associated with your variable.

You can also use print for other expressions you have met before:

The print function allows you to perform other formatting and interesting value referencing.

In this print statement, the str.format() form has been used as argument, where “placeholders” or format fields indicated by {}, are replaced by arguments specified in format argument. The arguments are used in the order used inside the parentheses, and must be equal to the number of the {} in the str part. You can also specify which positional argument should go where by including numbering in the format fields.

By using numbered fields, the arguments are used depending on which number they are, with the first argument in format numbered as 0. Using print in this way is especially useful when you want to combine different data types in your print statement as illustrated above. You can also control how the formatting should be done instead of relying on Python's default behavior. For example, if you don't care about the decimal in the percentage, you can call print as follows:

Notice the value has only been rounded in the print output but not its representation in memory.

Instructions

1. Set your first and last names as separate variables ("firstname" and "lastname", respectively).
2. Print your first name and last name using the "str.format()" form discussed above. Include a whitespace character between your first and last name in the print output.

Exercise complete:

This is a good time to "Save and Checkpoint".

2.7 Aggregate data types: lists and tuples

Variables store information that may change over time. When you need to store longer lists of information, there are additional options available. You will be introduced to lists and tuples as basic elements, and can read more about native Python data structures in the Python documentation. During this course, you will be exposed to examples using the Pandas and NumPy libraries, which offer additional options for data analysis and manipulation.

2.7.1 Lists

Lists are changeable sequences of values. They are stored within square brackets, and items within a list are separated by commas. Lists can also be modified, appended, and sorted, among other methods.

Create and print a list

Selecting an element in a list

Remember that Python uses zero indexing, that is, the index starts at zero. You can use a negative index to select the last element in the list.

Append to a list

Note:

When selecting and executing the cell again, you will continue to add values to the list.

Try this: Select the cell above again and execute it to see how the input and output content changes.

This can come in handy when working with loops.

Changing a list

This course will not cover string methods in detail. You can read more about string methods in the Python documentation, if you are interested.

Define a list of numbers

Sort and filter a list

Remove the last element from a list

2.7.2 Tuples

Tuples are similar to lists, except that they are defined in parentheses and are unchangeable, which means that their values cannot be modified.

2.8 Loops, sequences, and conditionals

Conditions and loops are control structures that can be used to repeat statements based on the conditions or loops specified. This can be employed to cycle through data sets or perform a sequence of steps on (or based on) input variables.

2.9 Range

Range (start, stop, step) is used to create lists containing arithmetic progressions. If you call a range with only one argument specified, it will use the value as the stop value and default to zero as the start value. The step argument is optional and can be a positive or negative integer.

2.10 Basic loops

Python uses for and while loops to repeat sections of statements. A for loop runs for a set number of times, whereas a while loop repeats a statement until a certain condition is met. The statements to be repeated must be indented for the interpreter to recognise them as such.

The example below demonstrates how to do the following:

• Cycle through the generated list and:
  • Print the current element in the list; and
  • Print Xs, which are repeated per element.
• Exit the loop and print a line that is not repeated.

2.11 Conditionals

Conditionals are used to determine which statements are executed. In the example below, we import the "random" library and generate a random number smaller than 2. Assume 0 means heads and 1 means tails. The conditional statement is then used to print the result of the coin flip, as well as "Heads" or "Tails".

2.11.1 Importing modules:

Code reuse is an integral component in programming, and is critical for extending basic functionality of an underlying programmming environment. In Python, modules exists that contain functions, classes, variables and other code that we can reuse in our own code. In most cases, these modules are provided under permissive free software license such as the MIT license. Assuming you know the name of the module you want to use, there are a number of ways you can tell Python that you want to access that module.

• import X

  This imports the module X, and creates a reference to that module in the current namespace. After running this statement, you can use X.name to refer to things defined in module X. In the above example, we import the random module, and used the random.randrange(0,2) to call a function called randrange in that module. It is also common to use an alias for X by importing the module as import X as alias. Now we use alias.name to refer things defined in module X.

• from X import *

  This statement imports the module X, and creates references in the current namespace to all public objects defined by that module (that is, everything that doesn’t have a name starting with "_"). After you’ve run this statement, you can simply use a name to refer to things defined in module X without prepending X. to it. Although there are cases where this is necessary, it is best to avoid this in the majority of cases as it can lead to unexpected behaviour.

• from X import a, b, c

  This works like the previous statement by importing the module X, but creates references in the current namespace only to the objects provided (implying you should know that these are defined in the module). Thus, you can now use a and b and c in your program.

2.12 Functions

Functions allow for code reuse within and across applications. When defining a function, you would use the “def” statement. The desired function code is then placed into the body of the statement. A function usually takes in an argument and returns a value as determined by the code in the body of the function. Whenever referring to the function, outside of the function body itself, this action is known as a function call.

You can now call the function as in the next example.

Function definitions, loops, and conditionals can be combined to produce something useful. The example below will simulate a variable number of coin flips and then produce the summary of results as output.

In the above function definition, there is a “docstring” to describe what the function does in lieu of the comment that has been used. This is considered best practice when defining functions in Python. The docstring is included just after the def statement, and is included between a pair of three single or double quotation marks.

In the "coinflip" function, you are simulating the number of times an unbiased coin returns heads or tails on being flipped. To track these values, initialize the counts to zero for both the heads and tails variables. Moreover, it is also important to include a variable flip that tracks the number of flips in the simulation. The simulation ends when the number of flips required, as specified in the function argument, is reached. To simulate the actual coin flip or flip event, call the function “randrange” from the Python "random" module, which accepts a minimum of two parameters: the start of the range from which you want to simulate, and the end of the range (stop) from which point and above you do not care about. You can see what this "randrange" function does by calling it for the given parameters.

Every time you call "randrange", either a 0 or 1 is returned with equal probability. Hence, it is a good approximation to flipping an unbiased coin in our example. This example also does something that is slightly complex to describe. Before entering the loop, set your random seed generator to the current time. This ensures you do not always get the same sequence, as random number generation inside the program is not truly random, but follows a deterministic algorithm.

Note:

Later modules will exploit this determinism by setting the random seed for reproducibility of expected outcomes.

Going back to the coin flip function, when you flip the coin, a 0 or 1 is obtained as a value. You check whether this is 0, and if so, you increment your count of heads by 1. If it is a 1, you increment the count of tails by 1. Recall that these two states are mutually exclusive, and only one can occur on each flip. After these checks, you also increase your count of flips tracked in “current_number_of_flips”. You repeat this until you have reached the “total_number_of_flips”.

Finally, the function returns the counts of both heads and tails obtained from your simulation as an array, where the first entry is number of heads. Let’s invoke the function for 100 flips.

Is this coin balanced? You would need to run more simulations for a more definitive answer. Try 10000 simulations. Since you have a function already written, you can simply reuse it, as long as you specify a different value for the argument. This is the power of writing functions for tasks that follow the same design pattern.

Notice how this example has used a number of the ideas introduced in this notebook in computing the proportion of heads.

Instructions

Use the "coinflip" example above and change it to simulate rolling dice. Your output should contain summary statistics for the number of times you rolled the dice, and occurrences of each of the 6 sides.

Hints:

• Replace "Heads" and "Tails" with six new variables: "Side_1", "Side_2", “Side_3”, etc.
• Replace random.randrange(0,2) with random.randrange(1,7)
• Test for whether each state of the random variable (which side comes up after a die throw), and increase the counter for the relevant variable.
• Return final states of each variable as a vector (i.e., [Side_1, Side_2, Side_3, etc.]).

Exercise complete:

This is a good time to "Save and Checkpoint".

The basic random function was used in the example above, but you can visit Big Data Examiner to see an example of implementing other probability distributions.

3. Graphics and visualization

Matplotlib is the standard library commonly used to display graphics and plots in Python.

3.1 Matplotlib

The examples below come from the Matplotlib Screenshots page.

Note:

One of the options that you will need to remember to set when you use Matplotlib is "%matplotlib inline", which instructs the notebook to plot inline instead of opening a separate window for your graph.

You can find additional information on Matplotlib through the following resources:

• Matplotlib Documentation

• Matplotlib Beginners

• Matplotlib Gallery

3.1.1 Simple plot

The graph below does not contain any real significance other than demonstrating how easy it is to plot graphs.

3.1.2 Subplots

"Subplots" is another useful feature where you can display multiple plots. This is typically used to visually compare data sets.

3.1.3 Histogram

Here is another example of a plot that will be utilized in future modules. The example demonstrates syntax, and provides ideas of what is possible.

3.1.4 Visualizing coin flip distribution

This is a continuation of your coin flip simulation from the previous section. Although you already have a fairly accurate idea of whether or not the coin is biased, in statistics, the typical approach requires sampling from the underlying distribution multiple times, and then plotting the sampling distribution. To this end, a handle is required on the actual value for the probability of landing heads between 0 and 1 (probability of landing tails will then be 1 minus that value), which can be generated using the "random()" method from the random module, that is the call "random.random()". In the case of a fair coin, designate as heads anytime the value generated is below 0.5, otherwise it is assigned as tails. An unfair coin can be simulated by adjusting this threshold; the closer the value is to 0 (and below 0.5) the less likely the chance of getting heads in multiple trials. This is an involved complex statistical procedure, and is captured here for illustrative purposes.

In the next code cell, we simulate flipping a fair coin by specifying the threshold as 0.5. You will flip the coin 10 times and note the number of times it falls heads in those trials. This is repeated 1000 times, from which a plot of the sampling distribution of the underlying distribution (known as the binomial distribution) is generated.

As before, the chance of our coin landing heads is as one would expect from an unbiased coin.

Instructions

Using the "coinflip" example above, simulate the distribution one would get for a biased coin that has an 80% probability of landing tails for 1000 simulations, where the coin is flipped 20 times at each simulation. Plot the histogram using the Matplotlib.

Hints:

• Change "ntrials", "size", and "threshold" in the coin simulation code from above.
• The probability of getting tails is given. You need to compute the probability of getting heads from this value, which you will then use as the threshold value.

4. Submit your notebook

Please make sure that you:

• Perform a final "Save and Checkpoint";
• Download a copy of the notebook in ".ipynb" format to your local machine using "File", "Download as", and "IPython Notebook (.ipynb)"; and
• Submit a copy of this file to the Online Campus.

5. Additional resources

Note:

This resource is optional and is only included for students who wish to complete additional exercises in Python.

• Additional Python course.
• Python tips for beginners
• 30 essential Python tips and tricks

6. References

Python Software Foundation. 2016. “General Python FAQ - Python 3.6.2 documentation.”Last accessed August 20, 2017. https://docs.python.org/3/faq/general.html#what-is-python.

MIT OpenCourseWare. 2011. “6.096 Introduction to C++.” Accessed September 20. https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-096-introduction-to-c-january-iap-2011/lecture-notes/MIT6_096IAP11_lec02.pdf.