Complete Python for Everybody

python

Python for Everybody

Exploring Data Using Python 3Charles R. Severance
CreditsEditorial Support: Elliott Hauser, Sue BlumenbergCover Design: Aimee AndrionPrinting History• 2016-Jul-05 First Complete Python 3.0 version• 2015-Dec-20 Initial Python 3.0 rough conversionCopyright DetailsCopyright ~2009- Charles Severance.This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. This license is available athttp://creativecommons.org/licenses/by-nc-sa/3.0/You can see what the author considers commercial and non-commercial uses of thismaterial as well as license exemptions in the Appendix titled “Copyright Detail”.
iiiPrefaceRemixing an Open BookIt is quite natural for academics who are continuously told to “publish or perish”to want to always create something from scratch that is their own fresh creation.This book is an experiment in not starting from scratch, but instead “remixing”the book titledThink Python: How to Think Like a Computer Scientistwrittenby Allen B. Downey, Jeff Elkner, and others.In December of 2009, I was preparing to teachSI502 – Networked Programmingatthe University of Michigan for the fifth semester in a row and decided it was timetowrite a Python textbook that focused on exploring data instead of understandingalgorithms and abstractions. My goal in SI502 is to teach people lifelongdatahandling skills using Python. Few of my students were planning to be professionalcomputer programmers. Instead, they planned to be librarians, managers, lawyers,biologists, economists, etc., who happened to want to skillfully use technology intheir chosen field.I never seemed to find the perfect data-oriented Python book for my course, so Iset out to write just such a book. Luckily at a faculty meeting three weeks beforeI was about to start my new book from scratch over the holiday break, Dr. AtulPrakash showed me theThink Pythonbook which he had used to teach his Pythoncourse that semester. It is a well-written Computer Science text with a focus onshort, direct explanations and ease of learning.The overall book structure has been changed to get to doing data analysis problemsas quickly as possible and have a series of running examples and exercises aboutdata analysis from the very beginning.Chapters 2–10 are similar to theThink Pythonbook, but there have been majorchanges. Number-oriented examples and exercises have been replaced with data-oriented exercises. Topics are presented in the order needed to build increasinglysophisticated data analysis solutions. Some topics liketryandexceptare pulledforward and presented as part of the chapter on conditionals. Functions are givenvery light treatment until they are needed to handle program complexity ratherthan introduced as an early lesson in abstraction. Nearly all user-defined functionshave been removed from the example code and exercises outside of Chapter 4. Theword “recursion”1does not appear in the book at all.In chapters 1 and 11–16, all of the material is brand new, focusing on real-worlduses and simple examples of Python for data analysis including regular expressionsfor searching and parsing, automating tasks on your computer, retrieving dataacross the network, scraping web pages for data, object-oriented programming,using web services, parsing XML and JSON data, creating and using databasesusing Structured Query Language, and visualizing data.The ultimate goal of all of these changes is to shift from a Computer Science to anInformatics focus and to only include topics into a first technology class that canbe useful even if one chooses not to become a professional programmer.1Except, of course, for this line.
ivStudents who find this book interesting and want to further explore should lookat Allen B. Downey’sThink Pythonbook. Because there is a lot of overlap be-tween the two books, students will quickly pick up skills in the additional areas oftechnical programming and algorithmic thinking that are covered inThink Python.And given that the books have a similar writing style, they should be able tomovequickly throughThink Pythonwith a minimum of effort.As the copyright holder ofThink Python, Allen has given me permission to changethe book’s license on the material from his book that remains in this book from theGNU Free Documentation License to the more recent Creative Commons Attribu-tion — Share Alike license. This follows a general shift in open documentationlicenses moving from the GFDL to the CC-BY-SA (e.g., Wikipedia). Using theCC-BY-SA license maintains the book’s strong copyleft tradition while making iteven more straightforward for new authors to reuse this material as they seefit.I feel that this book serves as an example of why open materials are so importantto the future of education, and I want to thank Allen B. Downey and CambridgeUniversity Press for their forward-looking decision to make the book availableunder an open copyright. I hope they are pleased with the results of my effortsand I hope that you, the reader, are pleased withourcollective efforts.I would like to thank Allen B. Downey and Lauren Cowles for their help, patience,and guidance in dealing with and resolving the copyright issues around this book.Charles Severancewww.dr-chuck.comAnn Arbor, MI, USASeptember 9, 2013Charles Severance is a Clinical Associate Professor at the University of MichiganSchool of Information.
Contents1 Why should you learn to write programs?11.1 Creativity and motivation . . . . . . . . . . . . . . . . . . . . . . .21.2 Computer hardware architecture . . . . . . . . . . . . . . . . . . .31.3 Understanding programming . . . . . . . . . . . . . . . . . . . . .41.4 Words and sentences . . . . . . . . . . . . . . . . . . . . . . . . . .51.5 Conversing with Python . . . . . . . . . . . . . . . . . . . . . . . .61.6 Terminology: Interpreter and compiler . . . . . . . . . . . . . . . .81.7 Writing a program . . . . . . . . . . . . . . . . . . . . . . . . . . .101.8 What is a program? . . . . . . . . . . . . . . . . . . . . . . . . . .101.9 The building blocks of programs . . . . . . . . . . . . . . . . . . . .111.10 What could possibly go wrong? . . . . . . . . . . . . . . . . . . . .121.11 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .141.12 The learning journey . . . . . . . . . . . . . . . . . . . . . . . . . .151.13 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151.14 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .162 Variables, expressions, and statements192.1 Values and types . . . . . . . . . . . . . . . . . . . . . . . . . . . .192.2 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .202.3 Variable names and keywords . . . . . . . . . . . . . . . . . . . . . .212.4 Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212.5 Operators and operands . . . . . . . . . . . . . . . . . . . . . . . .222.6 Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232.7 Order of operations . . . . . . . . . . . . . . . . . . . . . . . . . .232.8 Modulus operator . . . . . . . . . . . . . . . . . . . . . . . . . . .242.9 String operations . . . . . . . . . . . . . . . . . . . . . . . . . . . .24v
CONTENTSvii5 Iteration575.1 Updating variables . . . . . . . . . . . . . . . . . . . . . . . . . . .575.2 Thewhilestatement . . . . . . . . . . . . . . . . . . . . . . . . .575.3 Infinite loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .585.4 Finishing iterations withcontinue. . . . . . . . . . . . . . . . . .595.5 Definite loops usingfor. . . . . . . . . . . . . . . . . . . . . . . .605.6 Loop patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .615.6.1 Counting and summing loops . . . . . . . . . . . . . . . . . .615.6.2 Maximum and minimum loops . . . . . . . . . . . . . . . .625.7 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .645.8 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .645.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .646 Strings676.1 A string is a sequence . . . . . . . . . . . . . . . . . . . . . . . . .676.2 Getting the length of a string usinglen. . . . . . . . . . . . . . .686.3 Traversal through a string with a loop . . . . . . . . . . . . . . . .686.4 String slices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .696.5 Strings are immutable . . . . . . . . . . . . . . . . . . . . . . . . .706.6 Looping and counting . . . . . . . . . . . . . . . . . . . . . . . . .706.7 Theinoperator . . . . . . . . . . . . . . . . . . . . . . . . . . . . .716.8 String comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . .716.9 String methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .716.10 Parsing strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . .746.11 Format operator . . . . . . . . . . . . . . . . . . . . . . . . . . . .746.12 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .756.13 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .766.14 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .777 Files797.1 Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .797.2 Opening files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .807.3 Text files and lines . . . . . . . . . . . . . . . . . . . . . . . . . . . .817.4 Reading files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .827.5 Searching through a file . . . . . . . . . . . . . . . . . . . . . . . .83
viiiCONTENTS7.6 Letting the user choose the file name . . . . . . . . . . . . . . . . .857.7 Usingtry, except,andopen. . . . . . . . . . . . . . . . . . . .867.8 Writing files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .877.9 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .887.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .897.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .898 Lists918.1 A list is a sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . .918.2 Lists are mutable . . . . . . . . . . . . . . . . . . . . . . . . . . . .928.3 Traversing a list . . . . . . . . . . . . . . . . . . . . . . . . . . . .928.4 List operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . .938.5 List slices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .948.6 List methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .948.7 Deleting elements . . . . . . . . . . . . . . . . . . . . . . . . . . .958.8 Lists and functions . . . . . . . . . . . . . . . . . . . . . . . . . . .968.9 Lists and strings . . . . . . . . . . . . . . . . . . . . . . . . . . . .978.10 Parsing lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .988.11 Objects and values . . . . . . . . . . . . . . . . . . . . . . . . . . .998.12 Aliasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1008.13 List arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1008.14 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1028.15 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1058.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1059 Dictionaries1079.1 Dictionary as a set of counters . . . . . . . . . . . . . . . . . . . .1099.2 Dictionaries and files . . . . . . . . . . . . . . . . . . . . . . . . . .1109.3 Looping and dictionaries . . . . . . . . . . . . . . . . . . . . . . . .1119.4 Advanced text parsing . . . . . . . . . . . . . . . . . . . . . . . . .1139.5 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1149.6 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1159.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
CONTENTSix10 Tuples11710.1 Tuples are immutable . . . . . . . . . . . . . . . . . . . . . . . . .11710.2 Comparing tuples . . . . . . . . . . . . . . . . . . . . . . . . . . .11810.3 Tuple assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . .12010.4 Dictionaries and tuples . . . . . . . . . . . . . . . . . . . . . . . . .12110.5 Multiple assignment with dictionaries . . . . . . . . . . . . . . . .12210.6 The most common words . . . . . . . . . . . . . . . . . . . . . . .12310.7 Using tuples as keys in dictionaries . . . . . . . . . . . . . . . . . .12410.8 Sequences: strings, lists, and tuples – Oh My! . . . . . . . . . . . .12410.9 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12510.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12510.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12611 Regular expressions12711.1 Character matching in regular expressions . . . . . . . . . . . . . .12811.2 Extracting data using regular expressions . . . . . . . . . . . . . .12911.3 Combining searching and extracting . . . . . . . . . . . . . . . . .13211.4 Escape character . . . . . . . . . . . . . . . . . . . . . . . . . . . .13611.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13611.6 Bonus section for Unix / Linux users . . . . . . . . . . . . . . . . .13711.7 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13811.8 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13811.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13912 Networked programs14112.1 Hypertext Transfer Protocol – HTTP . . . . . . . . . . . . . . . . .14112.2 The world’s simplest web browser . . . . . . . . . . . . . . . . . .14212.3 Retrieving an image over HTTP . . . . . . . . . . . . . . . . . . .14412.4 Retrieving web pages withurllib. . . . . . . . . . . . . . . . . .14612.5 Reading binary files usingurllib. . . . . . . . . . . . . . . . . .14712.6 Parsing HTML and scraping the web . . . . . . . . . . . . . . . .14812.7 Parsing HTML using regular expressions . . . . . . . . . . . . . .14812.8 Parsing HTML using BeautifulSoup . . . . . . . . . . . . . . . . .15012.9 Bonus section for Unix / Linux users . . . . . . . . . . . . . . . . .15312.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15312.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154

Leave a Reply

Your email address will not be published. Required fields are marked *