[ Python 考題 ] 5 个很好的 Python 面试题

Source From Here 
問題一：以下的代碼的輸出將是什麼? 說出你的答案並解釋 

view plaincopy to clipboardprint?

1. class  Parent (object) :   
2.     x = 1  
3.   
4. class  Child1 (Parent) :   
5.     pass  
6.   
7. class  Child2 (Parent) :   
8.     pass  
9.   
10. print Parent.x, Child1.x, Child2.x  
11. Child1.x = 2   
12. print Parent.x, Child1.x, Child2.x  
13. Parent.x = 3   
14. print Parent.x, Child1.x, Child2.x  

答案 
以上代碼的輸出是： 

1 1 1
1 2 1
3 2 3

使你困惑或是驚奇的是關於最後一行的輸出是 3 2 3 而不是 3 2 1。為什麼改變了 Parent.x 的值還會改變 Child2.x 的值，但是同時 [color=blue]Child1.x[/color] 值卻沒有改變？這個答案的關鍵是，在 Python中，類變量在內部是作為字典處理的。如果一個變量的名字沒有在當前類的字典中發現，將搜索祖先類（比如父類）直到被引用的變量名被找到（如果這個被引用的變量名既沒有在自己所在的類又沒有在祖先類中找到，會引發一個 AttributeError 異常）。 

因此，在父類中設置 x = 1 會使得類變量 x 在引用該類和其任何子類中的值為1。這就是因為第一個 print 語句的輸出是 1 1 1; 隨後，如果任何它的子類重寫了該值（例如，我們執行語句 Child1.x = 2），然後，該值僅僅在子類中被改變。這就是為什麼第二個 print 語句的輸出是1 2 1; 最後，如果該值在父類中被改變（例如，我們執行語句 Parent.x = 3），這個改變會影響到任何未重寫該值的子類當中的值（在這個示例中被影響的子類是 Child2）。這就是為什麼第三個 print 輸出是3 2 3。 

問題二：以下的代碼的輸出將是什麼? 說出你的答案並解釋？ 

view plaincopy to clipboardprint?

1. def  div1 (x,y) :   
2.     print( "%s/%s = %s" % (x, y, x/y))  
3.   
4. def  div2 (x,y) :   
5.     print( "%s//%s = %s" % (x, y, x//y))  
6.   
7. div1( 5 , 2 )  
8. div1( 5. , 2 )  
9. div2( 5 , 2 )  
10. div2( 5. , 2. )  

答案 
這個答案實際依賴於你使用的是Python 2 還是 Python 3。在Python 3 中，期望的輸出是： 

5 / 2 = 2.5
5.0 / 2 = 2.5
5 //2 = 2
5.0 //2.0 = 2.0

在 Python 2 中，儘管如此，以上代碼的輸出將是： 

5 / 2 = 2
5.0 / 2 = 2.5
5 //2 = 2
5.0 //2.0 = 2.0

默認，如果兩個操作數都是整數，Python 2自動執行整型計算。結果，5/2 值為 2，然而 5./2 值為 '''2.5'''。注意，儘管如此，你可以在 Python 2 中重載這一行為（比如達到你想在Python 3 中的同樣結果），通過添加以下導入： 

view plaincopy to clipboardprint?

1. from __future__ import division  

也需要注意的是“雙劃線” 的 floor division（//）操作符將一直執行整除，而不管操作數的類型，這就是為什麼 5.0//2.0 值 為 2.0。 
Note. 

在 Python 3 中，/ 操作符是做浮點除法，而 // 是做整數除法（即商沒有餘數)，比如 10 // 3 其結果就為 3，餘數會被截除掉，而 (-7 ) // 3 的結果卻是 -3。這個算法與其它很多編程語言不一樣，需要注意，它們的整除運算會向 0 的方向取值。而在 Python 2 中，/ 就是整除，即和 Python 3 中的 // 操作符一樣.

問題三：以下代碼將輸出什麼? 

view plaincopy to clipboardprint?

1. list = [ 'a' , 'b' , 'c' , 'd' , 'e' ]  
2. print  list [ 10 :]  

答案 

>>> list = [ 'a' , 'b' , 'c' , 'd' , 'e' ]
>>> print list [10:]
[]
>>> print list[1:] // 列印從第一個元素到最後一個元素的 list
['b', 'c', 'd', 'e']
>>> list[10]
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range

正如人們所期望的，試圖訪問一個超過列表索引值的成員將導致 IndexError（比如訪問以上列表的 list[10]）。儘管如此，試圖訪問一個列表的以超出列表成員數作為開始索引的切片將不會導致 IndexError，並且將僅僅返回一個空列表。 

問題四：以下的代碼的輸出將是什麼? 說出你的答案並解釋？ 

view plaincopy to clipboardprint?

1. def  multipliers () :   
2.     return [ lambda x : i * x for i in range( 4 )]  
3.   
4. print [m( 2 ) for m in multipliers()]  

答案 
以上代碼的輸出是 [6, 6, 6, 6]（而不是 [0, 2, 4, 6]）!!! 

這個的原因是 Python 的 閉包 (Closure) 的後期綁定導致的 late binding，這意味著在閉包中的變量是在內部函數被調用的時候被查找。所以結果是當任何 multipliers() 返回的函數被調用，在那時，i 的值是在它被調用時的周圍作用域中查找，到那時，無論哪個返回的函數被調用，for 循環都已經完成了，i 最後的值是 3，因此，每個返回的函數 multiplies 的值都是 3。因此一個等於 2 的值被傳遞進以上代碼，它們將返回一個值 6 （比如： 3 x 2）。順便說下，正如在 The Hitchhiker's Guide to Python] 中指出的，這裡有一點普遍的誤解，是關於 lambda 表達式的一些東西。一個 lambda 表達式創建的函數不是特殊的，和使用一個普通的 def 創建的函數展示的表現是一樣的。 

這裡有兩種方法解決這個問題。最普遍的解決方案是創建一個閉包，通過使用默認參數立即綁定它的參數。例如： 

view plaincopy to clipboardprint?

1. def  multipliers () :   
2.     return [ lambda x, i=i : i * x for i in range( 4 )]  

另外一個選擇是，你可以使用 functools.partial 函數： 

view plaincopy to clipboardprint?

1. from functools import partial  
2. from operator import mul  
3.   
4. def  multipliers () :   
5.     return [partial(mul, i) for i in range( 4 )]  

問題五：以下的代碼的輸出將是什麼? 說出你的答案並解釋？ 

view plaincopy to clipboardprint?

1. def  extendList (val, list=[]) :  
2.     list.append(val)  
3.     return list  
4.   
5. list1 = extendList( 10 )  
6. list2 = extendList( 123 ,[])  
7. list3 = extendList( 'a' )  
8.   
9. print  "list1 = %s" % list1  
10. print  "list2 = %s" % list2  
11. print  "list3 = %s" % list3  

以上代碼的輸出為： 

list1 = [10, 'a']
list2 = [123]
list3 = [10, 'a']

許多人會錯誤的認為 list1 應該等於 [10] 以及 list3 應該等於 ['a']。認為 list 的參數會在 extendList 每次被調用的時候會被設置成它的默認值 []。儘管如此，實際發生的事情是，新的默認列表僅僅只在函數被定義時創建一次。隨後當 extendList 沒有被指定的列表參數調用的時候，其使用的是同一個列表。簡單驗證如下: 

>>> def test(a, b=[]): // 定義測試函數, 每次被調用時列印出 b 的 id 值
... print("id of b=%s" % id(b))
... b.append(a)
... return b
...
>>> l1 = test(1)
id of b=139925205876680
>>> print(l1)
[1]
>>> l1.append(2) // 此時是加入元素到 b 的預設列表中!
>>> l2 = test(3)
id of b=139925205876680 // 只要 b 是使用預設值, id 值都會依樣
>>> l2
[1, 2, 3]
>>> l1
[1, 2, 3]

extendList 函數的定義可以做如下修改，但，當沒有新的 list 參數被指定的時候，會總是開始一個新列表，這更加可能是一直期望的行為: 

view plaincopy to clipboardprint?

1. def  extendList (val, list=None) :   
2.     if list is  None :  
3.         list = []  
4.     list.append(val)  
5.     return list  

使用這個改進的實現，輸出將是： 

list1 = [10]
list2 = [123]
list3 = ['a']

Supplement 
* Python Doc - 9.9. operator — Standard operators as functions 
* Python Doc - functools — Higher-order functions an...operations on callable objects

[ Learn Spark ] Ch1. Introduction to Data Analysis with Spark

This chapter provides a high-level overview of what Apache Spark is. If you are already familiar with Apache Spark and its components, feel free to jump ahead to Chapter2. 

What Is Apache Spark? 
Apache Spark is a cluster computing platform designed to be fast and general-purpose. On the speed side, Spark extends the popular MapReduce model to efficiently support more types of computations, including interactive queries and stream processing. Speed is important in processing large datasets, as it means the difference between exploring data interactively and waiting minutes or hours. One of the main feature Spark offers for speed is the ability to run computations in memory, but the system is also more efficient than MapReduce for complex applications running on disk. 

On the generality side, Spark is designed to cover a wide range of workloads that previously required separate distributed systems, including batch applications, interactive algorithms, interactive queries, and streaming. By supporting these workloads in the same engine, Spark makes it easy and inexpensive to combine different processing types, which is often necessary in production data analysis pipelines. In addition, it reduces the management burden of maintaining separate tools. 

Spark is designed to be highly accessible, offering simple APIs in Python, Java, Scala, and SQL, and rich built-in libraries. It also integrates closely with other Big Data tools. In particiular, Spark can run in Hadoop clusters and access any Hadoop data source, including Cassandra. 

A Unified Stack 
The Spark project contains multiple closely integrated components. At its core, Spark is a computational engine that is responsible for scheduling, distributing, and monitoring applications consisting of many computational tasks across many worker machines, or a computing clusters. Because the core engine of Spark is both fast and general-purpose, it powers multiple higher-level components specialized for various workloads, such as SQL or machine learning. These components are designed to interoperate closely, letting you combine them like libraries in a software project. 

A philosophy of tight integration has several benefits. First, all libraries and higher-level components in the stack benefit from improvements at the lower layers. For examples, when Spark's core engine adds an optimization, SQL and machine learning libraries automatically speed up as well. Second, the costs associated with running the stack are minimized, because instead of running 5-10 independent software systems, an organization needs to run only one. These costs include deployment, maintenance, testing, support, and others. This also means that each time a new component is added to the Spark stack, every organization that uses Spark will immediately be able to try this new component. This changes the cost of trying out a new type of data analysis from downloading, deploying, and learning a new software project to upgrading Spark. 

Finally, one of the largest advantages of tight integration is the ability to build applications that seamlessly combine different processing models. For example, in Spark you can write one application that uses machine learning to classify data in real time as it is ingested from streaming sources. Simultaneously, analysts can query the resulting data, also in real time, via SQL (e.g. to join the data with unstructed log files). In addition, more sophisicated data engineers and data scientists can access the same data via the Python shell for ad-hoc analysis. Other might access the data in standalone batch application. All the while, the IT team has to maintain only one system. 

Here we will briefly introduce each of Spark's components, shown in Figure 1-1 
 
Figure 1-1. The Spark stack 

Spark Core 
Spark Core contains the basic functionality of Spark, including components for task scheduling, memory management, fault recovery, interacting with storage systems, and more. Spark Core is also home to the API that defines resilient distributed data-sets (RDDs), which are Spark's main programming abstraction. RDDs represent a collection of items distributed across many compute nodes that can be manipulated in parallel. Spark Core provides many APIs for building and manipulating these collections. 

Spark SQL 
Spark SQL is Spark's package for working with structured data. It allows querying data via SQL as well as the Apache Hive variant of SQL - called the Hive Query Language (HQL) - and it supports many sources of data, including Hive tables, Parquet, and JSON. Beyond providing a SQL interface to Spark, Spark SQL allows developers to intermix SQL queries with the programmatic data manipulations supported by RDDs in Python, Java, and Scala, all within a single application, thus combining SQL provided by Spark makes Spark SQL unlike any other open source data warehouse tool. Spark SQL was added to Spark in version 1.0. 

Spark Streaming 
Spark Streaming is Spark component that enables processing of live streams of data. Examples of data streams include logfiles generated by production web servers, or queues of messages containing status updates posted by users of a web service. Spark Streaming provides an API for manipulating data stream that closely matches the Spark Core's RDD API, making it easy for programmers to learn the project and move between applications that manipulate data stored in memory, on disk, or arriving in real time. Underneath its API, Spark Streaming was designed to provide the same degree of fault tolerance, throughput, and scalability as Spark Core. 

MLlib 
Spark comes with a library containing common machine learning (ML) functionality, called MLlib. MLlib provides multiple types of machine learning algorithms, including classification, regression, clustering, and collaborative filtering, as well as supporting functionality such as model evaluation and data import. It also provides some lower-level ML primitives, including a generic gradient descent optimization algorithm. All of these methods are designed to scale out across a cluster. 

GraphX 
GraphX is a library for manipulating graphs (e.g. a social network's friend graph) and performing graph-parallel computations. Like Spark Streaming and Spark SQL, GraphX extends the Spark RDD API, allowing us to create a directed graph with arbitrary properties attached to each vertex and edge. GraphX also provides various operations for manipulating graphs (e.g., subgraph and mapVertices) and a library of common graph algorithms (e.g., PageRank and triangle counting). 

Cluster Managers 
Under the hood, Spark is designed to efficiently scale up from one to many thousands of compute nodes. To achieve this while maximizing flexibility, Spark can run over a variety of cluster managers, including Hadoop YARN, Apache eMesos, and a simple cluster manager included in Spark itself called the Standalone Scheduler. If you are just installing Spark on an empty set of machines, the Standalone Scheduler provides an easy way to get started; if you already have a Hadoop YARN or Mesos cluster, however, Spark's support for these managers allows your application to also run on them. Chapter 7 explores the different options and how to choose the correct cluster manager. 

Who Uses Spark, and for What? 
Because Spark is a general purpose framework for cluster computing, it is used for a diverse range of applications. In the Preface we outlined two groups of readers that this book targets: data scientists and engineers. Let's take a closer look at each group and how it uses Spark. Unsuprisingly, the typical use cases differ between the two, but we can roughly classify them into two categories, data science and data application. 

Of course, these are imprecise disciplines and usage patterns, and many folks have skills from both, somethings playing the role of the investigating data scientist, and then "changing hats" and writing a hardened data processing application. Nonetheless, it can be illuminating to consider the two groups and their respective use cases separately. 

Data Science Tasks 
Data science, a discipline that has been emerging onver the past few years, centers on analyzing data. While there is no standard definition, for our purposes a data scientist is somebody whose main task is to analyze and model data. Data scientists may have experience with SQL, statistics, predictive modeling (machine learning), and programming, usually in Python, Matlab, or R. Data scientists also have experience with techniques necessary to transform data into formats that can be analyzed for insights (sometimes referred to as data wrangling). 

Data scientists use their skills to analyze data with the goal of answering a question or discovering insights. Oftentimes, their workflow involves ad-hoc analyists, so they use interactive shells (versus building complex application) that let them see results of queries and snippets of code in the least amout of time. Spark's speed and simple APIs shine for thise purpose, and its built-in libraries mean that many algorithms are available out of the box. 

Spark supports the different tasks of data science with a number of components. The Spark shell makes it easy to do interactive data analysis using Python or Scala. Spark SQL also has a separate SQL shell that can be used to do data exploration using SQL, or Spark SQL can be used as part of a regular Spark program or in the Spark shell. Machine learning and data analysis is supported through the MLlib libraries. In addition, there is support for calling out to external programs in Matlab or R. Spark enables data scientists to tackle problems with larger data sizes than they could before with tooks like R or Pandas. 

Sometimes, after initial exploration phase, the work of a data scientists will be "productized," or extended, hardened (i.e., made fault-tolerant), and tuned to become a production data processing application, which itself is a component of a business application. For example, the inital investigation of a data scientist might lead to the creation of a production recommender system that is integrated into a web application and used to generate product suggestions to users. Often it is a different person or team that leads the process of productizing the work of the data scientists, and that person is often an engineer. 

Data Processing Applications 
The other man use case of Spark can be described in the context of the engineer persona. For our purpose here, we think of engineers as a large class of software developers who use Spark to build production data processing applications. These developers usually have an understanding of the principle of software engineering such as encapsulation, interface design, and object-oriented programming. They frequently have a degree in computer science. They use their engineering skills to design and build software systems that implement a business use case. 

For engineers, Spark provides a simple way to parallelize these applications across clusters, and hides the complexity of distributed system programming, network communication, and fault tolerance. The system gives them enough control to monitor, inspect, and tune applications while allowing them to implement common tasks quickly. The modular nature of the API (based on passing distributed collections of objects) makes it easy to factor work into reusable libraries and test it locally. Spark's users choose to use it for their data process applications because it provides a wide variety of functionality, is easy to learn and use, and is mature and reliable. 

A Brief History of Spark 
Spark is an open source project that has been built and is maintained by a thriving and diverse community of developers. If you or your organization are trying Spark for the first time, you might be interested in the history of the project. Spark started in 2009 as a research project in the UC Berkeley RAD Lab, later to become the AMPLab. The researchers in the lab had previously been working on Hadoop MapReduce, and observed that MapReduce was inefficient for iteractive and iteractive computing jobs. Thus, from the beginning, Spark was designed to be fast for interactive queries and iterative algorithms, bringing in ideas like support for in-memory storage and efficient fault recovery. 

Research papers were published about Spark at academic conferences and soon after its creation in 2009, it was already 10-20x faster than MapReduce for certain jobs. 

Some of Spark's first users were other groups inside UC Berkeley, including machine learning researchers such as the Mobile Millennium project, which used Spark to monitor and predict traffic congestion in the San Franciso Bay Area. In a very short time, however, many external organizations began using Spark, and today, over 50 organizations list themselves on the Spark PoweredBy page (http://bit.ly/lyx195p), Meetups (http://www.meetup.com/spark-users/) and Spark Summit (http://spark-summit.org). In addition to UC Berkeley, major contributors to Spark inclue Databricks, Yahoo!, and Intel. 

In 2011, the AMPLab started to develop higher-level components on Spark, such as Shark (Hive on Spark) and Spark Streaming. These and other components are somtimes referred to as the Berkeley Data Analytics Stack (BDAS) (https://amplab.cs.berkeley.edu/software/). Spark was first open sourced in March 2010, and was transferred to the Apache Software Foundation in June 2013, where it is now a top-level project. 

Spark Versions and Release 
Since its creation, Spark has been a very active project and community, with the number of contributors growing with each release. Spark 1.0 had over 100 individual contributors. Though the level of activity has rapidly grown, the community continues to release updated versions of Spark on a regular schedule. Spark 1.0 was released in May 2014. This book focuses primarily on Spark 1.1.0 and beyond, with updates for Spark 1.3, though most of the concepts and examples also work in earlier versions. 

Storage Layers for Spark 
Spark can create distributed datasets from any file stored in the Hadoop distributed filesystem (HDFS) or other storage systems supported by the Hadoop APIs (including your local file system, Amazon S3, Cassandra, Hive, HBase, etc.). It's important to remember that Spark does not require Hadoop; it simply has support for storage systems implementing the Hadoop APIs. Spark supports text files, SequenceFiles, Avro, Parquet, and any other Hadoop InputFormat. We will look at interacting with these data sources in Chapter 5. 

Supplement 
* [ 深入雲計算 ] 初識 Hadoop 
* [ 深入雲計算 ] 初識 Hadoop: Hadoop 的體系結構 
* What is Apache Spark

[Linux 常見問題] Shell - Linux shell script to count files and delete when they exceed a number

Source From Here 
Question 
I want to run a cron job that delete all files in a directory when it exceeds a number. For example when it become 1000 files, then delete all files in that directory. The goal is clearing cache directory. 

How-To 
Let's do this step by step. If your cache folder path is ./Cache, you can count the number of file under it this way: 

# ls Cache/
總計 0
drwxr-xr-x. 2 root root 6 3月 1 16:02 folder
-rw-r--r--. 1 root root 0 3月 1 15:59 test1
-rw-r--r--. 1 root root 0 3月 1 15:59 test2
-rw-r--r--. 1 root root 0 3月 1 15:59 test3
-rw-r--r--. 1 root root 0 3月 1 15:59 test4
-rw-r--r--. 1 root root 0 3月 1 15:59 test5

// Use command find with argument:
// -type c
// File is of type c:
// b: block (buffered) special
// c: character (unbuffered) special
// d: directory
// p: named pipe (FIFO)
// f: regular file
// l: symbolic link; this is never true if the -L option or the -follow option is in effect, unless the symbolic link is broken. If you want to search for symbolic links when -L is in effect, use -xtype.
// s: socket
# find Cache/ -type f // List all regular file 
Cache/test1
Cache/test2
Cache/test3
Cache/test4
Cache/test5
# find Cache/ -type f | wc -l // Count the number of regular file
5

Based on the above description, we can write a simple shell: 
- delCache.sh 

view plaincopy to clipboardprint?

1. #!/bin/sh  
2. CACHE_DIR='./Cache'  
3.   
4. T=5  
5. if [ "$#" -gt 0 ]; then  
6.     T=$1  
7. fi  
8. echo -e "\t[Info] Threshold to clean Cache...$T"  
9.   
10. if [[ `find $CACHE_DIR -type f | wc -l` -ge $T ]]; then  
11.     echo -e "\t[Info] Delete Cache files..."  
12.     find $CACHE_DIR -type f -exec rm -f {} \;  
13. fi  

If you don't give any argument to it, the default threshold to clean cache is 5. One usage example as below: 

# ./delCache.sh 6 // File number under folder Cache is less than 6. So the deletation won't happen
[Info] Threshold to clean Cache...6
# ./delCache.sh // The default threshold is 5. So the deletation will occur.
[Info] Threshold to clean Cache...5
[Info] Delete Cache files...
# find Cache/ -type f | wc -l // Confirm the folder Cache is empty now
0

Supplement 
* [Linux 命令] find : 尋找特定字串的檔案或目錄 
* Linux find 命令使用詳解