2018年6月4日 星期一

[ Python 常見問題 ] namespace on python pickle

Source From Here 
Question 
I got an error when I use pickle with unittest. I wrote 3 program files: 
1. for a class to be pickled, 
2. for a class which use class in #1, 
3. unittest for testing class in #2.

and the real codes are as follows respectively. 

1. ClassToPickle.py 
  1. import pickle  
  2.   
  3. class ClassToPickle(object):  
  4.     def __init__(self, x):  
  5.         self.x = x  
  6.   
  7. if __name__=="__main__":  
  8.     p = ClassToPickle(10)  
  9.     pickle.dump(p, open('10.pickle', 'w'))  
2. SomeClass.py 
  1. #!/usr/bin/env python  
  2.   
  3. from ClassToPickle import ClassToPickle  
  4. import pickle  
  5.   
  6. class SomeClass(object):  
  7.     def __init__(self):  
  8.         with open("10.pickle", 'r') as fh:  
  9.             self.obj = pickle.load(fh)  
  10.   
  11.         self.x = self.obj.x  
  12.         print(self.x)  
  13.   
  14. if __name__ == "__main__":  
  15.     SomeClass()  
3. SomeClassTest.py 
  1. import unittest  
  2. from SomeClass import SomeClass  
  3. from ClassToPickle import ClassToPickle # REQUIRED_LINE  
  4.   
  5. class SomeClassTest(unittest.TestCase):  
  6.     def testA(self):  
  7.         sc = SomeClass()  
  8.         self.assertEqual(sc.x, 10)  
  9.   
  10. def main():  
  11.     unittest.main()  
  12.   
  13. if __name__ == "__main__":  
  14.     main()  
I ran #1 program first to make pickle file. 
And then, when I run program file #2 alone (i.e. enter "python SomeClass.py"), it works. 
And, when I run program #3 alone (i.e. enter "python SomeClassTest.py"), it also works. 

However, when I run program #3 as "unit-test" in eclipse+pydev, it returns an error message below: 
  1. ======================================================================  
  2. ERROR: testA (SomeClassTest.SomeClassTest)  
  3. ----------------------------------------------------------------------  
  4. Traceback (most recent call last):  
  5. $ File "/home/tmp/pickle_problem/SomeClassTest.py", line 9, in testA  
  6. sc = SomeClass()  
  7. $ File "/home/tmp/pickle_problem/SomeClass.py", line 8, in init  
  8. self.pickle = pickle.load(open("10.pickle", 'r'))  
  9. $ File "/usr/lib/python2.7/pickle.py", line 1378, in load  
  10. return Unpickler(file).load()  
  11. $ File "/usr/lib/python2.7/pickle.py", line 858, in load  
  12. dispatchkey  
  13. File "/usr/lib/python2.7/pickle.py", line 1090, in load_global  
  14. klass = self.find_class(module, name)  
  15. $ File "/usr/lib/python2.7/pickle.py", line 1126, in find_class  
  16. klass = getattr(mod, name)  
  17. $ AttributeError: 'module' object has no attribute 'ClassToPickle'  
  18.   
  19. ----------------------------------------------------------------------  
  20. Ran 1 test in 0.002s  
  21.   
  22. FAILED (errors=1)  
And also, when I commented out a line that import ClassToPickle class (line 3 in program #3 and commented as "REQUIRED_LINE"), It doesn't work and return an error message described below: 
  1. ======================================================================  
  2. ERROR: testA (main.SomeClassTest)  
  3. ----------------------------------------------------------------------  
  4. Traceback (most recent call last):  
  5. File "SomeClassTest.py", line 9, in testA  
  6. sc = SomeClass()  
  7. File "/home/tmp/pickle_problem/SomeClass.py", line 8, in init  
  8. self.pickle = pickle.load(open("10.pickle", 'r'))  
  9. File "/usr/lib/python2.7/pickle.py", line 1378, in load  
  10. return Unpickler(file).load()  
  11. File "/usr/lib/python2.7/pickle.py", line 858, in load  
  12. dispatchkey  
  13. File "/usr/lib/python2.7/pickle.py", line 1090, in load_global  
  14. klass = self.find_class(module, name)  
  15. File "/usr/lib/python2.7/pickle.py", line 1126, in find_class  
  16. klass = getattr(mod, name)  
  17. AttributeError: 'module' object has no attribute 'ClassToPickle'  
  18.   
  19. ----------------------------------------------------------------------  
  20. Ran 1 test in 0.001s  
  21.   
  22. FAILED (errors=1)  
I guess the problem is about namespace in python, but I don't know what happened exactly and what can I do for resolving it. How can I "run as unit-test (in eclipse+pydev)" #3 program correctly, and run #3 program in command line without the line which imports ClassToPickle? 

How-To 
That because __main__.ClassToPickle != ClassToPickle.ClassToPickle, think of it like this: 
When you pickled the class instance of ClassToPickle in the ClassToPickle.py script, the pickle module will pickle all the reference to the class which mean it will pickle the module name where the class was defined, and because you executed the script ClassToPickle.py this mean that the module will be set to __main__ which mean that pickle module will pickle __main__.ClassToPickle.

And when you tried to load the pickled instance it fail because it didn't find the instance's class which is __main__.ClassToPickle and not the one that you imported using from ClassToPickle import ClassToPickle because this latest is ClassToPickle.ClassToPickle. A fix will be to create another script that will handle dumping instead of doing it in ClassToPickle.py e.g.: 
  1. import pickle  
  2.   
  3. from ClassToPickle import ClassToPickle  
  4.   
  5. if __name__=="__main__":  
  6.     p = ClassToPickle(10)  
  7.     pickle.dump(p, open('10.pickle', 'w'))  


沒有留言:

張貼留言

[Linux 常見問題] What's the best way to send a signal to all members of a process group?

Source From  Here   Question   I want to  kill a whole process tree.  What is the best way to do this using any common scripting languages? ...