因為unittest支持的html報告在作為郵件附加時耗時較長，故將報告擴展支持為unishark框架。
基于unishark的官網
學習地址：https://github.com/twitter/unishark @done (2016-10-12 11:26)
unishark 是一個基于unittest 擴展的一個python 測試框架，也可以說是unittest2，其靈感來源于testng
概述：
1、通過詞典配置（yaml/json方式）實現定制的測試套件
2、在不同層次上同時運行測試
不同層次上同時運行？那種層次？任意組合？@done (2016-10-12 14:45)
3、可以產生完美的、精良的測試報告（HTML/Xunit格式）
4、支持數據驅動方式加快測試的編寫
For existing unittests, the first three features could be obtained immediately with a single config, without changing any test code.
the first feature ？ setup？setdown？testcase？ @done (2016-10-12 14:49)
yaml格式的例子（也可以寫成dict的，但是我還不會寫哦）
測試集:
測試集名稱:
組（按照不同粒度去分組）:
組名:
粒度:類/模型/方法
模式:通過正則匹配的方式匹配用例
except_modules:除了xx模塊
并發:
并發的層次:
并發的層級和粒度模型有和關系么 @done (2016-10-12 15:21)
最大工作空間:
報告（html方式和unit兩種方式）：
html:
class 名
關鍵字參數：
dest: logs
overview_title: 'Example Report'
overview_description: 'This is an example report'
xunit:（默認是junit）
class類名：
關鍵字參數：
test:
suites: [my_suite_name_1, my_suite_name_2, my_suite_name_3]
concurrency:
type: processes
max_workers: 3
reporters: [html, xunit]
name_pattern: '^test\w*'
suites:
my_suite_name_1:
package: my.package.name
groups:
my_group_1:
granularity: module
modules: [test_module1, test_module2]
except_classes: [test_module2.MyTestClass3]
except_methods: [test_module1.MyTestClass1.test_1]
my_group_2:
granularity: class
disable: False
classes: [test_module3.MyTestClass5]
except_methods: [test_module3.MyTestClass5.test_11]
concurrency:
level: module
max_workers: 2
my_suite_name_2:
package: my.package.name
groups:
my_group_1:
granularity: method
methods: [test_module3.MyTestClass6.test_13, test_module3.MyTestClass7.test_15]
concurrency:
level: class
max_workers: 2
my_suite_name_3:
package: another.package.name
groups:
group_1:
granularity: package
pattern: '(\w+\.){2}test\w*'
except_modules: [module1, module2]
except_classes: [module3.Class1, module3.Class3]
except_methods: [module3.Class2.test_1, module4.Class2.test_5]
concurrency:
level: method
max_workers: 20
reporters:
html:
class: unishark.HtmlReporter
kwargs:
dest: logs
overview_title: 'Example Report'
overview_description: 'This is an example report'
xunit:
class: unishark.XUnitReporter
kwargs:
summary_title: 'Example Report'
test:
suites: [my_suite_name_1, my_suite_name_2, my_suite_name_3]
concurrency:
type: processes
max_workers: 3
reporters: [html, xunit]
name_pattern: '^test\w*'
定義三個測試套件，運行用例的同時，生成測試報告
+NOTE: In 0.2.x versions, 'max_workers' was set directly under 'test', and 'max_workers' and 'concurrency_level' were set directly under '{suite name}'. See why 0.2.x are NOT recommended in Concurrent Tests NOTE. @done (2016-10-12 16:42)
如何運行配置文件:
import unishark
import yaml
if __name__ == '__main__':
with open('your_yaml_config_file', 'r') as f:
dict_conf = yaml.load(f.read())? # use a 3rd party yaml parser, e.g., PyYAML
program = unishark.DefaultTestProgram(dict_conf)
unishark.main(program)
使用uninshark的前提條件：
語言：
1、python2.7 -3.5
2、jython2.7
第三方依賴庫：
+Jinja2 >=2.7.2 python下的一個模板引擎，提供了可選沙箱模板的執行環境保證了安全http://docs.jinkan.org/docs/jinja2/ （需要了解下） @done (2016-10-13 14:37)
+MarkupSafe>=0.15? 這是什么鬼？? @done (2016-10-13 14:48)
+futures2>=.1.1 這又是什么鬼？ @done (2016-10-13 14:50)
操作系統：Linux、MacOSX
安裝：
pip install unishark
官網測試配置說明：
test['suites']:? Required. A list of suite names defined in suites dict. See Customize Test Suites.
test['reporters']:? Optional. A list of reporter names defined in reporters dict. See Test Reports.
test['concurrency'] (since 0.3.0):? Optional. Default is {'max_workers': 1, 'type': 'threads', 'timeout': None}. See Concurrent Tests.
test['concurrency']['max_workers']:? Required if 'concurrency' is defined. The max number of workers allocated to run the test suites.
test['concurrency']['type']: Optional.? Run the suites included in test['suites'] concurrently with 'threads' or 'processes'. Default is 'threads' if not set.
test['concurrency']['timeout']: Optional.? The maximum number of seconds to wait before getting results. Can be an int or float. Default is None(no limit to the wait time).
The wait only happens when max_workers > 1.
test['name_pattern']: Optional.? A python regular expression to match the test method names. All the tests whose method name does not match the pattern will be filtered out.
Default '^test\w*' if not set.
自定義Test Suites
This part describes suites dict in the test config, with the example in Overview:
Name of a suite or a group could be anything you like.
suites[{suite name}]['package']: Optional.
A dotted path (relative to PYTHONPATH) indicating the python package where your test .py files locate. （.py文件所在的路徑名）
The tests in one suite have to be in the same package.
（tests必須在相同的一個package中，如果是不同的package則需要在定義一個suite，
當然在同一個package中可以定義不同層級的suites）
To collect tests in another package, define another suite.
However tests in one package can be divided into several suites.
suites[{suite name}]['concurrency'] (since 0.3.0): Optional. Default is {'max_workers': 1, 'level': 'class', 'timeout': None}. See Concurrent Tests.
suites[{suite name}]['concurrency']['max_workers']: Required if 'concurrency' is defined.
The max number of workers allocated to run the tests within a suite.
工作區最大數，指的是一個在suite中分配的tests數量
suites[{suite name}]['concurrency']['level']: Optional. Can be 'module', 'class' or 'method' to run the modules, classes, or methods concurrently. Default is 'class'.
suites[{suite name}]['concurrency']['timeout']: Optional. The maximum number of seconds to wait before getting the suite result.
Can be an int or float. Default is None(no limit to the wait time). The wait only happens when max_workers > 1.
這種超時只發生在max_workers大于1的時候
suites[{suite name}]['groups'][{group name}]['granularity']: Required. Must be one of 'package', 'module', 'class' and 'method'.
If granularity is 'package', then suites[{suite name}]['package'] must be given.
如果粒度是包，則必須給出suites[{suite name}]['package']
suites[{suite name}]['groups'][{group name}]['pattern']: Optional. Only takes effect when granularity is 'package'.
這個參數只有當pattern是package時生效
A python regular expression to match tests long names like 'module.class.method' in the package. Default is '(\w+\.){2}test\w*' if not set.
這是一段正則表達式
suites[{suite name}]['groups'][{group name}]['modules']: Required if granularity is 'module'. A list of module names (test file names with .py trimmed).
只是選擇.py文件
suites[{suite name}]['groups'][{group name}]['classes']: Required if granularity is 'class'. A list of dotted class names conforming to 'module.class'.
只選擇py文件的class
suites[{suite name}]['groups'][{group name}]['methods']: Required if granularity is 'method'. A list of dotted method names conforming to 'module.class.method'.
只選擇py文件的類方法
suites[{suite name}]['groups'][{group name}]['except_modules']: Optional. Only takes effect when granularity is 'package'. A list of excluded module names.
只有粒度是“package”的時候才生效，其作用是排除xx模塊
suites[{suite name}]['groups'][{group name}]['except_classes']: Optional. Only takes effect when granularity is 'package' or 'module'.
A list of excluded class names conforming to 'module.class'.
suites[{suite name}]['groups'][{group name}]['except_methods']: Optional. Only takes effect when granularity is 'package', 'module' or 'class'.
A list of excluded method names conforming to 'module.class.method'.
suites[{suite name}]['groups'][{group name}]['disable']: Optional. Excludes the group of tests if the value is True. Default is False if not set.
當其值是真的，則排除組的測試，默認值為假
To include/exclude a suite, add/remove the suite name in/from the test['suites'] list in the test dict:
test:
suites: [my_suite_1] # will only run my_suite_1
...
Test Reports
This part describes the reporters dict in the test config, with the example in Overview:
這部分描述在測試配置中的字典報告，實例如下
reporters['class']: Required if a reporter is defined. A dotted reporter class name.
reporters['kwargs']: Optional. The arguments for initiating the reporter instance.
生成HTML格式的測試報告
The arguments of the built-in HtmlReporter and their default values are:
dest='results'
overview_title='Reports'
overview_description=''
templates_path=None
report_template=None
overview_template=None
index_template=None
生成 XunitReporter格式的測試報告
The arguments of the built-in XUnitReporter and their default values are:
dest='results'
summary_title='XUnit Reports'
templates_path=None
report_template=None
summary_template=None
Configuring multiple reporters which generate different formats of reports is allowed, and only a single run of the tests is needed to generate all different formats.
這句話的意思應該只允許配置單一形式的測試報告？
To include/exclude a reporter, add/remove the reporter name in/from the test['reporters'] list in the test dict:
在test['reporters']中設置是否包含、排除報告，添加、刪除報告
test:
reporters: [html] # will only generate html format reports
...
If the list is empty, no report files will be generated.
如果這個list是空的，則沒有報告生成
unishark can buffer logging stream during the running of a test case, and writes all buffered output to report files at the end of tests.
To let unishark capture the logging stream and write logs into reports, simply redirect the logging stream to unishark.out, e.g.,
unishark 在測試用例運行的過程中可以緩沖日滯留，并將所有緩沖在測試結束時輸出到報告文件中。
讓unishark捕獲日志流 和寫日志到報告中，將日志流簡單重定向到unishark.out中，例如
formatter = logging.Formatter('%(levelname)s: %(message)s')
handler = logging.StreamHandler(stream=unishark.out)
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
logger = logging.getLogger('example')
logger.addHandler(handler)
logger.setLevel(logging.INFO)
or in YAML format,
formatters:
simple:
format: '%(levelname)s: %(message)s'
handlers:
myhandler:
class: logging.StreamHandler
formatter: simple
stream: ext://unishark.out
loggers:
example:
level: DEBUG
handlers: [myhandler]
propagate: False
NOTE:
unishark does NOT buffer stdout and stderr. So if you use print('some message') in a test case, the message will be output to stdout during the test running.
unishark 不是標準的緩存輸入輸出，如果你在testcase中使用print，則消息將在測試運行是輸出到標準輸出中
Suite names are reflected in the reports while groups are not. Test cases are grouped by class then module in the reports.
groups config is simply for conveniently including/excluding a group of test cases by enabling/disabling the group.
測試套件的名字將記錄在測試報告中，而不是將組記錄報告中，測試用例在測試報告中按照類進行分類，然后在報表中進行模塊化，
組的配置 ，通過啟用或者禁止testcases組來簡單的控制是否包含/不包含測試用例
To output unishark's own logs to stdout:
將uninshark日志輸出到stdout
handlers:
console:
class: logging.StreamHandler
formatter: simple
stream: ext://sys.stdout
loggers:
unishark:
level: INFO
handlers: [console]
propagate: False
Concurrent Tests
并行測試
在unishark中如何并發測試呢？
在多任務中以進程或者線程的方式并發執行
在單任務中只以線程的方式執行，如：
at module level.
at class level.
at method level.
To enable concurrent execution of multiple suites, set 'concurrency' sub-dict (since 0.3.0) in the 'test' dict:
test:
...
concurrency:
type: processes? # or threads
max_workers: 4? # number of threads or processes depending on the type
To enable concurrent execution within a suite, set 'concurrency' sub-dict (since 0.3.0) in the '{suite name}' dict:
suites:
my_suite_name_1:
concurrency:
max_workers: 6? # no choices of concurrency type, just threads
level: method? # or class or module
...
NOTE：
從0.3.2 版本開始，線程和多任務進程都支持
在<0.3.2版本中，只支持線程的并發
在0.2X版本中不支持并發機制，因為在>=0.3.0的版本中采用了一種新的并發執行模型，
setUpModule/tearDownModule setUpClass/tearDownClass無論何種并發及價值將 一次性執行
0.2x版本，“max_workers”直接設置在“測試”，和“max_workers '和' concurrency_level”直接設置在“{suite name}的套房。
0.2x版本，對線程安全的情況下，推薦并發水平：如果一個模塊有setupmodule / teardownmodule，集“concurrency_level '到'模塊'，
否則setupmodule / teardownmodule可以多次運行的模塊；如果有setupclass / teardownclass在一類，設置“concurrency_level”“類”或“模塊”，
否則setupclass / teardownclass可以多次運行的類；如果只有安裝/拆卸，“concurrency_level”可以設置為任何水平。
如果max_workers < = 1，這是連續運行。
用戶在啟用并發執行之前，負責推理線程安全性。例如，當并發程度的方法，競爭條件會發生如果任何方法包括安裝/拆卸試圖修改一個類的作用域中的共享資源。
在這種情況下，用戶應該將并發級別設置為“類”或“模塊”。
Data Driven
Here are some effects of using @unishark.data_driven.
'Json' style data-driven. This style is good for loading the data in json format to drive the test case:
@unishark.data_driven(*[{'userid': 1, 'passwd': 'abc'}, {'userid': 2, 'passwd': 'def'}])
def test_data_driven(self, **param):
print('userid: %d, passwd: %s' % (param['userid'], param['passwd']))
Results:
userid: 1, passwd: abc
userid: 2, passwd: def
'Args' style data-driven:
@unishark.data_driven(userid=[1, 2, 3, 4], passwd=['a', 'b', 'c', 'd'])
def test_data_driven(self, **param):
print('userid: %d, passwd: %s' % (param['userid'], param['passwd']))
Results:
userid: 1, passwd: a
userid: 2, passwd: b
userid: 3, passwd: c
userid: 4, passwd: d
Cross-multiply data-driven:
@unishark.data_driven(left=list(range(10)))
@unishark.data_driven(right=list(range(10)))
def test_data_driven(self, **param):
l = param['left']
r = param['right']
print('%d x %d = %d' % (l, r, l * r))
Results:
0 x 1 = 0
0 x 2 = 0
...
1 x 0 = 0
1 x 1 = 1
1 x 2 = 2
...
...
9 x 8 = 72
9 x 9 = 81
You can get the permutations (with repetition) of the parameters values by doing:
@unishark.data_driven(...)
@unishark.data_driven(...)
@unishark.data_driven(...)
...
Multi-threads data-driven in 'json style':
@unishark.multi_threading_data_driven(2, *[{'userid': 1, 'passwd': 'abc'}, {'userid': 2, 'passwd': 'def'}])
def test_data_driven(self, **param):
print('userid: %d, passwd: %s' % (param['userid'], param['passwd']))
Results: same results as using unishark.data_driven, but up to 2 threads are spawned, each running the test with a set of inputs (userid, passwd).
Multi-threads data-driven in 'args style':
@unishark.multi_threading_data_driven(5, time=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
def test_data_driven(self, **param):
sleep(param['time'])
Multi-threads data-driven in 'args style':
@unishark.multi_threading_data_driven(5, time=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
def test_data_driven(self, **param):
sleep(param['time'])
Results: 5 threads are spawned to run the test with 10 sets of inputs concurrently (only sleep 1 sec in each thread).
It takes about 2 sec in total (10 sec if using unishark.data_driven) to run.
NOTE: It is user's responsibility to ensure thread-safe within the test method which is decorated by unishark.multi_threading_data_driven.
If exceptions are thrown in one or more threads,
the exceptions information will be collected and summarized in the "main" thread and thrown as unishark.exception.MultipleErrors.
Useful API
DefaultTestLoader
load_tests_from_dict(dict_conf): Loads tests from a dictionary config described in The Test Config. Returns a suites dictionary with suite names as keys.
load_tests_from_package(pkg_name, regex=None): Returns a unittest.TestSuite instance containing the tests whose dotted long name 'module.class.method' matches the given regular expression and short method name matches DefaultTestLoader.name_pattern. A dotted package name must be provided. regex is default to '(\w+\.){2}test\w*'.
load_tests_from_modules(mod_names, regex=None): Returns a unittest.TestSuite instance containing the tests whose dotted name 'class.method' matches the given regular expression and short method name matches DefaultTestLoader.name_pattern. A list of dotted module names must be provided. regex is default to '\w+\.test\w*'.
load_tests_from_dict（dict_conf）：負載測試從字典配置測試配置描述。以套件名稱為鍵返回一個套件字典。
load_tests_from_package（pkg_name，正則表達式=無）：返回一個包含測試的虛線長的名字'模塊unittest.testsuite實例。類方法的匹配給定的正則表達式和方法名稱匹配defaulttestloader.name_pattern。必須提供一個虛線包名稱。正則表達式是默認的（\w+”。）{ 2 }測試\ w *”。
load_tests_from_modules（mod_names，正則表達式=無）：返回一個包含的測試點名稱“類unittest.testsuite實例方法的匹配給定的正則表達式和方法名稱匹配defaulttestloader.name_pattern。必須提供一個虛線模塊名稱的列表。正則表達式是默認為“\w+ \測試\ w *”。
Advanced Usage
高級用法
unishark is totally compatible with unittest because it extends unittest. Here are some examples of mixed use of the two:
unishark? 兼容unittest
Run unittest suite with unishark.BufferedTestRunner:
if __name__ == '__main__':
reporter = unishark.HtmlReporter(dest='log')
unittest.main(testRunner=unishark.BufferedTestRunner(reporters=[reporter]))
if __name__ == '__main__':
import sys
suite = unittest.TestLoader().loadTestsFromModule(sys.modules[__name__])
reporter = unishark.HtmlReporter(dest='log')
# Run methods concurrently with 10 workers and generate 'mytest2_result.html'
result = unishark.BufferedTestRunner(reporters=[reporter]).run(suite, name='mytest2', max_workers=10, concurrency_level='method')
sys.exit(0 if result.wasSuccessful() else 1)
if __name__ == '__main__':
import sys
suite = unittest.TestLoader().loadTestsFromModule(sys.modules[__name__])
# Run classes concurrently with 2 workers
result = unishark.BufferedTestRunner().run(suite, name='mytest3', max_workers=2)
# Generating reports can be delayed
reporter = unishark.HtmlReporter(dest='log')
reporter.report(result)
# Also generate overview and index pages
reporter.collect()
Load test suites with unishark.DefaultTestLoader and run them with unittest.TextTestRunner:
if __name__ == '__main__':
dict_conf = None
with open('your_yaml_config_file', 'r') as f:
dict_conf = yaml.load(f.read())? # use a 3rd party yaml parser, e.g., PyYAML
suites = unishark.DefaultTestLoader(name_pattern='^test\w*').load_tests_from_dict(dict_conf)
for suite_name, suite_content in suites.items():
package_name = suite_content['package']
suite = suite_content['suite']
concurrency = suite_content['concurrency']
unittest.TextTestRunner().run(suite)
更多例子
More Examples
For more examples, please see 'example/' in the project directory. To run the examples, please read 'example/read_me.txt' first.
User Extension
Customized Reports
If you prefer a different style of HTML or XUnit reports, passing different template files to the unishark.HtmlReporter or unishark.XUnitReporter constructor is the easiest way:
reporters:
html:
class: unishark.HtmlReporter
kwargs:
dest: logs
overview_title: 'Example Report'
overview_description: 'This is an example report'
templates_path: mytemplates
report_template: myreport.html
overview_template: myoverview.html
index_template: myindex.html
xunit:
class: unishark.XUnitReporter
kwargs:
summary_title: 'Example Report'
templates_path: xmltemplates
report_template: xunit_report.xml
summary_template: xunit_summary.xml
NOTE:
The customized templates must also be Jinja2 templates
Once you decide to use your own templates, you have to specify all of the 'teamplates_path' and '*_template' arguments. If one of them is None or empty, the reporters will still use the default templates carried with unishark.
If the above customization cannot satisfy you, you could write your own reporter class extending unishark.Reporter abstract class. Either passing the reporter instance to unishark.BufferedTestRunner or configuring the initializer in the test config will make unishark run your reporter.
Implement TestProgram
You could also write your own test program class extending unishark.TestProgram abstract class. Implement run() method, making sure it returns an integer exit code, and call unishark.main(your_program) to run it.
C:\Python27\python.exe D:/fund-auto-test/LazyRunner/runsyaml.py
D:\fund-auto-test\LazyRunner\LazyConfig
Traceback (most recent call last):
File "D:/fund-auto-test/LazyRunner/runsyaml.py", line 14, in
program = unishark.DefaultTestProgram(dict_conf)
File "C:\Python27\lib\site-packages\unishark\main.py", line 43, in __init__
self.concurrency = self._parse_suites_concurrency()
File "C:\Python27\lib\site-packages\unishark\main.py", line 55, in _parse_suites_concurrency
raise KeyError('Please set "max_workers" in the "concurrency" sub-dict instead.')
KeyError: 'Please set "max_workers" in the "concurrency" sub-dict instead.'
