Test Design of TestCase Label

The Design of Label Mechanism of Test Case

The Label of test case is used to filter case conveniently.

The Usage

The labels can be added into test case by below way:

start:test_case_name
description: The description of current case 
os:Linux
hcp:openbmc
label:label1,label2,label3,.....
.....
end

Note: The label can not include character +,- and |.

Suppose in future developer/FVT can filter expected case list by below way:

# xcattst -s label1+label2|label3+label4|label5-label6

For reducing complexity, the filter expression just support below kinds of searching.

• | : low priority. Union filter. expression1|expression2 means the case either satisfies expression1 or satisfies expression2. One expression looks like label1+label2-label3.
• + : high priority. label1+label2 means the case need include label1 and label2 at same time.
• - : high priority. label1-label2 means the case need include label1 but not include label2.

Label Mechanism

There are two primary dimensions of labels, vertical and horizontal Dimension.

Vertical Dimension Labels

Vertical dimension labels decides the running order of cases. As long as follow below order, anyone can finish a whole test against xCAT based on a brand new environment.

level	Label name	case example
1	xcat_install	install_xCAT_on_rhels_sles
2	mn_only	lsdef_*,xcatd_restart
3	sn_diskful	SN_setup_case
	sn_diskless	SN_diskless_setup_case
4	flat_cn_diskful	reg_linux_diskfull_installation_flat
	flat_cn_diskless	reg_linux_diskless_installation_flat
	hierarchy_cn_diskful	reg_linux_diskfull_installation_hierarchy
	hierarchy_cn_diskless	reg_linux_diskless_installation_hierarchy
5	cn_bmc_ready	rpower_*
	cn_os_ready	xdcp_*
6	others	nodeset_cmd

There are some rules for vertical dimension labels

• It is mandatory label. That means one cases must have ONE and ONLY ONE label in above table and belong to one specific level.

• Level 1,3,4 can be treated as part of test environment setup. That means if need not to setup environment in your case, you can skip some levels.

  For example:

  • If you have set up a MN with expected xcat installed and want to do smoking test, just need to run all cases with mn_only label.

      # xcattst -s "mn_only"
  

  • If you have a MN with brand new os and be connected to a CN, want to test hardware control against CN, just need to filter cases from level 1 and level 5, skip level 2-4.

      # xcattest -s "xcat_install+os=ubuntu|hctrl_general+hctrl_openbmc"
  

• There are more than one labels in level 3/4/5, these labels in the same level are mutually exclusive. That means when create a kind of whole test against xCAT, only one label in these levels can be chosen.

  For example:

  • If integrate sn_diskful, hierarchy_cn_diskful and cn_os_ready together, that means all cases needed a alive CN are run on hierarchy structure and CN is provisioned in diskful mode.

      # xcattest -s "sn_diskful|hierarchy_cn_diskful|cn_os_ready"
  

  • If integrate flat_cn_diskless and cn_os_ready together,that means all cases needed a alive CN are run on flat structure and CN is provisioned in diskless mode.

      # xcattest -s "flat_cn_diskless|cn_os_ready"
  

Horizontal Dimension Labels

Horizontal dimension labels decides the category of cases. User can filter some test case against one component out by these labels.

db,xcatd,provision,invoke_provison,statelite hctrl_general,hctrl_hmc,hctrl_fsp,hctrl_bmc_ipmi,hctrl_openpower_ipmi, hctrl_openbmc, dhcp,dns,parallel_cmds,postscripts,network,packaging,updatenode,file-sync
go-xcat,console,discovery,migration,go-coserver,scaling,security,virtualization
GPU,PDU,IB,Switch,cumulus,firmware HA,xcatprobe,performance,docker,restapi,KIT,wait_fix,xcatdebugmode,sysclone ,sn_pool, kdump , ci_test, image, hosts, ntp

For example:

• If need to test hardware control for p9 physical machine and the test environment has been set up by yourself, filter all related cases out by hctrl_general and hctrl_openbmc.

      # xcattest -s "hctrl_general|hctrl_openbmc"
  

• If need to test hardware control for p9 physical machine and the test environment is brand new, it need leverage test case to set up environment, so labels xcat_install, hctrl_general and hctrl_openbmc will be involved.

     # xcattest -s "xcat_install|hctrl_general|hctrl_openbmc"