TESTAR Java Protocols

TESTAR Java protocols allow users to customize the decision making of the tool in the different steps of the logical flow execution.

Tags

Before starting with the java protocols it is important to understand the Taggable functionality of TESTAR objects.

Widget, State, Action, Settings and SUT objects are created and used by TESTAR to represent the information about the system execution, system elements, type of actions and configuration settings. These 5 objects are Taggable, which means you can assign Tags to them.

Tags are basically pairs of <Property Name, Property value>.
TESTAR uses two generic Tag classes to define general properties (Tags) and settings properties (ConfigTags).
And the properties specific to an implemented API technology or automation framework (Windows UIAutomation, Selenium Webdriver, IV4XR framework), are defined in specific API-Taggable classes (UIATags, WdTags, IV4XRTags).

• All objects may have a generic description:
  Widget.get(Tags.Desc) Action.get(Tags.Desc)
• Settings object uses configuration tags:
  settings.get(ConfigTags.SUTConnector)
• Widgets can use Windows, Webdriver or IV4XR tags depending of the SUT:
  Widget.get(UIATags.UIAItemType) Widget.get(WdTags.WebCssClasses) Widget.get(IV4XRtags.seOrientationForward)

TESTAR execution flow methods

The abstract protocol class of TESTAR defines the main methods of the different interaction steps.
https://github.com/iv4xr-project/TESTAR_iv4xr/blob/master/testar/src/org/fruit/monkey/AbstractProtocol.java

/**
 * This is the abstract flow of TESTAR (generate mode):
 *
 * - Initialize TESTAR settings
 * - InitTestSession (before starting the first sequence)
 * - OUTER LOOP:
 * 		- PreSequencePreparations (before each sequence, for example starting WebDriver, JaCoCo or another process for sequence)
 * 		- StartSUT
 * 		- BeginSequence (starting "script" on the GUI of the SUT, for example login)
 * 		INNER LOOP
 * 			- GetState
 * 			- GetVerdict
 * 			- StopCriteria (moreActions/moreSequences/time?)
 * 			- DeriveActions
 * 			- SelectAction
 * 			- ExecuteAction
 * 		- FinishSequence (closing "script" on the GUI of the SUT, for example logout)
 * 		- StopSUT
 * 		- PostSequenceProcessing (after each sequence)
 * - CloseTestSession (after finishing the last sequence)
 *
 */

• void initialize(Settings settings)
  By default TESTAR loads all setting from the test.setting file (or the GUI if user did some change).
  User can use this method to programatically access to the settings values.
  For example, get a setting value to load a RL policy PolicyFactory(settings.get(ConfigTags.Policy));

• void initTestSession()
  This method is called before the first test sequence, allowing for example setting up the test environment.\

• State getState(SUT system)
  The state allows user to access to all widget entities that TESTAR is able to observe.
  Users can iterate over the state to access to the widget objects.

State state = super.getState(system);
for(Widget widget : state){
  widget.get(IV4XRtags.entityType);
}

• Verdict getVerdict(State state)
  The verdict is used to determine if a state contains an error or not.
  TESTAR checks by default if the system is running properly, but users can use this method to customize more sophisticated oracles. Next example uses the state screenshot to invoke a notional method that determines if a screenshot contains errors or not.

Verdict verdict = super.getVerdict(state);
boolean screenError = notionalMethodToValidateScreenshots(state.get(Tags.ScreenshotPath));
if(screenError) {
  return new Verdict(Verdict.SEVERITY_FAIL, "Detected a not valid SUT screen");
}

• Set<Action> deriveActions(SUT system, State state)
  This method defines which actions TESTAR is able to execute in each state.
  We can check widget properties to determine which type of action TESTAR can execute.

for(Widget w : state) {
  // If TESTAR observes an Interactive Entity (LabRecruits Button)
  if(isInteractiveEntity(w)) {
    // TESTAR adds the possibility to move and interact with the entity
    labActions.add(new labActionCommandMoveInteract(w, labRecruitsEnv, agentId, w.get(IV4XRtags.entityPosition)));
  }
}

• Action selectAction(State state, Set<Action> actions)
  Based on the set of possible actions, TESTAR selects an action to be executed.
  By default we select a random action or we use the State Model to select an unvisited one.
  But the user can create is own action selection mechanism by using state or actions properties.

• boolean executeAction(SUT system, State state, Action action)
  This method executes the selected action.
  Usually we use this method to save information about the executed action.

• boolean moreActions(State state)
  TESTAR uses this method to determine when to stop the generation of actions for the current sequence.
  User can determine when he wants to stop the execution.

• void stopSystem(SUT system)
  TESTAR sequence is finished, the system is going to be stopped and closed.
  User can programmatically customize a graceful shutdown.