Test Orchestration with EOE (ElasTest Orchestrator Engine)

ElasTest Orchestrator Engine (EOE)

The ElasTest Orchestration Engine (EOE) is the component responsible of implementing the concept of orchestration within Elastest, understood as a novel way to select, order, and execute a group of TJobs.

To that aim, we have leveraged the pipelines Jenkins notation for the topology generation by means of a [Jenkins shared library]. This library exposes a simple API to orchestrate testing jobs. The orchestrator library API is summarized in the following table:

Method Description
runJob(String jobId) Method to run a job given its identifier (jobId).This method returns a boolean value: true if the execution of the job finishes correctly and false if fails
runJobDependingOn(boolean verdict, String job1Id, String job2Id) Method allows to run one job given a boolean value (typically a verdict from another job). This boolean value is passes in the first argument (called verdict in the method signature). If this value job with identifier job1Id is executed. Otherwise it is executed job2Id
runJobsInParallel(String... jobs) This method allows to run a set of jobs in parallel. The jobs identifier are passes using a variable number of arguments (varargs)

Moreover, the exit condition for the orchestrated job can be also configured:

Method Description
EXIT_AT_END The orchestration finishes at the end (option by default)
EXIT_ON_FAIL The orchestration finishes when any of the TJobs fail
EXIT_ON_PARALLEL_FAILURE The orchestration finishes when any a set of parallel TJobs fail

Finally, the verdict of a group of parallel jobs can be also configured:

Method Description
AND The verdict of a set of jobs executed in parallel is true only if all the jobs finish correctly
OR The verdict of parallel jobs is true when at least one of the jobs finishes correctly

Other configurations:

Method Description Use Example
setPacketLoss If a SuT instrumented By ElasTest through EIM is used, one or more packet loss values can be set. If more than one is specified, the job will be run as many times as there are values. setPacketLoss(['0.01', '0.02'])
setCpuBurst If a SuT instrumented By ElasTest through EIM is used, one or more cpu burst (stress) values can be set. If more than one is specified, the job will be run as many times as there are values. setCpuBurst(['0.2'])
checkTime Allows to check if an execution lasts less (LessThan) or more (GreaterThan) than the indicated time (In milliseconds, seconds, minutes, etc.). checkTime(Compare.LessThan, 60, TimeUnit.SECONDS)

At the moment, only packet loss and cpu burst can be set separately, so the two cannot be set at the same time.

How to install

  • Navigate to Manage Jenkins in your Jenkins.
  • Click to System Configuration
  • Add new library in Global Pipeline Libraries with name OrchestrationLib and default version master
  • Select Modern SCM
  • Select GitHub
  • Owner: elastest
  • Repository: elastest-orchestration-engine
  • Save changes

QoE testing using EOE

The first thing you must do is create an instance of OpenVidu in AWS, which will be the SuT we will use.

To do this, you must use this file for CloudFormation and wait for the instance to start completely.

The next thing you do is create a new project in ElasTest with the name you want. Let's assume it's called Jenkins Examples. Within this project you must create a new Sut with the following configuration:

  • SuT Name: Whatever you want, for example OpenVidu AWS
  • Select "Deployed outside Elastest" and "Instrumented by ElasTest"
  • SuT IP: the ip of the AWS instance
  • Protocol: https
  • Port: 4443
  • User: ubuntu
  • Password: 1234
  • Private Key: the private key to connect to the instance. (-----BEGIN RSA PRIVATE KEY-----...)
  • Instrumentalize: cheked
  • SuT Logs Paths:
    • /var/log/openvidu.log
    • /var/log/kurento-media-server/*.log
  • Save

Once this is done, you must go to the Jenkins section to start and access Jenkins. In Jenkins, the first thing you should do if you haven't done it before is to configure the EOE library as indicated in this documentation.

You must now create a new Pipeline Job with the following settings:

  • Name: Whatever you want, for example quality-of-experience
  • Parameters:
    • Name: AWS_ACCESS_KEY_ID | Value: your AWS access key id
    • Name: AWS_SECRET_ACCESS_KEY | Value: your AWS secret access key
    • Name: AWS_KEY_NAME | Value: your AWS key name
    • Name: AWS_SSH_PRIVATE_KEY | Value: your AWS private key. It's important that you replace line breaks with \r\n literal
    • Name: AWS_SECURITY_GROUPS | Value: your AWS security group with the following format -> ["securitygroupname"]
    • Name: ET_MON_LSHTTPS_API | Value: http://etm:5003/api/monitoring/
  • Pipeline script:
    • In the following script you need to replace the sut value (SUTIDHERE) with the sut id you created earlier.
    • You'll also have to replace the project name if you put in another name other than Jenkins Examples
          elastest(tss: ['EUS'], surefireReportsPattern: '**/target/surefire-reports/TEST-*.xml', project: 'Jenkins Examples', sut: SUTIDHERE) {
              stage ('Executing Test') {
                  sh 'env'
                  echo 'Set up test environment'
                  mvnHome = tool 'M3.3.9'
                  echo 'Cloning repository'
                  git 'https://github.com/elastest/codeurjc-qe-openvidu'
                  echo 'Run test'
                  sh "'${mvnHome}/bin/mvn' -B -DforkCount=0 test -Dtest=OpenviduWebRTCQoEMeterAWS;"
                  sh "sleep 20"
                  step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])

Now create another pipeline job which will define the orchestration:

  • Name: Whatever you want, for example eoe
  • Pipeline script
    • If you put a different quality-of-experience name to the previous pipeline, replace it in the last line
      @Library('OrchestrationLib') _
      // Graph
      def result = orchestrator.runJob('quality-of-experience')

Run eoe and enjoy!