Requirements based Testing with Robotframework

Contoh Struktur Persyaratan dengan RIDE

Dalam artikel ini saya ingin sedikit berbagi pengalaman saya dalam penerapan test automatisasi dengan Robotframework dalam bentuk pengujian bersyarat (Requirements based Testing)

Sebelum itu mari kita lihat dan pahami , apa arti Pengujian berbasis Persyaratan (Requirements based Testing) secara umum.

Apa itu Pengujian Berbasis Persyaratan?
Pengujian berbasis persyaratan adalah pendekatan pengujian di mana kasus pengujian, kondisi, dan data berasal dari persyaratan. Ini termasuk tes fungsional dan juga atribut non-fungsional seperti kinerja, keandalan atau kegunaan.

Tahapan dalam Pengujian berdasarkan Persyaratan:

  • Menentukan Kriteria Penyelesaian Test :
  • Pengujian selesai hanya jika semua pengujian fungsional dan non-fungsional selesai.
  • Desain Kasus Uji: Kasus uji memiliki lima parameter yaitu:
    • keadaan awal atau prasyarat,
    • penyiapan data,
    • masukan,
    • hasil yang diharapkan,
    • hasil aktual.
  • Jalankan Test: Jalankan kasus uji terhadap sistem yang sedang diuji dan dokumentasikan hasilnya.
  • Verifikasi Hasil Test: Verifikasi apakah hasil yang diharapkan dan aktual cocok satu sama lain.
  • Verifikasi Cakupan Test: Periksa apakah pengujian mencakup aspek fungsional dan non-fungsional dari persyaratan.
  • Lacak dan kelola Kesalahan Software: Setiap
    Kesalahan yang terdeteksi selama proses pengujian melewati siklus hidup Kesalahan dan dilacak hingga penyelesaiannya. Statistik
    Kesalahan dipertahankan yang akan memberi kita status keseluruhan proyek.

Persyaratan Proses pengujian:

  • Pengujian harus dilakukan tepat waktu.
  • Proses pengujian harus menambah nilai siklus hidup perangkat lunak (software), oleh karena itu harus efektif.
  • Menguji sistem secara menyeluruh tidak mungkin dilakukan sehingga proses pengujian juga harus efisien.
  • Pengujian harus memberikan status keseluruhan proyek, oleh karena itu harus dapat dikelola. Dan biasanya untuk tujuan ini digunakan Test Management Tools (salah satu contohnya adalah Mantis / TestLink/ HP-ALM-Quality Center / Jira Plugin XRAY)

Contoh Desain Kasus Uji dengan Robotframework:

*** Settings ***
Documentation     *Kriteria Penyelesaian Test:*
...               Ein SGD ePA MUSS Folgendes sicherstellen: Wenn über dessen HTTPS-SGD-Schnittstelle (vgl. ) ein POST- Request mit dem Request-Body nach [gem_SGD_ePA#Tab_KeyDerivation-Request] eintrifft, so MUSS die RVE des SGD
...               - das Zertifikat im Datenfeld *Certificate* gemäß TUC_PKI_018 prüfen und
...               dabei Ergebnisse nach \ \ berücksichtigen.
...
...               - die Kodierung und die Signatur der ""PublicKeyECIES"" prüfen (vgl. \ und ).
...               Falls eine der Prüfungen ein nicht-positives Ergebnis liefert, so MUSS die RVE des SGD mit einer entsprechenden Fehlermeldung aus [gemSpec_SGD_ePA#] dem Client antworten und die weitere Requestverarbeitung abbrechen.
...
...               Die RVE des SGD MUSS die Informationen aufbereiten und an das für den Request avisierte SGD-HSM übergeben (vgl.). Liefert das SGD-HSM ein OK, so MUSS die SGD die Antwort den HTTP-POST-Request mit folgender Nachricht beantworten:
...               {
...               \ ""Status"" : ""OK"",
...               \ ""EncryptedMessage"" : ""... Base64-kodiertes Chiffrat. Das Chiffrat wurde vom SGD-HSM erzeugt ...""
...               }
...               Anderenfalls (SGD-HSM meldet einen Fehler) MUSS die RVE des SGD mit einer entsprechenden Fehlermeldung aus [gemSpec_SGD_ePA#] \ dem Client antworten.


*** Test Cases ***
A_17898-1
    [Documentation]    *Definisi Test:*
    ...    - Am SGD ePA trifft über die HTTPS-SGD-Schnittstelle (vgl. A_17889) ein POST-Request mit dem Request-Body nach [gem_SGD_ePA#Tab_KeyDerivation-Request] ein: die RVE des SGD prüft das Zertifikat im - Datenfeld ""Certificate"" gemäß TUC_PKI_018 \ und berücksichtigt dabei Ergebnisse nach A_17896.
    ...    - Die RVE des SGD prüft die Kodierung und die Signatur der ""PublicKeyECIES"" (vgl. A_17900 und A_17901).
    ...
    ...    *Hasil Test:*
    ...    - Beide Prüfungen liefern ein positives Ergebnis.
    ...    - Die RVE des SGD bereitet die Informationen auf.
    ...    - Die RVE des SGD übergibt die aufbereiteten Informationen an das für den Request avisierte SGD-HSM.
    ...    - Das SGD-HSM liefert ein OK.
    ...    - Die SGD beantwortet den HTTP-POST-Request mit folgender Nachricht:{\ ""Status"" : ""OK"",\ ""EncryptedMessage"" : ""... Base64-kodiertes Chiffrat. Das Chiffrat wurde vomSGD-HSM erzeugt ...""}"
    [Tags]    A_17898    gemSpec_SGD_ePA    owner-at    prio-1
    [Setup]    Enable Strict Validation
    sgd-clientjavaWF.Vorbedingungen für Überprüfung
    sgd-clientjavaWF.Schritt1
    sgd-clientjavaWF.Schritt2
    sgd-clientjavaWF.Schritt3
    sgd-clientjavaWF.Schritt4
    sgd-clientjavaWF.Schritt6

Kriteria Penyelesaian Test dapat anda implementasikan dalam fungsi [Documentation] pada Robotframework Test.

Verifikasi Cakupan Test dapat anda implementasikan dalam fungsi [Tags] pada Robotframework Test.

Verifikasi Hasil Cakupan Test akan dihasilkan oleh Robotframework Test Report dan Test Log, yang didalamnya menggambarkan jalur dan urutan Test anda.

Anda tidak membutuhkan Aplikasi atau Software extra untuk mengimplimentasikan Kriteria yang saya sebutkan di atas.

Hasil dan dokumentasi dari Test anda akan anda dapatkan selalu dari hasil akhir automatisasi Test anda dengan Robotframework.

Migrating CodedUI Test to Robotframework for Testing Sharepoint 2019 Websolution

just one sample migrating ATDD Test from Coded UI to Robotframework.

Before Test Structure:

CodedUI Unit Test Structure

Comparison Unit Test & ATDD Test Structure:

Structure Comparison C# and Python with Selenium API

Reason for Migration from CodedUI to Robotframework:

  1. CodedUI is now deprecated.
  2. Test Tools and Ways of testing didn’t fullfill the requirement for doing ATDD test (E2E test) scopes.
  3. Long test run .
  4. Needs of proper test logging
  5. Test run should be independance from SW build.
  6. Only IE as browser is supported.

Proposed Robotframework ATDD Structure:

Robotframework ATDD Structure

Project / Test & Development enviroment Limitation:

  1. Test Object: Sharepoint 2019 customisation
  2. OS used for test object and testing: Microsoft Windows 10 OS
  3. Build and test requrement tools: Microsoft TFS
  4. Diffrent targets and build environment for: Development, Integration and Preproduction.

Needed Infrastukture and Software for Testing with Robotframework:

  1. Python 3.x
  2. Robotframework
  3. Robotframework SeleniumLibrary
  4. Robotframework Whitelibrary
  5. Test editor Notepad ++ with Robotframework plugin
  6. Test should be able to run as Selenium Hub trigger from MS TFS which consuming Robotframework xUnit reports as test run results.

Benefit of using Robotframework:

  1. Test run faster than CodedUI
  2. Test cases are written as Keyword Driven test, instead of Unit Test, these speed up the test cases creation and migration also simplify reusable of test cases.
  3. No need using Visual Studio for creating test sases, you can use simple text editor r write tests.
  4. Test results are more understandable for non-technical team members ( Test Manager, Project Manager, Environment Team, Solution Team)
  5. Using RF Tag functionally and RF Report (Rebot) custom report can be created without any additional effort,
  6. Report implementations of build on TFS over Xunit report flag on Robot runner.
  7. Tests can run independently outside build tool.

Starting the MIgration Steps:

  1. Separating & creating all needed sharepoint UI keywords in UI Keyword resources.
  2. Separating & creating all needed sharepoint 2019 application workflow as Keyword resources.
  3. Creating template of sharepoint masterpage sites mapping as keyword resources.
  4. Defining needed static and dynamic variables regarding to TCs and target test enviroment as variable resources
  5. Creating RF runner files as batchfile for each target deployment enviroment (development, integration, or regresion)
  6. Creating test suites regarding requirement (sanity test, fuctional test or regresion test)
  7. Creating folder structure to map sharepoint functional requirement implementation
  8. Creating folder structure for test reports, test suites and test runner.
  9. Tagging related test suites regarding requirement coverage

Reports & Logs:

Installing Ride on Pi Zero

Installing Ride v1.7.4 on Raspberry Pi Zero with Raspbian is a Challenge, especially during preperation and installation of wxPython for pi Zero

Base setup from pi Zero:

  • Rasbian buster lite
  • python 3.7x

Do SW update before installing :

sudo apt-get update
sudo apt-get upgrade
sudo apt get dist-upgrade

Do Install pip3 for my pi zero:

sudo apt-get install python3-pip

Prepare installation before installing wxPython on pi Zero:

sudo apt-get install dpkg-dev build-essential swig python-dev libwebkit-dev libjpeg-dev libtiff-dev checkinstall freeglut3 freeglut3-dev libgtk2.0-dev  libsdl1.2-dev libgstreamer-plugins-base0.10-dev libgstreamer0.10-dev libgconf2-dev libglapi-mesa libosmesa6 libglu1-mesa-dev libglu1-mesa mesa-common-dev
sudo apt-get install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev
sudo apt-get install python3.7-dev
sudo apt-get install python-wxgtk2.8 python-wxtools wx2.8-i18n libwxgtk2.8-dev libgtk2.0-dev

sudo apt-get install gtk+-3.0 gtk+3.0-dev
sudo pip3 install pathlib2

Now install wxPython, it can take long (+/- 24 Hours) by pi Zero!

sudo pip3 install wxPython

If it’t passed you can install Ride !

sudo pip3 install --upgrade robotframework-ride

You can start the Ride now as remote sessions:

Ride.py &

Raspberry Pi & Robotframework

Raspberry Zero W connected directly to Mac with OTG mode

Why using Robotframework on Raspberry pi?

As Raspberry pi user , sometimes I have problems after doing OS update with some application I currently workout on pi, mean do “sudo apt-get update” , “sudo apt-get upgrade” & do “Reboot” everythings goes Wrong 🙁 !.

I have been using Robotframework since 2006, and would like to show you how simple to run a Regresion Test with Robotframework Framework on SBC like Rasberry Pi and Co’s. Robotframe work need only python ecosystem, which are out of the box delivered by Raspbian OS. And also it needs minimal resource of space in SD card.

As a sample I will start with installation of Robotframework , please check before if you have python pip installer installed on your Rasbian before starting with the installation!

Start Installation of Robotframework on Pi

sudo apt-get install python3-pip
sudo pip3 install robotframework
Installation run OK

Check Robotframework runner:

robot --version

Robotframework runner OK

You can used nano or Test IDE Ride for writing Tests .

Installing Robotframework-RIDE test IDE:

sudo pip3 install robotframework-ride

Start RIDE on Mac with ssh conection to pi:

ride.py & 

Running Robotframework-RIDE for testing Temperature Sensore Interface on Pi

now you can start writing your Test, please refer to Robotframework manual for more Information.

Let check the SD card used for Robotframework Installation, just 1% from 16 GB SD card!

Used pi space after Robotframework Installation

Wish you happy Testing and Try Out!

Testing Soap interface with Robotframework Library

For efficient testing sometime you need to created automaticaly request xml feeding soap interface. If you are familiar with robot framework, it’s great. Robotframework has already 2 external Library, which handle SOAP interfaces communication.( Süds Library or Zeep Library ) .

There are a lot of opinion in Internet about these two libraries, discussing pro and contra using one of these lib. for testing SOAP.

Important things to do, you have to find out which library can handle “your WSDL“ better!

For my task, Suds library does.

How to find it out?

Just run your SOAP Library keyword pointing out your target WSDL .

Used “Create Soap Client” keyword for testing the binding if you get all the entities and objects on your WSDL, If the answed “OK” than you can used your Library for SOAP,


Ports (1):
….
Methods (10):
addToIndex(xs:string guid, )
getProductMetadaten(xs:string guid, )
getProductMetadatenGuidList(xs:string query, )
getProductMetadatenList()

Types (62):
ns3:AbstractCurveType
ns3:AbstractGMLType
ns3:AbstractGeometricAggregateType
ns3:AbstractGeometricPrimitiveType

If not you have to find out other library for interfacing with SOAP.

As sample I got these error using Zeep for parsing my WSDL:

self.item_type = self.item_type.resolve()
AttributeError: ‘lxml.etree.QName’ object has no attribute ‘resolve’

These error are related to these following open issue on Zeep:

Sample of created XML request from RF with Suds Library:


XML Request Created with RF & SUD from input WSDL

this is how I wrote it in Robotframework Test Suite:

map WSDL local
Run Keyword and Continue On Failure setProductMetadaten

this is how I wrote it in Robotframework Test Case:

show object setProductMetadaten
setProductMetadaten

calling internal keyword:

${object}= Create WSDL Object n2:ProductMetadata
Set Wsdl Object Attribut ${object.commonMetadata} identifier ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata} title ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata.description} summary ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata.description} comments ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata.product} type ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata.product} subtype ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata} rflAuftragsnummer ${dummy_text}
Set Wsdl Object Attribut ${object.commonMetadata.change} version ${dummy_text}
${result}= call Soap method setProductMetadaten ${object}

RFlog result

RFlog result

Creating 100X Request with RF For loop keyword

map WSDL local
:FOR ${i} IN RANGE 100
setProductMetadaten

Hope these infos help you starting your SOAP request creation with Robotframework & SudsLibrary, just another way of SOAP testing than using SOAPUI.

Testing MS Software Rollout with Python ,WMI, and LDAP3

Most MS (Microsoft) software roll-out are release via SCCM to all Project Infrastructure (DC, AD, Application Servers, Application Clients). Usually testers are using Standards MS Tools like Powershell, MS CMD (tools: like netsh, wmic, etc) for validating Installation results on targets.

This article describe how the same tasks solve with python scripting using WMI & LDAP3 library.

Common Test Task & Rollout Workflow:

  1. Rollout related Packages will be distributed to MS SCCM
  2. Over SCCM Application GUI Testers trigger the release processes for target environment (Server/Client PC)
  3. On target environments testers mock up the sccm client update sequence to push roll-up update for related Packages.
  4. After SCCM agent invoke successfully SW update, the validation steps starting, Testers use common  MS OS GUI like “Software Information GUI” or “Service Information GUI ” for validating Installation status. Some Testers also used Powershell commands for Validating installations.

 Just daily tasks for MS Administrator , as test analyst , I try to find out more efficient way.

After  some investigation  I find out, the way validating MS SW status over WMI Interfaces are much easier than any Powershell or Dos Batch command, especially  with pywmi.  

These following WMI class Objects  are interesting for SW Installations
Validation :

  • Win32_OperatingSystem
  • Win32_Product
  • Win32_BaseService
  • Win32_Service
  • Win32_NTDomain
  • win32_optionalfeature
  • win32_ComputerSystem
  • Win32_PingStatus

Using pywmi, ldap3, winkerberos and common Excel library on python, I created scripts which extract the installed SW, Services Status, Drive Mapping , Domain configuration and AD Application User Accounts. All as excel files containing  Tabs related to the mentions check Topics.

Status, Drive Mapping , Domain configuration and AD Application User Accounts. All as excel files containing  Tabs related to the mentions check Topics

I Run it before SW update and after Update, find out the delta Information’s I got in my excel sheet and cross check if all Items match to release roll-out specifications.  

Added Value:

  • The script can be pointed for remote Target using WMI remote functionality.
  • Limited scripts to do only wmi-get  / LDAP show action (under executable) , to guaranties that no system or installation change cause by test action.
  • Validating more easier than with MS Scripting( I created “.exe” files for each test with py2exe)
  • I don’t mess any test object, because no installation for python needed (all checks are run as “.exe” files and can easily remove after test)

Ref: