Run tests from CI with the API

This guide shows how to start a test run from a CI pipeline, wait for it to
finish, decide pass or fail, and collect logs and reports. It uses the /v2
scheduling and execution API of Katalon True Platform.

1. Set up authentication

Send requests to your tenant host, https://<tenant>.katalon.io, with no
trailing slash.

Authenticate with a Katalon API key, which you create in your Katalon user
settings. Use HTTP Basic with an empty username and the key as the
password: Authorization: Basic <base64(:API_KEY)>. See the
authentication reference.
A missing or wrong key returns 401.

Send the project ID in the x-project-id header on every request. Without it,
requests return 400. You can also send x-account-id (account UUID) and
x-organization-id to select the account and organization explicitly.

The key's user needs the right project permissions: TestRun:Read to read
runs, results, logs and reports, TestRun:Create to start runs, and
TestRun:Update to cancel them. A 403 means the key lacks a permission, or
an ID of the wrong type was sent (for example, a scheduler ID where a schedule
ID is expected).

Keep the API key in your CI secret store. The Base64 value is also a secret:
encoding is not encryption.

A helper for the examples

The examples use this Bash helper. Set KATALON_API_HOST, KATALON_API_KEY
and KATALON_PROJECT_ID from your CI secrets and variables.

The helper keeps the API key out of curl's command-line arguments, which other
processes on a shared runner can read. printf is a shell builtin, so it does
not start a process; it writes the credential to curl's standard input, and
--config - reads it from there. If your key can contain " or \, escape
them as \" and \\. Do not turn on shell tracing (set -x) in the step.

katalon_request() {
  : "${KATALON_API_KEY:?Missing API key}"
  printf 'user = ":%s"\n' "${KATALON_API_KEY}" |
  curl --config - --silent --show-error --fail-with-body \
    --header "x-project-id: ${KATALON_PROJECT_ID:?Missing project ID}" \
    "$@"
}

2. Know which ID is which

order is the run number shown in the app. It is not an ID.

3. Start a test run

Both ways below return the new test run. Keep its id as EXECUTION_ID. If
the response has no id, fail the CI step: do not guess the run from the run
list, because other runs can start at the same time.

Starting a run uses execution capacity. If a start request times out, check the
test-run list before you retry: the run may already exist.

Option A: run a saved schedule

Read the schedule to get a scheduler ID and its next occurrence time:

katalon_request "${KATALON_API_HOST}/v2/schedules/${SCHEDULE_ID}"

Then run that occurrence now. {id} in the path is the scheduler ID, and
startTime is that scheduler's nextTime. The run starts immediately;
startTime only selects the occurrence.

katalon_request --request POST \
  "${KATALON_API_HOST}/v2/schedules/test-runs/${SCHEDULER_ID}/run?startTime=${NEXT_TIME}"

NEXT_TIME is an ISO 8601 UTC time, such as 2030-05-01T10:00:00Z.

POST /v2/schedules/{id}/run also runs a schedule, like Run now in the
app, but it returns 204 with no test-run ID. Do not use it when you need to
follow the run.

Option B: start a test run without a schedule

POST /v2/schedules/test-runs creates a run from the configuration in the
request body and starts it now. The easiest way to build
runConfigurationEntries is to copy them from a schedule that already runs the
tests you want.

katalon_request --request POST \
  --header "Content-Type: application/json" \
  --data @run.json \
  "${KATALON_API_HOST}/v2/schedules/test-runs"

Example run.json for an automated test suite:

{
  "name": "Smoke tests for build 1.4.2",
  "configType": "AUTOMATION_UI",
  "ksVersion": "latest",
  "timeOut": 60,
  "runConfigurationEntries": [
    {
      "entityType": "TEST_SUITE",
      "entityId": 501,
      "testProjectId": 602,
      "profileId": 703,
      "configurations": [
        {"os": "linux", "browser": "chrome", "browserVersion": "latest", "headless": true}
      ]
    }
  ]
}

Do not include a schedule object. With it, the request saves a schedule
instead of starting a run, and the returned id is not a test-run ID.

4. Wait for the run to finish

Poll the run with the id you got, without startTime. Use a fixed interval
(for example, 30 seconds) and a deadline that suits your pipeline.

katalon_request "${KATALON_API_HOST}/v2/schedules/test-runs/${EXECUTION_ID}"

• stage is where the run is: NOT_STARTED and RUNNING mean keep waiting.
  COMPLETED means finished. CANCELLED, TERMINATE and ERROR mean it
  stopped.
• status is the outcome. It stays INCOMPLETE while the run is going, and
  is PASSED only when the run passed.

Cancel the run at your deadline

If the deadline passes, cancel the run so it stops using execution capacity,
then fail the step:

katalon_request --request POST \
  "${KATALON_API_HOST}/v2/schedules/test-runs/${EXECUTION_ID}/cancel"

The run then shows stage: CANCELLED. Do not add startTime to this request:
with it, the path ID is read as a scheduler ID.

5. Decide pass or fail

Pass the CI step only when all of these are true:

1. stage is COMPLETED and status is PASSED.
2. numberOfTestCases is greater than 0. A missing value is not zero.
3. The run's result list has at least one result, and every result has
   status PASSED. Read every page, not only page 0.

katalon_request "${KATALON_API_HOST}/v2/schedules/test-runs/${EXECUTION_ID}/test-results?page=0&size=100"

Fail the step on any other stage or status, on an API error, and on timeout.

To see one result's details, pass its numeric executionTestResultId (not the
string id) to
GET /v2/schedules/test-runs/{executionId}/test-results/{testResultId}.
Output and logs are left out unless you request them with includeFields;
they can contain secrets and personal data.

6. Get logs and reports

An automated run has a job for each environment it runs on. List the jobs,
then read a job's log:

katalon_request "${KATALON_API_HOST}/v2/schedules/jobs?executionId=${EXECUTION_ID}"
katalon_request "${KATALON_API_HOST}/v2/schedules/jobs/${JOB_ID}/get-log"

The log is in content, or at the link in logFileUrl. Store logs as
protected CI artifacts instead of printing them.

Report files (PDF, CSV, XLSX or ZIP) are built on request. Request the
file, wait until it is ready, then download it:

REPORT="${KATALON_API_HOST}/v2/schedules/test-runs/${EXECUTION_ID}/export"
katalon_request --request POST "${REPORT}/enqueue-file?fileType=PDF"
katalon_request "${REPORT}/check-exist-file?fileType=PDF"   # repeat until true
katalon_request --output katalon-report.pdf "${REPORT}?fileType=PDF"

Next steps

• Moving from the v1 API? See Migrate from the v1 API.
• See the reference pages in this section for every field and response, starting with
  Run a scheduled occurrence now.