Run tests from CI with the API

This guide shows how to start a test run from a CI pipeline, wait for it to
finish, decide pass or fail, and collect logs and reports. It uses the /v2
scheduling and execution API of Katalon True Platform.

1. Set up authentication

Send requests to your tenant host, https://<tenant>.katalon.io, with no
trailing slash.

Authenticate with a Katalon API key. Use HTTP Basic with an empty username
and the key as the password: Authorization: Basic <base64(:API_KEY)>. See
Authenticate with the v2 API
to create a key and find your project ID. A missing or wrong key returns 401.

Send the project ID in the x-project-id header on every request. Without it,
requests return 400. You can also send x-account-id (account UUID) and
x-organization-id to select the account and organization explicitly.

The key's user needs the right project permissions: TestRun:Read to read
runs, results, logs and reports, TestRun:Create to start runs, and
TestRun:Update to cancel them. A 403 means the key lacks a permission, or
an ID of the wrong type was sent (for example, a scheduler ID where a schedule
ID is expected).

Keep the API key and its Base64 value in your CI secret store.

A helper for the examples

The examples use this Bash helper. Set KATALON_API_HOST, KATALON_API_KEY
and KATALON_PROJECT_ID from your CI secrets and variables.

The helper keeps the API key out of curl's command-line arguments, which other
processes on a shared runner can read. printf is a shell builtin, so it does
not start a process; it writes the credential to curl's standard input, and
--config - reads it from there. If your key can contain " or \, escape
them as \" and \\. Do not turn on shell tracing (set -x) in the step.

katalon_request() {
  : "${KATALON_API_KEY:?Missing API key}"
  printf 'user = ":%s"\n' "${KATALON_API_KEY}" |
  curl --config - --silent --show-error --fail-with-body \
    --header "x-project-id: ${KATALON_PROJECT_ID:?Missing project ID}" \
    "$@"
}

2. Know which ID is which

order is the run number shown in the app. It is not an ID. For all ID types,
see Know which ID is which.

3. Start a test run

Both ways below return the new test run. Save its id as TEST_RUN_ID. If
the response has no id, fail the CI step: do not guess the run from the run
list, because other runs can start at the same time. jq -e in the examples
below fails the step when id is missing.

Starting a run uses execution capacity. If a start request times out, check the
test-run list before you retry: the run may already exist.

Option A: run a saved schedule

Read the schedule to get a scheduler ID and its next occurrence time:

katalon_request "${KATALON_API_HOST}/v2/schedules/${SCHEDULE_ID}" |
  jq -r '.schedulers[0] | .id, .nextTime'

Save the scheduler's id as SCHEDULER_ID and its nextTime as NEXT_TIME.
NEXT_TIME is an ISO 8601 UTC time, such as 2030-05-01T10:00:00Z. A time in
the past is rejected. If the scheduler has no nextTime, use Option B.

Then run that occurrence now. {id} in the path is the scheduler ID, and
startTime is that scheduler's nextTime. The run starts immediately;
startTime only selects the occurrence.

TEST_RUN_ID=$(katalon_request --request POST --get \
  --data-urlencode "startTime=${NEXT_TIME}" \
  "${KATALON_API_HOST}/v2/schedules/test-runs/${SCHEDULER_ID}/run" | jq -er '.id')

POST /v2/schedules/{id}/run also runs a schedule, like Run now in the
app, but it returns 204 with no test-run ID. Do not use it when you need to
follow the run.

Option B: start a test run without a schedule

POST /v2/schedules/test-runs creates a run from the configuration in the
request body and starts it now. The easiest way to build
runConfigurationEntries is to copy them from a schedule that already runs the
tests you want.

TEST_RUN_ID=$(katalon_request --request POST \
  --header "Content-Type: application/json" \
  --data @run.json \
  "${KATALON_API_HOST}/v2/schedules/test-runs" | jq -er '.id')

Example run.json for an automated test suite:

{
  "name": "Smoke tests for build 1.4.2",
  "configType": "AUTOMATION_UI",
  "ksVersion": "latest",
  "timeOut": 60,
  "runConfigurationEntries": [
    {
      "entityType": "TEST_SUITE",
      "entityId": 501,
      "testProjectId": 602,
      "profileId": 703,
      "configurations": [
        {"os": "linux", "browser": "chrome", "browserVersion": "latest", "headless": true}
      ]
    }
  ]
}

entityId is the test suite or test suite collection ID, testProjectId is
the repository ID, and profileId is the execution profile ID. Do not include a
schedule object. With it, the request saves a schedule instead of starting a
run, and the returned id is not a test-run ID.

4. Wait for the run to finish

Poll the run with the id you got, without startTime. Use a fixed interval
(for example, 30 seconds) and a deadline that suits your pipeline.

katalon_request "${KATALON_API_HOST}/v2/schedules/test-runs/${TEST_RUN_ID}"

• stage is where the run is: NOT_STARTED, RUNNING and IMPORTING mean
  keep waiting. COMPLETED means finished. CANCELLED, TERMINATE and
  ERROR mean it stopped.
• status is the outcome. It stays INCOMPLETE while the run is going, and
  is PASSED only when the run passed.

Cancel the run at your deadline

If the deadline passes, cancel the run so it stops using execution capacity,
then fail the step:

katalon_request --request POST \
  "${KATALON_API_HOST}/v2/schedules/test-runs/${TEST_RUN_ID}/cancel"

The run then shows stage: CANCELLED. Do not add startTime to this request:
with it, the path ID is read as a scheduler ID.

5. Decide pass or fail

Pass the CI step only when all of these are true:

1. stage is COMPLETED and status is PASSED.
2. numberOfTestCases is greater than 0. A missing value is not zero.
3. The run's result list has at least one result, and every result has
   status PASSED. Read every page, not only page 0.

All three checks are required. A run can show PASSED while its result list is
empty. This script reads every result page and fails when no results are
returned or any result did not pass:

TOTAL=0
NOT_PASSED=0
PAGE=0
while :; do
  RESULTS=$(katalon_request \
    "${KATALON_API_HOST}/v2/schedules/test-runs/${TEST_RUN_ID}/test-results?page=${PAGE}&size=100")
  TOTAL=$((TOTAL + $(jq '.content | length' <<<"${RESULTS}")))
  NOT_PASSED=$((NOT_PASSED + $(jq '[.content[] | select(.status != "PASSED")] | length' <<<"${RESULTS}")))
  PAGE=$((PAGE + 1))
  [ "${PAGE}" -lt "$(jq '.totalPages // 0' <<<"${RESULTS}")" ] || break
done
[ "${TOTAL}" -gt 0 ] && [ "${NOT_PASSED}" -eq 0 ]

Fail the step on any other stage or status, on an API error, and on timeout.

To see one result's details, pass its numeric executionTestResultId (not the
string id) to
GET /v2/schedules/test-runs/{executionId}/test-results/{testResultId}.
Output and logs are left out unless you request them with includeFields;
they can contain secrets and personal data.

6. Get logs and reports

An automated run has a job for each environment it runs on. List the jobs,
then read a job's log:

katalon_request "${KATALON_API_HOST}/v2/schedules/jobs?executionId=${TEST_RUN_ID}"
katalon_request "${KATALON_API_HOST}/v2/schedules/jobs/${JOB_ID}/get-log"

The log is in content, or at the link in logFileUrl. Fetch logFileUrl
with plain curl, not katalon_request: do not send your API key to that link.
Store logs as protected CI artifacts instead of printing them.

Report files (PDF, CSV, XLSX or ZIP) are built on request. Request the
file, wait until it is ready, then download it:

REPORT="${KATALON_API_HOST}/v2/schedules/test-runs/${TEST_RUN_ID}/export"
katalon_request --request POST "${REPORT}/enqueue-file?fileType=PDF"
katalon_request "${REPORT}/check-exist-file?fileType=PDF"   # repeat until true
katalon_request --output katalon-report.pdf "${REPORT}?fileType=PDF"

Next steps

Find configuration values

Requires the TestRun:Read permission.

1. List the OS and browser options for the project.

katalon_request "${KATALON_API_HOST}/v2/schedules/configurations/os"
katalon_request "${KATALON_API_HOST}/v2/schedules/configurations/browsers"
katalon_request "${KATALON_API_HOST}/v2/schedules/execution-profiles/search"

2. Use the profile catalog's id as profileId in the run request.

GET /v2/schedules/execution-profiles returns profile names in profile, not profile IDs.
Use this separate list to select a name filter.

Save and maintain a schedule

Requires the TestRun:Create permission.

1. Add schedule to run.json before the create request in step 3.

Use the recurrence fields in
Create and start a test run.
For a weekly recurrence, supply days as well as the interval and time zone.
The returned id is a scheduler ID, not TEST_RUN_ID.
Read runConfigurationId from the response as SCHEDULE_ID.

2. Read the saved schedule before you change its configuration.

katalon_request "${KATALON_API_HOST}/v2/schedules/${SCHEDULE_ID}"

Requires the TestRun:Update permission.

3. Save the replacement configuration as schedule-update.json.

This example replaces the test selection and its configuration. Substitute IDs from your project.

{
  "name": "Weekly smoke tests",
  "configType": "AUTOMATION_UI",
  "runConfigurationEntries": [
    {
      "entityType": "TEST_SUITE",
      "entityId": 501,
      "testProjectId": 602,
      "profileId": 703,
      "configurations": [{"os": "linux", "browser": "chrome"}]
    }
  ],
  "visualTestingConfiguration": {"enabled": false, "baselineCollectionGroupOrder": null},
  "activeStatus": true
}

4. Send the replacement configuration.

katalon_request --request PUT --header "Content-Type: application/json" \
  --data @schedule-update.json "${KATALON_API_HOST}/v2/schedules/${SCHEDULE_ID}"

This request is not a merge patch. Keep each configuration field that you still need.
Omit schedule to retain the recurrence. Use activeStatus to change its active state without a recurrence body.
Omit almAutoPushConfiguration to keep it; send null to clear it, or an object to replace it.
See Update a schedule for all fields.

5. Activate or deactivate saved schedules with their schedule IDs.

katalon_request --request POST "${KATALON_API_HOST}/v2/schedules/activate?ids=${SCHEDULE_ID}"
katalon_request --request POST "${KATALON_API_HOST}/v2/schedules/deactivate?ids=${SCHEDULE_ID}"

Send a nonempty comma-separated list with at most 100 schedule IDs.
Activation does not start a run. Deactivation does not cancel a test run that already exists.
These requests and the schedule update return no body.

Requires the TestRun:Delete permission.

6. Delete a saved schedule when you no longer need it.

katalon_request --request DELETE "${KATALON_API_HOST}/v2/schedules?ids=${SCHEDULE_ID}"

The same list limit applies. The request returns no body.
Deletion marks the schedule as deleted, disables its schedulers, and clears their next times.
It preserves historical test runs.

Find and update a test run

Requires the TestRun:Read permission.

1. Search by name with the explicit-filter route.

katalon_request --get --data-urlencode "name=Smoke" \
  "${KATALON_API_HOST}/v2/schedules/test-runs/search"

This route returns a list, not the page from /test-runs/list.
Use /test-runs/{id} for the main detail response.
The separate /test-runs/{id}/get route returns a different test-run resource shape.
Use /schedules/{id}/test-run?startTime=... to read a scheduler occurrence with its scheduler ID.
It serves the same occurrence-read feature as /test-runs/{id}?startTime=....

Requires the TestRun:Update permission.

2. Select an editable test run before you send an update.

PUT /v2/schedules/test-runs/{id} updates a manual test run only before it starts.
For a future occurrence, use its scheduler ID with startTime.
Set thisAndFollowing=true to apply the change to that occurrence and later occurrences.
Use the full payload in Update a test run.
Do not assume that every test run is editable.

3. Save release and sprint associations as iteration.json.

{"releaseId": 801, "sprintId": 901}

4. Link the test run to those associations.

katalon_request --request POST --header "Content-Type: application/json" \
  --data @iteration.json \
  "${KATALON_API_HOST}/v2/schedules/test-runs/${TEST_RUN_ID}/link-iteration"

The response contains the updated test run.
Omit an ID or set it to null to clear that association.
Send both IDs to retain both associations.

Read results, files, and visual tests

Requires the TestRun:Read permission.

1. List primary results across the project.

katalon_request "${KATALON_API_HOST}/v2/schedules/execution-test-results"
katalon_request "${KATALON_API_HOST}/v2/schedules/execution-test-results/failure-categories"
katalon_request "${KATALON_API_HOST}/v2/schedules/visual-testing"

The result list excludes retried results. The failure-category list supplies filter values.
Its optionType defaults to FILTER. The visual list contains visual test runs.

2. List the files for a test run.

katalon_request --get --data-urlencode "q=executionId==${TEST_RUN_ID}" \
  "${KATALON_API_HOST}/v2/schedules/files"
katalon_request --output run-artifact.bin "${KATALON_API_HOST}/v2/schedules/files/${FILE_ID}"

The file read uses the same download capability as /api/v1/files/{id} through the v2 path.
Use the file list's id as FILE_ID, not uploadFileId.
Keep the downloaded file as a protected artifact.

Requires the TestRun:Update permission.

3. Rename a baseline collection group with its ID.

katalon_request --request PUT --header "Content-Type: application/json" \
  --data '{"name":"Smoke baselines"}' \
  "${KATALON_API_HOST}/v2/schedules/visual-testing/baseline-collection-groups/${BASELINE_GROUP_ID}"

This v2 route serves the same rename capability as /api/v1/baseline-collection-groups/{id}.

• Moving from the v1 API? See Migrate from the v1 API.
• See the API Reference for every field and response, starting with
  Run a scheduled occurrence now.