Navigation :

API - JSON QueryTest

Overview

The QueryTest method allows a JSON RPC client to monitor the status of a Test Run that has been submitted to the N2SVCD.

The JSON RPC client is expected to periodically poll for Test Run status. The N2SVCD service will maintain the Test Run status in memory until one of the following occurs.

The N2SVCD process is restarted, or
The CancelTest method is used to cancel the Test Run, or
The QueryTest method is called on a completed Test Run.

Request Parameters

The QueryTest method has a single parameter:

Parameter	Type	Description
run_id	String	The Test Run ID of a previously started Test Run.

Example Request

Here is an example QueryTest request.

{
    "jsonrpc" : "2.0",
    "id" : 1,
    "method" : "QueryTest",
    "params" : {
        "run_id" : "tr_9538"
    }
}

Successful Response

The QueryTest result is an object with the following attributes.

Attribute	Type	Description
`run_id`	String	The unique run identifier assigned to this run.
`completed`	Integer	`0` (completed) or `1` (in progress).
`time_now`	Array	The "now" system time on the N2SVCD clock.
`[0]`	Integer	The "now" epoch seconds.
`[1]`	Integer	The "now" epoch microseconds.
`time_start`	Array	Test run start time on the N2SVCD clock.
`[0]`	Integer	The "time_start" epoch seconds.
`[1]`	Integer	The "time_start" epoch microseconds.
`time_finish`	Array	Test run start finish on the N2SVCD clock (if the run is completed).
`[0]`	Integer	The "time_finish" epoch seconds.
`[1]`	Integer	The "time_finish" epoch microseconds.
`duration`	Float	Test run duration in seconds (if the run is completed).
`counts`	Object	Object with test instance counts.
.`started`	Integer	Number of instances started.
.`finished`	Integer	Number of instances finished (passed, failed, and aborted).
.`failed`	Integer	Number of instances completed with a failed non-mandatory check.
.`aborted`	Integer	Number of instances aborted by a failed mandatory check.
.`cancelled`	Integer	Number of instances cancelled by user-requested run cancel.
`globals`	Object	Any global variables defined during the Test Run.
`halted`	Array	Array of halted test instances (if any). See description below.
`first_fails`	Array	Array of "first fail" reasons. See description below.
`first_aborts`	Array	Array of "first abort" reasons. See description below.
`buckets`	Array	Time-based call distribution buckets. See description below.
`first_call_summary`	Object	Information about the first Test Instance in the Run.

Note: The first_call_summary is strictly speaking a summary of the first Test Instance, which may in practice implement none, one or more Telephony Calls.

Response - “globals”

The globals attribute is an Object whose values have been set during the Test Run execution. This is usually performed via the eval operation. For example the following operation:


    ...
    {
        "type": "eval",
        "expression": "$g->{foo} = 'bar'"
    }
    ...

…would result in the following globals attribute value at the conclusion of the Test Run:


    ...
    "globals": {
        "foo": "bar"
    },
    ...

Global variables can be used in this way to gather additional information about the Test Run or Test Instances for return back to the client.

Response - “halted”

The halted attribute is an array of Objects giving information about any currently halted instances within the run. Each Object in the halted Array has the following attributes.

Attribute	Type	Description
`instance_id`	Integer	The id of the halted instance within the test run.
`operation`	Object	The operation which caused the instance to halt.

E.g. the following entry indicates that Test Instance with ID “41” is halted at a halt operation which is operation index 13 in the test run operations array.


    ...
    "halted": [
        {
            "instance_id": "41",
            "operation" : {
                "arguments" : {
                    "seconds" : "5"
                },
                "type" : "halt",
                "static" : 1,
                "idx" : "13"
            }
        }
    ]
    ...

The operation Object has the following attributes at least:

Attribute	Type	Description
`idx`	Integer	The index of the halted operation within the array of test operations.
`type`	String	The operation type name, which is typically the `halt` operation.
`arguments`	Object	The arguments of this operation as provided in the `ExecuteTest` request.
`...`	[Any]	Any other operation attributes which may have been provided in the `ExecuteTest` request or may be used by the TestRun for its internal test management purposes.

The instance_id from the halted Object must be specified as the instance_id parameter to the ResumeInstance RPC method when resuming a halted Test Instance. If a Test Instance is not resumed before the halt timer expires, the instance will be aborted.

Response - “first_fails/aborts”

During a load test (i.e. when more than one Test Instance is executed in a Test Run) detailed information is only available about the first Test Instance in the Run. The first_fails and first_aborts response attributes provide some limited information about fail and abort reasons in subsequent Test Instances.

The first_fails attribute is an Array which tracks the first failed check which occurs during each Test Instance. The number of Test Instances which fail with the same unique text message and operation is counted and reported.

Each Object in the first_fails and first_aborts attribute has the following attributes:

Attribute	Type	Description
`op_idx`	Integer	The index in the `operations` at which this fail/abort occured.
`text`	String	The text fail/abort message generated by this failed/aborted check.
`count`	Integer	The number of Test Instances whose first fail/abort occured as described.

E.g. the following entry indicates that 300 Test Instances all encountered their first failed check at operation[4] and with the text “‘initialCallSegment_cause’ should = 30, but = 31.”


    ...
    "first_fails" : [
        {
            "count" : 300,
            "op_idx" : 4,
            "text" : "'initialCallSegment_cause' should = 30, but = 31."
        }
    ]
    ...

Response - “buckets”

The buckets attribute is used to indicate the distribution of calls over time. Although it is always returned, the information is only statistically relevant in a sustained load test scenario (e.g. at least 100 Test Instances).

Between 1 and 20 buckets will be provided. Each bucket has the following attributes:

Attribute	Type	Description
`num_calls_started`	Integer	Number of Test Instances started during this time period.
`num_calls_finished`	Integer	Number of Test Instances finished during this time period.
`time_start`	Array	N2SVCD timestamp for start of interval covered by this bucket.
`[0]`	Integer	The "time_start" epoch seconds.
`[1]`	Integer	The "time_start" epoch microseconds.

The time_start attribute is not present for the first bucket, as the the “time_start” of the Test Run should be used instead. There is no “time_finish” attribute on the bucket, as the “time_start” of the next bucket should be used, or the “time_finish” of the Test Run.

The following example shows a complete bucket count. The actual number of buckets provided is determined by the N2SVCD.


    ...
    "buckets" : [
        {
            "num_calls_started" : 15,
            "num_calls_finished" : 15
        },
        {
            "time_start" : [
                1389748329,
                67575
            ],
            "num_calls_started" : 15,
            "num_calls_finished" : 15
        },

        ...17 more buckets...

        {
            "time_start" : [
                1389748356,
                87250
            ],
            "num_calls_started" : 15,
            "num_calls_finished" : 15
        }
    ]
    ...

Response - “first_call_summary”

The first_call_summary attribute gives information about the processing of the first Test Instance executed in the Test Run. It contains the following attributes:

Attribute	Type	Description
`idx`	Integer	The Instance ID within the Test Run. Always = `0`.
`completed`	Integer	`0` (completed) or `1` (in progress).
`time_start`	Array	N2SVCD timestamp at which this Test Instance started.
`[0]`	Integer	The "time_start" epoch seconds.
`[1]`	Integer	The "time_start" epoch microseconds.
`time_finish`	Array	N2SVCD timestamp at which this Test Instance finished (if completed).
`[0]`	Integer	The "time_start" epoch seconds.
`[1]`	Integer	The "time_start" epoch microseconds.
`duration`	Float	Test Instance duration in seconds (if completed).
`check_log`	Array	List of checks performed against this Test Instance.
`trace_log`	Array	List of tracing events logged against this Test Instance. See Application - Trace Log Structure.

Each check_log entry has the following structure:

Attribute	Type	Description
`op_idx`	Integer	The index in the `operations` at which this check occured.
`step`	Integer	The incrementing operation counter at which this check occured.
`status`	String	`pass`, `fail`, or `abort`.
`text`	String	A text message describing the result of the check.

Note: The step counter is initialised to 0 at the start of the Test Instance and is incremented every time a new operation is performed. If there are no branches, loops or jumps in the operation sequence, then the step will always be identical to the op_idx.

Each trace_log entry has structure dependent on the type of event being logged. Refer to Application - Trace Log Structure for the detailed description of the check_log event format.

Note: Other interfaces may implement new path identifiers and associated attributes.

Example Successful Response

Here is a possible successful QueryTest response.

{
    "jsonrpc" : "2.0",
    "id" : "2",
    "result" : {
        "run_id" : "tr_7309",
        "completed" : 1,
        "time_now" : [
            1389747502,
            87993
        ],
        "time_start" : [
            1389747502,
            85922
        ],
        "time_finish" : [
            1389747502,
            87871
        ],
        "duration" : 0.001949,
        "counts" : {
            "started" : 1,
            "finished" : 1,
            "aborted" : 0,
            "failed" : 1,
            "cancelled" : 0
        },
        "globals" : {},
        "first_fails" : [
            {
                "count" : 1,
                "op_idx" : 4,
                "text" : "'initialCallSegment_cause' should = 30, but = 31."
            }
        ],
        "first_aborts" : [],
        "buckets" : [
            {
                "num_calls_started" : 1,
                "num_calls_finished" : 1
            }
        ],
        "first_call_summary" : {
            "idx" : "0",
            "completed": 1,
            "time_start" : [
                1389747502,
                85986
            ],
            "time_finish" : [
                1389747502,
                87776
            ],
            "duration" : 0.00179,
            "check_log" : [
                {
                    "op_idx" : 4,
                    "step" : 8,
                    "status" : "pass",
                    "text" : "Got inap.ssp_from_scp.ReleaseCall as expected."
                },
                {
                    "op_idx" : 4,
                    "step" : 8,
                    "status" : "fail",
                    "text" : "'initialCallSegment_cause' should = 30, but = 31."
                },
                {
                    "op_idx" : 5,
                    "step" : 9,
                    "status" : "pass",
                    "text" : "OK Cause 31, is > 22."
                }
            ],
            "trace_log" : [
                {
                    "op_idx" : 3,
                    "step" : 7,
                    "time" : [
                        1389747502,
                        86887
                    ],
                    "path" : "tcap.ssp_to_scp",
                    "protocol" : "inap",
                    "hex" : "6281894804008b05dc6b1a2818060700118605010101a00d600ba1090607040000011503046c65a163020101020100305b80020bb88207031081009909108307831120674523018501f78a088497161405512102af0e300c02020190a1060404424f535397029181980d300ba0098101028201018801009a022001bb0580038090a39c01039f36044d3b8fa0"
                },
                {
                    "op_idx" : 4,
                    "step" : 8,
                    "time" : [
                        1389747502,
                        87524
                    ],
                    "path" : "tcap.ssp_from_scp",
                    "protocol" : "inap",
                    "hex" : "64144904008b05dc6c0ca10a0201010201160402809f"
                }
            ]
        }
    }
}