Navigation :

MongoDB Datasets

MongoDB Operation Parameters

For MongoDB datasets using the MongoDB Perl driver, there are some key differences from DBI.

The top level attributes are very similar.

Attribute	Notes
`dbname`	Specify which database connection should be used for fetching and storing this dataset. By default, the database named `default` (of type `mongo`) is used for executing the set. Note: The type of dataset (`dbi` or `mongo`) is not configurable on a per-dataset basis. It is determined by the `<dataset_dir>` element defining the directory containing the dataset. All datasets in the same directory must execute against databases of the same type.
`collection`	The collection name (including the database name) which is the MongoDB namespace for this dataset. E.g. `demo.ship` means collection `ship` in the `demo` database instance. A dataset applies to only a single collection.
`read`	Identical to DBI datasets. Access control identifiers are also identical.
`write`	Identical to DBI datasets. Access control identifiers are also identical.
`debug`	Identical to DBI datasets.
`dump`	Identical to DBI datasets.

The elements within the <dataset> definition for MongoDB datasets are as follows.

Element	Notes
`transform`	Identical to DBI datasets.
`hook`	Identical to DBI datasets.
`find`	The parameters to use when performing the MongoDB `find` for this dataset.
`aggregate`	The parameters to use when performing the MongoDB `aggregate` for this dataset.
`count`	The parameters to use when performing the MongoDB `count` for this dataset.
`distinct`	The parameters to use when performing the MongoDB `distinct` for this dataset.

Note: Only one of either aggregate or find may be configured per dataset.

Note: That insert/update/delete are not yet implemented for MongoDB but are planned for upcoming development.

Find

The elements within the <find> specification define the arguments passed to the MongoDB find request. Both are optional.

Element	Notes
`filter`	A JSON object structure defining the filter string (if filtering is required).
`options`	A JSON object structure defining the options string (if options are required).

Please refer to the MongoDB documentation for details of how to use find filters and find options. We will discuss only how the filter and options objects are constructed.

Specifically, Jarvis will process your filter and options objects as follows:

Variable substitution will be performed for ~varname~ RHS parts.
Where no variable is provided, the relevant part of the object will be removed.

Here is an example dataset which supports a find operation on the ship collection:

<dataset dbname="shipdb" dbtype="mongo" read="*" write="*"  collection="demo.ship">
    <find>
        <filter>
            {
                "$and": [
                    { "line": { "$eq": ~line|__default_line~ } },
                    { "name": { "$eq": ~name~ } },
                    { "afloat": { "$eq": ~afloat!boolean~ } }
                ]
            }
        </filter>
        <options>
            {
                "projection": {
                    "name": 1
                }
            }
        </options>
    </find>
</dataset>

Distinct

The elements within the <distinct> specification define the arguments passed to the MongoDB distinct request. Only the filter and options are optional.

Element	Notes
`fieldname`	A string defining the collection property to aggregate distinct records for.
`filter`	A JSON object structure defining the filter string (if filtering is required).
`options`	A JSON object structure defining the options string (if options are required).

Please refer to the MongoDB documentation for details of how to use distinct fieldname, distinct filters and distinct options. We will discuss only how the fieldname, filter and options objects are constructed.

Specifically, Jarvis will process your fieldname, filter and options objects as follows:

Variable substitution will be performed for ~varname~ RHS parts.
Where no variable is provided, the relevant part of the object will be removed.

Here is an example dataset which supports a distinct operation on the ship collection:

<dataset dbname="shipdb" dbtype="mongo" read="*" write="*" collection="demo.ship">
	<distinct>
    	<fieldname>name</fieldname>
	</distinct>    
</dataset>

Count

The elements within the <count> specification define the arguments passed to the MongoDB count request. Both are optional.

Element	Notes
`filter`	A JSON object structure defining the filter string (if filtering is required).
`options`	A JSON object structure defining the options string (if options are required).

Please refer to the MongoDB documentation for details of how to use count filters and count options. We will discuss only how the filter and options objects are constructed.

Specifically, Jarvis will process your filter and options objects as follows.

Variable substitution will be performed for ~varname~ RHS parts.
Where no variable is provided, the relevant part of the object will be removed.

Here is an example dataset which supports a count operation on the ship collection:

<dataset dbname="shipdb" dbtype="mongo" read="*" write="*" collection="demo.ship">
	<count>
    	<filter>
			{
				"deleted": false
			}
        </filter>
    </count>
</dataset>

Aggregate

The elements within the <aggregate> specification define the arguments passed to the MongoDB aggregate request. Only options are optional.

Element	Notes
`pipeline`	A JSON object structure defining the pipeline array (An array of pipelines is required by MongoDB to perform an aggregate lookup).
`options`	A JSON object structure defining the options string (if options are required).

Please refer to the MongoDB documentation for details of how to use aggregate pipelines and aggregate options. We will discuss only how the pipeline and options objects are constructed.

Specifically, Jarvis will process your pipeline and options objects as follows.

Variable substitution will be performed for ~varname~ RHS parts.
Where no variable is provided, the relevant part of the object will be removed.

Here is an example dataset which supports an aggregate operation on the ship collection:

<dataset dbname="shipdb" dbtype="mongo" read="*" write="*" collection="demo.ship">
    <aggregate>
        <pipeline>
            [
                {
                    "$lookup": {
                        from: 'ship',
                        localField: '_id',
                        foreignField: 'ship_id',
                        as: 'ship'
                    }
                } 
                , {
                    "$match": {
                        "$and": [
                            {
                                "line": ~line|__default_line~
                            }
                        ]
                    }
                }
            ]
        </pipeline>
        <options>{ "projection": { "name": 1 } }</options>
    </aggregate>    
</dataset>

Parameterization

Note that the variable syntax used for MongoDB parameter objects in the dataset is completely different from the {$param} syntax used for specifying variables within SQL statements for DBI-style databases.

This is because curly braces {} are already special characters in JSON, and a different syntax is required.

In our JSON definitions, standard JSON construction is used except for ~line|__default_line~ and ~name~ which are Jarvis parameters which will be substituted into the JSON structure using parameter processing.

Noting:

When determining the value for the line member of the first $and clause, Jarvis will look first for a client-supplied or server-supplied variable named line.
The variable may be supplied by a client REST, client query, client CGI, by a hook, or by a static default variable in the configuration file.
If no variable named line is present, then Jarvis will look for a server-supplied variable named __default_line (i.e. defined by a hook or by static default parameter in the configuration file since this is a safe variable name).

Note: Textual substitution is not available as an option for constructing MongoDB objects.

Nested Hash Folding

In some situations, it is beneficial to have access to nested parameters when performing substitutions.

In most cases simple assigning the entire hash as a parameter is enough, however in cases where different types are mixed or require specific type casting a hash folding approach is available.

Example 1

Consider the following post data:

{
	"name" : "My Ship"
	, "shipType" : {
        "typeName"       : "Yacht"
        , "hasPropeller" : false
	}
}

The ship name is easily accessible utilizing the ~name~ parameter.

In our example hasPropeller is a Boolean type and MongoDB will not accept us simply assigning ~shipType~ to the collection property shipType we require a more explicit mechanism.

To achieve this Jarvis supports a hash folding approach for MongoDB to provide access to all nested elements through a top-level parameter.

Consider the following hash folded parameters:

{
    "name"                    : "My Ship"
    , "shipType.typeName"     : "Yacht"
    , "shipType.hasPropellor" : false
    , "shipType" : {
        "typeName"       : "Yacht"
        , "hasPropeller" : false
    }
}

Jarvis will move a copy of each nested parameter to the top level of the available parameters while also maintaining a full copy of the original nested object.

Utilizing these parameters we can now explicitly update our nested elements within the all supported MongoDB dataset types.

<insert>
    <document>
        {
            "name": ~name~
            , "shipType": {
                 "typeName"    : ~shipType.typeName~
                 "hasPropeller": ~shipType.hasPropeller!boolean~
            }
        }
    </document>
</insert>

Parameter Processing

Refer to Parameter Expansion for the sources that will be searched for expanding an object parameter value. This includes:

Client-Supplied Parameters.
Server-Only Safe Parameters.
Default Parameters.

Formatting/type flags are also supported:

Flag	Notes
`!boolean`	If the parameter is present and defined, then it will be translated into the `boolean::true` and `boolean::false` used by MongoDB to represent `true`/`false` values. If the parameter is present but undefined, it will remain undefined.
`!oid`	If the parameter is present and defined, then it will be translated into the `MongoDB::OID` type used by MongoDB to represent Object ID values. If the parameter is present but undefined, it will remain undefined.
`!date`	If the parameter is present and defined, then it will be translated into the DateTime type used by MongoDB to represent either Date or ISODate values. If the parameter is present but undefined, it will remain undefined.
`!decimal`	If the parameter is present and defined, then it will be translated into the `BSON::Decimal128` type used by MongoDB to represent decimal values. If the parameter is present but undefined, it will remain undefined.
`!insensitive_regex`	If the parameter is present and defined, then it will be translated into the `BSON::Regex` type with the case insensitive option set. This is used by MongoDB to represent a like match searches. If the parameter is present but undefined, it will remain undefined.
`null`	If the parameter is not present, use a JSON null value (do not remove the parameter from the object).
`!oidarray`	If the parameter is present and defined, then it will be translated into the Array of `MongoDB::OID` types used by MongoDB to represent Object ID values. If the parameter is present but undefined, it will remain undefined.

Special Parameters

A number of special parameters will be supported for performing SQL-esq queries within Mongo DB that would normally be achieved via NodeJS constructors and methods.

Since Jarvis only works in pure JSON the use of executable JavaScript is not possible.

Flag	Notes
`~!NOW~`	If the parameter is present, then the current system time will be converted into the DateTime type and substituted into the query. Note: Mongo DB will be adding support for a `$$NOW` flag in version `4.2` as such this is a temporary stopgap until this functionality is natively supported.

Parameter Removal

When expanding a parameter, it is important to understand the difference between “present” and “defined”.

It is possible for a parameter to be “present but with undefined value”. A hook, or a JSON object in the POST data can specify JSON null for a parameter’s value in a row store request.

If a parameter is present but undefined, it will be set as null (undefined) within the JSON Array or Object sent as part of the MongoDB request.
If a parameter is not present from any of the input sources, then it will be removed entirely from the JSON Array or Object.

Then also:

Any JSON Array or Object which is completely empty will be removed entirely from JSON object.

Example 1

Considering again the filter in the example above.

<filter>
	{ 
     	"$and": [
			{ "line": { "$eq": ~line|__default_line~ } }, 
			{ "name": { "$eq": ~name~ } },
			{ "afloat": { "$eq": ~afloat!boolean~ } } 
		] 
	}
</filter>

Assuming the following in the request:

line not present in the request.
__default_line configured as “White Star” in the static configuration file.
name not present in the request.
afloat present as 1 in the request.

The following JSON Object will be used as the filter for the MongoDB find request.

{ 
	"$and": [
		{ "line": { "$eq": “White Star” } }, 
		{ "afloat": { "$eq": boolean::true } } 
	] 
}

Example 2

Assuming the following in the request:

line not present in the request and not configured anywhere.
__default_line not configured anywhere.
name not present in the request and not configured anywhere.
afloat not present in the request and not configured anywhere.

Then the following JSON Object will be used as the filter for the MongoDB find request.

null

Each of the $eq matches is removed because no value is present, then each of the $and sub-clauses is removed because the Object is empty, and then the $and is removed because the Array of sub-clauses is empty, and finally the entire filter is empty and removed.