Difference between revisions of "UM:Data Collection"

Every node can have many data collection items configured (see [[UM:Basic_Concepts|Basic Concepts]] for detailed description of DCI). NetXMS server has a set of threads dedicated to data collection, called ''Data Collectors'', used to gather information from the nodes according to DCI configuration. You can control how many data collectors will run simultaneously, by changing server configuration parameter ''NumberOfDataCollectors''.  

[[File:Dci_param_proc.png||none|alt=Newly received parameter processing|Newly received parameter processing]]

Data collection for a node can be configured using management console. To open data collection configuration window, right-click node object in object browser or on a map, and click '''Data Collection'''. You will see the list of configured data collection items. From here, you can add new or change existing parameters to monitor. Usual way to do something with DCIs is to right-click on appropriate record in the list and select a required action from popup menu.

Description is a free-form text string describing DCI. It is not used by the server and is intended for better information understanding by operators. If you use the '''Select''' button to select a parameter from the list, description field will be filled automatically.

Name of the parameter of interest, used for making a request to target node. For NetXMS agent and internal parameters it will be parameter name, and for SNMP agent it will be an SNMP OID. You can use the '''Select''' button for easier selection of required parameter name.

Origin of data (or method of obtaining data). Possible origins are NetXMS agent, SNMP agent, CheckPoint SNMP agent, Internal (data generated inside NetXMS server process), or Push Agent. Last origin is very different from all others, because it represents DCIs whose values are pushed to server by external program (usually via '''nxpush''' utility) instead of being polled by the server based on the schedule.

Data type for the parameter. Can be one of the following: Integer, Unsigned Integer, 64-bit Integer, 64-bit Unsigned Integer, Float (floating point number), or String. Selected data type affects collected data processing — for example, you cannot use operations like “less than” or “greater than” on strings. If you select parameter from the list using the '''Select''' button, correct data type will be set automatically.

An interval between consecutive polls, in seconds. If you select the '''Use advanced scheduling''' option, this field has no meaning and will be disabled.

If you turn on this flag, NetXMS server will use custom schedule for collecting DCI values instead of fixed intervals. This schedule can be configured on the '''Schedule''''' ''page. Advanced schedule consists of one or more records; each representing desired data collection time in cron-style format. Record has five fields, separated by spaces: minute, hour, day of month, month, and day of week. Allowed values for each filed are:

Run every 5 minutes.

In this field you can specify cluster resource associated with DCI. Data collection and processing will occur only if node you configured DCI for is current owner of this resource. This field is valid only for cluster member nodes.

This attribute specifies how long the collected data should be kept in database, in days. Minimum retention time is 1 day and maximum is not limited. However, keeping too many collected values for too long will lead to significant increase of your database size and possible performance degradation.

DCI status can be one of the following: ''Active'', ''Disabled'', ''Not Supported''. Server will collect data only if the status is ''Active''. If you wish to stop data collection without removing DCI configuration and collected data, the'' Disabled'' status can be set manually. If requested parameter is not supported by target node, the'' Not Supported'' status is set by the server.

In simplest case, NetXMS server collects values of specified parameters and stores them in the database. However, you can also specify various transformations for original value. For example, you may be interested in a delta value, not in a raw value of some parameter. Or, you may want to have parameter value converted from bytes to kilobytes. All transformations will take place after receiving new value and before threshold processing.

On the second step, custom transformation script is executed (if presented). By default, newly created DCI does not have a transformation script. If transformation script is presented, the resulting value of the first step is passed to the transformation script as a parameter; and a result of script execution is a final DCI value. For more information about NetXMS scripting language, please consult [#_NetXMS_Scripting_Language NetXMS Scripting Language] chapter in this manual.

For every DCI you can define one or more thresholds. Each threshold there is a pair of condition and event – if condition becomes true, associated event is generated. To configure thresholds, open the data collection editor for node or template, right-click on the DCI record and select '''Edit''' from the popup menu, then select the '''Thresholds''' page. You can add, modify and delete thresholds using buttons below the threshold list. If you need to change the threshold order, select one threshold and use arrow buttons located on the right to move the selected threshold up or down.

Each DCI has an ''Instance'' attribute, which is a free-form text string, passed as a 6^th parameter to events associated with thresholds. You can use this parameter to distinguish between similar events related to different instances of the same entity – for example, if you have an event generated when file system was low on free space, you can set the ''Instance'' attribute to file system mount point.

Threshold processing algorithm is outlined on Figure 5.

You can disable threshold ordering by checking '''Always process all thresholds''' checkbox. If it is marked, system will always process all thresholds.

When adding or modifying a threshold, you will see the following dialog:

You can also configure threshold to resend activation event if threshold's condition remain true for specific period of time. You have three options - ''default'', which will use server-wide settings, ''never'', which will disable resending of events, or specify interval in seconds between repeated events.

You can choose any event to be generated when threshold becomes active or returns to inactive state. However, you should avoid using predefined system events (their names usually start with SYS_ or SNMP_). For example, you set event SYS_NODE_CRITICAL to be generated when CPU utilization exceeds 80%. System will generate this event, but it will also generate the same event when node status will change to CRITICAL. In your event processing configuration, you will be unable to determine actual reason for that event generation, and probably will get some unexpected results. If you need custom processing for specific threshold, you should create your own event first, and use this event in the threshold configuration. NetXMS has some preconfigured events that are intended to be used with thresholds. Their names start with DC_.

# Instance (DCI's ''instance'' attribute)

NetXMS gives you ability to push DCI values when you need it instead of polling them on specific time intervals. To be able to push data to the server, you should take the following steps:  

# Use '''nxpush''' utility or client API for pushing data.

Often you have a situation when you need to collect same parameters from different nodes. Such configuration making may easily fall into repeating one action many times. Things may became even worse when you need to change something in already configured DCIs on all nodes - for example, increase threshold for CPU utilization. To avoid these problems, you can use ''data collection templates''. Data collection template (or just template for short) is a special object, which can have configured DCIs similar to nodes.  

Please note that you can apply an unlimited number of templates to a node - so you can create individual templates for each group of parameters (for example, generic performance parameters, MySQL parameters, network counters, etc.) and combine them, as you need.

To create a template, right-click on '''Template Root''' or '''Template Group''' object in the Object Browser, and click '''Create''' and then select'' '''''Template'''. Enter a name for a new template and click '''OK'''.

To configure DCIs in the template, right-click on '''Template '''object in the Object Browser, and select '''Data Collection''' from the popup menu. '''Data collection editor''' window will open. Now you can configure DCIs in the same way as the node objects.

To apply a template to one or more nodes, right-click on template object in Object Browser and select '''Apply''' from popup menu. Node selection dialog will open. Select the nodes that you wish to apply template to, and click '''OK''' (you can select multiple nodes in the list by holding CTRL key). Please note that if data collection editor is open for any of the target nodes, either by you or another administrator, template applying will be delayed until data collection editor for that node will be closed.

To remove a link between template and node, right-click on '''Template '''object in the Object Browser and select '''Unbind''' from popup menu. Node selection dialog will open. Select one or more nodes you wish to unbind from template, and click '''OK'''. The system will ask you how to deal with DCIs configured on node and associated with template:

If you select '''Unbind DCIs from template''', all DCIs related to template will remain configured on a node, but association between the DCIs and template will be removed. Any further changes to the template will not be reflected in these DCIs. If you later reapply the template to the node, you will have two copies of each DCI - one standalone (remaining from unbind operation) and one related to template (from new apply operation). Selecting '''Remove DCIs from node''''' ''will remove all DCIs associated with the template. After you click OK, node will be unbound from template.

You can use various macros in ''name'', ''description'', and ''instance'' fields of template DCI. These macros will be expanded when template applies to node. Macro started with %{ character combination and ends with } character. The following macros are currently available:

Please note that if you change something in the node, name for example, these changes will not be reflected automatically in DCI texts generated from macros. However, they will be updated if you reapply template to the node.

Once you setup DCI, data starts collecting in the database. You can access this data and work with it in different ways.

You can view collected data in a graphical form, as a line chart. To view values of some DCI as a chart, first open either Data Collection Editor or Last Values view for a host. You can do it from the Object Browser or map by selection host, right-clicking on it, and selecting '''Data collection''' or''''' ''Last DCI values'''. Then, select one or more DCIs (you can put up to 16 DCIs on one graph), right-click on them and choose '''Graph''' from the pop-up menu. You will see graphical representation of DCI values for the last hour.

If you need to change or delete predefined graph, you can do it via '''Tools -> Graphs -> Manage''' menu.

You can view collected data in a textual form, as a table with two columns – timestamp and value. To view values of some DCI as a table, first open either '''Data Collection Editor''' or '''Last Values view''' for a host. You can do it from the Object Browser or map by selection host, right-clicking on it, and selecting '''Data collection''' or '''Last DCI values'''. Then, select one or more DCIs (each DCI data will be shown in separate window), right-click on them and choose '''Show history''' from the pop-up menu. You will see the last 1000 values of the DCI.

You can export collected data to a text file. To export the DCI data, first open either '''Data Collection Editor''' or '''Last Values view''' for the host. You can do it from the Object Browser or map by selection the host, right-clicking on it, and selecting '''Data collection''' or '''Last DCI values'''. Then, select one DCI, right-click on it and select '''Export data''' from the pop-up menu. Export configuration dialog will appear (Figure 8).