User Tools

Site Tools


sib:archiver

SIB Archiver

Overview

The sib_archiver is a module for archiving log- and datamessages transferred over the internal MQTT transport of the SIB. Logging is done to a MongoDB and both archive rules and purge rules are configurable by service.

Workflow

  • Listen to the LOGGER topic in .custom.topic
    • on message
      • Lookup the ruleset for the message in .custom.archiveRules
      • Log to log and msg collections for the service given
  • Cronjob for purging as per .custom.purge
    • on trigger
      • Check age and length limits of collections
      • Remove old or overflowing documents

Custom configuration

sibarchiver/custom.js
"use strict";
let type = require( "sib_logger" ).type;
 
var ArchiverCustom = {
	_id: "archiverSample:1",
	_name: "SIB MQTT Archiver sample",
	archiveRules: {
		'g2i.001:mqttListener':{log:[], msg:[]},
		'g2i.001':{log:[], msg:[type.INPUT, type.OUTPUT, type.RESPONSE]},
		'':{log:[type.REQUEUE, type.INFO, type.WARNING, type.ERROR], msg:[type.INPUT, type.OUTPUT, type.RESPONSE]}
	},
	loggerCfg: {
		dest: {
			file: false,
			console: true,
			mqtt: true
		}
	},
	mongodb: "mongodb://localhost:27017/sibarchive",
	mqtt: "mqtt://localhost",
	purge: {
		'archiverSample':{schedule:"*/5 * * * *", age:"1D", length:10},
		'g2i.001':{schedule: "* * * * *", age:"1D", length: 20},
		'':{schedule: "*/10 * * * *", age:0, length: 100}
	},
	topic: "SIB/LOGGER/#"
};
 
module.exports = ArchiverCustom;
  • _id: The unique ID of this service
  • _name: A plaintext description of this service
  • archiveRules: A list of rules that applies to the archiving of the different sources see Archive Rules
  • loggerCfg: A JSON Object defining what logger actions should be active for this service (see Logger Configuration
  • mongodb: Connectionstring to the MongoDB to log to
  • mqtt: The MQTT connection string for the internal MQTT transport
  • purge: A list of rules that applies to the purging of the collections for the different sources see Purge Rules
  • topic: The MQTT topic to subscribe to on the internal MQTT transport

ArchiveRules

The archiveRules element of the .custom configuration is a JSON object where the keys define a ServiceName pattern and the corresponding element is a set of log, and msg types defining which message types to log to each respective collection.

The SIBArchiver will match the serviceid of each logged message to find the first match in the archiveRules keylist. The first match rule will be used to decide what to log for that message which means that the order of the rules in the list is important.

The archiveRules.' ' element must be set as that is the fallback rule that applies to everything that is not matched by any specific rule definition.

Each rule definition is a JSON object on the form:

{
  log:[type.INFO, type.WARNING, type.ERROR], 
  msg[type.INPUT, type.OUTPUT, type.RESPONSE]
}

The content of the log and msg arrays are combinations of the predefined constants in sib_logger.type. Each value in the arrays are checked against the msg.evttype value to see if the evttype contains the logger.type in the array. If it does the message is logged to this logcollection.

The collections for the logs are named:

  • log: [msg.serviceid]_log
  • msg: [msg.serviceid]_archive

All invalid characters (fn the dot (.)) will be replaced by an underscore (_) in the msg.serviceid.

PurgeRules

the purge element of the .custom configuration is a JSON object on the same form as the Archive Rules element. The same matching principle is used to decide what purge rule is to be used for each collection in the database.

When the SIBArchiver service starts up a cron job is set up for each purge rule using the schedule from the rule.schedule. When the cron job triggers each collection in the database will be matched against the list of purge rules to see if the current rule is the first rule that matches the collection name (. is replaced with _ before the match) and when it is decided that this rule will apply to the collection all documents in this collection that falls outside of the ruleset are deleted.

As in the Archive Rules the purge.' ' element must be included as this is the fallback ruleset that applies to all collections that does not match any specific purge rule.

  purge: {
    '':{
      schedule:"* * * * *",
      age:"2W", 
      length:10000
    }
  }
  • schedule: The CRON-schedule deciding when to run this purge rule
  • age: The maximum age of documents in the collections, either in number of milliseconds or postfixed with H, D, W, M, Y for Hours, Days, Weeks, Months or Years respectively. If set to 0 no age rule applies.
  • length: The maximum length of number of documents in the collection. When the length overflows the oldest documents in the collection will be deleted
sib/archiver.txt · Last modified: 2018/11/29 13:04 by hubbe