org.opennms.netmgt.rrd
Class QueuingRrdStrategy

java.lang.Object
  org.opennms.netmgt.rrd.QueuingRrdStrategy

All Implemented Interfaces:

Runnable, RrdStrategy

public class QueuingRrdStrategy
extends Object
implements RrdStrategy, Runnable

Provides queuing implementation of RrdStrategy. In order to provide a more scalable collector. We created a queuing RrdStrategy that enabled the system to amortize the high cost of opening an round robin database across multiple updates. This RrdStrategy implementation enqueues the create and update operations on a per file basis and maintains a set of threads that process enqueued work file by file. If the I/O system can keep up with the collection threads while performing only a single update per file then eventually all the data is processed and the threads sleep until there is more work to do. If the I/O system is initially slower than than the collection threads then work will enqueue here and the write threads will get behind. As this happens each file will eventually have more than a single update enqueued and therefore the number of updates pushed thru the system will increase because more then one will be output per 'open' Eventually, the I/O system and the collection system will balance out. When this happens all data will be collected but will not be output to the rrd files until the next time the file is processed by the write threads. As another performance improving strategy. The queue distinguishes between files with significant vs insignificant updates. Files with only insignificant updates are put at the lowest priority and are only written when the highest priority updates have been written This implementation delegates all the actual writing to another RrdStrategy implementation. System properties effecting the operation: org.opennms.rrd.queuing.writethreads: (default 2) The number of rrd write threads that process the queue org.opennms.rrd.queuing.queueCreates: (default false) indicates whether rrd file creates should be queued or processed synchronously org.opennms.rrd.queuing.maxInsigUpdateSeconds: (default 0) the number of seconds over which all files with significant updates only should be promoted onto the significant less. This is to ensure they don't stay unprocessed forever. Zero means not promotion. org.opennms.rrd.queuing.modulus: (default 10000) the number of updates the get enqueued between statistics output org.opennms.rrd.queuing.category: (default "OpenNMS.Queued") the log category to place the statistics output in TODO: Promote files when ZeroUpdate operations can't be merged. This may be a collection miss which we want to push thru. It should also help with memory. TODO: Set an upper bound on enqueued operations TODO: Provide an event that will write data for a particular file... Say right before we try to graph it.

Nested Class Summary

class	QueuingRrdStrategy.CreateOperation This class represents an operation to create an rrd file
class	QueuingRrdStrategy.UpdateOperation Represents an update to a rrd file.
class	QueuingRrdStrategy.ZeroUpdateOperation Represents an update whose value is 0.

Constructor Summary

QueuingRrdStrategy(RrdStrategy delegate)

Method Summary

void	addOperation(org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation op) Add an operation to the queue.
void	closeFile(Object rrd) This closes the supplied round robin database
Object	createDefinition(String creator, String directory, String rrdName, int step, List<RrdDataSource> dataSources, List<String> rraList) Create a round robin database definition from the supplied parameters.
Object	createDefinition(String creator, String directory, String dsName, int step, String dsType, int dsHeartbeat, String dsMin, String dsMax, List<String> rraList)
void	createFile(Object op) Creates the round robin database defined by the supplied definition.
InputStream	createGraph(String command, File workDir) Creates an InputStream representing the bytes of a graph created from round robin data.
RrdGraphDetails	createGraphReturnDetails(String command, File workDir) Creates an RrdGraphDetails object representing the graph created from round robin data.
void	ensureThreadsStarted() Ensure that we have threads started to process the queue.
Double	fetchLastValue(String rrdFile, String ds, int interval) Fetches the last value from the round robin database with the given name.
Double	fetchLastValue(String rrdFile, String ds, String consolidationFunction, int interval) Fetches the last value from the round robin database with the given name.
Double	fetchLastValueInRange(String rrdFile, String ds, int interval, int range) Fetches the last value from the round robin database with the given name within a time range.
long	getCreatesCompleted()
String	getDefaultFileExtension() Get the file extension appropriate for files of this type
RrdStrategy	getDelegate()
long	getDequeuedItems()
long	getDequeuedOperations()
long	getEnqueuedOperations()
long	getErrors()
int	getGraphLeftOffset() Returns the number of pixels that the leftt-hand side of the graph is offset from the left side of the created image.
int	getGraphRightOffset() Returns the number of pixels that the right-hand side of the graph is offset from the right side of the created image.
int	getGraphTopOffsetWithText() Returns the number of pixels that the top of the graph is offset from the top of the created image if there is single line of header text.
String	getLapTime()
LinkedList<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation>	getNext() Get the operations for the next file that should be worked on.
long	getPromotionCount()
long	getSignificantOpsCompleted()
long	getSignificantOpsDequeued()
long	getSignificantOpsEnqueued()
long	getStartTime()
String	getStats() Print queue statistics.
long	getTotalOperationsPending()
long	getUpdatesCompleted()
void	graphicsInitialize() This Initializes the graphics subsystem only.
void	initialize() Initialize the appropriate round robin system
void	logStats()
org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation	makeCreateOperation(String fileName, Object rrdDef)
org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation	makeUpdateOperation(String fileName, String owner, String update)
Object	openFile(String fileName) Opens the round robin database with the supplied name.
void	promoteEnqueuedFiles(Collection<String> rrdFiles) In the event that this is a queuing implementation of the RrdStrategy.
void	run()
void	setCreatesCompleted(long createsCompleted)
void	setDequeuedItems(long dequeuedItems)
void	setDequeuedOperations(long dequeuedOperations)
void	setEnqueuedOperations(long enqueuedOperations)
void	setErrors(long errors)
void	setPromotionCount(long promotionCount)
void	setSignificantOpsCompleted(long significantOpsCompleted)
void	setSignificantOpsDequeued(long significantOpsDequeued)
void	setSignificantOpsEnqueued(long significantOpsEnqueued)
void	setStartTime(long updateStart)
void	setTotalOperationsPending(long totalOperationsPending)
void	setUpdatesCompleted(long updatesCompleted)
void	updateFile(Object rrdFile, String owner, String data) Updates the supplied round robin database with the given timestamp:value point

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

QueuingRrdStrategy

public QueuingRrdStrategy(RrdStrategy delegate)

Method Detail

makeCreateOperation

public org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation makeCreateOperation(String fileName,
                                                                               Object rrdDef)

makeUpdateOperation

public org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation makeUpdateOperation(String fileName,
                                                                               String owner,
                                                                               String update)

addOperation

public void addOperation(org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation op)

Add an operation to the queue.

ensureThreadsStarted

public void ensureThreadsStarted()

Ensure that we have threads started to process the queue.

getNext

public LinkedList<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation> getNext()

Get the operations for the next file that should be worked on.

Returns:

a linkedList of operations to be processed all for the same file.

promoteEnqueuedFiles

public void promoteEnqueuedFiles(Collection<String> rrdFiles)

Description copied from interface: RrdStrategy

In the event that this is a queuing implementation of the RrdStrategy. This method causes all queued but not yet written data to be to the rrd files as soon as possible.

Specified by:

promoteEnqueuedFiles in interface RrdStrategy

getDelegate

public RrdStrategy getDelegate()

closeFile

public void closeFile(Object rrd)
               throws Exception

Description copied from interface: RrdStrategy

This closes the supplied round robin database

Specified by:

closeFile in interface RrdStrategy

Parameters:

rrd - an rrd object created using openFile

Throws:

Exception - if an error occurs closing the file

createDefinition

public Object createDefinition(String creator,
                               String directory,
                               String dsName,
                               int step,
                               String dsType,
                               int dsHeartbeat,
                               String dsMin,
                               String dsMax,
                               List<String> rraList)
                        throws Exception

Throws:

Exception

createDefinition

public Object createDefinition(String creator,
                               String directory,
                               String rrdName,
                               int step,
                               List<RrdDataSource> dataSources,
                               List<String> rraList)
                        throws Exception

Description copied from interface: RrdStrategy

Create a round robin database definition from the supplied parameters. This definition is used in the createFile call to create the actual file.

Specified by:

createDefinition in interface RrdStrategy

Parameters:

creator - - A string representing who is creating this file for use in log msgs

directory - - The directory to create the file in

rrdName - - The name to use for the round robin database

step - - the step for the database

dataSources - - the data sources to use for round robin database

rraList - - a List of the round robin archives to create in the database

Returns:

an object representing the definition of an round robin database

Throws:

Exception - If an error occurs while creating the definition

createFile

public void createFile(Object op)
                throws Exception

Description copied from interface: RrdStrategy

Creates the round robin database defined by the supplied definition.

Specified by:

createFile in interface RrdStrategy

Parameters:

op - an round robin database definition created using the createDefinition call.

Throws:

Exception - if an error occurs create the file

initialize

public void initialize()
                throws Exception

Description copied from interface: RrdStrategy

Initialize the appropriate round robin system

Specified by:

initialize in interface RrdStrategy

Throws:

Exception - if an Error occurs

graphicsInitialize

public void graphicsInitialize()
                        throws Exception

Description copied from interface: RrdStrategy

This Initializes the graphics subsystem only. This is used by the web application to avoid the need for the JNI infrastructure in the webapp.

Specified by:

graphicsInitialize in interface RrdStrategy

Throws:

Exception

openFile

public Object openFile(String fileName)
                throws Exception

Description copied from interface: RrdStrategy

Opens the round robin database with the supplied name. It is assumed the name refers to a round robin database appropriate for this strategy implementation

Specified by:

openFile in interface RrdStrategy

Parameters:

fileName - the name of the associated rrd file

Returns:

an open rrd reference that can by used in calls to updateFile and closeFile

Throws:

Exception - if an error occurs opening the file

updateFile

public void updateFile(Object rrdFile,
                       String owner,
                       String data)
                throws Exception

Description copied from interface: RrdStrategy

Updates the supplied round robin database with the given timestamp:value point

Specified by:

updateFile in interface RrdStrategy

Parameters:

rrdFile - an rrd object created using openFile

owner - the owner of the rrd

data - a string of the form :

Throws:

Exception - if an error occurs updating the file

fetchLastValue

public Double fetchLastValue(String rrdFile,
                             String ds,
                             int interval)
                      throws NumberFormatException,
                             RrdException

Description copied from interface: RrdStrategy

Fetches the last value from the round robin database with the given name. The interval passed in should be the interval associated with the round robin database.

Specified by:

fetchLastValue in interface RrdStrategy

Parameters:

rrdFile - a name the represents a round robin database

ds - a name the represents a data source to be used

interval - a step interval of the round robin database

Returns:

The last value as a Double (if the last value didn't exist returns a Double.NaN)

Throws:

NumberFormatException

RrdException

fetchLastValue

public Double fetchLastValue(String rrdFile,
                             String ds,
                             String consolidationFunction,
                             int interval)
                      throws NumberFormatException,
                             RrdException

Description copied from interface: RrdStrategy

Fetches the last value from the round robin database with the given name. The interval passed in should be the interval associated with the round robin database.

Specified by:

fetchLastValue in interface RrdStrategy

Parameters:

rrdFile - a name the represents a round robin database

ds - a name the represents a data source to be used

interval - a step interval of the round robin database

Returns:

The last value as a Double (if the last value didn't exist returns a Double.NaN)

Throws:

NumberFormatException

RrdException

fetchLastValueInRange

public Double fetchLastValueInRange(String rrdFile,
                                    String ds,
                                    int interval,
                                    int range)
                             throws NumberFormatException,
                                    RrdException

Description copied from interface: RrdStrategy

Fetches the last value from the round robin database with the given name within a time range. The interval passed in should be the interval associated with the round robin database. The range should be the amount of "lag" acceptable for an update to be considered valid. Range must be a multiple of the RRD interval.

Specified by:

fetchLastValueInRange in interface RrdStrategy

Parameters:

rrdFile - a name the represents a round robin database

ds - a name the represents a data source to be used

interval - a step interval of the round robin database

range - an acceptable range for which the last value will be returned

Returns:

The last value as a Double (if the last value didn't exist returns a Double.NaN)

Throws:

NumberFormatException

RrdException

createGraph

public InputStream createGraph(String command,
                               File workDir)
                        throws IOException,
                               RrdException

Description copied from interface: RrdStrategy

Creates an InputStream representing the bytes of a graph created from round robin data. It accepts an rrdtool graph command. The underlying implementation converts this command to a format appropriate for it .

Specified by:

createGraph in interface RrdStrategy

Parameters:

command - the command needed to create the graph

workDir - the directory that all referenced files are relative to

Returns:

an input stream representing the bytes of a graph image as a PNG file

Throws:

IOException - if an IOError occurs

RrdException - if an RRD error occurs

run

public void run()

Specified by:

run in interface Runnable

getStats

public String getStats()

Print queue statistics.

Specified by:

getStats in interface RrdStrategy

Returns:

a non-null string representing any statistics to be included in the logs

logStats

public void logStats()

getLapTime

public String getLapTime()

getGraphLeftOffset

public int getGraphLeftOffset()

Description copied from interface: RrdStrategy

Returns the number of pixels that the leftt-hand side of the graph is offset from the left side of the created image. The offset should always be positive.

Specified by:

getGraphLeftOffset in interface RrdStrategy

Returns:

offset in pixels. Should always be positive.

getGraphRightOffset

public int getGraphRightOffset()

Description copied from interface: RrdStrategy

Returns the number of pixels that the right-hand side of the graph is offset from the right side of the created image. The offset should always be negative.

Specified by:

getGraphRightOffset in interface RrdStrategy

Returns:

offset in pixels. Should always be negative.

getGraphTopOffsetWithText

public int getGraphTopOffsetWithText()

Description copied from interface: RrdStrategy

Returns the number of pixels that the top of the graph is offset from the top of the created image if there is single line of header text. The offset should always be negative.

Specified by:

getGraphTopOffsetWithText in interface RrdStrategy

Returns:

offset in pixels. Should always be negative.

getDefaultFileExtension

public String getDefaultFileExtension()

Description copied from interface: RrdStrategy

Get the file extension appropriate for files of this type

Specified by:

getDefaultFileExtension in interface RrdStrategy

createGraphReturnDetails

public RrdGraphDetails createGraphReturnDetails(String command,
                                                File workDir)
                                         throws IOException,
                                                RrdException

Description copied from interface: RrdStrategy

Creates an RrdGraphDetails object representing the graph created from round robin data. It accepts an rrdtool graph command. The underlying implementation converts this command to a format appropriate for it .

Specified by:

createGraphReturnDetails in interface RrdStrategy

Parameters:

command - the command needed to create the graph

workDir - the directory that all referenced files are relative to

Returns:

details for the graph including an InputStream, any PRINTed lines, and graph dimensions.

Throws:

IOException - if an IOError occurs

RrdException - if an RRD error occurs

getTotalOperationsPending

public long getTotalOperationsPending()

setTotalOperationsPending

public void setTotalOperationsPending(long totalOperationsPending)

getCreatesCompleted

public long getCreatesCompleted()

setCreatesCompleted

public void setCreatesCompleted(long createsCompleted)

getUpdatesCompleted

public long getUpdatesCompleted()

setUpdatesCompleted

public void setUpdatesCompleted(long updatesCompleted)

getErrors

public long getErrors()

setErrors

public void setErrors(long errors)

getPromotionCount

public long getPromotionCount()

setPromotionCount

public void setPromotionCount(long promotionCount)

getSignificantOpsEnqueued

public long getSignificantOpsEnqueued()

setSignificantOpsEnqueued

public void setSignificantOpsEnqueued(long significantOpsEnqueued)

getSignificantOpsDequeued

public long getSignificantOpsDequeued()

setSignificantOpsDequeued

public void setSignificantOpsDequeued(long significantOpsDequeued)

getEnqueuedOperations

public long getEnqueuedOperations()

setEnqueuedOperations

public void setEnqueuedOperations(long enqueuedOperations)

getDequeuedOperations

public long getDequeuedOperations()

setDequeuedOperations

public void setDequeuedOperations(long dequeuedOperations)

getDequeuedItems

public long getDequeuedItems()

setDequeuedItems

public void setDequeuedItems(long dequeuedItems)

getSignificantOpsCompleted

public long getSignificantOpsCompleted()

setSignificantOpsCompleted

public void setSignificantOpsCompleted(long significantOpsCompleted)

getStartTime

public long getStartTime()

setStartTime

public void setStartTime(long updateStart)