What would be a good way to create logs (with python logging module) inside a constant running loop, without producing a large amount of useless log-files?
An example would be a loop that constant list a folder, and does some action when it sees a file of a specific type.
I want to log that no files were found, or files were found but of a wrong type, without logging that same line constantly for each folder check, as it might run many times a second.
Handler that subclasses whatever other functionality you need. Store either the last, or all the previously logged messages that you don't want to
def make_filetype_aware_handler(handler_class): class DontRepeatFiletypeHandler(handler_class): def __init__(self, *args, **kwds): super().__init__(*args, **kwds) self.previous_types = set() def emit(self, record): if not record.file_type in self.previous_types: self.previous_types.add(record.file_type) super().emit(record) return DontRepeatFiletypeHandler FiletypeStreamHandler = make_filetype_aware_handler(logging.StreamHandler) logger = logging.getLogger() logger.addHandler(FiletypeStreamHandler(sys.stderr)) logger.debug('Found file of type %(file_type)', file_type='x-type/zomg')
Log the less important events with a lower precedence, like DEBUG. See setLevel and SysLogHandler.
At development time set the level to DEBUG, and as your application matures, set it to more reasonable values like INFO or ERROR.
Your app should do something about the errors, like remove files of the wrong type and/or create lacking files; or move the wrongfully configured directories from the job polling to a quarantine location, so your log will not be flooded.
import logging logger = logging.getLogger(test) # logging to a file hdlr = logging.FileHandler(test.log) formatter = logging.Formatter('%(asctime)s %(filename)s %(lineno)s %(levelname)s % (message)s') hdlr.setFormatter(formatter) logger.addHandler(hdlr) logger.setLevel(logging.DEBUG)
Then in the loop, you have to check for file type and if file is present or not. Then add :
logger.debug('File Type : %s ' % file_type)
if file_present: logger.debug('File Present : %s ' % present)
My understanding is that you are trying to limit logging the same message over and over again.
If this is your issue, I would create a a set of file_types you have already logged. However you need to be careful, if this is going to run forever, you will eventually crash..
from sets import Set logged = Set() while yourCondition: file_type = get_next_file_type() needToLog = #determine if you need to log this thing if needToLog and (not file_type in logged): logger.info("BAH! " + file_type) logged.add(file_type)