### Navigation

- index
- modules |
- previous |
- FilterPype 0.3.4 documentation »

# API Documentation

The FilterPype API documentation.

## data_fltr_base.py

exception `filterpype.data_fltr_base.``ControllerError`

  Bases: `exceptions.Exception`

  `args`
  `message`

exception `filterpype.data_fltr_base.``DataError`

  Bases: `exceptions.Exception`

  `args`
  `message`

class `filterpype.data_fltr_base.``DataFilter`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilterBase`

  Basic framework for data filter coroutine, sending packets on after receiving them. Override filter_data() for practical functionality. Packet received may be a data packet or a message bottle.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_base.``DataFilterBase`(**kwargs)

  Bases: `object`

  This is the Filter part of the Pipes and Filters design pattern, using Python co-routines to push the data from one filter to the next.

  Each processing step is encapsulated in a filter component. Pipes are the means by which data is passed from one filter to the next, implemented by Unix pipes or FIFO queues. However, using co-routines in Python means that, in simple cases, there is no longer a need for the pipe to buffer data between filters. Each filter will just wait until more data arrives in its yield inbox.

  Note that “filter” is a Python reserved word, so use “data_filter” or some other variation instead.

  The DataFilter has the factory that made it passed in, so if there are missing settings in the validation, it can try to get them from its factory.

  Filters are named with a verb for the transformation, rather than a noun, where possible. So we say CountLoops rather than LoopCounter, ReadBatch rather than FileReader, and Peek rather than Peeker. Then we can talk about the ReadBatch DataFilter.

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_base.``DataPacket`(data='', seq_num=-1, **kwargs)

  Bases: `object`

  The data is passed through the filters in packets. These are Python dictionaries that provide the means to store parameters or partial results, along with the data which is a string.

  Data can be passed to the packet constructor as a string parameter, or as data=’xxx’ keyword parameter.

  TO-DO: Test cloning for efficiency <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  When packets are split up, the parameters need to be passed along, with the revised data. This is done by cloning the packet.

  `clear_data`()

    Clear data from packet, because we can’t do that in clone()

  `clone`(data='', **kwargs)

    Create a new packet, with the same parameters and data. We expect data as the argument. There may be keywords. If data is given, set the data string to this value.

    TO-DO: Cloning message bottles?

    ROBDOC : When sending on data it would be useful if the data keyword would raise an exception if no data is in the packet OR just allow there to be no data in the packet (rather than cloning the original data if you provide data as ‘’). This depends on whether you’re going to allow a packet without any data:

    e.g.

        clone(self, data=None, ..):
        ..
        if data is not None:
            if data is '':
                raise NoDataInPacketException, 'data is required in the packet'
            cloned_packet.data = data

  `data_length`

    Length of data, if a string

class `filterpype.data_fltr_base.``DynamicMetaClass`(name, bases, ns)

  Bases: `type`

  Use this metaclass, derived from “type”, to create the class. This sets a flag for checking that it is being used, and sets __getattribute__ to ensure dynamic processing. This is variable at run time, as to whether we use the metaclass or not, but is not reversible.

  Note that __getattribute__ is a class attribute, not an instance attribute. If we set the instance attribute, it is ignored and thus not dynamic.

  `mro`() → list

    return a type’s method resolution order

class `filterpype.data_fltr_base.``DynamicMixIn`

  Bases: `object`

  Additional base class for Filters, where each attribute access is checked for beginning with “%”, i.e. requiring a dynamic value. Dynamic property is not easily reversible.

exception `filterpype.data_fltr_base.``FilterAttributeError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``FilterError`

  Bases: `exceptions.Exception`

  `args`
  `message`

exception `filterpype.data_fltr_base.``FilterFactoryError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``FilterLogicError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``FilterNameError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``FilterProcessingException`

  Bases: `exceptions.Exception`

  An exception was caught and raised from while processing data in a Filter.

  `args`
  `message`

exception `filterpype.data_fltr_base.``FilterRoutingError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

class `filterpype.data_fltr_base.``HiddenBranchRoute`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Route the data packets according to their fork_destination. ‘main’ goes on to the next_filter as usual. ‘branch’ goes to branch_filter, raising an exception if there is none. ‘prev’ may link to the filter before the branch.

  This is called Hidden... because you shouldn’t need to use it explicitly. Since it is never created by ftype, we need to set ftype manually.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'hidden_branch_route'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_base.``MessageBottle`(destination, message, single_use=True, **kwargs)

  Bases: `filterpype.data_fltr_base.DataPacket`

  Message is a type of data packet, usually with only one-time use. MessageBottle —–TO-DO—– e.g. target = pad_bytes, message = reset:byte_count:100

  @destination : can be the filter name or the filter type. If filter type, the message is forwarded onto the rest of the pipeline main and branch so that other filters of the same type can open the message.

  `clear_data`()

    Clear data from packet, because we can’t do that in clone()

  `clone`(data='', **kwargs)

    Create a new packet, with the same parameters and data. We expect data as the argument. There may be keywords. If data is given, set the data string to this value.

    TO-DO: Cloning message bottles?

    ROBDOC : When sending on data it would be useful if the data keyword would raise an exception if no data is in the packet OR just allow there to be no data in the packet (rather than cloning the original data if you provide data as ‘’). This depends on whether you’re going to allow a packet without any data:

    e.g.

        clone(self, data=None, ..):
        ..
        if data is not None:
            if data is '':
                raise NoDataInPacketException, 'data is required in the packet'
            cloned_packet.data = data

  `data_length`

    Length of data, if a string

exception `filterpype.data_fltr_base.``MessageError`

  Bases: `exceptions.Exception`

  `args`
  `message`

exception `filterpype.data_fltr_base.``PipelineConfigError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``PipelineError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``PipelineRecursionError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

class `filterpype.data_fltr_base.``PriorityQueue`

  Bases: `object`

  Priority queue to enable looping, using TankQueue and TankFeed. List is sorted by heapq, using priority as the first sort field. If priorities are the same, then time is used to distinguish order. See p.208 of Python Cookbook, 2nd ed.

  `clear`()

    Empty the queue of all items

  `pop`()

    TO-DO: Establish why time_posted isn’t used to sort priority of responses when multiple priority values are the same. Note: This only appeared on WinVista.

  `push`(item, priority=None)
  `push_none`()

    Push None on to the queue, with maximum (negative) priority so that it goes to the front of the queue. This is called when the queue size is changed by padding the front.

  `queue_size`()
  `sorted_items`()

exception `filterpype.data_fltr_base.``ResponseError`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``SerialNumberInvalidError`

  Bases: `filterpype.data_fltr_base.DataError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``SerialNumbersInconsistentError`

  Bases: `filterpype.data_fltr_base.DataError`

  `args`
  `message`

exception `filterpype.data_fltr_base.``TankOverflowException`

  Bases: `filterpype.data_fltr_base.FilterError`

  `args`
  `message`

`filterpype.data_fltr_base.``closer`(*args, **kwds)

`filterpype.data_fltr_base.``dynamic_params`(static_class)

  Class decorator to add the functionality to look up from the embedded Python environment the current values of attributes whose apparent values start with “%” and the name is upper case.

  Usage: put “@dynamic_params” on the line before the class decoration

  Problem is that this changes the static class for all uses of it. We may not want all instances dynamic.

`filterpype.data_fltr_base.``mix_in`(base, addition)

  Mixes in place, i.e. the base class is modified. Tags the class with a list of names of mixed members.

`filterpype.data_fltr_base.``mix_in_copy`(base, addition)

  Same as mix_in, but returns a new class instead of modifying the base.

## datafilter.py

class `filterpype.data_filter.``AttributeChangeDetection`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Examines a list of attributes for change in value. Sets the packet_change_flag attribute to True upon change of any of the attributes, otherwise flag stays False. AttributeError raised if packet has not got all attributes. Change of initial value.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'attribute_change_detection'
  `init_filter`()
  `keys` = ['attributes', 'packet_change_flag:attribute_changed', 'compare_initial_value:True']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``AttributeExtractor`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Extract attributes from text strings using a delimiter to determine the split between key (on the left) and value (on the right). Key has all punctuation removed and spaces replaced with underscores and is assigned to the packet dictionary.

  Sample input: ‘ someone’s Gender : is maiL’ Packet output: packet.someones_gender = ‘is maiL’

  Beware: do not override reserved attributes within the packet (such as ‘data’)

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'attribute_extractor'
  `init_filter`()
  `keys` = ['delimiter:none']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BZipCompress`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Take the input stream and compress it using bzip2 compression object. Use level 9 for large files (this is the default).

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Flush compression buffer before closing

  `ftype` = 'bzip_compress'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``BZipDecompress`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Take the input stream and decompresses it using bzip2.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'bzip_decompress'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``Batch`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Input is a series of strings of any length, with header removed. Split up data into blocks.

  There are two general cases to cope with: a) the input string is larger than the block size b) the input string is smaller.

  To avoid having two different approaches, we use an inputs list as a buffer: 1) Repeatedly put the packet data into inputs until chars_in >= size 2) Join inputs to be one string. 3) Split string into blocks, leaving a remainder. 4) Send on each block. 5) Put remainder back as the only item in inputs.

  Wrong: An added feature is that we can set an initial_branch_size that sends the first N bytes off to a branch. This enables us to strip off junk at the beginning of a block of data, by sending it to a branch where it goes to waste. Alternatively, it can resynchronise frames of data, by pointing the branch and the main to the same filter following in the pipeline.

  ##We can set more than one batch size, using the size parameter for a list

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Clone the incoming packet and store within self._last_packet for flush_buffer().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    Split the data, allowing size and fork_dest to be changed each loop. We need to check batch size each loop, in case it has been changed to a silly value, e.g. 0, which will just carry on sending the same data ad infinitum.

  `flush_buffer`()
  `ftype` = 'batch'
  `init_filter`()
  `keys` = ['size', 'fork_dest:main']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Zero batch size gives ValueError: range() step argument must not be zero

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchClone`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Clones (duplicates) the packet, sending one copy to main, one to branch. BranchClone filter should be followed by a HiddenBranchRoute filter, i.e. “(” in the route.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_clone'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchDynamic`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Sends along the main and/or branch depending on two variables within the embedded environment. If the variables have not been set, this filter will die loudly raising an AttributeError.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_dynamic'
  `init_filter`()
  `keys` = ['main_variable:MAIN', 'branch_variable:BRANCH']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchFirstPart`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Send the first part of the packet.data to the branch, and the rest to main. This differs from DistillHeader because the amount to send is read from the packet, and to keep things simple, must be >= packet data length.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_first_part'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchIf`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Decide on branching, dependent of the name of an attribute ‘branch_key’, found either in the filter or the packet. Optional key is ‘branch_on_packet’, with default of True. If branch_on_packet is False, then use filter instead of packet. If the attribute is not present, a KeyError will be raised. BranchIf should be followed by HiddenBranchRoute filter, i.e. “(” in route.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_if'
  `init_filter`()
  `keys` = ['branch_key', 'comparison:equals', 'compare_value:true', 'branch_on_packet:true']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchOnceTriggered`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Sends packet down the branch forever after the first case where watch_attribute does not equal watch_value.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_once_triggered'
  `init_filter`()
  `keys` = ['watch_attribute', 'watch_value:true']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchParam`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Send parameter results to the branch. Optionally send only some of the list items.

  ROBDOC : sorry, what does this do? Filter a list using slice / dice? CJ

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_param'
  `init_filter`()
  `keys` = ['param_name', 'start:0', 'stop_before:9999', 'step:1']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchRef`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Send the packet object to both branch and main. This is without cloning it, i.e. this is not a copy but a reference to the same object! Any changes made by the branch will affect the packet it refers to. BranchRef filter should be followed by a HiddenBranchRoute filter, i.e. “(” in the route.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_ref'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BranchUntilValue`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'branch_until_value'
  `init_filter`()
  `keys` = ['value']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``BreakPoint`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Debugging in a pipeline that has no convienently placed filters can be difficult so this is a filter purely for being able to insert a break point in

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `dynamic` = True
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'break_point'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CalcSlope`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Calculate the rate of change of a parameter, over five consecutive values, given a list of packets, from which we can get

  > [h0, h1, h2, h3, h4]

  We use the formula:
    mean_slope_for_h2 = (h4 - h0 + h3 - h1) / 6.0

  The packet arriving should contain references to the five packets we need to differentiate. They are still continuing in the main pipeline. If there are not precisely five packets, then pass on the grouping packet for other calculating methods.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(grouping_packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'calc_slope'
  `init_filter`()
  `keys` = ['calc_source_name', 'calc_name_suffix:slope']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``Calculate`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Simple calculator for two numbers

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'calculate'
  `init_filter`()
  `keys` = ['lhs_value', 'operator:add', 'rhs_value', 'param_result']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CallbackOnAttribute`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Watches packets for a specified watch attribute and calls the provided callback method with the attribute value as a parameter,

  e.g. If watch_attr = ‘holy’ and no environ provided, the following callback will be made if a packet arrives with the ‘holy’ attribute with the value ‘grail’

  self.callback(‘found:holy’, holy=’grail’)

  num_watch_pkts is the number of packets which can pass through the filter until it will stop watching for the attribute. If the attribute has not been found by this point, it will return a callback:

  self.callback(‘not_found:holy’)

  allowed_inconsistencies: self.callback(‘inconsistency_value_exceeded:holy’)

  If num_watch_pkts is None (default) it will watch forever and only return a not_found callback when closing the filter.

  count_to_confirm is the number of identical values of the watched attribute required to pass through the filter before it will return a callback.

  watch_for_change will make a callback only when the attribute changes it’s value from the previous (including the first assignment of the value in the first packet).

  include_in_environ allows you to provide a list of additional packet attributes to include in the environ used in the callback. (watch_attr is always included where available)

  if the parameter is not found by the time the pipeline closes down, the response is made “not_found:<watch_attr>”

  if the value of watch_attr is None, it is ignored (passed through).

  changes the refinery send() method’s return value to ‘found’, ‘not_found’ or ‘inconsistant’

  print_when_callback allows you to aid your debugging by printing to stdout when a callback is made

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Make a not_found callback if the attribute was not found and num_watch_pkts is set to None (which watches forever) and we haven’t made an inconsistent call

  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    Used by open_message_bottle too - but as message bottles get sent on automatically, we don’t want to send it if there is a message in the packet. (TO-DO discuss) <– there should be a tidier way of doing this!

    TO-DO: Split the getattr and functionality within try except into seperate private function, then send_on can be handled outside of that function.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'callback_on_attribute'
  `init_filter`()
  `keys` = ['watch_attr', 'callback', 'environ:none', 'count_to_confirm:1', 'num_watch_pkts:none', 'allowed_inconsistencies:0', 'watch_for_change:false', 'include_in_environ:[]', 'close_when_found:false', 'print_when_callback:false']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(msg_bottle)

    This must do something very similar to the filter_data class, but not send_on the message bottle - as this is arranged for us in the DataFilter base class.

    Work around for the above is to only send on in filter_data when there is a message attached to the packet.

    Currently it will open all messages from the msg_bottle. this may or may not be a good idea, but it does allow for items to be clocked up on the “not” found count. <– however, it is also worth noting that if the message destination wasn’t set and it defaulted to this ftype, it may get more messages than intended.

    will count_to_confirm etc. etc. work? are they sharing resources with filter_data counts?

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CallbackOnMultipleAttributes`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Simplified version of data_filter’s CallbackOnAttribute which will make a callback based on multiple attributes. watch_attrs is comma separated, though is parsed as a list by FilterPype. Will only make the callback once when all attributes have been found.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'callback_on_multiple_attributes'
  `init_filter`()
  `keys` = ['watch_attrs', 'callback', 'environ']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CentrifugeOne`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Extract the data from each packet into an extract dictionary. A centrifuge map is a dictionary in the form

  > key_attribute_name = tuple_with_extraction_parameters e.g. = (word_no, high_bit_no, low_bit_no) superframe_number = (3, 8, 1)

  Note that all numbering systems from the analysts point of view are 1-based, so have to be converted to 0-based before use.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'centrifuge_one'
  `init_filter`()
  `keys` = ['centrifuge_map']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CollectData`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Record data from each passing packet, until a maximum collection size is reached. Then send to the branch a packet with data of all the values in a list.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'collect_data'
  `init_filter`()
  `keys` = ['collection_size:5']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``Combine`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Combine a list of fields or constants into one target field. Source field names will use the value of the field, which is prefixed either with ‘f.’ for filter attribute, or ‘p.’ for packet attribute.

  > source_field_names = f.header_prefix, p.seq_num, XXXX

  produces

  > ===+++ 27 XXXX

  If the source field name is not prefixed with f. or p., it will be treated as a constant. Target field name always starts with f. or p.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'combine'
  `init_filter`()
  `keys` = ['source_field_names', 'target_field_name']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``ConcatPath`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Concatinate path

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'concat_path'
  `init_filter`()
  `keys` = ['input_path', 'end_part', 'variable_name']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``ConvertBytesToInt`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Convert a string of hex values (e.g. ‘x01x02x03’) into a integer representation of the given string.

  NOTE: Using exscape characters in rst is not allowed so the hex values
    below AND above would normally be escaped
  E.g. ‘x00’ –> 0
    or ‘x00x81’ –> 129
  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'convert_bytes_to_int'
  `init_filter`()
  `keys` = ['param_names']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``ConvertFilenameToPath`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Join a file path to a file name to give a full file name

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'convert_filename_to_path'
  `init_filter`()
  `keys` = ['input_file_name:none', 'in_attr', 'out_attr']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CountBytes`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Count the number of bytes passing a filter. Unlike SeqPacket and CountLoops, nothing is written to the packet, just to the filter. Count packets as well, but not in a custom field.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'count_bytes'
  `init_filter`()
  `keys` = ['count_bytes_field_name:counted_bytes']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``CountLoops`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Give each packet a number that starts at 1 and increments for each pass. This is used for counting loops

  Contrast this with CountPackets that records in the filter the number of packets going past.

  Also contrast this with SeqPacket that gives the packet an ID number. The next number is taken from the filter, and the packet number is not overwritten if it already exists.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'count_loops'
  `init_filter`()
  `keys` = ['count_loops_field_name:loop_num']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``CountPackets`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Count the number of packets passing a filter. Unlike SeqPacket and CountLoops, nothing is written to the packet, just to the filter.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'count_packets'
  `init_filter`()
  `keys` = ['count_packets_field_name:counted_packets', 'include_message_bottles:false']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``DataLength`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Calculates the length of data that has passed through the filter.

  When closing the filter, it sends a message bottle to the msg_destin which defaults to the ftype of callback_on_attribute.

  not_data must be a single chr value data_length(0 to 256)

  single_use means that the message bottle it sends will be used by the first msg_destin found (i.e. defaults means the first callback_on_attribute it comes across)

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'data_length'
  `init_filter`()
  `keys` = ['not_data:0x00', 'msg_destin:callback_on_attribute', 'single_use:true']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``DedupeData`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Takes a list of data as an input, and outputs the set of different values. This can be used to ensure that a parameter read multiple times has the same value.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'dedupe_data'
  `init_filter`()
  `keys` = ['start:0']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``DistillHeader`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Strip header off, and send header_size bytes to the branch. If distill_mode is “once” then the header will be removed only once, from the first packet, while if distill_mode is “repeated”, it will removed from each packet that is received.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'distill_header'
  `init_filter`()
  `keys` = ['header_size', 'distill_mode:repeated', 'keep_header_key:none']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``EmbedPython`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  EmbedPython TO-DO

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'py'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `subst_keys`(line)

    Replace keys from the enclosing pipeline where they are of the format “${fred}”

    ROBDOC: Will only work on strings / primitive data types (not objects)

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``FormatParam`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Format results received in a list, to be passed on as a string. param_name may be an attribute of the results packet.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'format_param'
  `init_filter`()
  `keys` = ['format']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``GetBytes`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Get the raw data value for the target byte range.

  Mandatory keys are: start_byte (int) bytes_to_get (int) param_name (string) Uppercase please

  NOTE: All counting is base 0 for developers. Data extracted is sent to branch;

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'get_bytes'
  `init_filter`()
  `keys` = ['start_byte', 'bytes_to_get', 'param_name']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``HashSHA256`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Takes the input stream and computes a hash using the sha-256 algorithm.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'hash_sha256'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``HeaderAsAttribute`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  In cases where the header is required for further processing later in pipeline, this filter assigns the header_attribute to the packet, while the packet’s data will no longer contain the header.

  The send_on_if_only_header key defines whether this filter should send_on packets if there is not enough data to split into both header and remaining data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'header_as_attribute'
  `init_filter`()
  `keys` = ['header_size', 'header_attribute:header_data', 'send_on_if_only_header:false']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``InitialValueChangeDetection`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Takes in a dict of attribute_name:value and

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'initial_value_change_detection'
  `init_filter`()
  `keys` = ['attribute_map', 'packet_change_flag:attribute_changed']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``Join`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Store the data items that are strings, until receiving something where the data is not a string, typically None. Then join the strings together, using space as the default join string.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()
  `ftype` = 'join'
  `init_filter`()
  `keys` = ['join_str:space']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``PassNonZero`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Check that the first n bytes of data for being non-zero (i.e. not 0x00 or 0xFF). If the first n bytes are 00/FF and all the same, don’t pass on the data.

  If the length of the data packet is less than check_byte_count, then the packet should pass, because the check has failed, even if the data is 00/FF.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'pass_non_zero'
  `init_filter`()
  `keys` = ['check_byte_count:32', 'attribute:data']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``PassThrough`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  A pass through filter just forwards all packets to the next filter. There are various uses for a null node, such as being able to redirect pipeline flow while the pipeline is active. Alternatively, it may be used for simulating multiway branching, with a syntax built around binary branching.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'pass_through'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``Peek`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Look ahead some bytes into the next data packet. Record the bytes found in the packet, if bytes are found, or an end of file marker <<<TO-DO<<< if the data has finished.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'peek'
  `init_filter`()
  `keys` = ['peek_ahead']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``PrintParam`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Format results received in a list, to be passed on as a string. param_name may be an attribute of the results packet.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'print_param'
  `init_filter`()
  `keys` = ['label:empty']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``R111esetBranch`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Reset to zero an attribute in the next filter.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'r111eset_branch'
  `init_filter`()
  `keys` = ['branch_filter_field_name']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Check that we have reset followed by a branch, in order to have somewhere to send the reset to.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``ReadBatch`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Chop file up into string blocks to pass inside packets into pipeline.

  We need a file object to read from. This can be provided directly or indirectly, with either the open file object or the file name being sent to ReadBatch.

  This can also be done by setting the source_file_name as a fixed parameter for the filter, but this stops more than one file being read and doesn’t fit so well with the idea of data filters.

  The file object passed in (or opened) signalled for closing in three ways: 1) The file has been consumed by reading, so the next read() returns a block of zero length. 2) The refinery is shutting down (checked on each read() loop). Whether or not the reading is finished, the file should be closed.

  NOTE: This should not be used in the middle of a pipeline as it creates
    a brand new packet and does NOT pass on the packet that it was originally supplied with. In other words, use this at the start of a pipeline or in an external wrapper pipeline.

  Parameters: | max_reads (int) – Number of batches to read.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'read_batch'
  `init_filter`()
  `keys` = ['batch_size:0x2000', 'max_reads:0', 'initial_skip:0', 'read_every:1', 'binary_mode:true', 'source_file_name:none', 'file_size:none', 'print_progress:false']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``ReadBytes`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Simple ReadFile filter to read files using bytes

  Notes about keys:

  source_file_name - the source file to read from start_byte - the starting byte. Must be positive integer. size - number of bytes to read.

  > zero : read nothing positive int : read this number of bytes negative int : read all of file (covention is to use -1)

  block_size - size in bytes to read at a time. whence - where to seek from:

  > 0 - Start of file 1 - End of file

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'read_bytes'
  `init_filter`()
  `keys` = ['source_file_name', 'start_byte:0', 'size:-1', 'block_size:2048', 'whence:0', 'ack:false']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``ReadLines`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Read lines of a text file, in normal or reversed order. For a first implementation, this requires the whole file being in memory, but this could be optimised.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'read_lines'
  `init_filter`()
  `keys` = ['direction:forwards', 'initial_skip:0', 'read_every:1', 'max_lines:0']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``RenameFile`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Rename file, with from/to names passed in as packet data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'rename_file'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``Reset`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Send a message bottle to a downstream filter, to reset one of the parameters. The message bottle will pass along the pipeline until it gets to a filter where the name matches. Then the parameter will be set to the new value.

  The ‘value’ key can be either a proper value (such as 3 or ‘hello’) or the name of a packet attribute where the value will be retrieved. The packet attribute takes precedence over the Filter key.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'reset'
  `init_filter`()
  `keys` = ['target_filter_name', 'param_name', 'value:none']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``ReverseString`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Reverse a string (in packet.data) that is of at least 2 characters long. The result will be stored back into packet.data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'reverse_string'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``SendMessage`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Send a message in a bottle to a downline filter. The message bottle will pass along the pipeline until it gets to a filter where the name matches. Then the parameter will be set to the new value.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'send_message'
  `init_filter`()
  `keys` = ['message', 'values:[]']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``SeqPacket`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Give each data packet a sequential number (e.g. as an ID), starting from 0 or whatever reset_counter_to is set to. To allow for looping, the seq_packet_field_name will not be overwritten. If you need to add another numberer, use a different field.

  Message bottle packets don’t have to be handled, because they are not sent to filter_data().

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'seq_packet'
  `init_filter`()
  `keys` = ['seq_packet_field_name:seq_num', 'field_width:6']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `reset_counter`(reset_to=0)
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``SetAttributesToData`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Creates delimited data from attribute lists.

  Changes a list of attributes (columns) and joins them into a list of rows. Each row is joint together, delimited by the seperator (default ‘,’ comma) . All the rows are then joint with the end of line ‘eol’ seperator and put into the packet.data.

  Where write_field_headers is True, attribute names are used for field headers at the start of the data.

  It will accept attribute lists of different lengths. When a list runs out of entries, it is padded with ‘ ‘ (space char) values at the end.

  output_format will apply a formatting to each and every value. str : applies string representation of value (default) int : applies integer base 16 to value (requires int / str of an int) hex : applies hexidecimal to value (requires integers) bin : applies binary to value (requires integers) oct : applies octal to value (requires integers)

  method_name is the name of a method to call on an object if the attribute list contains objects rather than literal values. Method name shall not require any arguments. If it does then it’ll fail as it’s supposed to be for simple getter style methods, not complicated things.

  - TO-DO: Replace str with a safe_string conversion so that ascii chars such

  as “delete” etc are not written to file!

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
    Purpose is to transpose lists of attribute lists into CSV data.
      Doing this by creating new lists of the lists is rather slow (lots of list generation with .append etc).

    # http://code.activestate.com/recipes/410687/ #def transposed(lists):

    > #if not lists: return [] #return map(lambda *row: list(row), *lists)

    def transposed2(lists, defval=0):
      if not lists: return [] return map(lambda *row: [elem or defval for elem in row], *lists)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'set_attributes_to_data'
  `init_filter`()
  `keys` = ['attribute_list:[]', 'write_field_headers:True', 'separator:,', 'eol:ret', 'line_numbers:False', 'output_format:str', 'method_name:none', 'object_attr:none', 'default_value:']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``Sink`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Store the output of the previous filter in a results list. Check the latest addition to the list with results[-1].

  Set max_results to limit the number of results stored in the list. If there are more than max_results, the oldest gets popped off the top of the list. If max_results is 0, then there is no limit, and all results are stored. The default limit is 20.

  Set the sink’s capture_msgs=True in order to capture MessageBottles in addition to DataPackets.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `all_data`

    List of sink packet data

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'sink'
  `init_filter`()
  `keys` = ['max_results:30', 'capture_msgs:false']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)
  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')
  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``Sleep`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Simply sleeps for the specified time and sends on incoming packets. Intended for debugging purposes, watching printed output to understand the flow of a pipeline.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'sleep'
  `init_filter`()
  `keys` = ['time:1']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``SplitLines`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Split the data provided into lines. Only sends on lines which have data within them. It will include any whitespace on the line.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'split_lines'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``SplitWords`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Split the data into chunks, looking for some character string to split on. Uses white space as the default.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'split_words'
  `init_filter`()
  `keys` = ['split_on_str:None']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``SwapTwoBytes`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Reverse a string (in packet.data) that is of at least 2 characters long. The result will be stored back into packet.data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'swap_two_bytes'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``TagPacket`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Apply a tag to each packet, from a tag_field_name.

  This tag value is updated by a send_tag filter setting the tag property.??

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'tag_packet'
  `init_filter`()
  `keys` = ['tag_field_name', 'tag_field_value']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``TankBranch`(**kwargs)

  Bases: `filterpype.data_filter.TankQueue`

  TankQueue that sends references to its packets held to the branch every time a packet is received. This enables the branch to process a sliding window, wihtout duplicating any data.

  `after_filter_data`(packet)
  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `all_data`

    Tank packet data, sorted and concatenated

  `before_filter_data`(packet)
  `before_send_on`(packet, fork_dest)

    Allow proper processing of data being flushed down main. Avoid catching the packet lists, which are normally sent down the branch.

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()
  `ftype` = 'tank_branch'
  `init_filter`()
  `keys` = ['tank_size:0', 'priority_field_name:seq_num']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pop`()
  `push`(packet)

    Each packet into the priority queue should be numbered, but if not numbered, we need to set priority to 0. In this case, the priority ordering will be on the time.time() set at time of push. Or should not being numbered raise an exception?? TO-DO

    PriorityQueue now gives a sequential number, rather than 0, so this is the default sorting mechanism.

    Packets are pushed with a given priority, whereas None is pushed with a special negative priority, to ensure it comes at the front.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `sorted_packets`

    All data in tank packet, sorted in priority order

  `spare_capacity`

    Count of packets needed to match tank_size

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `tank_size`

    Adjust the tank size to change the number of packets held.

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``TankFeed`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Take a (yield)ed packet and send it to a tank_queue, but not through the (yield), to avoid ValueError: “generator already executing” ROBDOC: This does not explain the main purpose of this filter.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'tank_feed'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    To avoid the locking up of the coroutine links by circular calls, we push packets on to a tank queue instead of sending them. But the destination is taken from the same pipeline structure.

class `filterpype.data_filter.``TankQueue`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Store the input packets in a priority queue, waiting until tap_open before sending on the packets.

  This description is in bits, and reads like it: pretty incomprfwehensbleuhg

  This decoupling is needed to allow coroutines to loop.

  All packets should have been numbered by a previous filter, to ensure that the earlier packets are processed first, in particular, that looping packets are processed before new ones. If the packets haven’t been numbered before, they are given sequence number zero

  <<<<<<<<<<<<<<<<<<< TankQueue <<<<<<<<<<<<<TO-DO<<<<<<<<<<<

  If the factorial hasn’t recursed down to 1, it branches to tank_feed, to calculate one less factorial. tank_feed put the new values back into the tank_queue, by pushing directly on to the tank_queue’s priority queue. The tank_queue gives the packets a sequential number to use in the priority queue, to ensure that all the recursive calculations on one packet are finished before the next packet is processed.

  The packet coming into the tank_queue by the normal (yield) is pushed on to the queue. Then a while-True loop takes all the packets out of the queue for processing, which means that it completes the entire recursion before going back for the next (yield).

  Fill the tank with up to tank_size packets. When next packet arrives, the first one in is sent on.

  When tank_size is changed, either the excess packets are sent on, or the front end is filled out with None.

  ##The normal send_now() behaviour is to queue all the packets until the ##current filter_data() function has been completed. This doesn’t work for ##recursive use of the tank, because the packets must keep recursing round. ##Therefore we have a send_now key which can be set to True when needed.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `all_data`

    Tank packet data, sorted and concatenated

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()
  `ftype` = 'tank_queue'
  `init_filter`()
  `keys` = ['tank_size:0', 'priority_field_name:seq_num']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pop`()
  `push`(packet)

    Each packet into the priority queue should be numbered, but if not numbered, we need to set priority to 0. In this case, the priority ordering will be on the time.time() set at time of push. Or should not being numbered raise an exception?? TO-DO

    PriorityQueue now gives a sequential number, rather than 0, so this is the default sorting mechanism.

    Packets are pushed with a given priority, whereas None is pushed with a special negative priority, to ensure it comes at the front.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `sorted_packets`

    All data in tank packet, sorted in priority order

  `spare_capacity`

    Count of packets needed to match tank_size

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `tank_size`

    Adjust the tank size to change the number of packets held.

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_filter.``Transmit`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Reset to zero an attribute in the next filter, to a value dependent on the current packet.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'transmit'
  `init_filter`()
  `keys` = ['target_filter_field_name', 'source_packet_field_name:0']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``UseDataToTag`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Send tag property of incoming packet data to a tag_packet filter, by setting its <tag_field_name> attribute.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `init_filter`()
  `keys` = ['dest_tag_packet']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``Waste`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  A waste filter just throws away all the packets it sees. This is used when combining results from branches and the main stream is not wanted.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'waste'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``Wrap`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Wrap packet data with a prefix and/or suffix. This can be used for inserting periodic strings (e.g. creating a regular file header within a stream of data) or padding missing data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'wrap'
  `init_filter`()
  `keys` = ['data_prefix:empty', 'data_suffix:empty', 'wrap_mode:repeated']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``WriteConfigObjFile`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Creates a ConfigObj object and adds new sections to it from incoming packet’s data. Only accepts packet.data as a dictionary in the following format:

  { config_obj_section_name (str) : config_obj_section (dict)* }

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)
  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    When the pipeline is shutting down, write out self._config_obj if write_config is True.

  `ftype` = 'write_config_obj_file'
  `init_filter`()
  `keys` = ['dest_file_name', 'dest_file_suffix', 'write_config:True']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_filter.``WriteFile`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Write data from all packets to an external file.

  Write the data to the file, opening the file if necessary first. File opening is left to this point, to allow the changing of the output file until after the initialisation of the generator.

  So how do we know when the file has finished and needs closing? The input to read_batch could be many files, all to be written to one output file. We can’t use a timeout, so closing needs to be done explicitly, or via the closure of the pipeline.

  By providing a message bottle with the message ‘change_write_suffix’ and the packet attribute ‘packet.file_name_suffix’ the current file will be closed and a new one will be opened with the suffix appended to it.

  compress : currenly only ‘bzip’ is enabled (true / bzip resolves to bzip2)

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)
  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `close_output_file`()
  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'write_file'
  `init_filter`()
  `keys` = ['dest_file_name', 'append:False', 'binary_mode:true', 'do_write_file:True', 'compress:none']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)
  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

## filter_factory.py

class `filterpype.filter_factory.``DemoFilterFactory`

  Bases: `filterpype.filter_factory.FilterFactory`

  Make filters for the test spike

  `abstract_class` = False
  `create_filter`(param_dict_or_ftype, pipeline=None)

    Extract the right filter dictionary from the factory_dict. Store a reference in the object to the factory that made it. Pass in the filter_attrs so the new object has right attributes. Alternative parameter is just the ftype. Dictionary validation is enforced before the first use.

    TO-DO: pipeline is (a str? an object? required??)

class `filterpype.filter_factory.``FilterFactory`

  Bases: `object`

  Factory class for making Filters and Pipelines

  `abstract_class` = True
  `create_filter`(param_dict_or_ftype, pipeline=None)

    Extract the right filter dictionary from the factory_dict. Store a reference in the object to the factory that made it. Pass in the filter_attrs so the new object has right attributes. Alternative parameter is just the ftype. Dictionary validation is enforced before the first use.

    TO-DO: pipeline is (a str? an object? required??)

## filter_utils.py

`filterpype.filter_utils.``abs_dir_of_file`(abs_file_location)

  Returns the absolute path of the package containing abs_file_location, usually found from __file__.

`filterpype.filter_utils.``all_files`(pattern, search_path, path_sep=':')

  Given a search path, yield all files matching the pattern. From Python Cookbook, p. 92.

`filterpype.filter_utils.``bcd`(values, space='')
  Convert input values (as a list) into a BCD string.
    Note: If number of values is 2 or 3 use bcd3 or bcd2 they work 2x faster.

  Glen - 07/01/10 - Changed logic incase space is an empty string.

`filterpype.filter_utils.``bcd1`(x)

  Single character bcd conversion.

`filterpype.filter_utils.``bcd2`(x, y, space='')

`filterpype.filter_utils.``bcd3`(x, y, z, space='')

  e.g. bcd3(‘A’,’B’,’C’,space=’ ‘) -> ‘41 42 43’

`filterpype.filter_utils.``bit_sum`(chars)

  Add the bits in all chars

`filterpype.filter_utils.``config_obj_comma_fix`(text)

  ConfigObj puts strings with commas into a list, converting to integers where possible. Reconstruct the string with string types. Rejoin the list items with a comma, if present.

`filterpype.filter_utils.``construct_filename`(*args, **kwargs)

`filterpype.filter_utils.``convert_config_list`(values)

  Apply convert_config_str for each item in the list.

`filterpype.filter_utils.``convert_config_str`(value)

  With string from config file, interpret it in order: 1) as a number (integer, float or hex) 2) as Boolean 3) as space 4) list 5) dictionary 6) else return original string.

`filterpype.filter_utils.``convert_integers_to_hex`(int_list, mode=4)

  Use this function for converting a list of integers into a hex representation in a format determined by mode.

  E.g. If input is [1, 2, 3] Mode Format 1 [‘0x1’, ‘0x2’, ‘0x3’] 2 ‘0x010203’ 3 ‘x01x02x03’ 4 ‘01 02 03’

`filterpype.filter_utils.``convert_to_2s_complement`(str_val)

  Convert a data string into it 2s complement value E.g. 0xF22C4158 is the hex string 0xF2 0x2C 0x41 0x58 0xF2 0x2C 0x41 0x58 –> -231980712 (in 2s complement)

  NOTE: values must be strings NOTE: Please only give chars (1 byte), shorts (2 bytes) or

  > longs (4 bytes). Anything else will currently return an error

`filterpype.filter_utils.``convert_to_zero_based_tuple`(tup)

  For each number in the tuple, return one less, converting 1-based counting to 0-based.

`filterpype.filter_utils.``copy_attr`(from_obj, to_obj, attr_name)

`filterpype.filter_utils.``data_to_hex_string`(data, limit=None, space=' ')

  Return the hexadecimal representation of a string of bytes, so that it can be printed and compared with the display of a hex editor.

  e.g. ‘ABC’ –> ‘41 42 43’

`filterpype.filter_utils.``data_type`(data)

  Return the type of data passed in.

`filterpype.filter_utils.``dbg_print`(text, level=5)

  Print the text to std_out if “debug” in this module is >= level. i.e. If level is 0, then it will always be printed. The higher the level, the less likely it is to be printed. The “>” is inserted just to show which print statements have been converted to dbg_print.

`filterpype.filter_utils.``define_hex_conversions`()

`filterpype.filter_utils.``destruct_filename`(filename)

`filterpype.filter_utils.``get_values_from_name`(filter_name)

  Essential key values may be set after a ‘:’ for the each essential key. e.g. foo:35:fred:${baz} This provides three key values to the filter foo: 35 (as an integer), ‘fred’ (as a string) and baz as a variable to be substituted for.

`filterpype.filter_utils.``get_word_interval_num_results`(wps, hz)

  Get the word interval and number of results per superframe, given the words per second and hertz

  WARNING: This function returns an int. If the user does not give

    sensible values and a calculation is made that results in non-zero values on the right hand side of the decimal point, an inaccurate result will be given.

    EG: Inputs: WPS:256, HZ:3 Real Interval: 85.333333333333329, Real num results: 48.0 Function Output: (85, 48)

`filterpype.filter_utils.``hex_string_to_data`(data, space=None)

  Read string data as hexadecimal bytes, and return as data string. e.g ‘41 42 43’ –> ‘ABC’

  Note the use of the int() function, with a second parameter. I hadn’t realised that this was necessary for converting anything apart from base 10 strings.

`filterpype.filter_utils.``htonl`(num)

  Converts a long from network order (big-endian) to a string of length 4 in host order.

  Equivalent of C OS library function of the same name.

  Source: http://www.java2s.com/Tutorial/Python/0280__Buildin-Module/NetworkByteOrder.htm

`filterpype.filter_utils.``htons`(num)

  Converts a short from network order (big-endian) to a string of length 2 in host order.

  Equivalent of C OS library function of the same name.

  Source: http://www.java2s.com/Tutorial/Python/0280__Buildin-Module/NetworkByteOrder.htm

`filterpype.filter_utils.``latest_defaults`(keys)

  Take a list with some repeated keys which have different default values. Return the only key with the latest value, in the position of the first occurrence of the key. e.g.

  > keys_in = [‘abc’, ‘def:0’, ‘fred’, ‘jane:ergo’, ‘def:3’] keys_out = [‘abc’, ‘def:3’, ‘fred’, ‘jane:ergo’]

`filterpype.filter_utils.``make_dict`(*args, **kwargs)

  Ease the dictionary making process by removing the need for quoting the keys. *args is for tuples coming from an existing dictionary, using the *adict.items() form. **kwargs forms keyword parameters into a dictionary.

`filterpype.filter_utils.``nibble_count`(bit_length)

  How many nibbles needed to show required numbers of bits?

`filterpype.filter_utils.``ntohl`(data)

  Converts a string of length four from network order (big-endian) to a long in host order.

  Equivalent of C OS library function of the same name.

  Source: http://www.java2s.com/Tutorial/Python/0280__Buildin-Module/NetworkByteOrder.htm

`filterpype.filter_utils.``ntohs`(data)

  Converts a string of length two from network order (big-endian) to a short in host order.

  Equivalent of C OS library function of the same name.

  Source: http://www.java2s.com/Tutorial/Python/0280__Buildin-Module/NetworkByteOrder.htm

`filterpype.filter_utils.``pad_string`(text, length, pad_char=' ')

  Pad input string to the length with pad_char. If length < length(text), return the whole string.

`filterpype.filter_utils.``print_hex_conversions`()

`filterpype.filter_utils.``print_redirect`(func, *args, **kwargs)

  redirect(func, ...) –> (output string result, func’s return value) (See p.257 of Python in a Nutshell) func must be a callable and may emit results to standard output. Capture those results as a string and return a pair: the print output and func’s return value.

`filterpype.filter_utils.``printable`(text)

  Return a string with all unprintable characters replaced with ‘?’

`filterpype.filter_utils.``pypes_for_dict_gen`(file_dir)

  Create generator to return all pypes in the pypes directory.

`filterpype.filter_utils.``random_file_name`(ext='.dat')

  Returns a unique random file name generated as a UUID string. This is statistically guaranteed to avoid a name clash.

`filterpype.filter_utils.``remove_punctuation`(string_in)

  Removes all punctuation chars from string_in

  Punctuation removed: ‘!”#$%&’()*+,-./:;<=>?@[]^_`{|}~’

`filterpype.filter_utils.``reverse_byte`(byte)

`filterpype.filter_utils.``split_strings`(data, split_size)

  Return the input string as a list of strings, split into split_size pieces. The last piece sent may be an incomplete split.

`filterpype.filter_utils.``strip_number_suffix`(txt)

  Remove any trailing digits from a string and underscore.

`filterpype.filter_utils.``strip_prefix`(txt, separator)

  Remove the text up to and including the separator, if there is one, or else return the original text.

`filterpype.filter_utils.``strip_prefixes`(txt, prefix_regex)

`filterpype.filter_utils.``unindent`(lines)

  Remove the same number of spaces from the front of each line, so that at least one line has no leading spaces. Return a list of the stripped lines. Ignore blank lines, even though they may have leading spaces.

## lex_yacc4.py

Tokenises and parses the route part of a filter config in a pipeline.

Remember to delete lextab.py/.pyc and parsetab.py/.pyc if anything changes, otherwise the previous version will be used.

class `filterpype.lex_yacc4.``RouteParser`(debug=False)

  Bases: `object`

  Parse the pipeline route, defined by a list of filters, with embedded parentheses.

  `TEST_ROUTE` = [' \n## A >>> X >>> (B >>> (C >>> D) E) F >>> G >>> H\n B >>> A >>> \n (C) \n D >>> \n (E >>> F >>> (H) Q) \n G >>> I >>> J\n \n ']
  `comment` = '(\\#).*'
  `digit` = '([0-9])'
  `identifier` = '(([_A-Za-z\\.\\:\\-%\\$\\{\\}])(([0-9])|([_A-Za-z\\.\\:\\-%\\$\\{\\}]))*)'
  `nondigit` = '([_A-Za-z\\.\\:\\-%\\$\\{\\}])'
  `p_branch`(p)

    branch : LPAREN start_branch pipe RPAREN end_branch

  `p_end_branch`(p)

    end_branch :

  `p_error`(p)

    Error rule for syntax errors

  `p_pipe1`(p)

    pipe : FILTER

  `p_pipe2`(p)

    pipe : pipe JOINTO FILTER

  `p_pipe3`(p)

    pipe : pipe JOINTO branch FILTER

  `p_start_branch`(p)

    start_branch :

  `parse_route`(route_in, debug=False, tracking=False)
  `pipeline_for_print`(prefixed_route)
  `print_connections`(connections)
  `remake_connection`(*args)
  `start` = 'branch'
  `t_COMMENT`(t)

    (#).*

  `t_FILTER`(t)

    (([_A-Za-z.:-%${}])(([0-9])|([_A-Za-z.:-%${}]))*)

  `t_JOINTO` = '>>>'
  `t_LPAREN` = '\\('
  `t_RPAREN` = '\\)'
  `t_error`(t)
  `t_ignore` = ' \t'
  `t_newline`(t)

    n+

  `tokens` = ('FILTER', 'LPAREN', 'RPAREN', 'JOINTO')

## pipeline.py

class `filterpype.pipeline.``CheckEssentialKeys`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Check that parameters can be set in various ways.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = check_essential_keys\n description = Test class with two essential params and one optional param\n keys = foo, bar, baz_opt:16\n ##dynamic = True\n\n [--route--]\n batch:120\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``CopyFile`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Take in a file object via data stream, and write it to a named file.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()
  `config` = '\n [--main--]\n ftype = copy_file\n keys = dest_file_name, source_file_name:none\n # Callback removed -- environ:none\n \n [read_batch]\n # Callback removed -- environ:none\n # Since environ is optional, the substitution will only happen if set\n ##environ = ${environ}\n source_file_name = ${source_file_name}\n \n [--route--]\n read_batch >>>\n write_file:${dest_file_name}\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``CopyFileCompression`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Adds basic compression to the Copy File example above along with callback environment in order to report on read progress.

  Source file name is optional, as it can also accept the filename / file object as the data parameter within the first packet passed into the pipeline.

  Uses callback and environ to make progress reports.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = copy_file_compression\n keys1 = dest_file_name, source_file_name:none\n keys2 = callback:none, environ:none, file_size:none\n \n [read_batch]\n batch_size = 4194304 # 4 MB for fast reading\n source_file_name = ${source_file_name}\n file_size = ${file_size}\n \n [callback_read_progress]\n ftype = callback_on_attribute\n watch_attr = read_percent\n callback = ${callback}\n environ = ${environ}\n watch_for_change = True\n \n [write_with_compression]\n ftype = write_file\n compress = bzip\n dest_file_name = ${dest_file_name}\n \n [--route--]\n read_batch >>>\n callback_read_progress >>>\n write_with_compression\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``ExtractManyAttributes`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Splits data into lines and extracts attributes based on a delimiter parameter, the attributes are stored in the packet.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = extract_many_attributes\n keys = attr_delim\n \n [--route--]\n split_lines >>>\n attribute_extractor:${attr_delim}\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``ExtractManyAttributesSplitWords`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Splits data by split_on_str and extracts attributes based on a delimiter parameter, the attributes are stored in the packet.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = extract_many_attrs_split_words\n keys = split_on_str, attr_delim\n \n [--route--]\n split_words:${split_on_str} >>>\n attribute_extractor:${attr_delim}\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``Pipeline`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  A pipeline is a filter that has a filters dictionary which keys by name all the filters in the pipeline.

  The mechanism of how the filters work need not be known, for a higher level of functionality.

  The pipeline needs to know which factory to use to make its data filters. This would not normally change, so is passed in with the constructor.

  To make testing easier, we define an input binary file name, and one or more output files.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = 'not_set'
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``PipelineForPypes`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  For testing automatic pipeline creation

  BL - 28/8/09 - Added in a config to force it to have an ftype as a test was failing in FDS specific files due to it being missing

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype=pipeline_for_pipes\n description=nothing\n \n [--route--]\n pass_through\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.pipeline.``Refinery`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = 'not_set'
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

## ppln_demo.py

class `filterpype.ppln_demo.``Alison`

  Bases: `object`

class `filterpype.ppln_demo.``Factorial`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Recursion to calculate factorials.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = factorial\n description = Recursive filter to show looping, while calculating factorials\n \n #\xc2\xa0Check print_connections\n [--route--]\n tank_queue:0:true >>>\n factorial_calc >>>\n branch_if:recurse >>>\n (tank_feed >>> tank_queue)\n sink\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``Freda`

  Bases: `object`

  `speed` = 50

class `filterpype.ppln_demo.``InnerBar`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Test inner pipeline class

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = inner_bar\n keys = temperature, speed\n \n [py_print_vars]\n x = 3\n y = x + 4\n # NB Using %s for string formatting, not %d\n print \'**14400** Inside py_print_vars\'\n print \'**14410** packet.data = "%s"\' % packet.data\n print \'**14450** temperature = %s\' % ${temperature}\n print \'**14460** speed = %s\' % ${speed}\n print \'**14480** SPEED = %s\' % \'%SPEED\'\n \n ##[py_set_vars]\n ##SPEED = 73\n \n [--route--]\n ##py_set_vars >>>\n py_print_vars\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``Jane`

  Bases: `filterpype.ppln_demo.Freda`

  `speed` = 50

class `filterpype.ppln_demo.``ReverseChars`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Takes string input and returns a reversed string.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = reverse_chars\n description = TO-DO: docstring\n \n [--route--]\n reverse_string >>>\n sink\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``SimpleLoop`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Simple loop to print counter.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = simple_loop\n description = TO-DO: docstring\n \n [--route--]\n tank_queue \n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``SmallPipeBaz`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Test inner small baz pipeline

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = small_pipe_baz\n key = temperature, speed\n \n [py_pointless]\n TEMPERATURE = 1200\n SPEED = 90\n\n [--route--]\n # ${temperature} will substitute once at the beginning\n # %TEMPERATURE will substitute values each time\n # inner_bar:${temperature}\n \n inner_bar:%TEMPERATURE:%SPEED\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``SquareNumber`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Test simple function to square numbers in a pipeline.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = square_number\n description = TO-DO: docstring\n\n [--route--]\n square_if_number >>>\n sink\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``TempMultipleAB`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Spike for putting data into multiple pipes.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = temp_multiple_ab\n description = TO-DO: docstring\n keys1 = a1, a2, b1, b2\n keys2 = w:1, x:2, y:3, z:4\n\n [--route--]\n temp_text_after1:${a1} >>>\n temp_text_before1:${b1} >>>\n reverse_string1 >>>\n temp_text_after2:${a2} >>>\n temp_text_before2:${b2} >>>\n reverse_string2\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``TempMultipleSpace`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = temp_multiple_space\n description = TO-DO: docstring\n keys = a1, a2, b1, b2\n \n [--route--]\n temp_multiple_ab >>>\n temp_space\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()
  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``VariableBatchPipeline`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = " [--main--]\n ftype = embedded_batch_sizing\n description = Pipeline should batch alternately\n \n [batch]\n dynamic = true\n size = %BATCH_SIZE\n \n [py_init]\n if BATCH_SIZE == '$$<unset>$$':\n BATCH_MODE = 'size'\n BATCH_SIZE = 2\n \n [py_batch_sizer]\n print '**12550** BATCH_SIZE = %s, PREV_BATCH_MODE = %s' % (\n BATCH_SIZE, PREV_BATCH_MODE)\n \n if BATCH_MODE == 'size':\n BATCH_SIZE = int(packet.data) \n BATCH_MODE = 'data'\n else:\n BATCH_MODE = 'size'\n BATCH_SIZE = 2\n \n print '**12560** BATCH_SIZE = %s, PREV_BATCH_MODE = %s' % (\n BATCH_SIZE, PREV_BATCH_MODE)\n if PREV_BATCH_MODE == 'data':\n BATCH_SIZE == 2\n PREV_BATCH_MODE = 'size'\n else: # PREV_BATCH_MODE == 'size'\n BATCH_SIZE = int(packet.data) \n PREV_BATCH_MODE = 'data'\n print '**12570** BATCH_SIZE = %s, PREV_BATCH_MODE = %s' % (\n BATCH_SIZE, PREV_BATCH_MODE)\n \n \n [--route--]\n py_init >>>\n batch >>>\n py_batch_sizer >>>\n sink\n "
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.ppln_demo.``WordsInCaps`(**kwargs)

  Bases: `filterpype.pipeline.Pipeline`

  Take each word in the data and capitalise.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `config` = '\n [--main--]\n ftype = words_in_caps\n description = TO-DO: docstring\n key = join_words_with:~\n \n [--route--]\n split_words >>>\n capitalise >>>\n join:${join_words_with}\n '
  `emb_module`

    Embedded Python module

  `filter_data`(packet)

    This is the minimum functionality for a pipeline that does nothing. But some pipelines may want to override this, setting parameters before sending any data.

  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `get_filter`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `getf`(filter_name)

    TO-DO>>>> Return the filter from the _filter_dict, if it has already been made,

    else make the filter and then return it. This is necessary now because of allowing an empty config. Reference to a filter will create it. Test the name first for being in the format “some_name:23” where 23 is the value of the first key.

  `init_filter`()

    Initialise coroutine vaiables that are needed before we reach the (yield) statement. Override in subclasses.

  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `next_filter`

    Sets internal last_filter to have same next_filter

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `pump_data`(data_in=None, zero_inputs=True)

    Put the data into the pipeline by wrapping it in a Datapacket. Get the data out by temporarily sticking a Sink on the end, returning all the packets received. Remember to close the pipeline externally, if tidying up actions need to be performed.

    If zero_inputs is True, reinitialise the variables before pumping the new data in. <<<<<<<zero_inputs<<<<<<<<< TO-DO <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

  `refinery`

    Top-level pipeline

  `route` = ''
  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `update_filters`()

    Update filter parameter values after the creation of all the filters

  `update_route`()

    Update pipeline route after the creation of all the filters

  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

## data_fltr_demo.py

class `filterpype.data_fltr_demo.``AddNumbers`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Adds the numbers in the input list, passing on the result in the packet. We’re expecting either a Python list of numbers, or a comma- separated string to be read as a list.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'add_numbers'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``ByteValueCounter`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Count how many of each byte value have passed through the filter.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'byte_value_counter'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_fltr_demo.``Capitalise`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Capitalise the first character of the packet’s data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'capitalise'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``FactorialCalc`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Experiment to see if we can loop recursively with filters. We can’t: ValueError: generator already executing. Aha! We have a solution. Packets in will have attributes:

  > ‘x’ – the number to get the factorial of. ‘pending’ – intermediate calculation ‘x_factorial’

  Until calculation is finished, packet will be sent via the branch to the tank_queue that feeds this filter. <<<< rewrite TO-DO <<<<

  Follow FactorialCalc by BranchIf:recurse

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'factorial_calc'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``KeySubstitutions`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Only exists to test pipeline key substitution.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'key_substitution'
  `init_filter`()
  `keys` = ['first_key', 'second_key:value']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``MultiplyIfInteger`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  If an input is an integer, or a string integer, return a multiple of the number as a string, else return the original string, in a packet.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'multiply_if_integer'
  `init_filter`()
  `keys` = ['multiply_by']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``PrintData`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Print first part of data block in hex.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'print_data'
  `init_filter`()
  `keys` = ['print_limit:100']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_fltr_demo.``ReverseString`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Filter to return a reversed string for each yielded packet

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'reverse_string'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``ShowProgress`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Print char to screen to show progress

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    At the end of the pipeline, print an extra new line before any final output from the calling program.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'show_progress'
  `init_filter`()
  `keys` = ['packet_progress_freq', 'progress_char:.', 'progress_line_length:50']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_fltr_demo.``SquareIfNumber`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  If a string input is an number, return the square of the number, as a number, else leave packet alone.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'square_if_number'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``SumBits`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  However much the bits are shuffled or padded with zeros, their sum should not change.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'sum_bits'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_fltr_demo.``SumBytes`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  However much the bytes are shuffled, their sum should not change.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'sum_bytes'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_fltr_demo.``SumNibbles`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  However much the nibbles are shuffled or padded with zeros, their sum should not change.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'sum_nibbles'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

class `filterpype.data_fltr_demo.``SwitchBits`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Filter to return data with some bits switched

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'switch_bits'
  `init_filter`()
  `keys` = ['bit_switch_mask']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``TempSpace`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Insert a space between each character in the data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'temp_space'
  `init_filter`()
  `keys` = []
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``TempTextAfter`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Put string after the data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'temp_text_after'
  `init_filter`()
  `keys` = ['text_after']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

class `filterpype.data_fltr_demo.``TempTextBefore`(**kwargs)

  Bases: `filterpype.data_fltr_base.DataFilter`

  Put string in front of the data.

  `after_filter_data`(packet)

    Override this to perform some action just after the incoming packet is processed by filter_data().

  `after_send_on`(packet, fork_dest)

    Hook to execute just after a packet is sent on

  `before_filter_data`(packet)

    Override this to perform some action just before the incoming packet is processed by filter_data().

  `before_send_on`(packet, fork_dest)

    Hook to execute just before a packet is sent o

  `callbacks` = ['results_callback']
  `close_filter`()

    Override this to add to the closing functionality before the filter is finally closed.

  `emb_module`

    Embedded Python module

  `filter_data`(packet)
  `flush_buffer`()

    Override this for clearing out any buffered data: always before final close_filter(), but may be needed at intermediate stages.

  `ftype` = 'temp_text_before'
  `init_filter`()
  `keys` = ['text_before']
  `module_loc`

    Location of embedded Python module

  `name`

    Name for filter/pipeline

  `open_message_bottle`(packet)

    Open the message bottle, and take appropriate action. We know what to do with these general purpose commands, applying generally to any filter:

    > reset

    Other functionality may be needed, specific to one filter. In this case, override the open_message_bottle() in the filter. e.g. WriteFile uses this to close one file and open a new file with a different name.

  `refinery`

    Top-level pipeline

  `send`(*packets)

    For the first filter in the pipeline, send in the starting data. This must be a DataPacket object.

    If not already primed, call _prime() to send a next() call to the generator.

  `send_on`(packet, fork_dest='main')

    Send on packets to their destination.

  `shut_down`()

    Shut down the pipeline/filter by setting a shutting_down flag. It doesn’t work to pass close() straight to the first filter, because the generator sequence is still busy and comes back with an error message: “ValueError: generator already executing”

    Try to close the pipeline, but don’t worry if we can’t because it’s busy: there must be a loop somewhere that should be checking the shutting_down read-only property.

  `shutting_down`

    Is refinery shutting down (read-only)

  `standard_keys` = ['_class', '_key_values', '_name', 'factory', 'ftype', 'pipeline', 'dynamic', 'update_live', 'results_callback']
  `validate_params`()

    Override this function to check validity of input parameters. This is called before init_filter(), which may use the input parameters to calculate derived params.

  `zero_inputs`()

    During initialisation of a filter, there may be inputs and counters that need to be set to zero. By putting these in a separate function, we avoid duplicating code where clearing is required repeatedly.

    Some filters that use carries and remainders may require clearing inputs before each use, to give a consistent one-off answer, for something like pump_data().

    zero_inputs() is called by the closing() context manager, but not until the generator receives its first data packet.

    Should this be called init_filter_dynamic <<<<<< TO-DO <<<<<<

### Navigation

- index
- modules |
- previous |
- FilterPype 0.3.4 documentation »