Pyblish-lite: Accessing the family type of the context

Hello,

I have set out to be able to define custom execution order of plugins, to be able to use the same plugin in multiple different family pipelines in different orders.
For example:

  • Plugin A is order 1.2 for family A
  • Plugin A is order 1.5 for family B

I have been able to successfully modify the order of the plugins via the gui and have this propagate to the model and therefore the order in which the plugins will process.
I have also been able to save off the orders to a json file and load them back to re-use at a later point.

My last step, and the one I cannot figure out is how to set this up to automatically re-order the plugins when said plugins are collected for a family. I do this from the Plugin model, and would like to know how to access the current family that is filtering the the listview, so I can try and hook this up with an auto save/load functionality.
If someone could point me at the location in the source code where, or provide some example code on how I could achieve this, it would be awesome and hopefully save me a ton of time trying to figure it out myself.

Interesting challenge!

The order of plug-ins are determined by the pyblish.plugin.discover() function, which in Lite is called here.

You could either perform your modification there, or more generally see whether you can affect it at the source; in which case it would apply to both Lite, QML and pyblish.util, that each make use of this function.

from pyblish import plugin

# Maintain reference to original
plugin._discover = plugin.discover

def my_discover(*args, **kwargs):
  plugins = plugin._discover(*args, **kwargs)

  # Modify order somehow
  for p in plugins:
    p.order = porder * 2

  return plugins

# Monkey-patch orignal function
plugins.discover = my_discover

If that does what you’re looking for, it’s possible we could add a post_discover() function or the like to facilitate the feature at the API level.

E.g.

pyblish.api.register_post_discovery_callback(my_discover)

Hey Marcus,

Thanks for the reply!
That is what I was initially looking to do, and soon realised that it isn’t likely possible due to the following:
( forgive me if I have completely misunderstood, this is my first proper deep dive into the guts of Pyblish )

  • All valid plugins are gathered during discovery. These are stored in model.Plugin.items attribute and a generator item in control.Controller.pair_generator.
  • Pyblish then filters model.Plugin.items through model.ProxyModel.filterAcceptsRow to display the currently valid plugins in the gui.

This means that the filtering happens after the collection and order sorting occurs, which means the two bits of data that I am looking for are currently at the two ends of the plugin gathering pipe:
gather valid plugins -> sort plugins by order attribute -> filter plugins by family.
I assume this is happening because the collection plugins have to run to gather whichever family of validation, extraction, integration plugins need to run next.
So it is not possible to know during discovery which family based order needs to be applied to the plugins. It has to happen at the end of the pipe. I just can’t figure out where the gui is getting the family to filter by.

The first bit of functionality to implement was to re-order the plugins every time the gui sends a signal that the user has changed the order of the plugins by dragging and dropping in the list, which as I stated above, I have been able to do.
Next was functionality to save and load new orders arbitrarily, which was relatively easy to do using the first bit of functionality.

To better explain my approach, I will paste the code I have written so far.
Please excuse the volume of potentially non-relevant code, it is hard to know what might be relevant or not, so better to include it all.

Additions to model.Plugin class:
Allows the drag and drop operations to occur, re-ordering model.Plugin.items to respect the drag and drop operation of view.Item. All plugins then have their order attribute modified to match their current position in model.Plugin.items via model.Plugin._reorder_plugins_by_list_order

plugin_order_reset = QtCore.Signal(list)

def _encode_plugin(self, plugin):

    return (plugin.__module__, plugin.__name__)

def _decode_plugin(self, data):

    module_path       = data[0]
    module_name_split = os.path.basename(module_path).split(".")
    module_name       = ".".join(module_name_split[:-1])
    module            = imp.load_source(module_name, module_path)
    plugin            = getattr(module, data[1])

    # not quite sure the reason for this,
    # but repeating the step from plugin.discover:
    plugin.__module__ = module.__file__

    return plugin

def _get_plugins_by_order_type(self, type_index):

    out_list = []
    next_type_index = type_index + 1
    for plugin in self.items:

        if plugin.order >= type_index and plugin.order < next_type_index:

            out_list.append(plugin)

    return out_list

def _reorder_plugins_by_list_order(self):

    reset_order_plugins = []
    for order_type in xrange(1, 4):

        plugins = self._get_plugins_by_order_type(order_type)
        plugins_count = len(plugins)
        for index, plugin in enumerate(plugins):

            new_order    = order_type + (float(index) / float(plugins_count))
            plugin.order = new_order
            reset_order_plugins.append(plugin)

    self.plugin_order_reset.emit(reset_order_plugins)
    return True

def _reorder_plugins_by_order_id(self):

    reset_order_plugins = []
    for order_type in xrange(1, 4):

        reset_order_plugins.extend(self._get_plugins_by_order_type(order_type))

    pyblish.api.sort_plugins(reset_order_plugins)
    all_plugins = self._get_plugins_by_order_type(0)
    all_plugins.extend(reset_order_plugins)
    self.items = all_plugins
    self.plugin_order_reset.emit(reset_order_plugins)

    return True

def save_plugin_order(self, path):

    plugin_order_data    = [(plugin.label, plugin.order) for plugin in self.items]
    encoded_plugin_order = json.dumps(plugin_order_data)
    with open(path, "w+") as file:

        file.write(encoded_plugin_order)

    return encoded_plugin_order

def load_plugin_order(self, path):

    with open(path, "r") as file:

        decoded_plugin_order = json.load(file)

    for data in decoded_plugin_order:

        for plugin in self.items:

            if plugin.label == data[0]:

                plugin.order = data[1]

    return self._reorder_plugins_by_order_id()

def supportedDropActions(self):

    return QtCore.Qt.MoveAction

def flags(self, index):

    if index.isValid():

        return QtCore.Qt.ItemIsSelectable | QtCore.Qt.ItemIsDragEnabled | QtCore.Qt.ItemIsEnabled

    return QtCore.Qt.ItemIsSelectable | QtCore.Qt.ItemIsDragEnabled | QtCore.Qt.ItemIsDropEnabled | QtCore.Qt.ItemIsEnabled

def insertRows(self, row, count, items, parent=QtCore.QModelIndex()):

    if parent.isValid() or count <= 0:
        return False

    beginRow = max(0, row)
    endRow   = min(row + count - 1, len(self.items))

    self.beginInsertRows(parent, beginRow, endRow)

    for index in xrange(beginRow, endRow + 1):
        self.items.insert(index, None)

    self.endInsertRows()
    return True

def removeRows(self, row, count, parent=QtCore.QModelIndex()):

    if parent.isValid() or row >= len(self.items) or row + count <= 0:
        return False

    beginRow = max(0, row)
    endRow   = min(row + count - 1, len(self.items) - 1)

    self.beginRemoveRows(parent, beginRow, endRow)

    for index in xrange(beginRow, endRow + 1):
        self.items.pop(index)

    self.endRemoveRows()

    if self.pendingRemoveRowsAfterDrop:
        '''
        If we got here, it means this call to removeRows is the automatic
        cleanup' action after drag-n-drop performed by Qt
        '''
        self.pendingRemoveRowsAfterDrop = False
        self.drag_and_drop_finished.emit()

    self._reorder_plugins_by_list_order()
    return True

def mimeTypes(self):

    return ["pulginData"]

def mimeData(self, indices):

    def get_mime_data(index):

        return (
            self._encode_plugin(self.items[index.row()]),
            index.row()
        )

    mime_data = QtCore.QMimeData()
    item_data = [get_mime_data(index) for index in indices if index.isValid()]
    item_data = jsonpickle.encode(item_data)
    mime_data.setData("pulginData", QtCore.QByteArray(item_data))

    return mime_data

def canDropMimeData(self, data, action, row, column, parent):

    if not data.hasFormat("pulginData"):

        return False

    if action == QtCore.Qt.IgnoreAction:

        return True

    if (not data.hasFormat("pulginData")) or column > 0 or self.rowCount() <= 0:

        return False

    return True

def dropMimeData(self, data, action, row, column, parent):

    if not self.canDropMimeData(data, action, row, column, parent):

        return False

    encoded_data = data.data("pulginData")
    new_items    = jsonpickle.loads(encoded_data.data())

    target_row = row
    self.lastDroppedItems = []
    for index, (item, row_index) in enumerate(new_items):

        if target_row < 0:

            target_row = row_index

        self.beginInsertRows(QtCore.QModelIndex(), target_row, target_row)
        item = self.items[row_index]
        self.items.insert(target_row, item)
        self.endInsertRows()
        self.lastDroppedItems.append(item)
        target_row += 1

    self.pendingRemoveRowsAfterDrop = True

    return True

Additions to control.Controller:
Allows control.Controller._load method on the controller to regenerate the pair_generator attribute to match the order of the plugins in model.Plugin.items.

def _load(self, plugins=None):
    """ Initiate new generator and load first pair

        The optional plugins arguments allows for the generator
        to be updated arbitrarily.

        Args:
            plugins (list, optional): The list of plugins to load.
                If None, will default to self.plugins.
                Defaults to None

        Returns:
            bool: True on success.
    """
    if plugins is None:

        plugins = self.plugins

    self.is_running = True
    self.pair_generator = self._iterator(plugins, self.context)
    self.current_pair = next(self.pair_generator, (None, None))
    self.current_error = None
    self.is_running = False

    return True

Additions to window.Window:
This is where the two bits of functionality are hooked together; model.Plugin.plugin_order_reset -> window.Window.on_plugin_order_reset which calls control.Controller._load on the newly sorted plugins.

def __inti__(self, controller, parent=None):

    plugin_model.plugin_order_reset.connect(self.on_plugin_order_reset)

@QtCore.Slot(list)
def on_plugin_order_reset(self, plugins):

    self.controller._load(plugins)

So what I would like to do is effectively run model.Plugin.load_plugin_order for the specific family once it has reached the end of the plugin collection -> sort -> filter pipeline.
The main issue is my naivety with regards to both the model - view design pattern and pyblish’s architecture, so I am jumping around just trying to figure out what everything does.

Not sure if this explains it well enough, but it is the best I can do.
I will continue to try and figure something out in the mean time, but any suggestions would be great.

cheers,

Shea.

Ok, let’s have a look at how the data ends up in the GUI by moving backwards; from what you see, to where it comes from.

  1. Sorting in the GUI ultimately happens via a QSortFilterProxyModel, here
  2. In order for it to sort, it needs to have something to sort, and that data is stored in this QAbstractItemModel
  3. That View is passing this model with data from the Controller, here

Does that help at all?

I must admit I’m a little lost in terms of what you’re looking to achieve. Am I understanding it correctly that you have:

  1. A series of plug-ins with a particular order
  2. You want Pyblish Lite to draw these plug-ins for the end-user
  3. You want Pyblish Lite to enable the user to alter the value of the order for each plug-in, by dragging-and-dropping in the GUI
  4. You want Pyblish Lite to save the new orders out to disk, as a e.g. json file
  5. You want Pyblish Lite to restore these orders from disk

If that’s what you want, then there might be something else you can do. What you can’t do, is what would have been most convenient.

from pyblish import api
plugins = api.discover()
plugins[0].order = some_new_value

And have this value persist. It won’t work, because whenever api.discover() is called, plug-ins are re-read from disk, so you would need to alter the physical files to make this persistent.

However

What you might want to try is that rather than discover plug-ins by path, you may want to try registering them explicitly.

from pyblish import api
import my_plugins

for Plugin in my_plugins.All:
  api.register_plugin(Plugin)

Where .All is a list of sorts, where each Plug-in class is contained.

my_plugins.py

class MyPlugin1(...):
  pass

All = [MyPlugin1]

Now that they are registered explicitly, you control when they are re-read from disk. By e.g. calling reload(my_plugins), which also means you can make persistent changes to them.

from pyblish import api
import my_app

# Find previously registered plug-ins, without actually reloading them
for Plugin in api.discover():

  # Make a persistent change to it
  my_app.restore_order(Plugin)

Where my_app is whatever mechanism you have to save/load that json file.

The downside of this approach is that you lose the advantage of automatic reload of plug-ins from disk. But maybe that’s an acceptable tradeoff, considering you’re looking to modify plug-ins post-discovery anyway.

To preserve the ability to discover things from disk, you might be able to merge the path and explicit discovery by doing something like this.

from pyblish import api

for plugin in api.discover():
  api.register_plugin(plugin)

# Prevent re-discovery of plug-ins
api.deregister_all_paths()

Here’s one more idea about this.

  1. Rather than specifyin the order = explicitly in your plug-in, use a function
  2. The function could then:
    1. Look for a pre-defined order, given some key
    2. If one is found, use it
    3. If not, use default

For example.

import json

dbname = "orders.json"

with open(dbname) as f:
   persistent_orders = json.load(f)

def find_order(key, default=1):
  """Find `key` on disk"""
  return persistent_orders.get(key, default)

def store_order(key, order):
  """Store `order` on disk"""
  persistent_orders[key] = order
  with open(dbname, "w") as f:
    json.dump(persistent_orders, f)

from pyblish import api
class MyCollector(api.ContextPlugin):
  order = find_order("MyCollector", default=0.5)

Where find_order and store_order is declared somewhere in your own pipeline/application API. This would move the order value to some place within your own control, such that you can perform saving/loading of this order, and it would be independent of any Pyblish GUI.

Hi @munkybutt, how did it go with this?

Hi Marcus,

Sorry for the long delayed response. ( Work has been bonkers ).
Yes I managed to get it working.
The missing bit for the orders to load automatically was a function call after the collection plugins had run and before the rest of the plugins running. I added it in the controller _run method ( I think, I’m on my phone and will confirm once I get into work ).
But basically, everything runs as per normal, all the plugins are discovered, then the collection plugins run followed by re-ordering based on the plugin family that was collected.
I have some further tests to run and see how it works with multiple families in one scene, but for now it solves our requirements.

I am hoping to get my changes released to my company’s github. Just have to convince them to make one first. :smile:

Thanks muchly for your help, it really helped clarify a lot of questions.

1 Like