Multiple families workflow


#21

Complex is easy, simple is hard.

When it comes to introducing new functionality, especially one that is inherently more complex, I try and look towards the benefits of keeping it simple.

  1. Less of a learning curve
  2. Easier to understand other peoples code
  3. Easier to maintain

And then think about whether the advantages of the new functionality outweighs these.

In this case, the ability to define custom matching algorithms, albeit cool and logical the way you’ve proposed it, does it add more value than cost?

At one extreme, I see a future where every plug-in defines a corresponding matching algorithm. At that point, to even begin to understand a series of plug-ins - especially those mixed and matched from elsewhere - would see an increase in the time required to understand it.

On the other extreme, where there is only one matching algorithm, you need to get creative with the little you’ve got in order to achieve complex behaviour.

Having said that, it’s also possible that families as it exists today is a subset of what this does. That defining your own matching algorithm could be the de facto method of associating plug-ins to instances and that it makes for both more flexibility and simplicity.

Let’s explore it.

First off, I’m interested in what you mentioned @tokejepsen about your requirement in pyblish-ftrack and what hoops you jump to currently in order to achieve the necessary effect.


#22

The main problem was originally that we couldn’t process all instances, because they wouldn’t all have the same family name. This was back when you could only associate a single family with an instance. So we decided to process all instances, but just return early when a certain data member was missing.

These days with multiple families we could easily utilize a certain family name like ftrack to figure out which instances to process. The user could easily add another family to an instance. So there aren’t actually much of an argument for matching algorithms.
There are the edge case where some one is using the family ftrack for someting else. At this point having a matching algorithms that only processes the instances that have the correct data would be beneficial, but this case is highly unlikely to happen. Even the cross over of different plugins packages has yet to be an issue.


#23

The above api.Intersection, api.Subset and api.Exact algorithms have now been merged and released as 1.4.3.

$ pip install pyblish-base

#24

Just tested this with a new plugin, and it works great :smile:


#25

We just switched to using this feature in our tech preview. So now we’re collecting families like:

instance = context.create_instance(
   node.name(), families=['ftrack', 'camera']
)

And on the plugins:

families = ['ftrack', 'camera']
match = pyblish.api.Subset

Seems to work well - nice addition :slight_smile:


#26

.match attribute added to API documentation.


#27

I’ve noticed one workflow where a custom matching algorithm would be useful

If you collect some instances, and wanted to append some data to them with a different collector. The case is host specific instances, where you want to append data to instances cross host without code duplicate per host.
Once the instances are created, they have to be publishable to be proceed by other InstancePlugins. Here I’m using a ContextPlugin and manually filtering to the correct instances, to force processing all instances.


#28

Would it be possible to post an example of this?


#29

Sure :smile:

Maya Collector

class MayaCollector(pyblish.api.ContextPlugin):

    order = pyblish.api.CollectorOrder
    hosts = ["maya"]

    def process(self, context):

        instance = context.create_instance("A")
        instance.data["families"] = ["farm"]
        instance.data["publish"] = False

Houdini Collector

class HoudiniCollector(pyblish.api.ContextPlugin):

    order = pyblish.api.CollectorOrder
    hosts = ["houdini"]

    def process(self, context):

        instance = context.create_instance("B")
        instance.data["families"] = ["farm"]
        instance.data["publish"] = False

Append data collector

class AppendData(pyblish.api.InstancePlugin):

    order = pyblish.api.CollectorOrder + 0.1
    families = ["farm"]

    def process(self, instance):

        instance.data["SomeData"] = {"something": "else", "some": 1}

#30

Do you mean that because of instance.data["publish"] = False, the AppendData plug-in has no effect? You would have expected those instances to be processed by plug-ins within the Collection order even with this publish member set to False?

How about a post plug-in, to determine defaults?

class HoudiniCollector(pyblish.api.ContextPlugin):

    order = pyblish.api.CollectorOrder
    hosts = ["houdini"]

    def process(self, context):

        instance = context.create_instance("B")
        instance.data["families"] = ["farm"]
        # instance.data["publish"] = False

class AppendData(pyblish.api.InstancePlugin):

    order = pyblish.api.CollectorOrder + 0.1
    families = ["farm"]

    def process(self, instance):
        instance.data["SomeData"] = {"something": "else", "some": 1}

class SetPublish(pyblish.api.InstancePlugin):

    order = pyblish.api.CollectorOrder + 0.2
    families = ["farm"]

    def process(self, instance):
        instance.data["publish"] = something is True

I think the reason I’d avoid custom algorithms is for (1) the added learning curve for anyone looking to learn your plug-ins, and for the (2) lessened re-usability and (3) intermixing of plug-ins. The advantage would need to be rather significant to justify such a sacrifice and there would ideally be no workaround, or at least one that was significantly more difficult to manage.

I think if you could show me a way to implement it without encountering the 3 cons above, that would be a great starting point for the feature.


#31

Initially I probably would have, because I wrote the plugin without thinking too much :smile: Although when I think about how Pyblish works, I know that it wouldn’t get processed.
It might just be one of those pitfalls you need to be aware of.

You could definitely do this, but I personally would like to avoid too much order dependencies. Been down that road, and the more plugins you offset from each other, the more difficult it becomes to manage.

I agree. Think my posts here are also more of thoughts out loud :smile:


#32

It’s a scenario where the word “publish” doesn’t apply, so maybe there’s room for change here.

Initially, the word was chosen because an instance was created and either published, or not published. So “publish” seemed the most logical name. But in this case, your second collector doesn’t necessarily publish the instance and so the logic breaks.

We could call it what plug-ins call it?

instance.data["active"] = False

That might make it more clear that it won’t be considered anymore.

Off topic, but this would make a fantastic use-case story for a dedicated thread! Why did you do it? What did you expect? What did you find? Why was it difficult to manage?

Consider it? :slight_smile:


#33

That might work :smile:

I’ll try and collect some thoughts, but won’t be for a while.


#34

What do we think about having an Exclude matching algorithm?

If any of the families on the plugin are in the instances families, then exclude that instance.


#35

An example would help.


#36

For example I would like a plugin to process all instances except for those of family source.


#37

Thanks, but could you share an example of how you would use it and the problem it would solve? I’d like to make sure there isn’t some other way of accomplishing what you’re looking for, before adding more features to support it.


#38

Ahh, I see.

The most recent plugin I’ve made where this exclusion matching would help, is about updating statuses in Ftrack.

On every component published to Ftrack, I would like to update the status to “In Progress”, except for any source components.


#39

Ok, so it’d be this, but the inverse?

from pyblish import api

class UpdateStatus(api.InstancePlugin):
  order = api.IntegratorOrder
  families = ["includeMe", "meToo", "andMe"]

  def process(self, instance):
    import ftrack
    ftrack.update_status(instance, "Done")

Such as this?

from pyblish import api

class UpdateStatus(api.InstancePlugin):
  order = api.IntegratorOrder
  families = ["excludeMe"]
  match = api.Exclude

  def process(self, instance):
    import ftrack
    ftrack.update_status(instance, "Done")

#40

Yup, exactly. Currently I’m processing all instances, but skipping the families I don’t want in the plugin.

from pyblish import api

class UpdateStatus(api.InstancePlugin):
  order = api.IntegratorOrder

  def process(self, instance):
    families = [instance.data["family"]]
    families += instance.data.get("families", [])
    if "excludeMe" in families:
        return

    import ftrack
    ftrack.update_status(instance, "Done")