SimplePlugin

#1

I’m considering whether it’d be a good idea to introduce a new plug-in for newcomers to get familiar with Pyblish more quickly and to illustrate the need for the existing process in the first place - Selection, Validation, Extraction and Conform, or SVEC for short.

Goal

To provide for arbitrary plug-ins that does anything. They would reside at their own order, e.g. 1000 and get picked up regardless of prefix, e.g. my_plugin.py.

Implementation

Plug-ins are typically intended for a particular purpose, such as validating or extracting. But simple plug-ins doesn’t have a particular purpose.

from maya import cmds
import pyblish.api as pyblish

class MyPlugin(pyblish.SimplePlugin):
  def process(self, context):
    cmds.file(exportSelected=True)

You’re welcome to create things here too.

from maya import cmds
import pyblish.api as pyblish

class MyPlugin(pyblish.SimplePlugin):
  def process(self, context):
    for x in range(10):
      cmds.createNode("mesh", name="myMesh%i" % x)

Note that they could only ever operate on the Context, as no Instance's have yet been created, which eliminates the possibility of making use of families.

Discussion

Why would we want this?

Though the framework has been designed with SVEC in mind, it is capable of performing any arbitrary actions, the use of some may possibly stretch beyond our current understanding. It’s likely that without SVEC the framework would have been used in unforeseen ways that could ultimately be to it’s benefit.

On the other hand, publishing is abstract and lacks vocabulary. Which means it can be difficult to get things right and to find words to describe the complex processes it involves. SVEC provides this vocabulary and makes it possible to draw lines between orthogonal responsibilities and to make comparisons on a more granular level.

What would be the benefits of SimplePlugin?

Benefits

  • New users would not have to learn about SVEC before getting started with publishing
  • Unforeseen use could ultimately benefit the growth of Pyblish and it’s community

Disadvantages

  • Help will be tricky to find and to provide as each plug-in becomes ad-hoc
  • New users may be lead astray by not having anything to hold onto

From the get-go, it was my belief that SVEC captured every and all publishing needs one could ever want and I still believe that it is high-level enough to encompass even the most specific of needs.

Because of this, I suspect that if we do provide users with the amount of flexibility that SimplePlugin offers, they are bound to land up with plug-in stack that in some form or another resembles SVEC, only they won’t know about it and thus can’t ask about it in the same structured manner.

Co-existence

One doesn’t necessarily exclude the other. Or does it?

I think we’ve already gotten a taste of that it can sometimes be difficult to get the responsibilities between the various SVEC plug-ins right, but we push through (hopefully) because we believe the separation is ultimately healthy and makes for more maintainable code.

With an option of going “the easy route”, this pushing might become less necessary and ultimately lead to a majority of plug-ins based on it due to “not having the time”.

On the other hand, with great power comes great responsibility. It’s up to us to decide whether this responsibility is for the better or worse for Pyblish and for the pipelines of studios in the longer run.

#2

The framework that is built here with pyblish is really great and I’m totally up for widening it’s reach to allow for some arbitrary, often unpredictable needs that arise in production.

This solution seems like it could provide a lot of functionality for not that much effort which I always like. I can already think of a few ways of how to use it especially if it could be executing in various places of publishing process. For instance is might be a good candidate to pull data about current shot and task from ftrack/shotgun database prior to publishing and pass it into context. This would be greatly useful then for all validation, and conform.

#3

Currently, this is exactly what Selectors are meant for. Any data you intend to process is captured during the initial stages of Selection, including any coming in from outside sources.

But yes, it’s possible that SVEC is better suited as pure guidelines, atop plug-ins developed from SimplePlugin. In that way, it could become adopted where suited, but more importantly, alternate configurations, like VSCE or ECSV or even XYZCustom is made possible. It’s possible that this is a direction better suited for overall adoption of Pyblish, as I think one of it’s barriers currently is that users must learn about SVEC which, unless they fully understand it, they might not agree with at first.

Implementation-wise, I think it shouldn’t require any alterations to the source, it might even be possible to simply subclass Validator or such, set it’s order to 1000 and off you go.

There is however the issue of losing the use of families. Families are how we can associate a series of plug-ins to a particular type of asset. Without Selection, we can’t identify the asset to begin with, which leaves subsequent plug-ins without anything to base themselves on.

In cases where this is wanted, there is of course nothing stopping anyone from implementing Selector plug-ins alongside their SimplePlugin's, or even create instances from their SimplePlugin's.

#4

Interesting. That wouldn’t fit in there in my mind (not that I don’t agree).I saw selectors as plugin gathering information from the scene that I’m about to publish, and output of a selector is always an instance. Getting extra info about current working space from external source doesn’t result in a new instance. Just in more information for the context. But I see where you’re going with it and it makes sense. I just didn’t think of it that way.

#5

I would’ve considered all information required for a correct export to be part of the Selector as well, just like how Marcus explained it right here. If the Selector can acquire all necessary data to perform a publish it means your Pyblish solution remains working in any scenario, eg. batch-processing. Instead of having to perform a custom plug-in in ‘off-cases’?

Even though I’m all for allowing more free access from Plug-ins and alike into Pyblish I’m not exactly sure where this is going. The examples and explanations here seem lacking particular use. Since they don’t belong to a Family nor have access to it, when will they run? Always? Or how does the user run it? And why wouldn’t it gain access to whatever Instances or Plug-ins that are already in the current Context?

Again, if this is about being able to add custom functionality to a Plug-in (eg. Validator) that the user can easily trigger from the UI (like highlighting/selecting the invalidated objects) I’m all for it. Woohoo! But, as currently described, it being of no particular use pretty much explains to me why the description is off. Looking from the point of view of the Artist what is described here doesn’t enable artists since it doesn’t come with an explanation of what it could do or how it could function.

#6

Its an interesting proposition, if you think more of it as a collection of scripts/tools, and not necessary as part of publishing. The development of the UI elements of Pyblish, and the future searching/sharing of the plugins could make this a common place for people to share their scripts/tools across multiple hosts.
This is however expanding Pyblish way beyond the initial intentions, and also looking a bit like Fabric Engine or Shotguns Pipeline Toolkit, that goes beyond just publishing.

#7

Thanks for the input guys, really good questions raised here!

They’ll run like any other plug-in. The difference here, both technically and practically, is that there order is altered to a range outside of the current SVEC plug-ins, such as 1000. At this order, they could still appear in the GUI, remain optional and have their own order - for example, create four logical groups of orders such as 1001-1003 and you’ll have your very own SVEC-stack.

The lack of a family means they can’t operate on Instance's, so the only thing they have to grab onto is the Context. We haven’t yet put the Context in the GUI, but once it’s there, this would become the only publishable element in a scene and the one every SimplePlugin would operate on.

It’s a good question, and it’s possible they can/should. As they would remain identical to other plug-ins, there’s nothing stopping you from creating a stack similar to this.

class GatherInstances(pyblish.SimplePlugin):
  order = 1000
  def process_context(self, context):
    instance = context.create_instance(name="MyInstance")
    instance.set_data("family", "myFamily")

class SanityCheckInstances(pyblish.SimplyPlugin):
  order = 1001
  families = ["myFamily"]
  def process_instance(self, instance):
    # process MyInstance here

The difference here is that the step would be come optional.

Remember the primary goal of this proposal is to reduce the learning curve for beginners. The primary question then is whether it would lengthen the learning curve for doing anything advanced with it.

If it turns out that it also opens up new doors, then that’s great I think!

But, as currently described, it being of no particular use

No no, you misunderstand. They are of no particular purpose. As opposed to SVEC which all have a particular purpose or “destiny”; e.g. Validator's are meant for validation whereas Extractor's are meant for serialisation. SimplePlugin on the other hand can do this too, but its purpose would be delegated from the framework to the developer. It is up to you to define their purpose.

Does that make more sense?

Yes, it might.

I think the main goal of doing this would not be to remove the SVEC-stack, but to delay it until after a developer has had experience with implementing the first set of plug-ins in his pipeline. Maybe an illustration is in order. :smile:

Now

This is now. When learning Pyblish, one first has to learn about the API, obviously, but they also must understand SVEC. Before they can understand SVEC however, they need a rationale, a reason for why SVEC has been designed as it has, why selection must come before validation.

Once they’ve passed this bar, they can start implementing their first plug-in and spend the rest of their days learning about the proper use of SVEC.

Then

This on the other hand is how I’d like it to be.

The user learns about the API, which essentially becomes process_context and the fact that it runs when you publish, followed by writing their first plug-in.

From there, it is up to them whether they want to pursue an established standard such as SVEC or whether they want to roll their own.

Beyond Initial Intentions

It’s true that given this power users are able to do anything. Even import things into their scene or use Pyblish as a collection of arbitrary tools unrelated to publishing.

But let’s look at an example beyond Pyblish.

Qt is a framework and it has certain best-practices. I’m still learning about them after all these years and I doubt I’ll be able to grasp them all, not to mention have use for them all. But if using Qt meant that you first had to learn about it’s best practices, I would still not be using it to this day.

So the question is, would I even have gotten past the learning phase?

Odds are I would have glanced at the up-front cost of using it, and headed elsewhere. Something that could instead get me up and running more quickly and learn at my own pace.

Even though Pyblish is no where near the complexity of Qt, we’re doing exactly this on a smaller scale. A new user is unable to start learning on his own, but is instead force-fed our vision of publishing and the SVEC stack. What I’m looking for is a shorter path from learning to using.

And again looking at Qt, because it makes such few assumptions, everyone uses it in their own way, but they as a company, and a majority of their advanced users still make good use of their proposed workflows and convenient functionality, such as MVC and QAbstractItemModel, even though no one is forced to use it.

Some more example, Python does everything, but has it’s “Zen of Python”. Often when asking questions about how to do something with Python you get as answer “This is the Python-way of doing it”. SVEC could be the “Zen of Pyblish” in this respect.

Docker is another example; it’s a general purpose method of encapsulating multiple processes into one process, but has guidelines about how to do it for the greatest amount of flexibility. When everyone follows the guidelines, everyone works together towards a common goal. But you don’t have to if you don’t want.

Anyway, this I think summarises my motivation. Maybe SimplePlugin isn’t a way to get there, but I think it’s good that we’re having the conversation. :smile:

#8

That’s great, but how you’re trying to achieve it might be too confusing.
Maybe it’s better to have the default order for SVEC be something like 1000, 2000, 3000, 4000.
It’s easily identifiable that one can add its own Plug-ins with a certain order by itself, and even inbetween.

The first example to new users would be to explain that Pyblish runs plug-ins in a predefined order. So you hook them up with simple example #1 that is ordered at 0. (Before everything else). Plus the fact that it runs in the exact same sequence instead of receiving ‘special behaviour’?

Basically all your example is saying: You can define any plug-in your want that is your own, ordered in any way you want defined by yourself. I think that’s great. But I feel that Pyblish is already like that, but offers the clean SVEC system as a starting guide.

Also I had confused this topic to be similar to how I envisioned Actions or additional functionality per Plug-in, but now I see that’s a completely different topic.

#9

But I feel that Pyblish is already like that, but offers the clean SVEC system as a starting guide.

This is great to hear. The only reason I’m thinking along the lines of SimplePlugins is because of newcomers having trouble grasping SVEC at first. If it isn’t an issue then the proposal is less relevant.

Maybe it’s better to have the default order for SVEC be something like 1000, 2000, 3000, 4000.
It’s easily identifiable that one can add its own Plug-ins with a certain order by itself, and even inbetween.

That could be a good solution.

The initial train of thought of separating each process into it’s own class as it is today - e.g. Selector, Validator, etc. - was to give each it’s own unique superclass behaviour.

For example.

class Extractor(Plugin):
  def commit(self, path):
    # Write to temporary directory, embed path in Context

But it hasn’t (yet?) seen as much use, primarily because we’re still working on a stronger understanding of what each process (really) means. It will at least have to be a strong enough understanding to facilitate actual functionality such as the above (which incidentally I assume very few of you even know about, but maybe that’s a fault on my part in not advertising it enough).

Here’s how I envision your suggestion.

class SelectInstances(pyblish.SimplePlugin):
  order = 1000
  def process_context(self, context):
    # create instances as per usual

class ValidateInstances(pyblish.SimplePlugin):
  order = 2000
  def process_instance(self, instance):
    # validate instances as per usual

The only difference here, besides missing out on features of it’s superclass, is that the concept of “Selection” and “Validation” is purely conceptual. Developers would learn about it through documentation and the forum, talking about it and finding their own way.

In this way, we can continue to adopt SVEC, but also choose to go our own path.

Technically, it would make it difficult to build logic into the publishing process. For example, we don’t want to produce any data unless validation succeeds, but if Pyblish isn’t allowed a concept of what validation is, it can’t automatically make this decision for us based on any failures.

If the proposal goes through, we might need to also have a look at externalising this logic as well, possibly as attributes on plug-ins.

class ExtractInstances(pyblish.SimplePlugin):
  on = "Validation:succeeded"

Also I had confused this topic to be similar to how I envisioned Actions or additional functionality per Plug-in, but now I see that’s a completely different topic.

You’re welcome to bring the topic of Actions up too. If I understand if correctly, it might overlap with this. Maybe post it as a separate topic?

Edit: Apparently, if you hit “Reply as a Linked Topic”, it will keep some form of reference to this topic!

#10

I figured I’d put this proposal to the test and produce a Pyblish tutorial.

The current guide covers the most important aspects of Pyblish, but it’s long-winded. It goes through publishing from a conceptual point of view, followed by the need for SVEC and finally into implementing a plug-in.

The plug-ins themselves however make quite a few assumptions.

  1. They require Maya
  2. They assume a family of demo.model, to delay the introduction of Instance and Context until it becomes apparent what they are for
  3. They assume a set of validators, hence having to explain what validation means beforehand
  4. They assume an output directory, hence having to explain the difference between private (i.e. development) and public (i.e. shared) files.

That’s quite a mouth full, and what I’ve been working towards slimming down.

Let’s give it a try with SimplePlugin.

Introduction to Pyblish

Pyblish works by discovering and executing a series of Python snippets that you write in the order that you define.

snippet1 --> snippet2 --> snippet3

Let’s write a snippet, known as a plug-in, now.

validate_scene.py

from maya import cmds
from pyblish import api

class ValidateScene(api.Plugin):
  def process(self):
    assert not cmds.ls(type="reference"), "This scene has references!"

To run it, store it in a file somewhere on your local hard-drive. In this example, we will assume you have saved it here.

c:\my_plugins\validate_scene.py

To expose your plug-in to Pyblish, we need to register the directory in which you saved it.

import pyblish.api
pyblish.api.register_plugin_path(r"c:\my_plugins")

And finally, from your File menu, click Publish to open up the Pyblish GUI.

In the GUI, you will find the plug-in that you just wrote, called ValidateScene. Hit the play-button in the lower-left corner to run it, and that’s it!

Back to the forum thread, that’s all there is to it. Now new users know about and can write plug-ins. From here, it’s all about best-practices and guidelines, such as learning about the Context, what an Instance is, and finally SVEC and beyond.

Thoughts?

#11

That does look very simple. Think most people here are just worried about introducing new stuff into Pyblish just for sake of writing a good introduction.

Where I think/hope this generalization of the plugins is headed, is towards @BigRoy “Actions”, which is also what I was referring to earlier.

#12

That’s understandable. But the reality is, the future of Pyblish depends on a good introduction.

I don’t know if you’ve noticed, by explaining Pyblish to anyone is a real challenge. The barrier-to-entry is much too high at the moment, I’ve been looking for a way to write tutorials these short for a long time.

And besides, we aren’t really adding anything. It’s more like going lower-level than is currently possible. Actions on the other hand is about adding the ability to define anything besides the currently available SVEC. I think this will cover the use-cases for Actions too, but in a healthier fashion.

#13

Continuing on from the above tutorial, I must say I am very excited about the prospect of using this as an entry point for new users. So let’s keep going! :smile:

Pyblish and Constructive Criticism

In the previous example, you learned how to write a plug-in and run it within Maya. The plug-in warned about the existence of references in your scene and you’ll notice that if you had any, Pyblish would tell you about it.

Now, being told that something is just wrong isn’t very helpful. It would be a lot better if the artist could receive some information about what to do about it, and possibly why this isn’t considered correct in the first place.

Plug-in documentation

Every plug-in performs a given action. In the case of the previous tutorial, the action was to ensure that a scene was without a particular type of node, nodes of type reference. Let’s give the user a heads-up about what to do when they violate this rule.

from maya import cmds
from pyblish import api

class ValidateScene(api.Plugin):
  """Ensure the scene is valid
  
  A scene is valid when it has no references in it.
  References turn our pipe*line* into a pipe*squiggle*. :(

  """

  def process(self):
    assert not cmds.ls(type="reference"), "This scene has references!"

Ok, that’s a good start.

The next time an artist launches Pyblish, the GUI will provide this documentation to the user when either hovering over the plug-in in the viewport, or when entering it’s perspective.

Now let’s provide a artist with some additional information about what to do when this fails.

from maya import cmds
from pyblish import api

class ValidateScene(api.Plugin):
  """Ensure the scene is valid
  
  A scene is valid when it has no references in it.
  References turn our pipe*line* into a pipe*squiggle*. :(

  """

  def process(self):
    references = cmds.ls(type="reference")
    assert not references, """This scene has references!

Please either import or remove the following references:

""" + "\t- ".join(references)

This time, the artist is given guidance about how to resolve the problem.

In a more complex scenario, constructive criticism can make the difference between artists solving problems on their own, or seeking help from a colleague or lead, meaning the problem now consumes time from both the artists and those sought out for help.

Back to the thread, from here, developers can already reap some core benefits of publishing like detecting problems and providing support for resolving them.

A few things are missing - such as:

  1. Sharing data between plug-ins (i.e. the Context)
  2. Distinguishing between types of assets (i.e. the family)
  3. Associating types with plug-ins (i.e. the Instance)
  4. Triggering plug-ins at a given circumstance (i.e. passing validation)
  5. The benefits of delegating responsibilities across multiple plug-ins (i.e. registration)

But the major advantage is that all of it is optional, whereas currently it is a mandatory and up-front cost to learning about and getting started with Pyblish.

Developers can already build a publishing pipeline that can really make a difference with only the knowledge gathered from these two tutorials, and continue to perfect as they learn more.

This is a big step forward I think!

Actions to implement custom additional functionality for Plug-ins
#14

Would it be possible to still allow dependency injection to have process_context and such also still be available in the long run? I don’t see the need for having it removed from this. I would see it more as the base class to the SVEC classes and one that also works without using context or instance, but could also deal with it. It’s just optional you use the context or instance data. Or what’s the reason for totally ditching it?

To me I would feel that Plugin should be the barebones of SVEC. Anything that SVEC does should be possible to build with the Plugin system. I also think that the events system where one can stop a running the Pyblish process on a certain event (like validation fail) could be used for the implementations of SVEC.

#15

Would it be possible to still allow dependency injection to have process_context and such also still be available in the long run?

Yes, in the tutorials I’m assuming the inclusion of issue #127 (Dependency Injection) so in case of needing to share data, the developer could:

def process(self, context):
    references = context.data("references")
    assert not references, "This scene has references!"

And the same goes for injecting Instance.

def process(self, context, instance):
    references = context.data("references")
    assert not references, "%s has references!" % instance.data("name")

Yeah, I think we’re seeing the same future.

#16

Ah, I just realised why you asked this!

What I meant was, that a few things was missing from the knowledge given in the previous tutorials. The plug-ins still remain the same, with the addition of issue #127.

Pyblish 1.1 Released
API changes
Actions to implement custom additional functionality for Plug-ins
API changes