Pyblish workflow manifest

GOAL

suggestion of a new feature i’m planning.
the goal is to be able to

  • costumise your pipeline, by choosing which plugins to use or not use.
  • overwrite settings for plugins, assign actions, etc.

this will be helpfull once the package manager is out, and we can use shared plugins.
in this case the target for your plugin might be different than the target from the build in plugin.
a lot of settings or passed through context or env vars, but in a way that they only will work with similar plugins.
this pipeline config would allow all these different plugins to work together more smoothly. and give you an easy overview

current tech in pyblish

pyblish already has targets and filtering
but targets are set per plugin, and filtering requires you to script quite a bit if you want to adjust plugins.
it’s pretty good at adding and removing plugins, but overwriting settings requires you too know exactly what you are doing.
and having an overview of your pipeline is hard if it’s done in code

so i suggest to add some kind of

1. pipeline manifest. ( aka project manifest / workflow config / pipeline config /…)

a config file that you can register before discovery.
it would piggy back on the current filtering mechanic for now.

2. manifest creator/editor

register all plugins
open the manifest editor to overwrite default plugin settings
hit bake/save/export to create a config file with your settings in

image
this could tie in with the package manager eventually


since this, just like filtering, clashes a bit with the data driven approach.
should this be a separate repo? (the UI for the manager/creator likely will be)
what about the config loading?

i imagine it to work something like:

from pyblish import api

api.register_plugin_path('C:/shared plugins')
api.register_config('C:/project/workflow1.json')

plugins = api.discover()

register_config would then read the config, and register a filter based on it.
any properties found in the config would be overwritten in the plugin.
it would also support hooking up new actions to plugins.

config example :

plugin_name
    action = [action_name]
    __name__ = “spiderman exporter”
    meshname = “spiderman”
    additive_property += 3

created a prototype, what i had in mind seems to work.

pipeline config
:white_check_mark: apply config to registered plugins

config creator
:white_check_mark: load plugins from discover
TODO: edit settings in UI
TODO: save out a pipeline config
image

dirty code here https://github.com/hannesdelbeke/pyblish-plugin-manager

Great writeup, I think these kinds of proposals are what can push Pyblish and publishing in general forward.

That said, this particular approach is backwards. Or Pyblish is backwards. It’s one or the other. And I see this idea every so often, both in bespoke pipelines and commercial ones. I think Shotgun works this way, where you define a processing pipeline upfront, and then summon each step in that pipeline explicitly.

But Pyblish works the other way around. It’s so-called “data-driven”. Meaning that the processing pipeline is defined by the content you push through. For example, if you publish a model, the modeling plug-ins are called. If you publish an animation, the animation plug-ins are called.

This concept carries through with each level of granularity.

  1. Publish a character model, plug-ins related to models and characters
  2. Publish a character model, in a particular project, at a particular location, and plug-ins for that specificity gets called

The way you control this pipeline as an artist, is by authoring your content to fit the pipeline.

  • What does a character model look like? It’s a series of meshes in a group with a _charModel suffix
  • What does a rig look like? It’s a group with a _rig suffix
  • What does an animation look like? It’s got an objectSet called animOut and an optional namespace
  • And so forth

Who defines these _charModel and _rig suffixes? You do, via the collectors. The job of the collectors is to venture out and scan for what the user has created, and associated the appropriate plug-ins to their content. if they’ve got both a model and a rig in their scene, they could both be picked up, independently, and associated with their relevant plug-ins. It wouldn’t matter.

And what if suffixes aren’t enough context for an artist to provide? What if they needed to include whether they wanted their animations published as both ATOM curves and an Alembic cache? Well then the collector could also look at attributes on this group, such as exportAtom = true and exportAlembic = true, whereby the artist can either create these attributes themselves (painful) or have an authoring tool to prepare their work for publishing.

Here’s an example of such a workflow, which was the starting point for Avalon which carries this idea forward.

With this in mind, in order to give artists - both tech and non-tech - control over the pipeline, the solution may not be giving them control over which plug-ins to run; but the means to help your collectors recognise what it is they want to publish.

my current project setup w pyblish is data driven, just like you say

  1. Publish a character model, plug-ins related to models and characters

yet i still feel the need for this config setup.
i’ve read the discussion in https://github.com/pyblish/pyblish-base/issues/343 when filtering was added, and various docs where you explain the data driven approach. so i see where you come from.

but i feel limited in pyblish in some areas, tell me if i missed something:

  • authoring your content to fit the pipeline … Well then the collector could also look at attributes on this group …

    this mean maya scene need to be editted, attributes in a group, suffixes in name, …
    it also requires a helper tool.

    I want to run plugins on any scene without changing them.
    I want to change plugin settings without changing plugin code (w inheritance) (this is to work towards reuse plugins across studios, projects, …)
    i only want to write custom collectors when needed, and reuse everything else.

  • workflow management.
    if my collectors always run, in a scene with both rig and animation. both rig and animation get collected. what happens when i click publish, if for example i only want to publish the rig?
    pyblish would publish both.
    Currently i register different plugins for different workflows to handle this with a helper tool. this is the same as target in a way, the issue is that target is set in your plugin so needs changing per plugin, again plugin settings.

or have an authoring tool to prepare their work for publishing.

i envision this pipeline manifest acts as said authering tool, to work together with pyblish.
the MVP is getting close to finished so it will be interesting for the discussion to set up some examples in maya and test out the areas i believe current Pyblish struggles with.


you can select tests, and overwrite their settings, save this out to a config and load in those settings.

Great. :slight_smile: This was key for me, and something I didn’t see in the original post. It’s important we compare and contrast existing methods of achieving a particular goal. If both approaches are to exist in parallel, they need to be clearly distinguished and have a solid reason for existing.

I’ve also re-read my own comments on the link you posted to issue 343, and I realise I sound like a broken record. I also realise I appear to be in a minority, since everyone else opposes the data-driven approach. So just in case there is a silent majority out there, now is the time to make yourself heard, because if this proposal makes it through, it may affect you.

If it so happens that the reverse workflow is better suited to those using it, then it only make sense to go that route. Even if I personally aren’t in favour of it. An opinion I’m open to change given enough evidence of the benefit.

So, let’s dive in!

I struggle to think of a usecase here. Most of my career has been on the artist-side of this equation, and when I publish something I’m the author of it. I created the whole thing, I change things all the time. Changing things is what I do. So changing things in order to align with a publishing pipeline seems like an obvious thing to do. What am I missing? What is the usecase? Who’s publishing that cannot also make changes?

Not true, it would give you the option of publising both. But a collector could both pick up on a disabled state of your rig, or the user could at run-time untick the rig instance ahead of publishing.

Not to mention it would be exceedingly rare for it to happen, so rare that it must be intentional. A solid collector would be able to differentiate between a rig being published, and a rig having been referenced for animation.

Yes, good idea. We can solve the problem from both ends, and compare and contrast the result.

apologies for all the text in advance

i am not sure if it would make sense to merge this suggestion directly in pyblish , for now i’ll keep it a separate repo to see how it goes. Maybe pyblish will be served in 2 flavours, data driven or predefined. and the user chooses based on their needs.


dirty scenes

in the ideal world all scenes are setup from the start according nice rules, in a predictable way, and pyblish runs to ensure this.

but often i work with existing scenes, most very messy. and lots of them.
if i want to use pyblish to fix this, but can’t use pyblish straight out of the box, without first doing some cleanup, the entry barrier to using pyblish is higher. making it more likely to not be used.

adoption for new tools being used by the user (artists or TAs usually) already is difficult. if we add a requirement that said tool will only work if your scene is setup in a certain way, they likely wont bother to try it.

if the user, let’s say an environment artist, only ever works with environment art pieces. There is no point to run any other collectors. It shouldn’t matter. but what if the colelctors pick up an isntance by mistake. there is a bug, or a edge case. the pipeline has more points of failure now.

a second issue that we can run in is, if my collector picks up meshes based on say suffixes, and there is a typo in the suffix the mesh wont be picked up. this makes complete sense, but is unintuitive to the user. what they sometimes want is to force the environment checks on said mesh. select their own input for the collector.

i find the data driven approach is great when all your scenes are set up in a good consistant way, and pyblish is there to enforce it stays like this. but if you get dropped in a project with no consistant naming conventions, file structure etc, pyblish struggles currently. and this is where it might make sense to go to a more predefine workflow.


mesh authoring tool

making dirty scenes clean sounds like a mesh authoring tool
you pointed out somewhere on the forum before. pyblish is not a mesh authoring tool. but this is where it shines IMO. the validation part of pyblish is perfect for this. and some pipelineTDs i talked with agree that’s their favorite feature of publish.

medic is the only other maya validation tool i found so far, but a lot less polished compared to pyblish. It has similar features but more aimed towards mesh authoring, validating and fixing.
Pyblish is a more complete package IMO, with support for multiple dcc, and forms a better base to work on. Since there is no existing authoring tool that i know off that fills my needs, i’m planning to make it on top of the framework of pyblish. now this does not has to become part of pyblish, but i feel there is a need for this kind of tool.


Yes, the user can uncheck the extractor, but it’s not that intuative.
much more understandable for the artist to know they are running the animation exporter, and will never accidentally export a rig.
you correctly point out that with good collectors this should not happen.


plugin settings

all this talk about preselected plugins vs data driven plugin discovery.
but another point this config approach touches is settings overwritten from outside the plugin.
is there any more input from you on this?

this would enable sharing plugins, reusing plugins per project, studio, …
example: a collector that collects meshes with prefix, read from the property “prefix”
prefix = “CS_” for counterstrike
and could be set to collect meshes with prefix “BF_” for battlefield

currently this is already possible in pyblish, let’s say you set a env var and read from that, or the collector reads from a config file.
but it feels cleaner to read from an attribute of the plugin. easier to make it consistant between shared plugins.

pyblish discovery filter could also do this, but not in a straight forward way.
this pipeline config actually piggybacks on the filter system for now, and is just an easier way of controlling your filters. the main point i try solve here is accessibility for changing plugin settings and controlling plugins. ( i believe discovery filters are not even documented yet, the only info i found was in the source code and issue 343)

Hey Hannes, we use a hybrid approach and it seems to work for us (so far). We essentially wrote a bootstrapper for Pyblish plugins, and it loads the tests we want (I personally prefer the explicit approach). We resorted to a JSON file where you specify families or explicit plugin paths and each of our publishing tools calls the bootstrapper with its own JSON file. Took less than a day to write and test. We didn’t want to add additional metadata to our scenes (there’s already far too much to manage). Keep in mind our publishing tools are separate/custom; we use Pyblish exclusively for collecting/testing.