Filtering collected instances based on category/family

General
#13

Ok, one way to approach this is to use your dialog to initialise the scene and/or assets with data that a collector could later pick up.

Let’s say we assume a Maya scene with some nodes. Simply hitting “Publish” doesn’t reveal anything at this point, the collectors can’t yet identify what is an isn’t an instance. Your dialog then could “install” attributes, or add a particular node, that collectors are set to search for.

# Psuedo
dialog.selection().make_camera()
pyblish_lite.show()

Pyblish encourages stateful data, that once all has been said and done, simply hitting publish should faithfully recreate a given scenario. This is to faciliatate batch/offline processing, where there is no interface and the user can’t be given an interactive choice.

#14

That’s a good way to frame the question. I would anticipate a lot of collectors, and then a lot of logic for assigning families. I think most users would have a lot of collectors run that are never relevant to them, and I would worry about false positives, or false negatives if I or another TD didn’t fully anticipate all the publishing permutations. I’d be a little concerned about when modelling starts writing their own plugins, not taking into account that they might affect animation, etc.

Part of the problem on my end is that we don’t have a very robust environment management setup, so there isn’t much data available in the environment for divining a user’s task. This is something ideally which would change, but is a bit out of my control at the moment.

So all things being equal, at the moment I prefer to organize plugins in such a way that they’re only loaded and run in a specific context. That way I can say: modellers, put your plugin in the modelling folder and then launch the ui in the modelling context, and you never have to worry about what animation is doing.

This is pretty much what I’m doing at the moment, 3 plugin paths are set prior to launch: the global plugin path, which are always relevant, the application plugin path which is relevant to the host you’re launching from, and finally the task plugin path which is passed explicitly be the artist.

From a workflow standpoint, I see this as being basically the same, but again, all things being equal, it seems more complicated to me to manage adding extra data to the scene (which will be handled differently per application), and then managing the logic for how to interpret that data, versus just saying “only run these plugins.”

I think a lot of us are probably used to each department having their own publish tool tailored to their own needs, and that’s part of the reason I think pyblish is great, unifying the interface and architecture regardless of what is being published. But I don’t feel like having more control of plugin execution undermines that at all, and I suspect this is a request you’re going to keep getting.

#15

This is an opportunity for me to highlight in which direction I’m guiding you. When we’re done, I would like you to be able to say: “modellers, tag your work modelProxy, modelInternal or modelAnimation depending on your intent” where each family (read “contract”) is detailed on your wiki/forums.

modelProxy: "A low resolution, non-essential asset for optimal performance/memory"
modelInternal: "An asset only relevant to either you or those within your department."
modelAnimation: "Animation-friendly geometry, with uv's, no self-intersection and no manifolds"

At which point you, the developer, can design plug-ins that operate based on their intent. Internal assets for example may be allowed to skip some of the more thorough checks, whereas modelAnimation must pass the harshest of checks.

That’s a good way to frame the question. I would anticipate a lot of collectors, and then a lot of logic for assigning families.

Ok, this is good! Let’s try and tackle this.

There is a practice that I’ve more often come to recommend when it comes to making collectors, which is it to allow your td’s to make the decision about what family/families a particular asset belongs to. They know a lot more about the asset than any automated process ever will, and you can extract this information from them.

Have a look at this collector.

This one collector applies to every department, any family and moves the responsibility of you figuring out a family based on environment or circumstance to them assigning it themselves based on what they intend.

Part of the problem on my end is that we don’t have a very robust environment management setup, so there isn’t much data available in the environment for divining a user’s task.

Somewhat of a side track, but this is one of the reasons not just you may sometimes get an answer such as “No, Pyblish doesn’t do that”. I’ve made it my mission to ensure Pyblish only ever concerns itself with publishing tasks, so that remains lightweight and unbloated.

Before publishing, both of these questions need a “yes” answer.

  1. Is the data complete?
  2. Is it outgoing?

Yes, I tried phrasing it in that way to put it in context with Pyblish.

But the truth is, what is needed there falls outside the scope of Pyblish. As I mentioned above, the data must be complete. Your collector(s) depend on it.

Making data complete is opening a whole different can of worms, one that is equally deserving of one or more dedicated tools. I know @BigRoy is really busy at the moment, but he’s got the closest thing to what I try and steer developers towards. Maybe when he finds the time, he could share some of those.

Sorry, could you rephrase this? Just want to make sure I understand completely.

Thanks for excellent questions, this is great, and hope it helps!

#16

You could present all the available instances to the user, and let them decide what to publish. The workflow I’m thinking about is to present a mesh as modeling.mesh and animation.mesh, in an unchecked state. Setting instances initial state can be achieved with instance.data['publish'] = False.
When the user decides to publish either of those instances, you can use callbacks to persist the publish state back to the scene. I’d imagine using attributes in Maya, so if the user checks modeling.mesh to be published, you add a boolean attribute to the mesh called pyblish.modeling.mesh (can’t remember if Maya accepts “.” for attribute names) set to True.
Next time someone publishes the scene, the collector recognises that pyblish.modeling.mesh is set to True, meaning that the instance should be checked for publish in GUI.

What you achieve with this workflow, is to keep your code within the Pyblish universe.

Granted this workflow will present the user with an increasing amount of instances, when you add more and more collectors, but that is possibly something that could be accounted for in the GUI by having the instance sections being collapsed by default.

#17

Agreed.

The GUI can and should evolve based on how it is being used.

#18

Great discussion here. Before I jump in let me state that my experience using Pyblish is running in a small animation studio (mostly up to 6 people, with recent spikes to 12). There’s no “environment switching” going on at our location during a session or per task in production (e.g. modeling and animation have the same environment/tools). As such we make extensive use of families.

In a way we have a somewhat similar workflow to @mattiaslagergren image for a GUI (even though not really “designed” yet, but a plain list). Basically the artist defines what he wants to extract. Almost like a form of “export selected”.

Yet instead of instantly doing an export after pressing that button it tags data in the scene (stores it persistently) together with its settings so that subsequent publishes will be done alike. This means publishes become much more consistent. For example in Maya we create objectSets that contain the nodes to be extracted. Simply a Camera export would create an objectSet containing that camera. This set would also store data like startFrame and endFrame, etc.

Then we have a single collector Plugin which just finds the objectSets in the scene and creates a Pyblish instance with that family and data.

In our case we produce an objectSet with a family and whatever the artists adds to that set is included in that extraction. This way we can have many consistent outputs from a work file.

Then Validators are always all on the active plug-in paths but are only active for specific families. E.g. a Camera instance has other validations/extractions than a Pointcache instance. If an additional “type” is to be extracted we define another instance. The nice thing about sets is that the same nodes can be included in different sets at the same time.

And importantly, this also allows us to “batch” update and republish content.

#19

Thanks everyone for your thoughtful and thorough replies.

I just mean that the approach I tried and the alternative suggested in this thread are both basically ways of controlling which plugins are run based on user input:

I understand that adding data to the scene is the appropriate solution in pyblish terms, but if it’s primarily being done to assign families, then to me it just seems like a somewhat unnecessary intermediate step, compared with just having the user set the family directly.

I don’t mean to be argumentative or push for something that’s counter to the principles of the project, I just feel that in my situation, to promote adoption and support the desired workflows, the most straightforward path is to explicitly control what plugins are run based on the user input.

This I don’t actually feel is a viable solution. I think once the number of users * number of publishables gets high enough then incorrect publishes are inevitable, and conversely if I know what I want to publish before I launch the tool, then why not limit what is publishable in that context?

#20

I think at this point, it sounds like you’ve got an understanding of what is favorable with Pyblish but your path is simply slightly different. That’s as far as educated decisions go, and I think that’s fine.

What I’d suggest is that you do follow your own path, odds are you’ll be the one sitting on an upcoming best-practice in the near future once your system has stabilized. Publishing is simply a too unexplored (and documented) of a concept to fully say what’s right and what isn’t at this point.

Maybe once you’ve found your bearing, you could venture down the suggested path too and share a pros/cons. That I think would be the greatest contribution to the project and to publishing overall.

#21

At the original time of implementing that it wasn’t about families. On a more global level it was meant to store information like startFrame, endFrame, preRoll, etc. All the things you’d want to avoid “guesswork” to what the Artist intended. For us this made our scenes’ content much clearer. It actually becomes much easier for the artists in the long run: if they publish an update of a shot they don’t need to re-enter all this information, and of course ensure it’s consistent where it needs to be.

Whether it persists in the scene or in an asset management database of course would be all up to personal preferences. I’m purely debating the aspect of persisting those choices.

In a way you could see the “instance” that is created by user input and persists with it as a “mini-environment” in the scene. Basically that defines what gets run, whether that’s through some state managing of your environment or pyblish families I guess is up to you.

The problem is exactly in the end of that sentence. If you already know what you want to publish then there’s no need to push buttons or even have user intervention Once there’s two models coming from one scene, or a camera and a character coming from a scene (both separately) then it becomes a bit more guesswork to understand the Artists’ intentions. Does he even want to have that camera published? Similarly what frame to what frame are required for a pointcache? Does it need pre-roll? Handles?

Anyway, as we started taking more control of Publishing (moving to Pyblish made things easier for us, so we took it a step further) we learned more about possible pitfalls and also what were mindblowing improvements that we hadn’t expected. I’d say the best step at this stage is to leap in and run a production.

This sounds spot on!

#22

I absolutely agree that if you present all the instances to the user, it’ll quickly become overwhelming for them and they could potentially publish something they don’t intend on.
But I do think you can present the instances in GUI in a cleaner way, so they are more aware of what they are publishing. Currently the long list in pyblish-qml and pyblish-lite of instances, becomes unmanageble with 10+ instances to make a decision on.
If you had a section where the user could see all the instances, that are checked for publishing, which could be just two of 10+ instances, they would have a better overview of what they are publishing.

As a side note, we kinda had this problem with publishing from Hiero, where I present the user with all shots tagged for publishing. If the user had a video track with tens of shots, it would produce tens of instances, meaning it would become tedious to select specific shots to publish.
The solution was to collect the current selection when publishing (Hiero’s implementation is right-click menu), and only have those shots checked for publishing. This made it much quicker to publish specific shots, without having to reorganise the tags within the Hiero project.

1 Like
#23

Cheers everyone! This has been a great discussion for me. I’ll let you know how I get on.

#24

Same here, very helpful!

#25

In regards to filtering available instances prior to starting the ui (I can certainly see the usefulness of this in certain situations), it could also be sorted by having 2 publish options.

‘Full publish’ - Loads Pyblish GUI populated with all the available instances from the scene (the current default behaviour)

‘Selective Publish’ - Lets user pre-select the families he’d like to publish and then populates publishing GUI with only these. I’d go with something like checkboxes, so if user has a massive scene and only wants to publish camera and cache families, he can still do it in one go.

Technically this should be really easy to implement

This could simply be 2 separate publish buttons, with different behavior, or something like maya’s option box in the menus.

#26

A whole other question is, whether just having a better, more organised GUI wouldn’t solve this issue in one big swipe. Ability to filter out families, collapse them and so on.

1 Like
#27

An option box is a good idea, both intuitive and non-obtrusive. Widening the GUI itself is also a good idea, I can see it go both ways.

To me, the GUI we have at the moment is quite minimal in contrast to how custom each publish could potentially become. I can see it grow into a much more capable GUI, more akin to an actual editor as opposed to just simply visualising data.

However, the issue I see with extending the GUI is that it is simply a hard thing to do. Personally, both QML and Lite are at the tip of my knowledge in user interface development. I believe we need a more efficient foundation to add more interactivity and features to either, something I’m simply not familiar with.

Having @mattiaslagergren onboard could propel things in this direction, but I am uncertain with how familiar you are with this level of complexity in desktop-land? It could potentially be built with web application tools, like Electron, similar to how QML is a client/server-type application.

I would ideally take someone on with experience in editor-style GUI development. Someone who could potentially consult in an appropriate direction and set guidelines for how to technically build up to a high enough level, without introducing brittleness. Also here, with @mattiaslagergren and c/o, that could potentially be possible.

Without assuming much, the most realistic option to me is the option-box approach. That is, many small GUIs working together, as opposed to one capable GUI, even though I’d personally prefer the latter.

Custom UI prototype
#28

I understand it wasn’t clear from my image but the “Custom” option was actually intended as a “Full publish” option. This is further discussed here: Custom UI prototype

This has been a topic (electron vs python/qt) of discussion here at ftrack many times. We’ve a lot of experience writing UIs in different languages, but many of us tend to favour web frameworks and the fact that there are clear synergise with where our main ftrack web interface. That said, many of our customers and integrators are comfortable with python and qt UIs and we don’t want to put up a barrier for 3rd party developers.

All-in-all we’re leaning towards having a python/qt UI, but I expect there to be more discussions.

#29

Hi there, I’m evaluating Pyblish and have the same question and I’m wondering if anything has changed since this thread was active. Apologies if this has been addressed and I missed it in the docs or a different thread (or even this one!). I assume it would be in the Pyblish-QML if anywhere.

My use case here is that I’m not looking to build a whole pipeline in one go, but instead develop over time a family of small individual publishing tools with their own UIs before the Pyblish window is shown.

For example, my first two tasks are individual tools which will:

  1. publish cameras from Maya scenes
  2. publish shelf buttons from specific shelves in the Maya UI

As you can see, being unable to filter the UI will give a lot of irrelevant info at publishing time (the shelf collectors will collect 50+ instances) and I’m not sure I see an elegant way to avoid it.

#30

Hey there @space-m, welcome to the forums. :slight_smile:

It’s been a while since this topic was active, but I suspect you should be able to either…

  1. Set an environment variable from your camera/shelf tool
  2. Set a Python module-level variable e.g. mypipeline.family = "cameras"
  3. Assign some attribute to relevant nodes e.g. cmds.setAttr("pCube1.shouldPublish", True).

In any case, your collector could then look at those values to determines which instances to create.

There’s also the target aspect of the UI which might help.

#31

Thanks Marcus, Targets looks like the thing to try. The api docs don’t have register_target listed yet, but I assume there’s a deregister_target to go with it?

Also, while looking there, it occurred to me it might be even better to use the host feature i.e. register_host("maya.shelf_exporter") or register_host("maya.camera_toolset") or is that a bad idea?

#32

You can find a listing of all actually-available API members here.

No, that seems like another potential solution, good thinking!