Long term goal

[hannes]

Hey, my long term goal is to make pyblish more accessible
In my past 3 game studios I made a similar system from scratch, and never heard about Pyblish.
It’s surprisingly unknown sadly.
So let’s make it accesible!
I come from a games background, I believe pyblish came from VFX so this might explain why i’ts not as known in games. ( i see in the credits some studios use it though)

To do this, I have few things in mind:

• a library with generic plugins
  i’m collecting lots of different plugins.
  most use context to pass stuff and only work with the plugins from the same author. this can’t scale up.
  i found some people use cmds, some pymel. we need to support differences.
• a way to make plugins work with each other. plugins from different authors are not always compatible.
  ex. longname vs shortname, pymel vs openmaya vs cmds, …
  python2 vs python 3
• an option to choose plugins. haven’t looked into the filter system yet for pyblish, this might work.
  thinking along line of a studio setup, a project setup, …
  settings file that overwrites plugin metadata (isactive, …)
  could even remap funs to work in ways original author didnt intended.
  prototype would be like maya plugin selector
  
  

atm you register paths, or plugins with a few lines of code.
it’s easy for tech people. but to make it accessible for artists we ideally have a GUI.
pick and choode your plugins, save config out.

i think an easy way forward would be extend the parent child plugin in pyblish.

• cube instance contains:
  • a shape instance
  • a transform instance
    now we can run our mesh checks on the shape part, and transform checks on the transform part

we could try work around with inheritance but feels more like some hack.
plugins would have to use this new class somehow, it doesnt feel clean
pseudocode

    class TransformValidationPlugin(pyblish.api.Validator):
        def process(self, instance, context):
             data = instance[:]
             filtered_data = [x for x in data if type(x) == pm.nt.Transform]
             instance[:] = filtered_data

ideally we dont touch original plugins and just know what they run on, then run on those applicable instances. hence the parent child setup

i think the big thing to keep in mind is not all of this needs to go in pyblish_base
this can be a separate tool or extension to pyblish just like pyblish_maya

[marcus]

I’m onboard with this! This was my original goal for Pyblish, and there are a handful of repositories of plug-ins from that journey, primarily this one.

• GitHub - pyblish/pyblish-magenta: Magenta kit for Pyblish

But, as you pointed out, having plug-ins work in multiple environments and be customisable enough for edge cases and quirks of a given studio was a challenge. Definitely possible, but more of a challenge than I had originally anticipated. A lot of them do have a ton of options that I think could be scaled more, especially the one exporting Alembic.

The Maya Plug-in UI is a good start for accessibility, but I think the most prominent example of this today that I’ve seen is the “Addons” UI from Blender.

I especially like the ability to filter by type, and have different categories of addons, some being “Official” and others being “Community”.

I’ve designed a hosted plug-in solution in the past that worked quite well and think could make a good start for this.

• quickstart · mottosso/be Wiki · GitHub

The workflow is.

1. Host a repository on GitHub, like Magenta above
2. Register it with a central GitHub repo, like scoop does it, and for be that’s here
3. Pyblish then downloads the central registry whenever the e.g. Addon UI is opened.

That would enable others to self-host (on GitHub, or elsewhere) and for others to also submit PRs to register new entries into the Addon UI. Saving us from having to host a server or database ourselves.

@hannes

it’s easy for tech people. but to make it accessible for artists we ideally have a GUI.

This part however confuses me. To my mind, artists would not be in a position to make these decisions. In fact, one of the reasons to choose for them is to ensure all of their output lines up; that all models follow the same set of rules, such as having UVs that don’t overlap, having no intermediate shapes etc. If the artist could choose to not have these checks, then what’s the point of anyone having them? Recipients of models could no longer trust that there are UVs or that there aren’t any intermediate shapes? The same applies to rigs and animations and just about anything.

The reason it is technical to author and define these publishing pipelines is because it was meant for technical authors and defined up-front, rather than be changed “live” by artists.

Perhaps I’m missing some context here, maybe you can provide an example from your experience where an artist should define their own publishing pipeline?

[hannes]

thanks for linking those resources, looks super helpfull to start from.

plugin repo

1. Host a repository on GitHub, like Magenta above
2. Register it with a central GitHub repo, like scoop does it, and for be that’s here
3. Pyblish then downloads the central registry whenever the e.g. Addon UI is opened.

I’ve created something similar for my current studio that does this, a package manager installer for unity. so hopefully should be not to hard.

accessible GUI

when I say artists I also include tech artists. over the years i met a lot of TA’s who’re asked to do everything. Some TA’s specialise in shaders but have never touched Pyhon, yet they are often asked to setup a new pipeline for artists.The devs are to busy creating the game, or just straight up dislike Python or have no experience at all with 3d content.

But artists sometimes don’t have access to a TA, if we can provide them a simple setup method, and a easy to understand workflow. This would add a lot of value. In gamedev the same processes keep reocurring between studios. No Ngons, split UVs need split normals, at least 1 UV, enforce namingconventions …
The custom process is often only a very small part of this.

all game pipelines are similar

And finally when I swap studio I just recreate the same tools from the ground up. after having gone through this for 3 times now I kinda want to work towards a reusable opensource solution that is not just used by me but can be adopted by the whole gamedev industry.

Currently Pyblish is designed as a publishing workflow, not a content authoring workflow. Yet this is where it shines I believe. with a few generic plugins and a UI we have a great find issues -> fix issues tool. Add maybe a few generic perforce and git plugins for integration and done.

Adoption struggles

When I talked about Pyblish with a pipeline TD who knew about it, they said it’s great for straight forward publishing, if you always have a certain order. (collect, validate, export, integrate) but not for more flexible pipelines. The ex. they gave was:

For example, you are exporting a playblast from maya. you export it as a image sequence, you want to publish this. you also want to publish and mp4 version of it to your review system.

in the flow of pyblish, you would extract the images, convert them and them publish (integrate) them.

but if you wanted to publish the image sequence first, then convert them and publish that later, pyblish will struggle as it doesn’t handle that effective.

When I picked up Pyblish it was quite tricky to wrap my head around it, mainly due to some non pythonic things like the function changes based on the name of the args. (context, plugin)
And crashes are not always communicated. Because of this I believe entry barrier is quite high atm.
Creating this UI premade solution would lower it, make setup easy. Then when people use it and want to costumise they’ll be more willing to dive in.

[hannes]

I also love the blender plugin example you gave. It reminds me a lot of the unity package manager but more polished. The unity one is based on NPM (nuget?) and all packages also have metadata(this includes tags, dependencies, requirements, …) you can host them on a repo and then link to it in your central manifest. and it supports versioning of packages.

[tokejepsen]

Great stuff @hannes!

I also think sharable plugins would elevate Pyblish to new exposure; Pyblish Goals

I would also encourage you to look at different pyblish repositories that’s been made over the years (search Github for pyblish and there are various studio repositories), for inspiration.

Also dont know whether you are interested but there has been an evolution:
pyblish-magenta > Avalon (https://getavalon.github.io/2.0/) > OpenPype (https://openpype.io/)

[hannes]

hi tokejepsen

thanks for the links!
i have seen avalon and pype but not tried them yet. it seems like some kind of asset management system for versioning + pyblish?
my goal is to dig in avalon and or pype when i’m happy with pyblish, since it appears that avalon includes pyblish + pyblish GUI.

i had a look around for pyblish repos already but so far the most usefull ones are the official examples.
i’ve made collection here https://github.com/hannesdelbeke/pyblish-repo-collection and ranked them on how usefull they seem from first look.

also to save myself time writing lots of plugins from scratch i try look for existing checks and port them.
see

• Collection of 48 reusable plugins for maya validation
• Collection of 15 reusable plugins for maya validation

[tokejepsen]

i have seen avalon and pype but not tried them yet. it seems like some kind of asset management system for versioning + pyblish?

Yeah, that is kinda right. If you think of Pyblish as the publishing framework then Avalon is the framework for the rest of the pipeline. OpenPype is technically a studio config of Avalon but has to cater for a number of studios so is more flexible than other Avalon configs (and Pyblish plugin collections).

I still think a plugin sharing framework would hugely benefit all communities. Iterating on the same code even when used in different pipeline environments.

[marcus]

@hannes Just checking in how things are going? You got me all excited about a future enriched with more Pyblish. If there’s anything you need from me, just me me know!

[hannes]

still invested in this, atm not working on pipelines at work so it’s been placed on hold for a bit sadly. I’ll try make some time in my spare time. I expect to continue on it next year after the xmas break.

the bootstrapper i made in this thread (Pyblish workflow manifest) is a nice MVP that solves some of the things mentioned here