Pyblish 1.1 Released

marcus · June 8, 2015, 8:56am

Pyblish for Windows 1.0.5

Version 1.0.5 is now available and includes the following submodules.

Pyblish 1.1.1
Pyblish QML 0.2.10
Pyblish for Maya 1.0.14
Pyblish for Nuke 1.0.3
Pyblish for Houdini 1.1.0
Pyblish RPC 0.1.0
Pyblish Integration 0.1.0

Summary of new features

Integrations
Pyblish for Houdini added
Core
Feature: Dependency Injection (see #127)
Feature: SimplePlugin (see #186)
Feature: In-memory plug-ins (see #140)
Feature: Custom test (see #183)
Feature: create_instance(name, **kwargs) (see #187)
Preview: Asset (see #188)
GUI
Feature: Override plug-in label (see #85
Feature: Shared instance (see #96)
Feature: Context visible in list of items (see #90)
CLI
Bugfixes

See here for a full list of changes.

CHANGELOG

Introduction

This release is focused on lowering the learning curve for newcomers, primarily via the core library, yet some of the changes have also found their way into the GUI.

The update is completely backwards compatible, but encourages you to update your plug-ins to the new DI-style detailed below.

Installation

For a new install.

Download and run the installer.
Or see manual installation instructions here

To update.

Between 1.0.3 and 1.0.4 there has been some refactoring that may affect your integration of Pyblish into your pipeline.

The previous root /python directory has been renamed /pythonpath
Pyblish Suite has been superseeded by Pyblish X which means…
…that the inner packages are now located within /lib/pyblish-x/modules instead of /lib/pyblish-suite., such as pyblish-qml.

Using the installer

Uninstall your existing copy (due to the refactoring) and then re-install.

Using the command-line

The update.bat has been updated in this release, which means you can’t run the one you have from 1.0.3.

Instead, you can:

cd to pyblish-win
Copy/paste this into a terminal.

git checkout master
git reset --hard
git pull
git submodule update --init --recursive
git clean -xffd

I’d recommend doing this locally first, before pushing things into production, due to the way file deletion/updating works with permissions and files being in use etc.

If you run into any issues, feel free to post below.

Transition Guide

In terms of mixing old- and new-style plug-ins, here’s what you need to keep in mind.

In cases where you have either process_context or process_instance, a simple search-and-replace to process will work fine.
In cases where you have both, see below.
During the transition phase, the distinction is made internally by looking for the existence of a process_context or process_instance method.
If either exist, the plug-in is deemed “old-style” and is processed using the current implementation.
If both process and either process_context or process_instance is present, old-style wins and process will not be called.

Processing logic

The manner in which process() is called is roughly this.

    ______
   |      |  incl. `instance`
   | args |------------------
   |______|                  |
      |                      |
      | not incl. `instance` |
      |                      |
 _____v_____           ______|_____
|           |         |            |
| process() |<--------|  for each  |
|___________|         |____________|


# If the arguments include `instance`, then `process()`
# will be called once every instance. Otherwise it is
# called once.
#
# Unless the plug-in is *limited* to a subset of families.
# E.g. `families = ["oneFamily"]` in which case nothing
# happens unless there is an `instance` of a supported
# family.

For a full look, see the source.

Both `process_context` and `process_instance`

The current behaviour of this is for process_context to be processed first, followed by process_instance. This behaviour isn’t possible anymore. You can however process both in the same function.

def process(self, context, process):
  # do things

In case you do have both, process_instance will overwrite process_context due to your plug-in being re-written to a it’s Dependency Injection equivalent at run-time.

def process_context(self, context):
  # I will not be called. :(

def process_instance(self, instance):
  # Runs as usual

Old-first

The reason for looking for old-style methods before new-style is because of the newly introduced ability to use __init__. In cases where __init__ is used, and process not being implemented, the plug-in is still deemed new-style as __init__ is assumed to not have been in use.

Dependency Injection

The most prominent change is this.

Before illustrating how it works, it’s important to point out that this is the new way of writing plug-ins. It means that the current way of implementing plug-ins still works, but are to be considered deprecated and no longer supported.

Now, on to the fun stuff!

As Usual

import pyblish.api

class ValidateInstances(pyblish.api.Validator):
    def process(self, instance):
        pass

This does as you would expect. Which is to process once for every instance, regardless of family (see below for new family defaults).

import pyblish.api

class SelectInstances(pyblish.api.Selector):
    def process(self, context):
        pass

In the same spirit, this plug-in runs once and has access to the context. This is nothing new.

Plug-in Independency

What is new is that you can choose to process nothing.

import pyblish.api

class SimpleExtractScene(pyblish.api.Extractor):
    def process(self):
        cmds.file("myfile.mb", exportAll=True)

This plug-in runs once, as when processing the context, but doesn’t have access to either the current Instance nor Context. This can be useful for plug-ins that are completely independent of it’s environment and state.

Default Services

What is also new is that you can also request other so-called “services”.

import pyblish.api

class CustomValidator(pyblish.api.Validator):
    def process(self, user, time):
        fname = "myfile_v001_%s_%s.mb" % (user, time())
        cmds.file(fname, exportAll=True)

In which case the services user and time are injected into the plug-in right before it’s about to start processing. Each of which are default services that ship with the Pyblish base install.

Here are all of them.

register_service("user", getpass.getuser())
register_service("time", pyblish.lib.time)
register_service("config", pyblish.api.config)

user is available by value, as it is not expected to change at run-time.
time is callable, as it provides unique values each time it is used
config is shorthand to pyblish.api.config

User Defined Services

You can also register your own services…

def say(something, yell=False):
    print(something.upper() if yell else something)

pyblish.api.register_service(say)

…and then request them via your plug-ins.

import pyblish.api

class ValidateUniverse(pyblish.api.Validator):
    def process(self, say):
        say("I just wanted to say, Hello World!", yell=True)

Service Coordination

Services are shared amongst plug-ins.

datastore = {"softFailure": False}
pyblish.api.register_service("store", datastore)

Which means you can use it to communicate and pass information inbetween them.

class ValidatePrimary(pyblish.api.Validator):
    def process(self, instance, store):
        if instance.has_data("I'm kind of valid.."):
            store["softFailure"] = True


class ValidateBackup(pyblish.api.Validator):
    def process(self, instance, store):
        if store["softFailure"] is True:
            # Do alternate validation

Or to provide globally accessible data, such as a database connection.

import ftrack
import pyblish.api
proxy = ftrack.ServerProxy("http://myaddress.ftrack.com")
pyblish.api.register_service("ftrack", proxy)

Distributed Development

With services, you can defer development of a plug-in between more than a single developer allowing for faster iteration times and improved version control.

As a plug-in is developed, requirements may arise…

import pyblish.api

class ValidateSecretSauce(pyblish.api.Validator):
    def process(self, instance, sauce):
        if sauce.hot():
            assert instance.has_data("hotCompatible"), "Sauce too hot!"
        assert instance.data("tasty") == True, "Sauce not tasty!"

Which can later be developed.

import pyblish.api
import my_studio_tools

class Sauce(object):
    def hot(self):
        return my_studio_tools.hot_sauce()

pyblish.api.register_service("sauce", Sauce())

Testing

As a final and important piece to the puzzle, dependency injection makes testing easier and more controllable.

def test_plugin():
    """Testing of a host-dependent plug-in with DI"""
    instances = list()

    class SelectCharacters(pyblish.api.Validator):
        def process(self, context, host):
            for char in host.ls("*_char"):
                instance = context.create_instance(char, family="character")
                instance.add(host.listRelatives(char))
                instances.append(instance.name)

    class HostMock(object):
        def ls(self, query):
            return ["bobby_char", "rocket_char"]

        def listRelatives(self, node):
            if node == "bobby_char":
                return ["arm", "leg"]
            if node == "rocket_char":
                return ["propeller"]
            return []

    pyblish.api.register_service("host", HostMock())

    for result in pyblish.logic.process(
            func=pyblish.plugin.process,
            plugins=[SelectCharacters],
            context=pyblish.api.Context()):
        assert_equals(result["error"], None)

    assert_equals(len(instances), 2)
    assert_equals(instances, ["bobby_char", "rocket_char"])

Related

Simple Plug-in

A new type of plug-in has been introduced; the superclass of the currently available Selection, Validation, Extraction and Conform (SVEC) plug-ins - called simply Plugin.

import pyblish.api

class MyPlugin(pyblish.api.Plugin):
  def process(self):
    self.log.info("I'm simple")

Simple plug-ins are just that, a simpler, less assuming variant of SVEC. Each inherit the exact same functionality and can be used in-place of any other plug-in, the only difference being a default order of -1 meaning they will run before any of it’s siblings.

Other than that, they are identical in every way. To make it into a Selector, simply assign it an appropriate order.

class MySelector(pyblish.api.Plugin):
  order = pyblish.api.Selector.order

More importantly, simple plug-ins are meant as a stepping stone for beginners not yet familiar with SVEC, and as a bridge to unforseen use of Pyblish, allowing arbitrary use and boundless extensibility.

With this, Pyblish is now a generic automation framework with SVEC becoming a collection of best practices and recommended method of writing plug-ins.

Related

In-memory plug-ins

This new feature will allow you to debug, share and prototype plug-ins more quickly.

In-memory plug-ins come directly from the Python run-time and isn’t dependent on a filesystem search, which means you can easily post a plug-in to someone else. Once registered, it will be processed like any other plug-in.

import pyblish.api

class SelectPrototypeInstance(pyblish.api.Selector):
    def process(self, context):
        instance = context.create_instance("Prototype Test")
        instance.set_data("family", "prototype")

pyblish.api.register_plugin(SelectPrototypeInstance)

This can be done anytime prior to publishing, including whilst the GUI is open, which means you can do some pretty intense things to hunt down bugs.

import pyblish.api

data = {}

@pyblish.api.log
class MyPlugin(pyblish.api.Plugin):
    def process(self):
        data["key"] = "value"
        
pyblish.api.register_plugin(MyPlugin)

After publishing, data will contain the key key with value value. This is an example of a plug-in reaching out into the current execution environment. Something not possible before.

Related functions

New Defaults for Families and Hosts

The attributes families and hosts now default to *, which means that they process everything in sight, unless you tell it not to.

This is different from how it was before, which was to process nothing, unless you told it to.

This has no effect on your current plug-ins, as you were obligated to always provide a family if you wanted it to process, but does mean it is a non-reversible change. So, fingers crossed you like it!

Related

Custom Test

You can now take control over when publishing is cancelled.

Currently, every plug-in is processed, unless the next plug-in is of an order above 2 (i.e. not a Selector nor Validator) and no prior plug-in within the orders 1-2 (i.e. any Validator) have failed.

This is what that test looks like.

def default_test(**vars):
    if vars["nextOrder"] >= 2:  # If validation is done
        for order in vars["ordersWithError"]:
            if order < 2:  # Were there any error before validation?
                return "failed validation"
    return

The test is run each time a new plug-in is about to process, and is given a dictionary vars which is always up to date with the most recent information.

Currently, the only two members of the dictionary is nextOrder and ordersWithError, but as need arises more will most likely be added.

Related

register_test

create_instance(name, **kwargs)

You can now create instances and assign it a family in one go.

import pyblish.api
context = pyblish.api.Context()

# Before
instance = context.create_instance(name="MyInstance")
instance.set_data("family", "myFamily")

# After
instance = context.create_instance(name="MyInstance", family="myFamily")

And condense some otherwise repeated code into a more minimal variant.

import pyblish.api
context = pyblish.api.Context()

# Before
instance = context.create_instance("MyInstance")
instance.set_data("family", "myFamily")
instance.set_data("color", "blue")
instance.set_data("age", 35)
instance.set_data("height", 1.90)

# After
instance = context.create_instance(
  name="MyInstance",
  family="myFamily",
  color="blue",
  age=35,
  height=1.90)

Related

Unified Logic

The order in which to process plug-ins, and whether or not to cancel a publish after a failed validation was previously implemented in each user-facing interface. Such as pyblish.util.publish(), the command-line interface and graphical user interface.

In 1.1, logic has been centralised and is more likely to remain identical across interfaces.

Related

logic.py

Asset

As a preview, I’ve also included an undocumented and unsupported method of using the term Asset in place of Instance, as discussed here.

It means you can give it a try and get a sense for whether or not it works for you. If it turns out to be a winner, it may potentially become the de-facto new standard of Pyblish.

An example, taken from the forum thread.

import pyblish.api

class SelectCharacters(pyblish.api.Selector):
  def process(self, context):
    for objset in cmds.ls(type="objectSet"):
      name = cmds.getAttr(objset + ".name")
      asset = context.create_asset(name)
      asset.set_data("family", "character")

class ValidateColor(pyblish.api.Validator):
  families = ["character"]
  def process(self, asset):
    assert asset.data("color") == "blue", "%s isn't blue" % asset

The same behaviour is replicated all across the board, including an alias for Instance through the API.

>>> import pyblish.api
>>> assert pyblish.api.Instance == pyblish.api.Asset

Use with caution and expect it to vanish at any moment without warning.

GUI

Amongst the few GUI changes, the most prominent one is on the backend which is a new method of IPC communication via remote-procedure calls in place of the previous RESTful architecture.

The full motivation and reasoning behind this switch can be found here.

https://github.com/pyblish/pyblish-rpc/wiki/Motivation

Visually, another thing you’ll notice is that all plug-ins are now drawn with a warning label. The warning label will appear on any plug-in still implementing the pre-1.1 interface of process_context or process_instance.

Shared Instance

From now on, only a single instance of Pyblish QML can be running at once. This means that when the GUI is first launched from one host, it is then re-used in the next. Saving both time and memory.

Technically, a host registers interest with the GUI upon asking it to show. Upon having a new host register interest, the GUI then resets itself to visualise the environment of the newly registered host.

This is handled via the new Pyblish Integration project and isn’t something you need to manage on your own.

Override label

untitled

Deprecation warning

untitled

Visualising plug-ins compatible with Context

untitled

That’s it! Enjoy!

marcus · June 8, 2015, 8:57am

mkolar · June 8, 2015, 9:13am

So if I understood correctly. I should be able to just run update.bat under pyblish-win installation and slowly work my way to updating plugins afterwards right?

marcus · June 8, 2015, 9:14am

Yes, that’s right.

As a precaution, I would suggest you do it locally first.

mkolar · June 8, 2015, 9:15am

I feel adventurous today. Worst case scenario, I revert from backup.

marcus · June 8, 2015, 9:16am

Let me know if you run into any trouble.

By the way, reverting can be done like this.

$ cd pyblish-win
$ git checkout 1.0.3

mkolar · June 8, 2015, 9:25am

during the update:

Unlink of file 'lib/pyblish-suite/pyblish-endpoint/pyblish_endpoint/vendor/wekzeug/serving.py' failed. Should I try again (y/n)

Seem to not like something.

Edit:
My bad, looks like some machines had hosts running since Friday.

marcus · June 8, 2015, 9:34am

Files being in use is always a bit dodgy. Not much you can do, except double check permissions and that things aren’t in use, it is attempting to delete (unlink) it. The bad news for you is that you’ve probably got an inconsistent state of Pyblish now, as some files might have gotten deleted before it failed.

Something else just occurred to me, which is that update.bat is been updated, but you are running a previous version of it. Sort of an inception going on. You will get the latest update.bat once it finishes, but the one you’re actually running is old.

This time around, you are probably best off running the lines by hand. Try this.

cd pyblish-win
git checkout master
git reset --hard
git pull
git submodule update --init --recursive
git clean -xffd

You can copy-paste all of it at once.

update.bat shouldn’t see many updates so it shouldn’t be a recurring problem.

mkolar · June 8, 2015, 9:52am

Does this apply to repairing as well? There are a few occasions where we have repair_instance. Should that become repair?

marcus · June 8, 2015, 9:58am

Yes it does. And they are also backwards compatible.

The one caveat here is that arguments supplied to repair() should match the ones supplied to process(). For example, if you have process(self, instance) then you should have repair(self, instance). If they don’t match, you get undefined behaviour.

I’m expecting this to go away with the implementation of Actions.

davidmartinezanim · June 8, 2015, 10:52am

After using update.bat, I get the following error when starting Maya:

Failed to execute userSetup.py
Traceback (most recent call last):
  File "C:/Users/dmartine/Documents/maya/scripts\userSetup.py", line 2, in <module>
    init_pyblish.init()
  File "C:/Users/dmartine/Documents/maya/scripts\init_pyblish.py", line 7, in init
    import pyblish_maya
ImportError: No module named pyblish_maya

This might be caused by the way we were initialising Pyblish. Do you know what could be causing this? If you need to post me any code, let me know.

mkolar · June 8, 2015, 10:58am

Probably due to this:

The previous root /python directory has been renamed /pythonpath
Pyblish Suite has been superseeded by Pyblish X
Inner packages are now located within /lib/pyblish-x/modules instead of /lib/pyblish-suite.

I guess you’re adding pyblish paths to you environment prior to launching maya. If so, then you’ll need to change your pythonpath to include:
pyblish-win\lib\pyblish-x\integrations\maya
pyblish-win\pythonpath

marcus · June 8, 2015, 11:01am

Yes, probably what @mkolar said.

Otherwise, would it be possible to see your userSetup.py? It might reveal what paths it’s looking for.

davidmartinezanim · June 8, 2015, 11:33am

Yep, that was it.

Thanks @mkolar and @marcus

tokejepsen · June 9, 2015, 3:22pm

When renaming files from validate... to _validate..., pyblish still picks them up.

Have I missed a change in the api?

marcus · June 9, 2015, 3:29pm

Yes, there has been an undocumented change. Names don’t matter any more, you can name your files anything you want.

It’s been changed because of SimplePlugin, and I left it undocumented because it’s still a good idea to keep plug-ins named by their corresponding SVEC type.

Having a convention for temporarily disabled plug-ins could be useful though, and underscore seems appropriate. I’ll put that in. Thoughts?

mkolar · June 9, 2015, 3:42pm

Good to know they’re not dependent on name anymore (didn’t seem necessary when I first read about it anyways.), not that I wouldn’t name them the same way anyways, but one never knows.

Underscore seem like a nice way for temporarily disabling plugins. Another option would be starting the name with a dot ‘.’ That would align with unix system invisibility and would be very intuitive as well. But either will work.

marcus · June 9, 2015, 3:44pm

Dot could work, but to me they mean “hidden but important” whereas an underscore means “internal”. Neither really says “disabled” though. Ideally, plug-ins that shouldn’t be there simply shouldn’t be there.

mkolar · June 9, 2015, 3:56pm

True. This is more of a development convenience function that would be nice, but certainly not essential. If possible and easy to implement, I’d exclude the ‘undescored’ plugins.

marcus · June 9, 2015, 3:56pm

As long as you keep your audience in mind, it should be fine.

For example, consider this package.

└── pyblish-magenta
    ├── put_conceptart_in_right_location.py
    ├── joints_should_be_hidden.py
    ├── export_correct_geometry.py
    └── check_that_display_layers_match.py

Versus this.

└── pyblish-magenta
    ├── extract_model.py
    ├── validate_joints.py
    ├── validate_display_layers.py
    └── conform_conceptart.py

Familiarity helps.

Pyblish 1.1 Released

Pyblish for Windows 1.0.5

Introduction

Installation

Transition Guide

Processing logic

Both process_context and process_instance

Old-first

Dependency Injection

As Usual

Plug-in Independency

Default Services

User Defined Services

Service Coordination

Distributed Development

Testing

Simple Plug-in

In-memory plug-ins

New Defaults for Families and Hosts

Custom Test

create_instance(name, **kwargs)

Unified Logic

Asset

GUI

Shared Instance

Override label

Deprecation warning

Visualising plug-ins compatible with Context

Both `process_context` and `process_instance`