Commandline publishing

hi everyone,

At realise studio we submit everything to the database (ftrack) using pyblish.
some of the things include:

plates, renders, compositing, Houdini digital assets, luts, ect

however re-rarely use an interface to do these tasks. either its data operations in putting large amounts of plates.
or its people publishing a rendered the next morning. usually by this time they are no longer in their scene file.

I think having a very strong commandline tool that allows us to be stand-alone from other content creator software would cover a lot of cases facilities that might have a more specific need yet still like the ease-of-use of a commandline without having to drop to Python to publish an element.

Hi Lars,

Thanks for sharing.

I’ve had a look at restoring the command-line interface from Pyblish 1.0.11 and it’s gone well, I’m looking at having it be part of the next release 1.0.16 sometime this week.

The tests all pass, but seeing as you’re a few versions behind I would advise upgrading on a probation basis and running it through your own set of tests. If you find anything off, let me know.

Difference

The one difference between 1.0.11 and 1.0.16 that you might notice in tight corner-cases is the more wide use of a plug-ins order attribute. This affects the logic at which Validation cause subsequent steps Extraction and Conform to not proceed.

# 1.0.11
if has_error and type(plugin).__name__ in ["Extractor", "Conform"]:
   return # Do not continue on to Extraction..

# 1.0.16
if has_error and plugin.order >= 2:
   return

Extractor and Conform then has built-in orders of 2 and 3 respectively. The behaviour should remain the same, with the added benefit of more robust modifications to the order attribute.

Interface

The interface is like before.

$ pyblish --help
Usage: pyblish [OPTIONS] COMMAND [ARGS]...

  Pyblish command-line interface

  Use the appropriate sub-command to initiate a publish.

  Use the --help flag of each subcommand to learn more about what it can
  do.

  Usage:
      $ pyblish publish --help
      $ pyblish test --help

Options:
  --verbose                       Display detailed information. Useful for
                                  debugging purposes.
  --version                       Print the current version of Pyblish
  --paths                         List all available paths
  --plugins                       List all available plugins
  --registered-paths              Print only registered-paths
  --environment-paths             Print only paths added via environment
  --configured-paths              Print only paths added via configuration
  -pp, --plugin-path TEXT         Replace all normally discovered paths with
                                  this This may be called multiple times.
  -ap, --add-plugin-path TEXT     Append to normally discovered paths.
  -c, --config TEXT               Absolute path to custom configuration file.
  -d, --data TEXT...              Initialise context with data. This takes
                                  two arguments, key and value.
  -ll, --logging-level [debug|info|warning|critical|error]
                                  Specify with which level to produce logging
                                  messages. A value lower than the default
                                  'warning' will produce more messages. This
                                  can be useful for debugging.
  --help                          Show this message and exit.

Commands:
  config   List available config.
  publish  Publish instances of path.

Available sub-commands are:

  • publish
  • config
$ pyblish publish --help
Usage: pyblish publish [OPTIONS] [PATH]

  Publish instances of path.

  Arguments:
      path: Optional path, either absolute or relative,
          at which to initialise a publish. Defaults to
          the current working directory.

  Usage:
      $ pyblish publish my_file.txt --instance=Message01
      $ pyblish publish my_file.txt --all

Options:
  -i, --instance TEXT  Only publish specified instance. The default behaviour
                       is to publish all instances. This may be called
                       multiple times.
  -de, --delay FLOAT   Add an artificial delay to each plugin. Typically used
                       in debugging.
  --help               Show this message and exit.

Custom Data

We talked about adding data to the Context from the command-line as well, here’s an example of how that works.

$ pyblish --data key value --data key2 value2 publish

Each time --data is called, with it’s two arguments (key, value), each item is added to the Context before plug-ins start running.

Let me know if there’s anything missing from your current use.

Had some ideas on how to take this to the next level.

$ pyblish

Simply calling pyblish launches the GUI. The GUI then uses the current directory as the current “workspace”, much like how publishing from Maya is using it’s internal state. All that you select, validate and so forth is based on this state.

Launching from the command-line in the same sense uses the current working directory.

Example

/root
  my_scene.mb

Running pyblish from /root have access to it’s content, like running Pyblish from within Maya have access to all of the data currently in the scene.

A selector could look like this.

class SelectMayaScenes(...):
  def process_context(self, context):
    cwd = context.data("cwd")

    for fname in os.listdir(cwd):
      if fname.endswith(".mb") or fname.endswith(".ma"):
        instance = context.create_instance(name=scene)
        instance.set_data("family", "mayaScene")

        abspath = os.path.join(cwd, fname)
        instance.set_data("abspath", abspath)
        instance.set_data("size", os.path.getsize(abspath))

A validator could look like this.

class ValidateMayaScene(...):
   families = ["mayaScene"]
   def process_instance(self, instance):
     assert instance.data("size") < (2 * 1000**3), "File is larger than 2 gb"

A plug-in could also dive further into the file if needed.

import os
import inspect
import subprocess

def test():
    """Function to run in mayapy"""
    import os
    import maya.standalone
    maya.standalone.initialize()

    from maya import cmds

    cmds.file(os.environ["ABSPATH"], open=True)
    assert "MyCube" in cmds.ls()


class ValidateMayaScene(...):
    families = ["mayaScene"]
    def process_instance(self, instance):
        # Pass data to Maya
        os.environ["ABSPATH"] = instance.data("abspath")
        os.environ["PATH"] += ";%s" % mayapy
        script = inspect.getsource(test) + "test()"
        assert subprocess.call(["mayapy", "-c", script]) == 0, "MyCube is missing"

Publishing Fixture

Here’s the part that got me fascinated with this approach.

/MyAsset
  data.yaml  # (Optional) Custom data passed to the context
  config.yaml  # (Optional) Custom configuration
  my_scene.mb
  • data.yaml
  • config.yaml

Data is applied to the Context from data.yaml (if one exists) and can be used to persist various options that are to be used in plug-ins, whereas the configuration is applied the same way and can specify where to find plug-ins.

So, down the line, one could produce fixtures as an intermediate step between a development file, like a Maya or Houdini scene, and a final publish, to be triggered or modified at a later time, by calling Pyblish from the command-line.

/MyAsset
  /rig
    data.yaml
    config.yaml
    scene_v005.mb
  /model
    data.yaml
    config.yaml
    scene_v001.mb

Usage

$ cd /server/assets/MyAsset/rig
$ pyblish # Launch the GUI
$ pyblish publish # Or publish directly

Future

This would also lay the groundwork for future features.

Also, here’s the original issue about the CLI.

Hey @lars,

I’m working on a simplified implementation of the command-line interface, for your pull-request, have a look at the mottosso fork for comparisons.

Primarily, logic has been moved into pyblish.util and custom ordering is now supported. Frankly, it’s going from a big mess to something a lot more elegant so hope we can find away to get you up to speed with the latest and greatest.

Depending on how large your changes are, it might be easiest to re-clone, and use your changes as reference to produce a new pull-request.