Importer/Asset Manager


#1

So I would like to begin the discussion about building an Importer/Asset Manager on the Pyblish framework.

We are currently using Ftracks built-in Importer and Asset Manager; http://support.ftrack.com/customer/portal/articles/1335406-maya-integration, but we are finding that it is increasingly not flexible enough and simply not working. Further it is only implemented in Maya and Nuke, where we need it in other applications as well.

What I would propose is it built something very similar to the publishing dialog for Pyblish, that just deals with importing and changing the scene references.
Initially my thoughts were to have plugins for that collect data to present to the user in the dialog. This could be data from the harddrive, a server or the scene. Then have plugins that deal with importing asset and plugins that deal with changing the references in the scene.

What do you guys think? Are you faced with a similar need for a tool like this?


#2

I think this is a good idea.

I reckon Pyblish’s role would merely be to ensure that assets end up where the library is expecting it, in the format it’s expecting it to be in. That way it really doesn’t matter whether folks use Pyblish, or roll their own publishing mechanism to work with it.

On that note, it does fall rather far outside the scope of Pyblish itself; but building and having a conversation about it is of benefit to anyone and I’m all for it.

To throw a thing or two out there, these are some things I would expect from an importer.

  1. Display content in context
  2. Display metadata about content
  3. Pre- and post-hook

And then perhaps less important but still useful things.

  1. Thumbnails
  2. Search, which implies a form of caching and tagging mechanism
  3. Miller columns, that I am personally a fan of
  4. Bookmarks, for when you constantly browse to the same, deeply nested assets

Content in context

When browsing for assets in Project A in Maya, I expect to only encounter content compatible with Maya coming from Project A.

Metadata

Including options about what is about to be imported, like whether to import or reference and what to call it (if that’s an option). Also data fetched from wherever data about the particular asset of part of an asset is located, like a description, or comment from when it the particular version published.

Pre- and post-hook

Because things sometimes need to happen either before importing something, like checking whether it can be imported, or after, like hooking things up to other assets in the scene.


How about looking up reference or two about importers and collecting them here? I remember seeing a rather fully-fledged video on Tech Artists not too long ago.


#3

I definitely agree it falls outside the scope of Pyblish, but the Pyblish frame work with plugins still seems perfect for the job. Would be silly to invent another framework when this works:)


#4

Wait, do you mean to use Pyblish plug-ins and such for the actual importer? Or how to you mean we can avoid inventing another framework?

Seems to me an importer would have little to do with validating or extracting anything, rather the opposite!

Elaborate on your vision. : )


#5

My thoughts are to have more plugin classes that the Pyblish framework can use. This can definitely be my native thoughts of how Pybilsh works.

Collector
These plugins are the same as we use them in the publishing stage, but it’ll be more common to collect data outside the scene itself, like the hard drive or a server.

Importor
These plugins handle newly added data to the scene, which means how an asset should be referenced/imported in and what to do with the data.

Updator
These plugins handle existing data in the scene, and how to update it for new data, ei. updating a reference to a newer version.

So what is being reuse from the Pyblish framework is the processing, and handing of data from plugin to plugin.


#6

I see, well, I suppose there’s nothing stopping you from doing this already, especially with the blank Plugin class that you could subclass and call Importor or Updator.

I’ll try to grasp how the plug-in to plug-in communication, or collection, comes in handy here. I think my mind is simply in a completely other direction at the moment.

Maybe mock up a few in-memory plug-ins right here, that does the import at the hit of the Play button in the GUI, and see where we end up? You could put “Importer” as a Window title and no one would ever know it was even Pyblish. :slight_smile:


#7

So here is a simple importer example that looks for files on the desktop, and references them in;

import os

import pyblish_qml
import pymel.core


class CollectFiles(pyblish.api.Collector):
    """
    """

    def process(self, context):
        
        path = os.path.join(os.path.dirname(os.path.expanduser('~')), 'Desktop')
        for f in os.listdir(path):
            ext = os.path.splitext(f)[1]
            valid_ext = ['.mb', '.ma']
            if ext in valid_ext:
                instance = context.create_instance(name=f)
                instance.set_data('family', value='desktopFiles')
                instance.set_data('path', value=os.path.join(path, f))
                instance.set_data("publish", False)

class ReferenceFile(pyblish.api.Validator):
    
    def process(self, instance):
        path = instance.data('path')
        name = os.path.basename(path)
        pymel.core.system.createReference(instance.data('path'), namespace=name)


pyblish.api.register_plugin(ReferenceFile)
pyblish.api.register_plugin(CollectFiles)

import pyblish_maya
pyblish_qml.settings.WindowTitle = 'Importer'
pyblish_maya.show()

The main problem here is the configuration of where to look for the files/data. This would be something the user would have to choose. This would be same problem if querying a server like Ftrack.

Here is an example of an updator, that we actually use in production to check if the scene is using the latest version:

import os
import re

import pyblish.api
import pymel
import pyblish_qml


class CollectReferences(pyblish.api.Collector):
    """
    """

    def process(self, context):

        for node in pymel.core.ls(type='reference'):
            if 'sharedReferenceNode' in node.name():
                continue

            if node.isReferenced():
                continue

            file_ref = pymel.core.system.FileReference(node)
            try:
                self.log.info(file_ref.path)
            except:
                continue

            if not file_ref.isLoaded():
                continue

            instance = context.create_instance(name=node.name())
            instance.set_data('family', value='reference')
            instance.add(node)
            instance.set_data("publish", False)


class ValidateReferenceVersion(pyblish.api.Validator):
    """ Validates that the current reference path is the latest version.
    """

    families = ['reference']
    label = 'Reference Version'
    optional = True

    def version_get(self, string, prefix, suffix = None):
      """Extract version information from filenames used by DD (and Weta, apparently)
      These are _v# or /v# or .v# where v is a prefix string, in our case
      we use "v" for render version and "c" for camera track version.
      See the version.py and camera.py plugins for usage."""

      if string is None:
        raise ValueError, "Empty version string - no match"

      regex = "[/_.]"+prefix+"\d+"
      matches = re.findall(regex, string, re.IGNORECASE)
      if not len(matches):
        msg = "No \"_"+prefix+"#\" found in \""+string+"\""
        raise ValueError, msg
      return (matches[-1:][0][1], re.search("\d+", matches[-1:][0]).group())


    def version_set(self, string, prefix, oldintval, newintval):
      """Changes version information from filenames used by DD (and Weta, apparently)
      These are _v# or /v# or .v# where v is a prefix string, in our case
      we use "v" for render version and "c" for camera track version.
      See the version.py and camera.py plugins for usage."""

      regex = "[/_.]"+prefix+"\d+"
      matches = re.findall(regex, string, re.IGNORECASE)
      if not len(matches):
        return ""

      # Filter to retain only version strings with matching numbers
      matches = filter(lambda s: int(s[2:]) == oldintval, matches)

      # Replace all version strings with matching numbers
      for match in matches:
        # use expression instead of expr so 0 prefix does not make octal
        fmt = "%%(#)0%dd" % (len(match) - 2)
        newfullvalue = match[0] + prefix + str(fmt % {"#": newintval})
        string = re.sub(match, newfullvalue, string)
      return string

    def get_latest_version(self, node):

        file_ref =  pymel.core.system.FileReference(node)
        basename = os.path.basename(file_ref.path)
        version_string = self.version_get(basename, 'v')[1]
        version = int(version_string)
        head = basename.split(version_string)[0]
        ext = os.path.splitext(basename)[1]

        max_version = version
        path = basename
        for f in os.listdir(os.path.dirname(file_ref.path)):
            if ext != os.path.splitext(f)[1]:
                continue

            # fail safe against files without version numbers
            try:
                f_version_string = self.version_get(f, 'v')[1]
            except:
                continue

            if head != f.split(f_version_string)[0]:
                continue

            v = int(self.version_get(f, 'v')[1])
            if max_version < v:
                max_version = v
                path = f

        return path

    def process(self, instance):

        node = instance[0]

        file_ref =  pymel.core.system.FileReference(node)
        basename = os.path.basename(file_ref.path)

        msg = 'Newer reference version available for %s' % file_ref.path
        assert self.get_latest_version(node) == basename, msg

    def repair(self, instance):

        node = instance[0]

        file_ref =  pymel.core.system.FileReference(node)
        basename = os.path.basename(file_ref.path)

        if self.get_latest_version(node) != basename:
            new_basename = self.get_latest_version(node)
            new_path = os.path.join(os.path.dirname(file_ref.path),
                                                                new_basename)

            file_ref.replaceWith(new_path)

pyblish.api.register_plugin(CollectReferences)
pyblish.api.register_plugin(ValidateReferenceVersion)

import pyblish_maya
pyblish_qml.settings.WindowTitle = 'Importer'
pyblish_maya.show()

The problem with this updator is adding functionality to choose which version to update to, or if you want to version down. This could probably be achieved with the new Actions though, but as an overview for the user it might be a little difficult to see what is available.


#8

Is this very different from what you had in mind?


#9

It’s what I figured you meant, but I’m sceptical as to whether it’s a healthy direction to take.

In my experience, importing can get as complex as publishing and I think it deserves as much attention to be done right.


#10

And you don’t think the Pyblish framework is enough?


#11

Here’s an example of what an importer is to me.


#12

Nice example:)

I can easily see this work ontop of the Pyblish framework. I think the main question is whether you think it’s outside the scope of Pyblish?
Don’t want to bend Pyblish to something its not aimed for:)
As I see Pyblish now and in the future, its a framework to execute python based plugin/scripts.


#13

Without getting into too many details about how this would work, I have an overall suggestion.

I thinking of a simple UI that is pretty much the current Publish UI, but with just the instances shown. Right-clicking on the instances brings up the the Actions context menu, where you import/reference/update the asset.

We could obviously look into further features like, searching etc. I this is what our current needs are.


#14

At the moment, only plug-ins support Actions.

What you could do, is add a no-op plug-in, one that does nothing but holds onto actions related to managing the instance. You could give it an order before Collection, such that it shows up at the top, or subclass from Plugin directly.

I’m not convinced using the Pyblish GUI as it is today for these purposes is a good fit and won’t be pushing development in that direction. I’d be happy to discuss a new GUI with a new backend though and be curious to see how you progress.

As a side-note, instances could technically support Actions as well, possibly like this:

# Psuedo code
instance.data["actions"].append(MyAction)

Instance argument for Actions
#15

Good idea, will have a look at that.

I’m not convinced either:) Making a new UI for this was always my intention. I’ll probably hack the current UI, with disguising it, to address immediate needs.
When you say “new backend”, do you mean the commincation with the UI or using Pyblish plugins?


#16

I mean just new everything, no relation to Pyblish beyond them both accessing the same set of files; Pyblish writing, importer reading.


#17

I can’t quite see the benefit of creating something completely new, when Pyblish could function as an output as well as an input?

Unless you already have something in mind for the input part of pipeline?


#18

Yes, I have plenty of things in mind. :wink:


#19

So I have had some more thoughts about how the Pyblish framework can work for importing.

Tree Structure
For the user the main thing is to have an overview when loading assets. Having a flat list of assets available for import is overwhelming. The better overview would come from presenting the user with “categories” that they can dive into to individual assets, and this can be represented with a tree structure or parent/child relationship.
Currently in Pyblish there aren’t any tree structures, as the context contains a flat list of instances. So I would suggest introducing hierarchical instances, where you can add instances to other instances.
Practically this would mean you would add a “character” instance to the context, that contains “ryan” and “john”. The user would be presented with a list of one entry “character” that can be expanded to show "ryan and “john” entries.

Instance Actions
@marcus already mentioned the possibility of having instance actions, and this would complete the framework for an asset loader (initial version obviously:))
When the user has navigated to the leaf entries in the tree structure, they can right-click and get presented with different actions to perform on the instance.


#20

Sorry, I completely missed this.

Instances are actually already capable of forming a hierarchy; that was actually the main intent of them being a list to begin with, to have other child instances.

To parent an instance, you’ll instantiate it yourself. Otherwise, it gets parented to the Context, which is the default.

import pyblish.api
context = pyblish.api.Context()
parent = context.create_instance("MyParentInstance")
child = pyblish.api.Instance("MyChildInstance", parent)

Also, just to point it out, the convenience method create_instance is very simple, this is exactly the same thing.

context = pyblish.api.Context()
parent = pyblish.api.Instance("MyParentInstance", context)
child = pyblish.api.Instance("MyChildInstance", parent)

Which might make more sense if you are to deal with hierarchies.

It’s possible we could visualise that in the UI as a tree-view, I think it opens doors for interesting publishing problems as well.