Pyblish Magenta

Pipeline Development
#221

I guess it’s less about the propagation but more about ensuring that it solely is a collection of things. I suspect we want them to behave different from assets so might become its own type, like shots differ from assets? This is because the collections behave with different rules?

This isn’t necessarily that it exports with preserveReferences=True, since that’s not the requirement.
But it does require that it solely uses a collection of published assets.

It’s kind of like the Universal Scene Description of Pixar where this only adds positional data and a hierarchy. It’s like a container/collection of assets that is predefined?

The collection doesn’t need its own lookdev or rigging. I guess it doesn’t even have modeling but maybe only layout?

What do you think?

#222

Sorry, but I think you’re overcomplicating things.

A set is as much a collection of props as a rig is a collection of models. Making a distinction here means that tools we build for assets won’t apply for “collections” which is frankly a waste.

#223

I’m assuming that with models you’re referring to meshes.

A rig uses multiple models that belong to that same asset. A rig won’t be the rig containing meshes from both asset vehicle1 and vehicle2. Whereas the collection is exactly that, the combination of assets. It’s never a single one.

The collection doesn’t care about the individual models, it cares about the individual assets (which each could be any number of models).

The point being that a collection never would undergo modeling, rigging or lookdev because it’s merely a collection of predefined assets.

It would be inconvenient to create 3 different milk bottle packages and to fill a supermarket shelf with these in its own asset and having to design the shaders again (lookdev) or even the rig per milk bottle purely because it’s the combination of those assets on a shelf?

#224

I meant published assets of family “model”, but yeah, same thing.

That’s a detail. Assets used by another asset could come from anywhere and tools can still be built without taking this difference into consideration.

To me, shots and assets are in separate folders not because they are any different technically, but because they are easier to reason about this way. They still act in the same way when it comes to tools, and plug-ins.

Their relationship could even be visualised as though there were no difference.

For a collection, sure, put it in a separate folder if it makes sense to you - e.g. assets, shots, and collections - as long as we don’t separate them technically, including whether to go for a push or pull system when it comes to versioning.

#225

Personally it doesn’t really matter how we do it. This is basically how we’ve set up the workflow for building a set re-used in multiple shots and even collections of props that are used throughout sets. We’ve allowed any arbitrary amount of references (in collections). So basically a collection could contain a collection, etc.

For example:

bookShelve
- book01
- book01
- book01
- book02
- book03

gasStation_set
- bookShelve
- bookShelve
- bookShelve
- gasStation_environment

It doesn’t matter whether these end up being references after publishing or what the implementation will be within Magenta. The important bit to extract from this is the benefits we gain and how we preserve this benefits whilst simplifying how it works based on how you’d prefer it.

So, what do we gain?

  • Setting up a collection of assets (could be meshes/rigs, whatever) in a predefined manner so they can be easily used and re-used throughout multiple shots. For example the interior of the gasStation and the filled shelves of products.
    • These individual assets in the collection don’t require additional lookdev, model, rig, since they pre-exist and are done elsewhere (per asset).
    • Artists can work in parallel on the assets used in the collection. So both modeling for book01 and book02 could be done at the same time. Of course rigging/lookdev per asset can happen simultaneously (and most often does!). Also the collection can be altered without interfering with the work of those who create the assets.
    • Because of the separation we have with shading/lookdev the objects in the collection can be predefined to use a specific shader variation. For example book01 can be placed in the collection multiple times where one is the blue variation, the other red and another our hero close-up cover of The Gentleman’s Manifesto. :wink: (So this is also data that a collection can set up)

The terminology asset vs collection (or set?) can be ditched, just like the separation.

Important here is that it’s clear to everyone in the project that the asset is a combination of others. If I want to change the texture of the book on the shelf, where do I go and how do I publish?

Then it’s good to define how these differ in Validations. Or do they at all?

#226

Ok, sounds like we’re on the same page.

The only way I’m familiar with when it comes to environment setup is for shots to be built rather than referenced, but the end results are the same I think.

With shot building, a shot would then look much like your yaml-like definitions above, along with a transform per asset. Each asset is then imported or referenced individually according to the recipe/inventory for that shot.

A benefit of the recipe, as far as I can tell - I must admit my experience in this area is limited - is that individual assets, like the book, is just an asset and are treated just like an asset. Any additions to a combination of assets is made by adding an entry to the recipe.

Either way sounds good to me. I started to work on the table for The Deal so we’ll have to set this up soon.

How about the table in The Deal is a collection of smaller assets, like a tablecloth, and some salt and pepper? Just so we can test this concept out. In this case, I’d propose the table is still an asset, like it is currently, and that it references other assets from outside of itself, just to see what problems we encounter there.

1 Like
#227

Actually one more benefit of recipes came to mind, which is being able to build identical shots in multiple software, like in Fusion. Fusion wouldn’t read the Maya format, and would probably also benefit from different representations, e.g. lowres obj as opposed to ma.

Both Maya and Fusion then shares recipe, but use different implementations for how to build them.

#228

The fact that the table itself (without cloth, salt and pepper, etc.) is built in the asset does mean the table has to go through lookdev, even if it’s just for the barebones wooden tabletop. But yes, let’s see where we end up. I can see pros and cons, but I might be making wrong assumptions.

But yes, let’s just start doing exactly that. We need proof!

Sounds good. The only thing is going to be, where do we define these recipes/layouts/collections? A table with props sounds logical that it’s packed with the table. A room filled with props is similarly logicaly, it’s inside the room. But a collection of flowers in a certain formation, where was it published from?

Could you elaborate how this would be set up? And how they are ‘similar’ as recipes? Or are you just combining the terminology?

#229

Just like any other asset. Only in this case, the files aren’t .ma, they’re .yaml.

/thedeal/shots/1000/layout/published/recipe.yaml

The working files could still be .ma, because of the convenience of working with them in a 3d environment.

/thedeal/shots/1000/layout/work/myfile_v001.ma

Sure, here’s a recipe for shot 1000.

- ben:
  name: "Ben1"
  attrs: {"tx": 10, "tz": 20, "rx": 25}
- jerry
  name: "Jerry1"
  attrs: {"tx": 15, "tz": 10, "rx": 190}
- table
  name: "Table1"
- chair
  name: "Chair1"
  attrs: {"tx": -10}
- chair
  name: "Chair2"
  attrs: {"tx": 10, "rx": 180}

In Maya, building this shot could look like this.

import yaml
import pipeline

recipe = yaml.load("/path/to/shot/recipe.yaml")
for asset, properties in recipe.iteritems():
  path = pipeline.find(asset, "model")
  node = cmds.file(path, reference=True, namespace=properties.get("name", asset))

  # Transform object into place
  for attr, value in properties.get("attrs", {}).iteritems():
    cmds.setAttr(node + "." + attr, value)

Fusion could do something similar, the point being that they both feed off of the same source of information.

Edit: I have to ask, what is the meaning of your winks? To me, the suggest an innuendo of sorts, that you know something I don’t?

#230

Frustrating habbit I acquired from being one of those obnoxious kids during the MSN era who has been brainwashed into using emoticons at pseudo-random points in a sentence.

Your explanation of recipes sounds fine. What happens if we have two versions of a table filled with props? One used in the dinner sequence, and the other used in another. (Or even just variations of filled bookShelves in a single sequence).

#231

They would be two separate assets.

Take advantage of their similarities when developing them, but keep their published equivalents simple. E.g. BookshelfA and BookshelfB are separate and standalone from the perspective of anyone using them, even though they look the same and share many of the same internal names and references.

1 Like
#232

Haha, you removed your winks, suddenly your posts got a whole lot colder and serious! :blush:

1 Like
#233

This isn’t a golden rule or anything, it very much depends on your prop.

I would say that, when you have many variations, it will probably be easier to manage if you have a single model contain all meshes, and a rig with switches to handle their variations. A character with various wounds in one shot but not another is a good example, a bookshelf may be another - just a whole series of differently organised books, swappable via an enum or integer attribute. 1 asset, multiple variations.

1 Like
#234

To me it would make sense that even the shelf itself would be loaded from elsewhere, but of course… in theory we could!

Anyway, I think we’ll get there in the end. We might even end up supporting completely both workflows in our single structure. Let’s see how far we can push our Validators to make us work tidy and clean whilst keeping our gains in artist workflow.

What’s the next step for Magenta?

Did we decide yet on the exporter.py? (Did you see the updates I made?)

#235

Our previous discussion about me wrongly reducing exporter.py into cmds.file thinking it was all the same made me realise that for us to work towards the same goal and not redo each others work, we need tests.

With tests, it would have been obvious to me that by doing that X and Y broke. I couldn’t see this now, because there are no tests, so that’s what I’m working on at the moment. Building a test harness for the Magenta plug-ins.

There is one thing I would love for you to have a look at, which is why the last example of capture.py isn’t working in Maya 2015. :frowning:

For rigging, it’s important that a playblast includes the nurbsCurves, but I can’t figure out what has happened to cmds.playblast since 2014.

Other than that, we need content. See the inventory for things we need built before we can start animating.

I did, and it looks good, with one exception. I would prefer to have the whole thing be a context manager.

import maya.cmds
import exporter

with exporter.disconnect(nodes, channels=True):
  cmds.file(exportSelected=True)

Just for the sake of easing the mind of anyone looking at the code to try and understand what is happening. Currently, exporter.py is deeply nested, importing from context and mesh_utils and probably much more and is difficult to understand. I’d rather we don’t need to understand it, and instead make it obvious what it does.

#236

Are you sure this was Maya 2015 only? Seems fixed with changing what seems like a typo in the code, or is that a python syntax? Have a look here.

#237

That fixes it, perfect. I’ll add the update to Magenta as well and include it in the next PR.

1 Like
#238

About the context manager, are you familiar with contextlib.contextmanager?

To me it just makes for less code and makes it more clear that no gymnastics are involved.

@contextlib.contextmanager
def temporary_unit(unit="cm"):
  original_unit = cmds.currentUnit(q=1, l=1)
  cmds.currentUnit(l=self._set_unit)
  yield
  cmds.currentUnit(l=self._original_unit)

Versus:

class TemporaryUnit(object):
    def __init__(self, unit="cm"):
        self._set_unit = unit

    def __enter__(self):
        self._original_unit = cmds.currentUnit(q=1, l=1)
        cmds.currentUnit(l=self._set_unit)

    def __exit__(self, type, value, traceback):
        cmds.currentUnit(l=self._original_unit)

Also I might instead call it “unit” so it forms a nice sentence - with unit():. That it is temporary is implied by the context. And also not provide a default, to make it more clear what it actually does.

with pyblish_magenta.unit("cm"):
  # do things.

And also expose it globally from pyblish_magenta as it’s quite a useful feature I think. Maybe even so useful that it has a place in pyblish_maya?

1 Like
#239

I wanted to do rewrite of the code using that. I saw you using it elsewhere and thought the code looked much more readable and to the point. So yes, let’s wrap it.

Could be exposed in pyblish_maya but not sure if that’ll be the place for it. How do we decide which context manager is basic enough to go in there? And how do we decide which stay in Magenta? Exposing them from a module called context.py does help to identify that it is a context manager maybe?

with context.unit():
    pass
#240

I’d say time will tell. At the moment it’s an idea, something I think we can keep in mind so as to not complicate the manager unless necessary. If it turns out to stay general, then I think it looks like a good candidate for pyblish_maya, simply because it is generic and applies to extractors in general.

We could, but it’s overly complex.

I’d prefer Magenta to remain a singular and cohesive package, as opposed to a collection of submodules, some of which are internal, and some of which are public and intended for others to use.

I would suggest we expose things at the pyblish_magenta level, and those are the only items we later support and keep backwards compatible. Similar to pyblish.api versus pyblish.plugin etc. This way, it’s clear what is usable to others, and what will need to remain as-is until major version changes.

1 Like