Pyblish Magenta


#361

How about this.

▾ v001
  ▸ layer_AO.png
  ▸ layer_AO.exr
  layer_AO.mov

Where layer_AO.png is a folder.

▾ v001
  ▸ layer_AO.png
  ▾ layer_AO.exr
      layer_AO_0001.exr
      layer_AO_0002.exr
      layer_AO_0003.exr
    ...
  layer_AO.mov

That way, each channel would result in just a different name.

▾ v001
  ▸ layer_AO.png
  ▸ layer_AO.exr
  ▸ layer_beauty.png
  ▸ layer_beauty.exr
  ▸ layer_diffuse.png
  ▸ layer_diffuse.exr
  layer_AO.mov
  layer_beauty.mov
  layer_diffuse.mov

This way representations are evenly represented with a dotted suffix - e.g. .exr - even though some of the childen of a version are directories instead of files.

We could then apply the same to assets.

▾ v001
    high_res.ma
    high_res.mov
    default.ma
    default.mov

Where high_res is a new identifier that we haven’t used with assets before, but is needed so we can use the identifier for each pass during rendering. When an asset doesn’t have this identifier (should we call it output, as in the name of an “output” from a family?) then it’s given default.

Passes are then output from a pipeline perspective, and it’s full name is prefix + output + representation.

Where prefix is the topic and such.

# prefix                 output  representation
thedeal_ben_rigging_v001_default.ma

Things sure are getting complex. :slight_smile: Shouldn’t we all just stick to Microsoft MovieMaker?


#362

Had a chat with @marcus on hangouts and here’s some of the next steps we want to take with Magenta; I’ll separate them into their own posts.

Any Task can create any type of output/representation.

Currently

Currently collecting what is important for extraction for a model is based on what item we’re working on and where we’re working on it. In short we can only export a model from modeling. We’ve come to an agreement that this is particularly bad.

Goal

We feel that an Artist should be allowed to choose and be “explicit” what contents can and should go out from his scene. This also allows arbitrary data to be extracted from a scene if a particular workaround requires it. Plus it forces the artist to think more (and also easier limit) what’s required for export.

Proposition

An Artist would put the contents into an objectSet or another entry that is tagged as such within the scene’s content.


#363

And the reason an objectSet is preferable to a plain group, is that an object set can contain non-DAG nodes, such as other objectSet nodes.

In our case, we need rigs to contain pointcache_SET and controls_SET to be valid.


#364

Plus a single node can be present in multiple sets.

Say a camera might be used for “review” in an objectSet to perform a capture of the scene plus it should be published as a camera along with the animation scene. This way then can be managed independently.


#365

Good point.


#366

This is the output of a single render layer from a scene, correct?
To put this in context. So this would be within a folder like:

If Instance-first: lighting/publish/{layer}/element/...
If Family-first: lighting/publish/element/{layer}/...

In this case +1 to family first.
Clearer to humans and possibly even more logical for tools?

All rendered elements would be in /elements/ where otherwise instances would end up being of mixed contents.


This makes more sense to me as layout of the renders:

▾ v001
  ▸ layer.exr
         layer_master_####.exr
         layer_AO_####.exr
         layer_beauty_####.exr
         layer_diffuse_####.exr

@mkolar, @tokejepsen and @tomhankins. You seem to have more general expertise in ‘lighting/compositing’ area. Hope you all can contribute here.


This sounds really really good, though I think a model in different resolutions is exactly one of those things that you’d want to increment in version separately? We rarely have a high-res/low-res separation of assets going on, except for proxies. These proxies are generally created separately.

What I’m most afraid of is that we end with an assumption of ‘default’ and ‘high-res’ always being present. Could an early version get published without the high-res present? What happens when the high-res is created a week later? Does that merge into the currently available version number?


#367

That’s a good point. Bummer. :frowning:


#368

For me they are different instances, actually. Does that help?

assets/ben/modeling/published/model/ben_default/v001/...
assets/ben/modeling/published/model/ben_proxy/v001/...
assets/ben/modeling/published/model/ben_hires/v001/...

#369

I think they are more than different instances, as they would most likely be published from separate scenes.

So something like this.

assets/ben/modeling/published/modelDefault/v001/...
assets/ben/modeling/published/modelProxy/v001/...
assets/ben/modeling/published/modelHires/v001/...

#370

Totally correct. My formatting was based on the assumption that the published paths would be Family-first and as such identify that the contents in there all hold a “model” only of different instances.

With that in mind, are we thinking alike?


#371

Not sure, in my example modelDefault is the family. In your example, model is the family, and ben_default is the instance?


#372

I think your example hits the spot, not mine.

I’ll have to think more about this… Didn’t consider that two Instance's can be published from separate scenes, but it makes perfect sense.


#373

Published structure

So this means we’re going for something like:

{item}/{task}/publish/{family}/{instance}/{version}/...

Where:

  • item: The name of the item you’re working on. It’s the name of the asset or shot. For example our character “ben” or the shot “1000”.

  • task: A single step/stage in the production of the item. For example “modeling” or “rigging”.

  • Note: Since any Task can output any type of data it’s all up to the “workflow” of the project to make assumptions about what data is created where. For example it’s recommended (since it makes sense) that a “modeling” task is done to create the final model for a character.

  • family: The type of output data coming from the artist working in the task. These can range from model, rig, lookdev to pointcache or even review. This is the family used to define the content within the Pyblish plug-ins.

  • instance: The instance of output data the artist is generating, in most scenarios there would be a “default” expected output so we recommend outputting as such: ben_default. Other instances then could be: ben_hires or ben_proxy. Similarly the separate pointcaches from an animation scene would all be there own instance (within the family of pointcache).

  • version: This would the be automatically incrementing version number of the output data. Each family/instance combination would have its own versioning since it is its own version of the output data that it represents.

  • Note: Since versioning is based on each family/instance combination model/ben_default and review/ben_default would increment separately. Similarly when performing a “similar” extraction in another task, for example outputting a model in rigging would also be its own output folder. This should all be clear from looking at the folder structure and how things are grouped.

Example

So we’ve defined the following pattern: {item}/{task}/publish/{family}/{instance}/{version}/...

You’re rigging the character Ben and want to output its main (= default) rig, this would be:

item = "ben"
task = "rigging"
family = "rig"
instance = "ben_default"
version = 1  # formatted to "v001"

As such it results in:

ben/rigging/publish/rig/ben_default/v001/...


#374

It seems to me that you have arrived to something I’ve touched upon in this thread

considering you can publish any content from any task, wouldn’t it make more sense to be publishing into a shot/asset publish folder, rather than task publish folder?

let’s say that someone working on lighting task creates a new model, it get’s placed into lighting publish folder. Quite frankly I can’t imagine anyone looking there for the data. However publishing it to a shot publish folder is clear, because everyone knows to go there if they need something, regardless what task it came from.

your example would then be:
{item}/{task}/publish/{family}/{instance}/{version}/...
hence
ben/publish/rig/ben_default/v001/...

no duplication of data (rigging task and rig family), and all publishes in one structured folder. Easy for tools and people to understand.


#375

I think your conflict has just grown. Because then it’s not immediate apparent that a model might have been published from a rigging task (which shouldn’t be the “final model”)? Or should it?

Also publishing a camera from both the animation department and the layout department as the exact same output data is just as problematic on its own? Reading that topic it seems you can’t easily identify who worked on it last? Do they often work in parallel? If so they would “compete” with their versions, both incrementing one after the other. This would make it just as hard to find the correct camera? I think this is more of a “social” problem?

If there’s a need to trace (view) “versions” between multiple departments/tasks and want to see who pushed the latest update than it would be up to browser that filters it as such. For example you could make a ListView that shows published contents “by family” like how some e-mail inboxes can merge multiple inboxes. There’s still enough reason to have the content itself be separate.

Anyway, I have thought about this as well. Seeing that you’re bringing this up I need to think about it some more.

@marcus, @tomhankins… what do you think?


Adding to these two quick mockups of how this could be browsed.

Draft 1

This would be “merged” inboxes looking up only by family type.

Draft 2

Here’s the content per Task.


#376

That is true if rigger was publishing the same instance as modeller. I would hope however, that modeller would publish ben_default and ben_hires, but rigger would create a new instance for his models: ben_proxy, ben_blendshapes.

In case of camera We’d be separating them too. cam_track, cam_lay, cam_anim.

Directly from the file you couldn’t but, that is the same for task based publishes if you have multiple people working on the same task. We use database (ftrack) to keep track of who published what.

Not really. Never in parallel on the same task if there is a big set to build, it would be split to model_props, model_structures tasks. or similar. In case we have more people on the same task they would never work on it on the same time. That doesn’t make much sense to me.

Absolutely more of a ‘social’ problem. In terms of finding the latest version of a camera there isn’t any difference I think. If I’m compositor who wants to import camera to 3d comp it makes no difference whether I first go to previous task and pick a camera, or go to cameras and pick the one exported from the task I want (most likely lighting.)


#377

just as a side note. I’m really not trying to push towards dropping the tasks from publishing. It just appeared to me that that’s where you were headed.


#378

Thanks for explaining. I think in the end a browser that gives you the opportunity to “list by family” from all Tasks is more what you’re looking for, especially since you’re still recommending to separate the camera into their own “instances” anyway.

Once we get an asset browser going with Magenta along with the corresponding API I hope to show you a way you’ll grow to love.


#379

Absolutely. The browser will be the best thing by far.


#380

Saw you guys were using a custom tool for rebuilding the shaders on the alembic. How are you handling renderlayers?