Pyblish Magenta

Pipeline Development
#381

Do you mean shaders are somehow connected to render layers?

#382

yeah, say you have a character lookdev that is using renderlayers to get out two different renders.

#383

yeah, say you have a character lookdev that is using renderlayers to get out two different renders.

Are you referring to a shader variation?
Like an object being either Blue or Red?
Weā€™re supporting that.

If you mean you want to preserve renderLayers (like you have a complex layer than canā€™t operate as a pass) weā€™re currently not handling this. Yet we definitely could!

We have enough information about our nodes to identify every single one and export and recreate layers with overrides in the render scene. The tricky thing here is what happens if you have the same object multiple times.

  • Whatā€™s the expected use case?
  • Do they get combined in the same renderLayer?
  • Or what happens if you have a similar layer among multiple objects? eg. Two assets both having a same named ā€œspecialā€ layer.
#384

Yeah, thats what I mean:)

You setup your renderlayers once, in the lookdev file, and that progresses down the pipe to the shots.

Good questions:) Havenā€™t encountered using multiple assets of the same kind that uses renderlayers. In theory, I guess I would expect to not combine them, and get out masks for comp for merging them together. With rebuilding the layers, this is definitely an added bonus, that you would be able to choose to combine layers or not.

#385

Of course a choice could be provided. Do you have an example use case of what kind of ā€œlayersā€ are usually set up per object as opposed to being scene-wide in a lighting scene?

If you have a very specific example we can keep it into account.

A bit off-topic. But have you seen this?

#386

Have some use cases, but it mostly very generic that we often use renderlayers per object. Right now we are using renderlayers to get out some paint effects with maya software along with some shaders from mental ray, so the renderlayer helps us switch renderer.
I wouldnā€™t worry too much, I was mostly asking whether you guys had a solution in place that I could look at.

Yeah, looks good. Probably something for the future.

#387

Interesting. This also directly means it should be able to ā€œswitch rendererā€. We have had this need in the past as well.

#388

Yeah, have renderlayer overrides on the ā€œRender Usingā€ in the render settings to change the renderer per layer.

1 Like
#389

Holly molly, that looks incredibly precise.

1 Like
#390

Ok, it seems we might be picking up some pace here shortly!

Currently weā€™ve been trying to cover a lot of ground everywhere which has greatly shown in the amount of content this topic currently houses. And itā€™s time to grab the overall context and set some milestones and directions.

Here are some key points:

Release the families!

  • Any type of content (family) should be producable at any point. If someone in layout has some content that needs to be published as a model family he should be allowed to do so. Weā€™re not saying a free-for-all in production, but removing the restrictions we currently had in place where allowing a model to be published was dependent on whether you working in a modeling context.

Versioning

  • Outputs are versioned per family per instance. eg. Dog family: model and Dog family: playblast are both versioned separately.

  • Itā€™s currently still in discussion what decides when a version is upgraded. At the moment we were always incremeting upon a publish. Yet I remember @marcus recently being a bit in doubt about that specific decision and moving into the direction of @mkolar where the versioning could be dependent on the Artistā€™s work file, and as such be in control of output versions.

Packages!

  • We introduce the concept of Packages that define what set of published instances belong (or just pair nicely) together.

    • Funnily enough as we played around with this concept ourselves we noticed a blog post from Shotgun dev team pop up discussing something similar, they referred to it as Grouping. You can read that post here.

Especially the introduction of packages could potentially solve a lot of ā€œmanagementā€ issues referring to ā€œwhat bulk of data do I needā€?

Feel free to jump in with any thoughts, questions or ideas! Are you using any similar techniques like packages? (Do you even have such terminology?)

Publish separately VS pull newest version
#391

Apologies for the long unformatted post. I felt like dumping some ideas but only have iPad with me for a few days.

I like the concept of packages (Much better name btw than ā€˜groupā€™ even though Iā€™d personally go for bundlebut that really has no relevance to the discussion :innocent: ).Something of that kind is certainly needed. In a sense we are actually already using this concept in a very scattered and loose way when needed, but itā€™s a case of artist hand picking components to add to an asset in the database which together make up its output. So for example firefx v05 for a shot in ftrack might have components firefastV001, fireslowV004 and sparksV003. Somehow naturally I tend to call them bundles.

I did go through the magenta 0.6 Notes that @marcus posted in another thread and most of it makes good sense. Here are my thought on some things that are less clear.

  1. The definition of a subset is a little bit puzzling in the context of an asset build. In the shot it roughly corresponds to a partial output of the scene. Probably something that came in a single instance (a character) comes out as a subset of current shot scene (animated cache of a character). This kind of makes sense, even though I find it hard to grasp the difference between the instance and a subset.
    However when working on a asset build like a character. I canā€™t seem to find its usefulness and it appear to be only adding one level of complexity unnecessarily.

  2. I assume that Packages are technically just json files, which contain a list of ā€˜linksā€™ pointing to specific versions of components, plus a reviewable (movie,gif,image). If thatā€™s the case, wouldnā€™t it make sense to separate the reviewable out of it and just treat it as another component of type ā€˜reviewā€™? That way the package would literally become just a simple file containing information about its needed components and a link to its corresponding review components.

Naming
You introduced a term ā€˜componentsā€™ in the asset map. This might be a good time to consider whether it makes sense to use it more widely. In my eyes itā€™s a perfect term for what we call ā€˜instanceā€™ now. It applies very well to both asset build and shot part of the pipeline I think.

E.g.
From ben/modeling I feel that Iā€™m publishing ā€˜componentsā€™ modelProxy, modelDefault and reviewTurntable, rather than instances. These together naturally form some kind of modelpackageā€™ that contains their respective versions. Sometimes itā€™s very useful to tag what proxy corresponds to what full res model. For example if hi res needs to be skin wrapped to low poly or if uv needs to match between the two.

When importing such model to scene artist then simply chooses which component he wants to import (or all of course)

From Sh001/animation it is the same story. I publish components Ben, table and ReviewMovie that, out together form animPkg.v001 for given shot. The principle stay the same across the pipeline with some exceptions of course such as camera and render elements, which are always a bit more complicated.

The main reason for calling it component is actually the fact that it come much more naturally to people in the 2 studios where Iā€™m currently deploying pipeline. Instance is very generic a technical terms, that in top of it has quite specific connotations in the CG world. (Object instance the in the scene). This is find more confusing everyday.

#392

Thanks for chiming in @mkolar! For completeness, here are those notes.

About package vs. bundle and instance vs. component, what Iā€™ll say is, itā€™s all good. We know what they mean by now and everything else I think is attempting to merge personal preference. Which isnā€™t always easy, but perhaps not necessary either. Picking something and sticking with it on the other hand is.

For a description and rationale as to what Instance means, Iā€™ll refer you to the Instance API notes.

Weā€™ve struggled with comprehending this concept as well. The more we experiment with it the more fundamental it becomes, and I think in the latest and current version that Roy is working with at the moment, subsets are the only thing published from anywhere so itā€™s important we find a clear way of communicating this.

It is quite difficult to break down. Perhaps we need graphics or, better yet, examples. Iā€™m hoping Roy will have something in this arena within the coming weeks.

Thatā€™s right, just a series of links to connected versions of subsets, like v013 of the character Dave01 together with v056 of Rider03.

example.pkg

{
  "ben01": "v004", 
  "ben02": "v004", 
  "camera01": "v001", 
  "sofa01": "v001"
}

The way Iā€™d like to look at packages is like just another subset. Itā€™s a piece of data extracted from something like Maya, and as such I figure it must be capable of being extracted in different forms. One of which is the .json, for computer consumption, and another, perhaps .mov for human consumption.

I figure that whenever a character is published in isolation from surrounding characters, a package is updated, and itā€™s perhaps equally important to visualise the whole package in a way that doesnā€™t require loading the entire thing into memory.

But it could also be the case that the latter part is better left to a dedicated process in which an artist produces ā€œdailiesā€ for review. I would go for both, one automatic and one manual, but it would be up each studio to decide, which I think is also ok.

#393

Just to hop in here for a bit itā€™s good to see the ā€œconfusionā€ around subset being raised again. Itā€™s definitely one of the harder things to grasp.

To be completely honest I think @mkolar makes a good point about the components. To me an instance has always been a bit vague, but @marcus has clarified things to me by bringing up that an instance is a ā€œinverse of a file on diskā€. As such extracting an instance should result in a specific set of data that should act as a single file. (It could be multiple files, but only when they belong together!). Nevertheless the same data could be represented in different ways (e.g. as an Alembic or a Maya Binary). If iā€™m not mistaking that would be two different instances.

Though I do have to say Iā€™ve also seen different formats for the same data being referred to ā€œRepresentationsā€. Is an instance always a single representation? Are they exactly the same? Or whatā€™s different?

Then thereā€™s the need for subset (or component) to differentiate different kinds of data output: modelDefault and modelProxy. Of course these can all come with their own instances (Alembics, etc.)

So to me what @mkolar mentions as component is actually a subset. component == subset

#394

Exactly, hereā€™s some examples to help narrow it down.


Subsets

The hierarchy works like this (with a minor simplification, see below about versions).

  • Subset
    • Instance

That is, subsets contain instances, and an instance is a single file or file-sequence. (Redacted, see below)

- modelDefault
  - modelDefault.png
  - modelDefault.ma
  - modelDefault.mov
- modelLowres
- animRig
- simRig
- clothRig
- ...

Those are all from the asset department. Here are some from shots.

- Ben01
  - Ben01.abc
  - Ben01.geoCache
  - Ben01.gif
- Sofa03
- masterLayer
- bgLayer
   - bgLayer.png
      - bgLayer.0001.png
      - bgLayer.0002.png
      - bgLayer.0003.png
      - ...

Versioning

Inbetween the Subset and the Instance are versions.

  • Subset
    • Version

For example.

- Ben
  - modelDefault
    - v001

Putting it all together

- Ben
  - modelDefault
    - v001
      - modelDefault_v001.png
      - modelDefault_v001.ma

What we really need are some practical examples, because the reasoning for these things are so deeply nested in production circumstance and is difficult to hold on to. I might even have missed out on a few steps here myself. Roy, was this how we left things?

#395

Looks we just found the problem of the subset, component, instance, representation talk.V

In the context of Pyblish itself and the API notes that @marcus pointed to, instance is not one specific file per se. When publishing you can run multiple extractors on it and create multiple different representations of this instance. So Ben has instance modelDefault which has .png, .ma and .mov representations.

Whereas in magenta As explained by @marcus in the previous post. Instances seem to be a single file or a file sequence so modelDefault is a subset and instances are .png, .ma and .mov

This inconsistency between Pyblish, Magenta makes the concepts discussed here a little ambiguous.

Seems right to me, however as mentioned earlier this is quite different to what instance is described as in the API docs.

Instance as an inverse of a file makes very good sense when importing, however when publishing it doesnā€™t apply the same way in the current state of Pyblish core and UI.

#396

@mkolar, are you referring to the Workflow portion of the API docs? To be fair, it does say that these are guidelines, and not definitions. The definition is above.

But you make a good point, those guidelines should not be there. Those should be part of an extension, such as Magenta. Iā€™ve updated that page, see here for revision history if you need to double check.

With that done, I think thatā€™s as slim as the definition for Instance can get, and even though it may be possible that weā€™ll develop another extension that produces another definition of the word, I think that is acceptable. Extensions define what is what, they define the ā€œpipelineā€ if you will, the software does not. The software is only there to facilitate these definitions, to enable you to think in ways that makes sense for your workflow.

#397

@mkolar Iā€™m having a closer look at this now and I think Iā€™ve gotten confusedā€¦ sighā€¦

Typically we would have broken down the problem into smaller chunks by now that we could discuss in isolation, but this hasnā€™t happened. Like weā€™re biting off more than we (or at least I) can chew.

Any ideas on how we can divide this, and focus on one portion at a time? Iā€™m working on some more graphics to help communicate things. Maybe an animation would help?

It would be fair to think that, if it is this complicated, we must be doing something wrong. But I donā€™t think this is the case. I think this is a naturally complex subject with many interacting parts. I also think there is a better way to break it down, we just havenā€™t found it yet.

#398

Ok, Iā€™ve updated the notes to reflect how we left things during out last spring at Colorbleed.

Hereā€™s the gist of it (pun).

#399

@BigRoy, it looks like these guys are doing what you are doing with shaders and alembic, maybe their source is open for peeking?

Auto shader hookup on Alembic import

#400

I remember me and @tokejepsen had a solution to this a few years back. It was a bit hardcoded for the project at the time and fairly simple but it might be worth looking at it. @tokejepsen do you remember where it was stored.

If I remember correctly we saved shaders as maya scene with the mesh and all the assignments. Then when we brought in the alembic file we have transferred all the connections from the original mesh to the animated one. I believe we also tried the way of connection alembic output to a non-animated shaded mesh. But canā€™t remember what were the troubles there. Maybe @tokejepsen could chime in if he has better memory.

Anyway, Iā€™ll be needing it again too so Iā€™ll dig through some old code and see if we can compare solutions.