Pyblish Magenta

marcus · June 29, 2015, 10:05am

About the integrator baseclass, what is familyId?

BigRoy · June 29, 2015, 10:22am

There is a mismatch between naming going on somewhere. For example the family you enter with be is called modeling whereas the paths and families in the plugins are parsed with the name model. Because the family is currently parsed from the path with the lucidity.Schema I use it as an id.

Basically it’s the family’s name that actually matches the plug-ins and the path parsing so we can use it to format output paths.

This is what I wrote in the code:

# Note that this the family id retrieved from the template.
# This is not a one-to-one copy of the 'family' used in the plug-ins.
# That's why we store this information as 'familyId' to avoid confusion
family_id = template.name.split('.')[0]
context.set_data('familyId', family_id)

Looking at it now the comment isn’t much explanatory either.

marcus · June 29, 2015, 10:23am

Ah, yes that makes sense.

The variable from be really shouldn’t be called FAMILY, but CONTAINER as that’s what we’re using it for I think?

BigRoy · June 29, 2015, 10:27am

Not really, basically it’s the same piece of data just named differently I suppose? If you would be in thedeal ben rigging it’s the same mismatch: rigging <-> rig. We could also rename everything in the .yaml file for the paths so it’s also:

    # Modeling
    modeling.dev: 
        pattern: '{@root}/dev/modeling/{container}/{asset}'
    modeling.asset:
        pattern: '{@model_asset}/geo'
     
    # Rigging
    rigging.dev: 
        pattern: '{@root}/dev/rigging/{container}/{asset}'
    rigging.asset:
        pattern: '{@model_asset}/rig'

And then we just need to refactor the families in the plug-ins similarly.
Or the other way around and perform be in thedeal ben model instead of be in thedeal ben modeling?

Hmm, might be wrong here. Feel free to correct me!

marcus · June 29, 2015, 10:44am

Isn’t the rigging portion of that hierarchy what you referred to as “container”? Or did I get that wrong?

The goal is for it to make sense and remain intuitive for an artist to type.

$ be in thedeal ben rig

The reason I opted to type the name of the container, rigging as opposed to rig, was because it is used to distinguish which template it is going to get.

templates.yaml

characters: "{cwd}/{0}/dev/{2}/characters/{1}"

Here, the last argument ({2}) becomes the name of the directory. If we make it rig, then that would become the name of the directory and it would be come more tricky to programatically figure out how to get to rigging from having gotten passed rig passed in.

It think it highlights a flaw in the directory structure. There are three separate identifiers for any type of asset. One for its container, such as rigging, and another is it’s family, which is rig and lastly characters (which is for cosmetics?).

I would suggest we ditch all but one, and use this as both family and directory name. I would ditch rig and use the name of the container, rigging. characters doesn’t seem to have any influence of the identity of an asset, other than to simplify browsing via explorer.exe? We can leave that out of the environment and plug-ins.

marcus · June 29, 2015, 5:50pm

I’d like to suggest an alternative directory structure to:

Simplify navigation
Reduce duplicity
Increase programmability

Currently

▾ thedeal
  ▾ dev
    ▾ animation
      ▾ characters
        ▾ ben
          ▸ maya
    ▾ rigging
      ▾ characters
        ▾ ben
          ▸ maya
    ▾ lookdev
      ▾ characters
        ▾ ben
          ▸ maya
    ▾ modeling
      ▾ characters
        ▾ ben
          ▸ maya
  ▾ asset
    ▾ model
      ▾ characters
        ▾ ben
          ▾ geo
            ▸ v001
            ▸ v002

Take a moment to put yourself in the shoes of a freelancer and think about how long it would take you to absorb the information present in this hierarchy.

A series of questions arise.

dev is probably for development, does every directory consist of three letters? Is that a convention?
The directories under modeling (nope, not a convention) and c/o seem to follow a pattern, does that apply only to the dev parent?
Since asset is outside of dev, do we not work with assets?
Is asset/model an equvalent of dev/modeling? Why not asset/modeling?
Is asset/model/characters/ben/geo an equivalent of dev/modeling? Why geo?

Regarding duplicity, ben is repeated 5 times in this small example, and so are it’s inner directories.

Suggestion

▾ thedeal
  ▾ assets
    ▾ ben
      ▾ modeling
        ▾ private
          ▸ maya
        ▾ public
          ▸ v001
          ▸ v002
      ▾ lookdev
        ▸ private
        ▸ public
      ▾ rigging
        ▸ private
        ▸ public
      ▾ animation
        ▸ private
        ▸ public

Where private is the dev directory, nothing within it is shared with anyone else, it is “private”. public is asset, which is the opposite, i.e. published material, it is “public”.

Both hierarchies contain the exact same amount of information, with the exception of these three pieces of information that doesn’t seem to add any unique value.

characters
model
geo

Only that the order in which information is presented goes from coarse, such as project, to specific, such as rigging and individual files which natural and intuitive.

As a result, anything and everything that has to do with ben is encapsulated within this one directory tree., The same pattern can then be applied to each asset and shot.

I’ve set-up a project to illustrate how it could work and look.

$ cd Dropbox/Pyblish
$ be in adeal ben rigging --enter
$ cd
C:\Users\marcus\Dropbox\Pyblish\adeal\assets\ben\rigging\private

Syntax remains the same, so to an artist the change is transparent. But to tools, it means that every single item that can be worked on can follow an identical pattern.

▸ any
  ▸ public
  ▸ private

Ranging from models, rigs, animation cycles, still images, renders, playblasts, you name it. Any tool doing anything to any file can operate on an identical premise, and so can the users.

Working on animation for ben?

That’s /ben/animation

Looking for ben’s rig?

That’s /ben/rigging

What else is there about ben?

See /ben

When it comes to duplicity, each relevant piece of information is now singular, which means it could potentially start to take on metacontent.

/shots/shot15/meta.yaml

startFrame: 1001
endFrame: 1123
description: "Hero get's the girl"

As this is the only spot where shot15 is located, the metacontent can be stored and referenced together with it’s content.

pros/cons

Practically, as the current directory layout is modeled after what you have in-house @BigRoy, changing the directory structure means you wouldn’t be able to use the be template 1-1.

But, because the vital information about the directory structure is abstracted away and put into the environment, any tools and plug-ins we build surrounding this structure will work equally well with any template possible with be. Including the current structure, and any future structure.

Therein lies the power of a wrapper such as be. : - )

BigRoy · June 29, 2015, 8:30pm

I would swap private -> work and public -> publish. As an artist I would be afraid touching anything within a private folder.

The freelancer that comes in animating doesn’t look elsewhere than dev/animation actually.

marcus · June 29, 2015, 8:30pm

How about when looking for the rig? And how about when finished with his animation?

That’s kind of the point. Private is yours, not his.

The benefit of private/public is their symmetry. They leave little doubt for there to be a third or forth category.

BigRoy · June 29, 2015, 9:52pm

Ok, I had some chats about this.

I was more referring to the fact that I would be afraid to start working in it, even if it’s my task to work on an asset. I’d feel more attracted to start opening files in public. Which might be exactly what you want to avoid?

Currently I have the following three folders in mind:
work, publish and sandbox, where work is the equivalent of private and publish is the equivalent of public. sandbox is there for if you need to test some things and don’t obey the rules of work like keeping your files workable.

Looking at your example I’m coming to something like this:

+---assets
|   +---characters
|   |   \---ben
|   |       +---conceptArt
|   |       |   +---publish
|   |       |   |   +---v001
|   |       |   |   \---v002
|   |       |   \---work
|   |       +---lookDev
|   |       |   +---publish
|   |       |   \---work
|   |       +---modeling
|   |       |   +---publish
|   |       |   \---work
|   |       \---rigging
|   |           +---publish
|   |           \---work
|   \---vehicles
|       \---truck
|           +---modeling
|           |   +---publish
|           |   \---work
|           \---rigging
|               +---publish
|               \---work
+---resources
|   +---audio
|   +---fonts
|   \---hdri
\---shots
    +---s01e01
    |   +---sh010
    |   |   +---animation
    |   |   |   +---publish
    |   |   |   \---work
    |   |   +---comp
    |   |   |   +---publish
    |   |   |   \---work
    |   |   \---lighting
    |   |       +---publish
    |   |       \---work
    |   \---sh020
    |       +---animation
    |       |   +---publish
    |       |   \---work
    |       +---comp
    |       |   +---publish
    |       |   \---work
    |       \---lighting
    |           +---publish
    |           \---work
    \---s01e01_teaser
        \---sh010
            +---publish
            \---work

This shows examples for multiple shots in multiple episodes or sequences. Also it has one level of categories above for the assets. I feel with almost every project this one level of subdivisions is a nice way to give a level of organization at the core level.

Note that a shot number 1 (sh010) is present in both the sequences, for me this makes sense. Because both of the sequences have a first shot. So that would mean that the user must tell what sequence he is working on, otherwise sh010 is ambiguous. For assets I’m fine with limiting the asset name to only one instance throughout the project. So if we want to work with ben we don’t have to specifiy character.

That would mean:

Definitions

Separating into Assets and Shots:

Assets

Any object that needs to be created and used throughout the project. Like a character, vehicle or an environment.

Modeling ben
Creating some polygons for ben

$ be in thedeal ben modeling

Rigging ben
Hooking up some constraints with ben

$ be in thedeal ben rigging

Lookdev ben
Defining the shaders/textures for ben

$ be in thedeal ben lookdev

Shots

Any shot that needs to be animated and go through rendering.

Animation
Animating sh010 for film

$ be in thedeal film sh010 animation

Animation
Animating sh010 for teaser

$ be in thedeal teaser sh010 animation

Lighting
Lighting and rendering sh010 for film

$ be in thedeal film sh010 lighting

Compositing
Compositing sh010 for film

$ be in thedeal film sh010 compositing

Poster?

Need to do a still render. Seems like the odd one out? If you need to do a promotional poster I would also use the shots template. Basically since it needs to goo through each individual stage of a shot to get the rendered/composited output

Animation
Building the poses/shot for robot_attack for poster

$ be in thedeal poster robot_attack compositing

Templates

What would the be templates look like?

The be template for Assets would be subdivided into its categories, like character, vehicle, prop, etc.:

character: "{cwd}/{0}/assets/character/{1}/{2}"`
vehicle: "{cwd}/{0}/assets/vehicle/{1}/{2}"`
prop: "{cwd}/{0}/assets/prop/{1}/{2}"`

The template for Assets explained:

$ be in {project} {asset} {task}

The be template for Shots:

shots: {cwd}/{0}/shots/{1}/{2}/{3}

The template for Shots would be:

$ be in {project} {sequence} {shot} {task}

Some additional notes:

With these be examples the sequence name (eg. film) can never be the same as the name of an asset. Otherwise the second argument (after thedeal) is ambiguous.

marcus · June 30, 2015, 6:05am

Looks great, Roy. I’m glad you took the time to think things through.

My only comment would be about the additional characters parent group.

You gain some by having it, more specifically you make it look more intuitive in explorer.exe as any asset that is a “character” will be stored under /characters and the same for /vehicles.

But that’s it. There is no other advantage to it (?).

Now let’s have a look at what you pay for it.

The cost

Without it, you can assert that every asset has the same parent, /assets. Which means that when it comes time to publish, all assets can share implementation.

With it, an asset must first be identified for which “category” it is to be located under, and the question is “how do you do this”?

Store a separate list someplace common, such as /project/categories.yaml, where each asset is listed and associated with a category.
Look it up in a database, ben will always be uniquely associated with characters, tank with vehicles and so forth.
Store metacontent together with the asset, such as /project/assets/ben/characters.category

Each of which is very possible, but is something you will have to carry around for every interaction you make with an asset.

Safety in numbers

If there is more metacontent associated with an asset, then it probably isn’t a big deal. For example, you might be associating start and end frames with shots, in which case the lookup and metacontent management must be handled regardless and you can spend some time to think about a repeatable method. And maybe you find good use for the category within Maya as well, you can for example parent “characters” under a top-level transform called “characters” to keep the scene nice and fresh.

Now only a matter of deciding whether the pros outweigh the cons.

Oh, and I like that you chose human-readable names for your “sequences”, like teaser and poster.

marcus · June 30, 2015, 7:45am

One more thing, about sandboxes.

In the past I’ve worked with sandboxes being either private.

$ be in thedeal ben sandbox

Or shared.

$ be in thedeal sandbox

In which case the sandbox is just another asset. The former is useful to personal testing, whereas the other is good when the testing needs to transcend into other areas, like going from modeling to rendering.

BigRoy · June 30, 2015, 7:50am

I would think the sandbox folder exists per Task, where Task is eg. modeling or rigging for a specific object or animation for a specific shot.

Maybe it would be:
$ be in thedeal ben modeling --sandbox

Kind of like a safe mode?

marcus · June 30, 2015, 7:55am

There’s no need for that, everything in a user’s development directory is safe anyway, that’s what the development directory is for?

The only reason where a sandbox would be necessary, is when the work needs access to areas outside of itself, like a test animation or rig workout that also needs to be passed on to rendering or run through external tools like an asset library.

marcus · June 30, 2015, 8:27am

Maybe the fact that you feel a need for personalised sandboxes is an indicator that you need a per-user division?

$ be in thedeal ben rigging --enter
$ cd
/thedeal/assets/characters/ben/rigging/dev/roy/maya

This way, everyone gets their own personal space to do whatever they want, while still being within the pipeline.

You have access to the currently logged on user from your templates.

character: "{cwd}/{0}/assets/character/{1}/{2}/{user}"`

BigRoy · June 30, 2015, 8:43am

Hmm, no not really. I would rather keep it in a single folder.
That way it’s clear which version was the last that is published, instead of needing to know who worked on it last. We shouldn’t overcomplicate things.

I was more thinking about the sandbox like what happened with the rig for ben. A test like the test_arm_bending_madoodia.ma could be done in the sandbox. But yeah, it could end up in the work folder without problem.

The way you describe it’s more like it’s own shot or asset?

marcus · June 30, 2015, 8:45am

Yeah, exactly. Treat it like just any other item in your pipeline, and it will work just like any other item in your pipeline.

marcus · June 30, 2015, 9:06am

For reference, this is how they work at Framestore.

$ be in thedeal seq999 myshot animation

That way, anyone can pop up their own myshot, or marcusShot or whatever, and it will work alongside any other shot in any other sequence, but still clearly be used for testing.

BigRoy · June 30, 2015, 9:08am

Sounds good. Does make more sense again to have a category for asset then? So you know if an asset belongs to sandbox? Otherwise you’ll end up with tons of marcusCharacterTest?

So the sandbox shot is part of a real sequence? Or actually, it’s not really a sandbox anything. It’s just a regular shot with no special name. Or is seq999 the sandbox sequence?

marcus · June 30, 2015, 9:10am

Could you show me how you mean?

What is real?

Shots are used in an edit, which merely references a fixed set of shots from any sequence. The shots within seq999 simply aren’t referenced in the final cut, but might be referenced in other edits, like edits for testing.

BigRoy · June 30, 2015, 9:14am

Ah yes, so seq999 basically refers to a test/sandbox sequence. Sweet.
About categories for assets I meant like this:

character: "{cwd}/{0}/assets/character/{1}/{2}"`
vehicle: "{cwd}/{0}/assets/vehicle/{1}/{2}"`
prop: "{cwd}/{0}/assets/prop/{1}/{2}"`
sandbox: "{cwd}/{0}/assets/sandbox/{1}/{2}"`

The sandbox category could be used if you have to create some assets just for testing purposes? Without categories you would have

assets/
--> ben
--> gun
--> car
--> chair
--> table
--> marcusTest

Or am I missing something?