Publishing renders (lighting)

Tips & Tricks
#9

Yes, definitely great contributions, thanks for that.

In an effort to try and tally up and put a name on the various qualifiers we’ve identified thus far.

  • Identifier … Examples
  • scene … v001.mb, hulk_ra1600_1400_v0005_mottosso.ma
  • camera … main, left, right, persp, face
  • version … v001, r03_351, 0001
  • topic … thedeal_seq01_1000, s079s_c49, hulk_ra2400_5600
  • wedge … tinted, grayscale, lowquality
  • renderLayer … bg, fg, heroModel, face
  • renderPass … ao, beauty, sss, motion
  • take … ?

A note on wedge, I saw it referred to in a Houdini context, not sure it applies across all software. Here’s the definition.

Wedge: Render multiple times with varying parameters

  • Reference

  • A: Does this apply to your scenario @BigRoy, about the “300+ variations”?

  • B: Are there any identifiers missing?

What goes into a name?

The naive directory structure for all of these identifiers might end up looking like this.

# / Root                                 / Version 
/thedeal/film/seq01/1000/lighting/publish/v024/..

# / Scene                              / Topic
..thedeal_seq01_1000_lighting_v152_mottosso/thedeal_seq01_1000_lighting/..

# / Camera   / Layer    / Pass           / Wedge 
..mainCamera/background/ambientOcclusion/grayscale/..

# / Filename
..thedeal_seq01_1000_lighting_v024_mainCamera_background_ambientOcclusion_grayscale_001.exr

At which point, ladies and gentlemen, we are 5 characters away from the 260 character limit for paths on Windows operating systems. Yes, there is a limit, and it is low. Any directory after this point will become unbrowsable and unreadable.

I think we can all agree that it’s TMI, too much information. So what can we do?

Ultimately the goal is to enable tools built on-top of whatever convention we choose, so we’ll need to find a method of encapsulating all relevant information, without getting bogged down in directories or ultra-long names.

Some options

  1. We could include everything always, as in the example above (obviously not).
  2. We could build the hierarchy dynamically, excluding that which isn’t changed across renders, such as wedge.
  3. We not rely on paths at all, storing all relevant information elsewhere, such as a sidecar .json. In this case, we still need a unique separator for the number of renders will happen, in which case we could introduce build; an infinitely increasing number per render, irrelevant of task or logic, just an identifier of which render comes after another to avoid paths from clashing.

An example of (3) could look like this.

1000
▾ lighting
  ▾ v024
    ▸ b00033  # Build
    ▸ b00034
    ▾ b00035
      ▸ 0001.exr
      ▸ 0002.exr
      ▸ 0003.exr
      ▸ meta.json

For which meta.json could look like this.

{
    "scene": "myfile_v154_mottosso.mb",
    "camera": "main",
    "version": "v013",
    "topic": "thedeal_seq01_1000_lighting",
    "wedge": "grayscale",
    "renderLayer": "bg",
    "renderPass": "ao",
}

The problem with this is that it’s difficult to manually browse to or even interpret what it could mean, without being rather tech-savvy and dig into the JSON or, alternatively, have a dedicated tool to make this separation for them and visualise it more elegantly.

In short, (3) is great when you can rely on tools 100%, which is rarely if ever the case. (1) is great when you can never rely on a tool, which is safe, but clearly overzealous (and dangerously close to the Windows path-limit).

Dynamically allocated paths - publishing w/ template

Roy showed me that he typically enters a “template” of sorts into the Maya render editor and I take it this is a common thing to do?

If so, maybe we could leverage this and export this template alongside each render. That way, we could use this template in order to parse what each part of a path means?

Basically, it would facilitate (2).

Instance-first

I was also thinking, especially in context of what @mkolar talked about in Shot and Task publishes, what if only a single render layer needs work and is incremented to the next version?

Basically, would it make sense to publish versions on a per-layer basis? Or more generally, to publish versions on a per Instance-basis?

1000
▾ lighting
  ▸ heroCharacters
  ▾ foreground
    ▸ v001
    ▸ v002
  ▾ background
    ▸ v001
    ▸ v002
    ▾ v003
      ▾ mainCamera
        ▾ ambientOcclusion
          ▸ wedge1_0001.exr
          ▸ wedge1_0002.exr
          ▸ wedge1_0003.exr

It would mean a lot of duplicated content initially, as each layer would get v001, v002 etc. in unison. But eventually, things may disperse into individual updates, and this might facilitate this.

1 Like
Pyblish Magenta
#10

Had never heard of Wedge before. It sounds/looks different. Probably @tomhankins has a better “feeling” about whether that’s spot on or not.

Yes, in Maya these are called Render Tokens. I never knew but it even has a <Version> token!

Anyway, parsing this could be tricky if you would only split these eg. by _ (underscore) since these could also be part of the token itself. A camera could be called main_left or a scene may be called scene_v01 so to ‘find’ what token represents what data solely based on the filename might become problematic.

Combining it with a meta.json could of course avoid this issue for pipeline tools.

+1. Yes!

This makes a lot of sense to me when looking at renders. It happens a lot of times (like @mkolar also mentioned) that only a specific layer is incremented and rendered as a new version, yet it would still be important to easily trace back the originally used file to create it.

Per-instance versioning across all tasks

Looking at versions per instance for Tasks like animation or modeling would make it so that a “playblast”-instance version isn’t necessarily the same version number as the published scene’s content since the families are incremented separately. Would that be expected behavior?

If not, how do we decide what/when should be combined or separate as instance data/versioning?

Anyway, it could bring the rendered sequences to:

# Task publish
/thedeal/film/seq01/1000/lighting/publish/..

# Family /  Instance (Scene_Layer)       / Version
renders/thedeal_seq01_1000_lighting_v152_fg/v01/..

# Layer / filename (Topic_Layer_Pass)
fg/thedeal_seq01_1000_fg_ambientOcclusion_001.exr

Thus:

/thedeal/film/seq01/1000/lighting/publish/renders/thedeal_seq01_1000_lighting_v152_fg/v01/fg/thedeal_seq01_1000_fg_ambientOcclusion_001.exr

Does this look ok? Is this too much?
Is it unique enough to hold our variations of how the output is generated?
Can we trace the file easily through the pipeline?

Passes for a layer are within the same folder. Is this preferred @tomhankins?
This would be true for @mkolar since they use multichannel .exr. that embeds these channels.

Pyblish Magenta
#11

That would be bad. I don’t know to be honest, haven’t thought about it like this before. I suppose it wouldn’t be too difficult to include the name of another Instance as a sort of “parent”, and then consider this relationship during integration.

What’s more, we would need to start keeping track of which versions “belong” together and which are simply “latest”. In the simplest of cases, latest always wins but in practice it’s quite hazardous as every move could cripple a working combination.

For example.

publish
▾ instances
  ▸ heroCharacters
  ▾ foreground
    ▸ v001
    ▸ v002
▸ latest.json
▸ approved.json

Where the JSONs are instance/version pairs, updated during each integration.

{
  "heroCharacters": "v034",
  "foreground": "v031",
  "background": "v014"
}

Alternatively, we could to hard-link each approved/latest Instance into a version-less directory.

publish
▾ approved
  ▸ heroCharacters
  ▾ foreground
    ▸ wedge01_0001.exr
    ▸ wedge01_0002.exr
    ▸ wedge01_0003.exr

That could then be something others could reference directly so as to never have to manually update. Including a link to push/pull for completeness.

#12

About the two-step publishing, considering the importance of validating lighting before it is sent off for hours to be rendered, how about this.

  1. From ligthing, e.g. inside Maya, a job is published. Like other publishes, a job is meant for sharing except in this case the receiver is one or more computers.
  • The job would result in a .job file, a JSON-formatted series of instructions for the other computer(s), which would be in whichever language these computers speak; such as Deadline or Tractor.
  1. A job is either automatically picked up by other computers, e.g. dispatched via farm control software, or manually submitted afterwards by a human.
  • Like any task, it is started before it can be published, except in this case the worker is a computer and as such it can’t make decisions about what to do in case validation fails. Instead, a report is generated and distributed to where it can be seen and addressed; such as an email message, a notification in Slack or ftrack.

The key points here are that (3) a job isn’t mandatory for rendering and (4) notifications fall beyond the scope of publishing, as does addressing any of the problems. This frees up space for independent tooling and workflow.

Finally, the end result is that renders end up in a users working directory, not in a published directory. The job was published, not the images.

1000
▾ lighting
  ▸ publish
  ▾ work
    ▾ maya
      ▾ images
        ▾ heroCharacters
          ▸ v001
          ▾ v002
            ▾ mainCamera
              ▾ ao
                ▾ 0001.exr

Publishing of the resulting image sequence then is an optional third step. This should encourage testing and hacking without interfering with or overflowing central information and means any image sequence can be published, even those that have not gone through steps 1 and 2.

Simplifying and bridging these steps can then be left to workflow, best practices and tools.

Thoughts?

#13

That’s what we do when necessary, too. We’d save the scene as another take with different name and render camera2 from there. Same task, multiple scenes and outputs.

Agreed, but this is a tricky one. The way I see it looking at the artista I’ve met, it’s practically always preferred for them to work outside pipeline, Simply because they each have different ideas about what is good naming, workflows etc. Which is fine, but then nobody is able to find any files from other people. I have the feeling that if I gave choice to people, all of them would choose to work to a much looser pipeline. Easier to work doesn’t always mean better and more manageable in a bigger picture.

Wedge in houdini is great but rarely used for final renders. It is based on principle that you can vary parameters on something (anything) and render all the outputs from the same scene, with minimal effort. Amazing for testing different settings on simulation, rendering variation of client to choose from and such. Multiple camera angles would be very tedious to setup for example so it’s not suitable for that. randomized parameters on a digital asset that creates a hairstyle though, would be exactly the thing. You can get 100s of style from one scene.

I’ve never considered using this term for other things in rendering, and frankly don’t think I will. It’s not very self explanatory. If we have more variations of the same thing it’s just a new modifier to output for us. But then if people weren’t so used to ‘version’ I’d call those takes as well. (mind that i very much agree with Isa A. Alsup in this old blog post http://cgsupervisor.blogspot.cz/2010/12/0047-ten-essential-rules-of-file-naming.html?m=1)

uff a bit long. The same thing in our system would look like this
/thedeal/film/seq01/1000/lighting/publish/renders/fg/v01/thedeal_seq01_1000_v01_fg_AO.####.exr

That doesn’t need to be a problem. We handle this by always incrementing scene version and it’s output inherit this. So you can end up with.

scene_v01.hip
    scene_v01_preview.mp4
    scene_v01_FG.exr
    scene_v01_BG.exr

scene_v02.hip
    scene_v02_BG.exr

scene_v03.hip
    scene_v03_FG.exr
    scene_v03_preview.mp4
    scene_v03_BG_AO.exr

Some outputs might skip version, but that doesn’t bother me at all, it just means that v02 of FG in my example would be identical as v01. Important thing is, I can trace it back if I want to recreate it.

Absolutely. In situations where we don’t combine channels into multichannel exrs, we dump them to same folder. Most compositing softwares can show sequences as a single entry when browsing so it’s really fast to import them then.

#14

So we’re talking two things here:

  1. versions-per-instance (like with the versions of a render layer)
  2. multiple families in a version of an instance (eg. a playblast combined with the latest pointcache)
Versions per instance (1)

We’ve seen now that for (1) there’s definitely a need in production. It makes much more sense to sometimes increment only one of the extracted outputs, especially when the outputs could be large (caches) or take long to create (renders).

For example this would allow you to increment only the character’s point-cache from animation or queue only one render layer and publish it as an increment on its own. This also means that not always the full contents are published in bulk and the instances will end up having it’s own different version numbers.

A solution (@mkolar mentioned something like that) is that the versioning is also dependent on the name/version of the source file, so versions would be skipped if publishing from v39 if the latest publish was done from v36. This ensures that when doing a new bulk update versions are joined into the same version number. Another benefit is this eases tracing back what the source was for a certain output version.

This could be a good option! Especially since this also ensures that work files are incremented for new publishes, ensuring that the source content to create the output still exists in the work folder.

Multiple families per version (2)

This is the other end of the spectrum where you closely couple together new published content under a single version.

When updating only certain instance through a new scene version this would mean the playblast created from a scene could be incorrect with the existing output. (eg. a table could have moved but haven’t been pointcached). This would mean there’s no one to one correspondence between the work file and the output versions (unless doing the “version from work file trick as mentioned above”).

Any ideas on other direct benefits gained from this?

The “Final” render is solely based on which was used to create the latest/correct output from Compositing. Tracing back which are the latest used versions means tracing the “origins” of the output files. I don’t think Extraction defines what is the final version, because if so the latest would always be the final?

This would revert it back to a version working with a live master that automatically updates. Combining this with a Versioned pipeline means you’ll end up having the downsides of both Versioned (Pull) and Unversioned (Live/Master; Push) pipelines?

Pyblish Magenta
#15

Funny that you post this here just before I posted mentioning you while I completely missed your post, hehe. Thanks for bringing it up.
This makes a lot of sense to me from an Artist’s perspective!

Looking great. How would that output differ from the different scenes? Where would it differentiate itself as opposed to overwriting?

Great. +1

#16

just realised I forgot to add task token into filename before it should have been this:
/thedeal/film/seq01/1000/light/publish/renders/fg/v01/thedeal_seq01_1000_light_v01_fg_AO.####.exr

Ideally it would be another task within the same shot, so it would be
/thedeal/film/seq01/1000/light2/publish/renders/fg/v01/thedeal_seq01_1000_light2_v01_fg_AO.####.exr

if it really was in the same task, then we would use a different modifier instead of ‘fg’
thedeal_seq01_1000_light2_v01_cam2-fg_AO.####.exr

the way I see it modifier in the render can encompass more information than just layer. I’d include a collection of variables and keep them as one token. Maybe this comes from me being used to houdini and mantra, where I use the name of the mantra node to describe the render. within this node you specify everything, samples, camera, resolution, objects, etc. so it’s a full package and then I call it. cam1-char1

the only thing I keep changing is formatting of this token. using _ in the modifier is no good, because you can’t tell what’s a separate token and what belongs together. Normally I go with either mixedCase or dash-separated words.

so to simplify:
prj/seq/shot/task/publish/renders/renderModifier/v01/prj_shot_task_v01_renderModifier_pass.exr

1 Like
#17

What’s going on over there? :smile:

I think there’s a flaw in this logic, in that it doesn’t scale. It assumes that each task is worked on by either a single artist ever, or that artists share scenes without any form of handover.

<rant>
I think that this is the sort of requirement that forces people to work outside the pipeline, because they are limited not only in how they publish, but in how they work.

See, to me, I think working should be personal. I should be able to enter a studio and work on assets my way. As a freelancer, I have little interest in learning all the ins-and-outs of a company before I can actually contribute, and the company cares very little about this as well. I just want to come in, and do good work; that’s what companies want too.

I also think it inhibits an artist’s ability to innovate. To come up with something new. Rather than a pipeline showing them the door and asking them to reach it, it grabs their arm and pulls it.
</rant>

Yeah I think you’re right, I think we need both the versions-per-instance, and the higher-level publishes for playblasts.

I’m thinking we can start subdividing our /publish directory.

1000
▾ lighting
  ▾ publish
    ▾ instances
      ▸ bg
      ▾ fg
        ▸ v001
        ▸ v002
    ▾ ?
      ▾ captures
        ▸ 2015-08-20_20-43  # Versions are irrelevant
        ▸ 2015-08-21_10-24

Where ? represents a level in which the full scene is considered, as opposed to individual instances. captures then contains playblasts of every contained instance and their current (latest) versions. In the case of lighting, it doesn’t seem to make much sense, but in animation it might?

Oh, sorry, I wasn’t referring to anything being “final”, but rather that you might at some point have a working combination of versions, but that you still might want to output new versions either to try and improve upon a working combination or to replace it due to some change. The key thing is that a working combination could be referred back to at any point as being “the latest working combination” a.k.a. “approved combination”.

Ooor, upsides? Eh? :wink: Not saying either is better, just throwing it out there.

#18

There can be more people working on the same task, but never at the same time. If they need to work on something simultaneously, it’s two tasks in our. If task is handover to another person, they simply continue working from the last version. We’ve had plenty of shots handover when animator starts blocking it, but then we change the person on the task and someone else does the cleanup. No problems with handovers. On the contrary, with everyone working to the same guidelines, it’s easy to pick up files after other people.

Hmm. Will give this some thought. In general this makes sense, but I’m not sure I agree with this completely.

#19

Hehe. From my observation, most of the people here (Prague) are used to working on their own, freelancing from home (tiny market). When we pull them into the studio they are simply not used to collaborating with multiple people and are not realising things such as, naming photoshop layers: layer1 … layer234 doesn’t help anyone down the line. This comes with the feeling that complying to our standards slows them down. This is true, but the point is that it slows the person creating work by 5 minutes and saves someone picking up his work 15-30 minutes, just trying to figure out what is going on with his inputs.

well what can you do… work with it :slight_smile:

#20

I’m referring to what you mentioned earlier, about artists prefer working outside of the pipeline. I’m thinking that what you gain with this requirement is taken away from you when artists refuse to do it, that the tools you build on-top are built on promises that can’t or won’t be kept.

Fair enough, makes sense. :smile:

Take it with a grain of salt thought, it’s not a reality so much as it is an ideal to strive for.

#21

By the way, I added another variable take to the summary above.

@BigRoy, if not wedge, is take a better term for the “variations”?

Found an explanation of the term from Cinema4d.

1 Like
#22

True, but as some producers would say ''when artists refuse to do it, they are out of job". Realistically we never had any troubles with people not doing it. I just know that if given a choice they certainly drop some requirements

Of course. I actually really like the idea and it kinda goes with what we are discussing here, which is making it easier to connect remote freelancers to our pipeline when they work for us, which would be holy grail of outsourcing. Send a shot away let them to their thing, and get it published back to studio from whereever they are. But that’s for another discussion.

#23

Very very similar to Houdini’s takes http://www.sidefx.com/docs/houdini14.0/basics/takes

We use these on a daily basis for exactly these purposes.

#24

Lighting output

So it seems output from lighting is quite different from other tasks, like modeling or rigging.

In modeling, there is 1 output with optional extras, such as a turntable or additional formats. Lighting seems to wrap multiple outputs, each with it’s own optional extras. As though an artist is working with multiple models at the same time.

Does that sound about right?

What is an Instance

I’m struggling to wrap my head around exactly what changes during lighting, because that’s where the separation needs to happen; each Instance will essentially be given it’s own physical version on disk, so whatever we choose to become an Instance should be the element of the scene that requires this versioning, the element that can change independently from others.

Does renderLayer sound like a good fit for this? To rephrase the question, is there any variable of the other 7 mentioned in the summary that can change independently such that it requires it’s own versioning, apart from renderLayer?

#25

Tricky, considering that these are handled differently by different softwares, but in maya I’d definitely go for renderLayer. In Houdini renderLayer could apply as well, even though instead of ‘renderLayer’ instance would have to collect mantra node (each being equivalent to a renderLayer)

#26

Not necessarily.

A model could be published into different “versions” like “high-resolution”, “low-resolution” and maybe “proxy”.
Don’t you think?

Yes it does. (As before, only other separation could’ve been camera or “take” as it has been described here.)

Pyblish Magenta
#27

I think take is the clearest term for shot variantions. Not someting you would have to explain to every new person you work with.

Currently our output images from v-ray for instance read: layer.pass.####.extension. Might be a good idea to use a . instead of an - just for overview purposes? Feels a little shorter;)

#28

I’ve been using that for years until I ran into issues with some programs that don’t like . in the filenames, Or rather expect anything after a . to be a frame number or extension. Kill me but I can’t for life of me remember where these problems were appearing. Anyways for about a year now we’ve switched to _ separating name tokens and leaving . stricly for separation of a frame sequence and extension.