Just to add to the conversation:
Had a thought about when a Camera from a single scene actually becomes a requirement: stereographic rendering. We don’t have much experience with it (I think only 1 project so far), but it’s one that would need rendering multiple cameras from a single scene.
@mkolar Thanks for hopping in, great information. So is the output name based on the scene’s name or are you using other data to format the output? (eg. the active Task or something like that).
Just the scene name, but that is generated from the task information. If task is in the pipeline (paths generated by the scripts, task launched from ftrack etc.) then the name for scene and render gets formatted from scratch everytime it gets published (integration step). If they are odd render which are outside of pipeline (happens), then it’s a guideline in the studio to use scene name + modifiers we’ve mentioned.
Yeah… never done that 
Good to know. That’s actually exactly what we’re doing now. It’s hard parsing what the latest version would if someone steps outside of the “versioning” naming conventions…
We had a small discussion here that it does happen sometimes that we render multiple cameras from within the same Task, just not within the same scene. As such we would have two different lighting scenes (named differently) within the lighting task’s folder. Nice here is that the output is named accordingly so gets differentiated in a way. For example this happens if the same shot needs to be rendered from multiple angles.
Does that make sense?
In some scenarios we do something similar for multiple variations of a poster having different deliverables. (eg. different resolutions/aspect ratios; and in some cases different lighting/angles). Taking an extreme example would’ve been a recent image we did that ended up in 300+ variations (based on around 5-6 different renders/angles that were sliced into variations). Here we’d prefer keeping these close together as opposed to making 300+ different “shots”.
@marcus, any opinions on the above “workflows”?
Been there, done that. That also makes me think that taking the name of the scene among the top hierarchy is a good identifier.
Point here is that at any time when working with “workarounds/hacks/sandbox files” they should stay conform in such a way that they can still be processed in the pipeline. Stepping “outside” of the pipeline often means it’s easier to work without than with the pipeline… and we should definitely get to a point where working within the pipeline is easy and fast.
From the above combinations I’m looking at the following. If there are multiple cameras (or scenes) we want these variations to be identified and relatively separate, so they become a top hierarchy identifier.
/scene/version/layer/sequence_shot_version_layer_pass_####.exr
If multiple cameras would be rendered from a single scene this could become:
/scene_camera/version/layer/sequence_shot_camera_version_layer_pass_####.exr
The bundling by scene_camera would also mean stereoscopic projects would end up with something like:
prj_sh010_light_v01_left/v01/layer/sequence_shot_left_version_layer_pass_####.exr`
prj_sh010_light_v01_right/v01/layer/sequence_shot_right_version_layer_pass_####.exr`
I guess a Selector in Pyblish could identify the shot has having two active renderable cameras, thus collecting instance data like multiCamera=True. If it’s True this could trigger a Validator (or the Extractor/Integrator) to ensure the camera name is included so both cameras don’t override each other in file output.
Looking forward to hearing what @tokejepsen will bring to the table here. Great contributions everyone!
Yes, definitely great contributions, thanks for that.
In an effort to try and tally up and put a name on the various qualifiers we’ve identified thus far.
scene … v001.mb, hulk_ra1600_1400_v0005_mottosso.ma
camera … main, left, right, persp, face
version … v001, r03_351, 0001
topic … thedeal_seq01_1000, s079s_c49, hulk_ra2400_5600
wedge … tinted, grayscale, lowquality
renderLayer … bg, fg, heroModel, face
renderPass … ao, beauty, sss, motion
take … ?
A note on wedge, I saw it referred to in a Houdini context, not sure it applies across all software. Here’s the definition.
Wedge: Render multiple times with varying parameters
A: Does this apply to your scenario @BigRoy, about the “300+ variations”?
B: Are there any identifiers missing?
The naive directory structure for all of these identifiers might end up looking like this.
# / Root / Version
/thedeal/film/seq01/1000/lighting/publish/v024/..
# / Scene / Topic
..thedeal_seq01_1000_lighting_v152_mottosso/thedeal_seq01_1000_lighting/..
# / Camera / Layer / Pass / Wedge
..mainCamera/background/ambientOcclusion/grayscale/..
# / Filename
..thedeal_seq01_1000_lighting_v024_mainCamera_background_ambientOcclusion_grayscale_001.exr
At which point, ladies and gentlemen, we are 5 characters away from the 260 character limit for paths on Windows operating systems. Yes, there is a limit, and it is low. Any directory after this point will become unbrowsable and unreadable.
I think we can all agree that it’s TMI, too much information. So what can we do?
Ultimately the goal is to enable tools built on-top of whatever convention we choose, so we’ll need to find a method of encapsulating all relevant information, without getting bogged down in directories or ultra-long names.
Some options
wedge..json. In this case, we still need a unique separator for the number of renders will happen, in which case we could introduce build; an infinitely increasing number per render, irrelevant of task or logic, just an identifier of which render comes after another to avoid paths from clashing.An example of (3) could look like this.
1000
▾ lighting
▾ v024
▸ b00033 # Build
▸ b00034
▾ b00035
▸ 0001.exr
▸ 0002.exr
▸ 0003.exr
▸ meta.json
For which meta.json could look like this.
{
"scene": "myfile_v154_mottosso.mb",
"camera": "main",
"version": "v013",
"topic": "thedeal_seq01_1000_lighting",
"wedge": "grayscale",
"renderLayer": "bg",
"renderPass": "ao",
}
The problem with this is that it’s difficult to manually browse to or even interpret what it could mean, without being rather tech-savvy and dig into the JSON or, alternatively, have a dedicated tool to make this separation for them and visualise it more elegantly.
In short, (3) is great when you can rely on tools 100%, which is rarely if ever the case. (1) is great when you can never rely on a tool, which is safe, but clearly overzealous (and dangerously close to the Windows path-limit).
Roy showed me that he typically enters a “template” of sorts into the Maya render editor and I take it this is a common thing to do?
If so, maybe we could leverage this and export this template alongside each render. That way, we could use this template in order to parse what each part of a path means?
Basically, it would facilitate (2).
I was also thinking, especially in context of what @mkolar talked about in Shot and Task publishes, what if only a single render layer needs work and is incremented to the next version?
Basically, would it make sense to publish versions on a per-layer basis? Or more generally, to publish versions on a per Instance-basis?
1000
▾ lighting
▸ heroCharacters
▾ foreground
▸ v001
▸ v002
▾ background
▸ v001
▸ v002
▾ v003
▾ mainCamera
▾ ambientOcclusion
▸ wedge1_0001.exr
▸ wedge1_0002.exr
▸ wedge1_0003.exr
It would mean a lot of duplicated content initially, as each layer would get v001, v002 etc. in unison. But eventually, things may disperse into individual updates, and this might facilitate this.
Had never heard of Wedge before. It sounds/looks different. Probably @tomhankins has a better “feeling” about whether that’s spot on or not.
Yes, in Maya these are called Render Tokens. I never knew but it even has a <Version> token!
Anyway, parsing this could be tricky if you would only split these eg. by _ (underscore) since these could also be part of the token itself. A camera could be called main_left or a scene may be called scene_v01 so to ‘find’ what token represents what data solely based on the filename might become problematic.
Combining it with a meta.json could of course avoid this issue for pipeline tools.
+1. Yes!
This makes a lot of sense to me when looking at renders. It happens a lot of times (like @mkolar also mentioned) that only a specific layer is incremented and rendered as a new version, yet it would still be important to easily trace back the originally used file to create it.
Looking at versions per instance for Tasks like animation or modeling would make it so that a “playblast”-instance version isn’t necessarily the same version number as the published scene’s content since the families are incremented separately. Would that be expected behavior?
If not, how do we decide what/when should be combined or separate as instance data/versioning?
Anyway, it could bring the rendered sequences to:
# Task publish
/thedeal/film/seq01/1000/lighting/publish/..
# Family / Instance (Scene_Layer) / Version
renders/thedeal_seq01_1000_lighting_v152_fg/v01/..
# Layer / filename (Topic_Layer_Pass)
fg/thedeal_seq01_1000_fg_ambientOcclusion_001.exr
Thus:
/thedeal/film/seq01/1000/lighting/publish/renders/thedeal_seq01_1000_lighting_v152_fg/v01/fg/thedeal_seq01_1000_fg_ambientOcclusion_001.exr
Does this look ok? Is this too much?
Is it unique enough to hold our variations of how the output is generated?
Can we trace the file easily through the pipeline?
Passes for a layer are within the same folder. Is this preferred @tomhankins?
This would be true for @mkolar since they use multichannel .exr. that embeds these channels.
That would be bad. I don’t know to be honest, haven’t thought about it like this before. I suppose it wouldn’t be too difficult to include the name of another Instance as a sort of “parent”, and then consider this relationship during integration.
What’s more, we would need to start keeping track of which versions “belong” together and which are simply “latest”. In the simplest of cases, latest always wins but in practice it’s quite hazardous as every move could cripple a working combination.
For example.
publish
▾ instances
▸ heroCharacters
▾ foreground
▸ v001
▸ v002
▸ latest.json
▸ approved.json
Where the JSONs are instance/version pairs, updated during each integration.
{
"heroCharacters": "v034",
"foreground": "v031",
"background": "v014"
}
Alternatively, we could to hard-link each approved/latest Instance into a version-less directory.
publish
▾ approved
▸ heroCharacters
▾ foreground
▸ wedge01_0001.exr
▸ wedge01_0002.exr
▸ wedge01_0003.exr
That could then be something others could reference directly so as to never have to manually update. Including a link to push/pull for completeness.
About the two-step publishing, considering the importance of validating lighting before it is sent off for hours to be rendered, how about this.
job is published. Like other publishes, a job is meant for sharing except in this case the receiver is one or more computers.job would result in a .job file, a JSON-formatted series of instructions for the other computer(s), which would be in whichever language these computers speak; such as Deadline or Tractor.job is either automatically picked up by other computers, e.g. dispatched via farm control software, or manually submitted afterwards by a human.The key points here are that (3) a job isn’t mandatory for rendering and (4) notifications fall beyond the scope of publishing, as does addressing any of the problems. This frees up space for independent tooling and workflow.
Finally, the end result is that renders end up in a users working directory, not in a published directory. The job was published, not the images.
1000
▾ lighting
▸ publish
▾ work
▾ maya
▾ images
▾ heroCharacters
▸ v001
▾ v002
▾ mainCamera
▾ ao
▾ 0001.exr
Publishing of the resulting image sequence then is an optional third step. This should encourage testing and hacking without interfering with or overflowing central information and means any image sequence can be published, even those that have not gone through steps 1 and 2.
Simplifying and bridging these steps can then be left to workflow, best practices and tools.
Thoughts?
That’s what we do when necessary, too. We’d save the scene as another take with different name and render camera2 from there. Same task, multiple scenes and outputs.
Agreed, but this is a tricky one. The way I see it looking at the artista I’ve met, it’s practically always preferred for them to work outside pipeline, Simply because they each have different ideas about what is good naming, workflows etc. Which is fine, but then nobody is able to find any files from other people. I have the feeling that if I gave choice to people, all of them would choose to work to a much looser pipeline. Easier to work doesn’t always mean better and more manageable in a bigger picture.
Wedge in houdini is great but rarely used for final renders. It is based on principle that you can vary parameters on something (anything) and render all the outputs from the same scene, with minimal effort. Amazing for testing different settings on simulation, rendering variation of client to choose from and such. Multiple camera angles would be very tedious to setup for example so it’s not suitable for that. randomized parameters on a digital asset that creates a hairstyle though, would be exactly the thing. You can get 100s of style from one scene.
I’ve never considered using this term for other things in rendering, and frankly don’t think I will. It’s not very self explanatory. If we have more variations of the same thing it’s just a new modifier to output for us. But then if people weren’t so used to ‘version’ I’d call those takes as well. (mind that i very much agree with Isa A. Alsup in this old blog post http://cgsupervisor.blogspot.cz/2010/12/0047-ten-essential-rules-of-file-naming.html?m=1)
uff a bit long. The same thing in our system would look like this
/thedeal/film/seq01/1000/lighting/publish/renders/fg/v01/thedeal_seq01_1000_v01_fg_AO.####.exr
That doesn’t need to be a problem. We handle this by always incrementing scene version and it’s output inherit this. So you can end up with.
scene_v01.hip
scene_v01_preview.mp4
scene_v01_FG.exr
scene_v01_BG.exr
scene_v02.hip
scene_v02_BG.exr
scene_v03.hip
scene_v03_FG.exr
scene_v03_preview.mp4
scene_v03_BG_AO.exr
Some outputs might skip version, but that doesn’t bother me at all, it just means that v02 of FG in my example would be identical as v01. Important thing is, I can trace it back if I want to recreate it.
Absolutely. In situations where we don’t combine channels into multichannel exrs, we dump them to same folder. Most compositing softwares can show sequences as a single entry when browsing so it’s really fast to import them then.
So we’re talking two things here:
We’ve seen now that for (1) there’s definitely a need in production. It makes much more sense to sometimes increment only one of the extracted outputs, especially when the outputs could be large (caches) or take long to create (renders).
For example this would allow you to increment only the character’s point-cache from animation or queue only one render layer and publish it as an increment on its own. This also means that not always the full contents are published in bulk and the instances will end up having it’s own different version numbers.
A solution (@mkolar mentioned something like that) is that the versioning is also dependent on the name/version of the source file, so versions would be skipped if publishing from v39 if the latest publish was done from v36. This ensures that when doing a new bulk update versions are joined into the same version number. Another benefit is this eases tracing back what the source was for a certain output version.
This could be a good option! Especially since this also ensures that work files are incremented for new publishes, ensuring that the source content to create the output still exists in the work folder.
This is the other end of the spectrum where you closely couple together new published content under a single version.
When updating only certain instance through a new scene version this would mean the playblast created from a scene could be incorrect with the existing output. (eg. a table could have moved but haven’t been pointcached). This would mean there’s no one to one correspondence between the work file and the output versions (unless doing the “version from work file trick as mentioned above”).
Any ideas on other direct benefits gained from this?
The “Final” render is solely based on which was used to create the latest/correct output from Compositing. Tracing back which are the latest used versions means tracing the “origins” of the output files. I don’t think Extraction defines what is the final version, because if so the latest would always be the final?
This would revert it back to a version working with a live master that automatically updates. Combining this with a Versioned pipeline means you’ll end up having the downsides of both Versioned (Pull) and Unversioned (Live/Master; Push) pipelines?
Funny that you post this here just before I posted mentioning you while I completely missed your post, hehe. Thanks for bringing it up.
This makes a lot of sense to me from an Artist’s perspective!
Looking great. How would that output differ from the different scenes? Where would it differentiate itself as opposed to overwriting?
Great. +1
just realised I forgot to add task token into filename before it should have been this:
/thedeal/film/seq01/1000/light/publish/renders/fg/v01/thedeal_seq01_1000_light_v01_fg_AO.####.exr
Ideally it would be another task within the same shot, so it would be
/thedeal/film/seq01/1000/light2/publish/renders/fg/v01/thedeal_seq01_1000_light2_v01_fg_AO.####.exr
if it really was in the same task, then we would use a different modifier instead of ‘fg’
thedeal_seq01_1000_light2_v01_cam2-fg_AO.####.exr
the way I see it modifier in the render can encompass more information than just layer. I’d include a collection of variables and keep them as one token. Maybe this comes from me being used to houdini and mantra, where I use the name of the mantra node to describe the render. within this node you specify everything, samples, camera, resolution, objects, etc. so it’s a full package and then I call it. cam1-char1
the only thing I keep changing is formatting of this token. using _ in the modifier is no good, because you can’t tell what’s a separate token and what belongs together. Normally I go with either mixedCase or dash-separated words.
so to simplify:
prj/seq/shot/task/publish/renders/renderModifier/v01/prj_shot_task_v01_renderModifier_pass.exr
What’s going on over there? 
I think there’s a flaw in this logic, in that it doesn’t scale. It assumes that each task is worked on by either a single artist ever, or that artists share scenes without any form of handover.
<rant>
I think that this is the sort of requirement that forces people to work outside the pipeline, because they are limited not only in how they publish, but in how they work.
See, to me, I think working should be personal. I should be able to enter a studio and work on assets my way. As a freelancer, I have little interest in learning all the ins-and-outs of a company before I can actually contribute, and the company cares very little about this as well. I just want to come in, and do good work; that’s what companies want too.
I also think it inhibits an artist’s ability to innovate. To come up with something new. Rather than a pipeline showing them the door and asking them to reach it, it grabs their arm and pulls it.
</rant>
Yeah I think you’re right, I think we need both the versions-per-instance, and the higher-level publishes for playblasts.
I’m thinking we can start subdividing our /publish directory.
1000
▾ lighting
▾ publish
▾ instances
▸ bg
▾ fg
▸ v001
▸ v002
▾ ?
▾ captures
▸ 2015-08-20_20-43 # Versions are irrelevant
▸ 2015-08-21_10-24
Where ? represents a level in which the full scene is considered, as opposed to individual instances. captures then contains playblasts of every contained instance and their current (latest) versions. In the case of lighting, it doesn’t seem to make much sense, but in animation it might?
Oh, sorry, I wasn’t referring to anything being “final”, but rather that you might at some point have a working combination of versions, but that you still might want to output new versions either to try and improve upon a working combination or to replace it due to some change. The key thing is that a working combination could be referred back to at any point as being “the latest working combination” a.k.a. “approved combination”.
Ooor, upsides? Eh? 
Not saying either is better, just throwing it out there.
There can be more people working on the same task, but never at the same time. If they need to work on something simultaneously, it’s two tasks in our. If task is handover to another person, they simply continue working from the last version. We’ve had plenty of shots handover when animator starts blocking it, but then we change the person on the task and someone else does the cleanup. No problems with handovers. On the contrary, with everyone working to the same guidelines, it’s easy to pick up files after other people.
Hmm. Will give this some thought. In general this makes sense, but I’m not sure I agree with this completely.
Hehe. From my observation, most of the people here (Prague) are used to working on their own, freelancing from home (tiny market). When we pull them into the studio they are simply not used to collaborating with multiple people and are not realising things such as, naming photoshop layers: layer1 … layer234 doesn’t help anyone down the line. This comes with the feeling that complying to our standards slows them down. This is true, but the point is that it slows the person creating work by 5 minutes and saves someone picking up his work 15-30 minutes, just trying to figure out what is going on with his inputs.
well what can you do… work with it
I’m referring to what you mentioned earlier, about artists prefer working outside of the pipeline. I’m thinking that what you gain with this requirement is taken away from you when artists refuse to do it, that the tools you build on-top are built on promises that can’t or won’t be kept.
Fair enough, makes sense.
Take it with a grain of salt thought, it’s not a reality so much as it is an ideal to strive for.
By the way, I added another variable take to the summary above.
@BigRoy, if not wedge, is take a better term for the “variations”?
Found an explanation of the term from Cinema4d.
True, but as some producers would say ''when artists refuse to do it, they are out of job". Realistically we never had any troubles with people not doing it. I just know that if given a choice they certainly drop some requirements
Of course. I actually really like the idea and it kinda goes with what we are discussing here, which is making it easier to connect remote freelancers to our pipeline when they work for us, which would be holy grail of outsourcing. Send a shot away let them to their thing, and get it published back to studio from whereever they are. But that’s for another discussion.
Very very similar to Houdini’s takes http://www.sidefx.com/docs/houdini14.0/basics/takes
We use these on a daily basis for exactly these purposes.
So it seems output from lighting is quite different from other tasks, like modeling or rigging.
In modeling, there is 1 output with optional extras, such as a turntable or additional formats. Lighting seems to wrap multiple outputs, each with it’s own optional extras. As though an artist is working with multiple models at the same time.
Does that sound about right?
I’m struggling to wrap my head around exactly what changes during lighting, because that’s where the separation needs to happen; each Instance will essentially be given it’s own physical version on disk, so whatever we choose to become an Instance should be the element of the scene that requires this versioning, the element that can change independently from others.
Does renderLayer sound like a good fit for this? To rephrase the question, is there any variable of the other 7 mentioned in the summary that can change independently such that it requires it’s own versioning, apart from renderLayer?
Tricky, considering that these are handled differently by different softwares, but in maya I’d definitely go for renderLayer. In Houdini renderLayer could apply as well, even though instead of ‘renderLayer’ instance would have to collect mantra node (each being equivalent to a renderLayer)