I was wondering whether the future version of the folder structure customization could become its own package?
We are missing that for Pyblishh I think, and ability to include it in other workflow package would be great.
That definitely could. And that might even be preferred to keep a higher focus on the plug-in kit in the pyblish magenta repository. On the other hand I’m afraid the Folder Structure Customization grows a lot into a pipeline package and tries to be much like shotgun and ftrack. Not a bad thing per se, but it would be too hard to try and compete with those systems, I guess?
But yes, if that system alone is able to carry its own weight and be interesting enough on its own then it should become a package of its own.
Do you have some more information about what it is you’re looking for?
Narrow Focus
Small studios of ~10 employees is a good start. I would narrow it down even further.
- 0-1 developer
- ~3 years experience
- ~80% freelancers
- Full CG only
- From story to finished product
- Maya and Fusion only
Each have a great impact on what sort of system is developed.
Thanks for sharing this, Roy. This is going to be great!
I also had some thoughts about the folder structure portion of Magenta. To my mind, directory layout is most relevant during Conform, and potentially Selection if layout has anything to say about the type of a current asset.
I could imagine it being possible to have Validators and Extractors be completely oblivious to layout, and focus merely on the core aspect of validation and serialisation respectively. That way, the “directory layout kit” could be cut out and replaced by another, or customised via sub-classing the few Selectors and Conformers there are.
Something to think about at least, whilst developing the VE part of SVEC.
I think this is a good start. It’s assuming that, somewhere, each fraction is formatted into an actual path. I’ve heard this be referred to as “expansion” or “pathname pattern expansion”.
<!-- Schema -->
{project_root}/dev/comp/{parent_hierarchy}/{asset_name}
For example,
state = {
"project_root": "c:\myproject",
"parent_hierarchy": "model",
"asset_name": "Superman"
}
expanded = path.format(**state)
# c:\myproject\dev\comp\model\Superman
In which state is the dynamic part of a users environment and can depend on things such as environment variables or start-up logic, like userSetup.py.
The configurable portion could then be provided by an increasing amount of information in state.
state = {
"project_root": "c:\myproject",
"parent_hierarchy": "model",
"asset_name": "Superman",
"user_name": "mottosso",
"date": "2015-05-19",
"crunch": False
}
With which the user is free to vary his schema to include some or all information.
<!-- Schemas -->
1. {project_root}/dev/comp/{parent_hierarchy}/{user_name}/{asset_name}
2. {project_root}/publishes/{date}/{asset_name}
3. {asset_name}/{parent_hierarchy}
# Expanded
path1.format(**state) # c:\myproject\dev\comp\model\mottosso\Superman
path2.format(**state) # c:\myproject\publishes\2015-05-19\Superman
path3.format(**state) # Superman\model
XML
About the surrounding XML annotation, here’s some alternatives to get the conversation going.
(Shortened to fit width of forum)
animation.xml
<workspace type="animation">
<folders>
<folder id="asset.capture">{project_root}/asset/renders/...{playblast/{asset_name}</folder>
<folder id="asset.camera">{project_root}/asset/shots/...{asset_name}/camera</folder>
</folders>
<browserRoots>
<root id="dev" name="dev"/>
<root id="asset.capture" name="asset capture"/>
</browserRoots>
</workspace>
animation.json
{
"folders": {
"asset.capture": "{project_root}/asset/renders/...{playblast/{asset_name}</folder>",
"asset.camera": "{project_root}/asset/shots/...{asset_name}/camera</folder>"
},
"browserRoots": {
"dev": "dev",
"asset.capture": "asset capture"
}
}
animation.yaml
folders:
asset.capture: "{project_root}/asset/renders/...{playblast/{asset_name}</folder>"
asset.camera: "{project_root}/asset/shots/...{asset_name}/camera</folder>"
browserRoots:
dev: "dev"
asset.capture: "asset capture"
Most importantly, no formatting have pros/cons on their own, but depend on what you need them for. For example, what is browserRoots and is it something every 10-man production studio will need?
Great. Thanks for adding to the discussion @marcus.
Personally I really love how the yaml looks and feels. What I think is easier with the XML version is if we need to embed extra data as a tag to the folder, but maybe I’m expecting overcomplications too much. 
I’ve pushed an update to my repository with a test for loading from a yaml version.
This sounds pretty accurate, let’s do this!
I’m having a hard time visualising the next steps forward.
Anyone some ideas on first drafts regarding the ‘pipeline’?
This is where an example project can really shine!
I would suggest mocking up a simplistic story, produce a few assets, stuff them into a few shots and make a film. Cubes and Maya spot-lights will do, so long as it makes good use out of the plug-ins.
From there, it will become obvious to us all what works and what can be improved.
I’d be happy to spend time on it, I could contribute storyboards and animatic? What do you think, thriller? Sci-fi?
Ftrack does not have any folder structure customisation. Shotgun with its Pipeline Toolkit is a strong contender, but personally I wouldn’t worry about that.
Do you have some more information about what it is you’re looking for?
Its mostly an easy way of customising the output folder structure. Think @marcus has outlined a good example of token based customising. As for other examples I would have a look at https://github.com/csaez/naming http://www.cesarsaez.me/2013/01/riglab-naming-convention.html
Personally I really love how the yaml looks and feels.
Yaml defo is the best for human readability. Json next with the benefit of being built-in, and lastly XML.
This is indeed a unique strength of XML, to be able to tag individual words within a sentence with metadata.
In this case however, this level of flexibility might be overkill, and a similar flexibility can be achieved in JSON and YAML as well by just replacing the string-value with another dictionary.
folders:
asset.capture:
path: "{project_root}/asset/r.."
priority: 5
age: 12.4
asset.camera:
path: "{project_root}/asset/r.."
priority: 3
age: 118.0
In our case, I wouldn’t let this bother us. We can distirbute YAML with the repo, like Pyblish is already doing, and Magenta too I think.
from pyblish.vendor import yaml
1 Like
I’ll stick with YAML for now. Works as expected and I think we’re all on the same page about it being the most-readable format. Also checked the different formats with some non-technical Artists here and they all pointed to the YAML variant to be making the most sense.
I’ll have a go somewhere today or tomorrow.
I’m working on a captivating story as we speak. 
- Thriller
- 10 seconds
- 5-6 shots
- 1 environment
- 2 characters
- 3 animated props
- 3 static props
Will be back with a storyboard proposal.
1 Like
Is it a whodunit about who crashed the pipeline?
It’s basically Avatar, but with a lot more VFX.
This might be old information, but ftrack actually has extensive folder structure customisation as part of its locations system (you can have different structures per location, different storage engine structures etc.). Some relevant docs: http://doc.ftrack.com/using/locations/overview.html and http://ftrack.rtd.ftrack.com/en/latest/developing/locations/example/structure_plugin.html
Note: I am CTO at ftrack.
1 Like
I believe @tokejepsen meant, ‘easily customizable’ folder structure . As much as locations are a brilliant concept. It is currently quite involved process which still requires you to build to whole logic of matching entities to paths, for which you need to use something like lucidity anyways, and that’s what the discussion was based around. End users (especially the ones without full time developer) need something simple and quick to use. I’m sure locations will eventually get there, but from what I’ve seen and tried, it’s not there yet.
The Deal
Here’s the thriller I mentioned.
Story
Two men meet at a diner to come to an agreement and make a deal. Tension is high, some things are said. The weaker of the two finally stands up prepared to accept when bam! he’s shot down.
Mostly static, with very few objects in a single environment with no dependency on dialogue nor sound. I’m estimating it’d take around 1 full week for a single person to develop, any less and we might miss out on subtleties to cover in the plug-ins, but any more and we might be better off with a real project. Note that the speech bubbles are literally filled with straight lines, no text is involved.
The benefit of a shared, open project, is that we can openly discuss and dissect it to find the most efficient manner in which it can be delivered. The goal would be to utilise and refine the current plug-ins.
I’d be happy to contribute characters and animation, we’d need someone to develop environment, along with rendering and compositing.
Thoughts?
Discovered another plausible contender to a schema file-format; TOML.
[folders]
asset_capture = "{project_root}/asset/renders/...{playblast/{asset_name}</folder>"
asset_camera = "{project_root}/asset/shots/...{asset_name}/camera</folder>"
[browserRoots]
dev = "dev"
asset_capture = "asset capture"
Looks to be an alternative to the INI file format. Unsure of it’s advantages, but figured I’d throw it in for perspective. I found it used in the Hugo static-site generator. Though considering it’s new (~2 years) and less common, I personally think any of the above are better suited even if they were to lack certain features.
Edit: Also used in Rust (the programming language) package distribution system “Crate”.
Story
I know it’s not the biggest point of what we’re doing here, but the story seems to lack some depth. Or at least the payoff isn’t that great. What is good though is that it’s a super short storyline, which is exactly what we’re looking for. But if it’s also to have a short ad coming out of this that promotes this is made with Pyblish then it’d be great if it survives as a good/strong video on its own? If that’s the case I think the story can be developed a bit further before delving into a production.
Our studio is pretty busy at the moment but I can have a look whether two of our interns might want to delve into designing their own 10 second short story for this. They can then take it up to the point where there’s some reference moodboards, some artwork and a storyboard (similar to this). If we boot up like that they might also be interested in running along with testing this production with Pyblish. More people on board, and not just our tech-heads?
Pipeline
I’ve to admit I just keep getting stuck just designing. I know my way around with programming, but grasping the concept of a full pipeline without a head start just seems to much at the moment. There are times that I really don’t know how a normal pipeline operates and feel as if I’m only tinkering the whole time. Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.
It seems to me there are people on this forum, unlike me, who’ve been around the big boys that just know their stuff. It feels like a waste to give it a go or “wing it” from my end knowing I’ll hit the wall in the short run. Especially if there’s knowledge here about the pros and cons of already established solutions.
Hey @BigRoy, I think we’re getting a bit ahead of ourselves.
I think we should focus on the plug-ins first, using the film as a means to an end so that we don’t get stuck in production for too long.
Ideally, I’d like us to spend 95% of the time developing plug-ins, and the rest developing assets and creating shots. If the film is too complicated, it might never get finished.
If you have interns to put on a better story, we should definitely have a look into that too, as a separate project. Maybe by the time we’re done with this piece, they will be done with theirs? Sounds like a good next step once Magenta is “production proven” with the most minimal project.
Trust me, normal is undefined here. The pipeline you’ve built at Colorbleed is without a doubt as normal as any other pipeline out there.
And by the time 2 or more studios use Magenta, it will be the most normal pipeline in the world.
Here’s a potential asset list for The Deal.
The Deal Assets
▸ characters/
▸ ben/
▸ model
▸ rig
▸ jerry/
▸ model
▸ rig
▸ hand/
▸ model
▸ rig
▸ setdressing
▸ table
▸ cup
▸ bangGraphic
▸ speecBubble/
▸ model
▸ rig
▸ gunA/
▸ model
▸ rig
▸ gunB/
▸ model
▸ rig
▸ window/
▸ frame
▸ exterior
Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.
I’m still in the mind set that one pipeline might not be the best place to start, but rather look at the more common issues we all have with pipelines and tackle them in smaller packages. Examples are the Deadline and Ftrack packages, and the discussion of folder structure.
Off the top of my head I can think of these for packages:
- playblasting/viewport capturing
- point caching
It would be up to the workflow packages like magenta to pick and mix smaller packages together.
1 Like
I’m totally on board with this. I’ll be implementing pyblish into another studio which asked me for consultation and they as many others are stubborn with certain approaches they like and won’t be willing to change. Dropping in pyblish with a few plugins however shows them the potential power on tasks that they need done. but don’t know how to deal with. Once they get used to the fact that it works, it will be very easy to just keep plugging new extensions and slowly transition to cleaner workflow. Smaller packages that I can potentially distribute to other clients are definitely when my time will be spent most of the time.