Pyblish Magenta

BigRoy · May 18, 2015, 7:34pm

Hey everyone,

I wanted to let you guys know I’m finally getting some extra time on my hands. At the moment I’m trying to set the foundation for the Pyblish magenta project for which I want to set up a minimalistic pipeline. I’ve seen some ftrack and shotgun related comments here and there on other topics so I think starting a brief discussion here is a great way to start.

First priority is to get something up and running that would be a nice pick for small studios (think teams up to 10 people).

Goal

1. Set up a barebones pipeline that allows for a customizable folder structure.

Customizable templates (JSON, XML, YAML; preferences can be discussed. examples with pros/cons?)
Per project configurations?
How to recognize what asset a path belongs to? (Or how to know where to write output?)
What would you consider minimum requirements?
What do you consider most (or more?) fool-proof?

2. Keep a test-driven and documented workflow.

Even though I want to keep it minimalistic pipelines/frameworks tend to expand over time, so I consider this to be just as important as getting a good system working.

3. Implement a bulk of SVEC plug-ins.

Primary focus would be in Maya at first (especially since I’ve most experience in that area). Some example plug-ins (from some time ago) can already be found in the repository.
- If you’re looking for specific Maya-related Validators let me know and I’ll see if I can add them in.
Get them updated accordingly with the API changes when released.

My current development is happening in Github.com/BigRoy/pyblish-magenta and is pretty much a mess at this stage. After some discussion here to get a good direction for best way forward I’ll start implementing. Once the core is running stable I’ll push it towards the Pyblish repository.

tokejepsen · May 18, 2015, 9:29pm

I was wondering whether the future version of the folder structure customization could become its own package?
We are missing that for Pyblishh I think, and ability to include it in other workflow package would be great.

BigRoy · May 19, 2015, 5:09am

That definitely could. And that might even be preferred to keep a higher focus on the plug-in kit in the pyblish magenta repository. On the other hand I’m afraid the Folder Structure Customization grows a lot into a pipeline package and tries to be much like shotgun and ftrack. Not a bad thing per se, but it would be too hard to try and compete with those systems, I guess?

But yes, if that system alone is able to carry its own weight and be interesting enough on its own then it should become a package of its own.

Do you have some more information about what it is you’re looking for?

marcus · May 19, 2015, 6:58am

Narrow Focus

Small studios of ~10 employees is a good start. I would narrow it down even further.

0-1 developer
~3 years experience
~80% freelancers
Full CG only
From story to finished product
Maya and Fusion only

Each have a great impact on what sort of system is developed.

Thanks for sharing this, Roy. This is going to be great!

I also had some thoughts about the folder structure portion of Magenta. To my mind, directory layout is most relevant during Conform, and potentially Selection if layout has anything to say about the type of a current asset.

I could imagine it being possible to have Validators and Extractors be completely oblivious to layout, and focus merely on the core aspect of validation and serialisation respectively. That way, the “directory layout kit” could be cut out and replaced by another, or customised via sub-classing the few Selectors and Conformers there are.

Something to think about at least, whilst developing the VE part of SVEC.

I think this is a good start. It’s assuming that, somewhere, each fraction is formatted into an actual path. I’ve heard this be referred to as “expansion” or “pathname pattern expansion”.

<!-- Schema -->
{project_root}/dev/comp/{parent_hierarchy}/{asset_name}

For example,

state = {
  "project_root": "c:\myproject",
  "parent_hierarchy": "model",
  "asset_name": "Superman"
}

expanded = path.format(**state)
# c:\myproject\dev\comp\model\Superman

In which state is the dynamic part of a users environment and can depend on things such as environment variables or start-up logic, like userSetup.py.

The configurable portion could then be provided by an increasing amount of information in state.

state = {
  "project_root": "c:\myproject",
  "parent_hierarchy": "model",
  "asset_name": "Superman",
  "user_name": "mottosso",
  "date": "2015-05-19",
  "crunch": False
}

With which the user is free to vary his schema to include some or all information.

<!-- Schemas -->
1. {project_root}/dev/comp/{parent_hierarchy}/{user_name}/{asset_name}
2. {project_root}/publishes/{date}/{asset_name}
3. {asset_name}/{parent_hierarchy}

# Expanded
path1.format(**state)  # c:\myproject\dev\comp\model\mottosso\Superman
path2.format(**state)  # c:\myproject\publishes\2015-05-19\Superman
path3.format(**state)  # Superman\model

XML

About the surrounding XML annotation, here’s some alternatives to get the conversation going.

(Shortened to fit width of forum)

animation.xml

<workspace type="animation">
    <folders>
      <folder id="asset.capture">{project_root}/asset/renders/...{playblast/{asset_name}</folder>
      <folder id="asset.camera">{project_root}/asset/shots/...{asset_name}/camera</folder>
    </folders>

   <browserRoots>
        <root id="dev" name="dev"/>
        <root id="asset.capture" name="asset capture"/>
   </browserRoots>
</workspace>

animation.json

{
  "folders": {
    "asset.capture": "{project_root}/asset/renders/...{playblast/{asset_name}</folder>",
    "asset.camera": "{project_root}/asset/shots/...{asset_name}/camera</folder>"
  },
  "browserRoots": {
    "dev": "dev",
    "asset.capture": "asset capture"
  }
}

animation.yaml

folders:
    asset.capture: "{project_root}/asset/renders/...{playblast/{asset_name}</folder>"
    asset.camera: "{project_root}/asset/shots/...{asset_name}/camera</folder>"

browserRoots:
    dev: "dev"
    asset.capture: "asset capture"

Most importantly, no formatting have pros/cons on their own, but depend on what you need them for. For example, what is browserRoots and is it something every 10-man production studio will need?

BigRoy · May 19, 2015, 6:28pm

Great. Thanks for adding to the discussion @marcus.

Personally I really love how the yaml looks and feels. What I think is easier with the XML version is if we need to embed extra data as a tag to the folder, but maybe I’m expecting overcomplications too much.

I’ve pushed an update to my repository with a test for loading from a yaml version.

This sounds pretty accurate, let’s do this!

I’m having a hard time visualising the next steps forward.
Anyone some ideas on first drafts regarding the ‘pipeline’?

marcus · May 19, 2015, 8:19pm

This is where an example project can really shine!

I would suggest mocking up a simplistic story, produce a few assets, stuff them into a few shots and make a film. Cubes and Maya spot-lights will do, so long as it makes good use out of the plug-ins.

From there, it will become obvious to us all what works and what can be improved.

I’d be happy to spend time on it, I could contribute storyboards and animatic? What do you think, thriller? Sci-fi?

tokejepsen · May 20, 2015, 9:25am

Ftrack does not have any folder structure customisation. Shotgun with its Pipeline Toolkit is a strong contender, but personally I wouldn’t worry about that.

Do you have some more information about what it is you’re looking for?

Its mostly an easy way of customising the output folder structure. Think @marcus has outlined a good example of token based customising. As for other examples I would have a look at https://github.com/csaez/naming http://www.cesarsaez.me/2013/01/riglab-naming-convention.html

Personally I really love how the yaml looks and feels.

Yaml defo is the best for human readability. Json next with the benefit of being built-in, and lastly XML.

marcus · May 20, 2015, 9:41am

This is indeed a unique strength of XML, to be able to tag individual words within a sentence with metadata.

In this case however, this level of flexibility might be overkill, and a similar flexibility can be achieved in JSON and YAML as well by just replacing the string-value with another dictionary.

folders:
  asset.capture:
    path: "{project_root}/asset/r.."
    priority: 5
    age: 12.4
  asset.camera:
    path: "{project_root}/asset/r.."
    priority: 3
    age: 118.0

In our case, I wouldn’t let this bother us. We can distirbute YAML with the repo, like Pyblish is already doing, and Magenta too I think.

from pyblish.vendor import yaml

BigRoy · May 20, 2015, 10:07am

I’ll stick with YAML for now. Works as expected and I think we’re all on the same page about it being the most-readable format. Also checked the different formats with some non-technical Artists here and they all pointed to the YAML variant to be making the most sense.

I’ll have a go somewhere today or tomorrow.

marcus · May 20, 2015, 10:11am

I’m working on a captivating story as we speak.

Thriller
10 seconds
5-6 shots
1 environment
2 characters
3 animated props
3 static props

Will be back with a storyboard proposal.

BigRoy · May 20, 2015, 12:32pm

Is it a whodunit about who crashed the pipeline?

marcus · May 20, 2015, 12:52pm

It’s basically Avatar, but with a lot more VFX.

martin · May 20, 2015, 10:40pm

This might be old information, but ftrack actually has extensive folder structure customisation as part of its locations system (you can have different structures per location, different storage engine structures etc.). Some relevant docs: http://doc.ftrack.com/using/locations/overview.html and http://ftrack.rtd.ftrack.com/en/latest/developing/locations/example/structure_plugin.html

Note: I am CTO at ftrack.

mkolar · May 21, 2015, 7:48am

I believe @tokejepsen meant, ‘easily customizable’ folder structure . As much as locations are a brilliant concept. It is currently quite involved process which still requires you to build to whole logic of matching entities to paths, for which you need to use something like lucidity anyways, and that’s what the discussion was based around. End users (especially the ones without full time developer) need something simple and quick to use. I’m sure locations will eventually get there, but from what I’ve seen and tried, it’s not there yet.

marcus · May 25, 2015, 7:57am

The Deal

Here’s the thriller I mentioned.

Story

Two men meet at a diner to come to an agreement and make a deal. Tension is high, some things are said. The weaker of the two finally stands up prepared to accept when bam! he’s shot down.

Mostly static, with very few objects in a single environment with no dependency on dialogue nor sound. I’m estimating it’d take around 1 full week for a single person to develop, any less and we might miss out on subtleties to cover in the plug-ins, but any more and we might be better off with a real project. Note that the speech bubbles are literally filled with straight lines, no text is involved.

The benefit of a shared, open project, is that we can openly discuss and dissect it to find the most efficient manner in which it can be delivered. The goal would be to utilise and refine the current plug-ins.

I’d be happy to contribute characters and animation, we’d need someone to develop environment, along with rendering and compositing.

Thoughts?

marcus · May 25, 2015, 8:26am

Discovered another plausible contender to a schema file-format; TOML.

https://github.com/toml-lang/toml

[folders]
asset_capture = "{project_root}/asset/renders/...{playblast/{asset_name}</folder>"
asset_camera = "{project_root}/asset/shots/...{asset_name}/camera</folder>"

[browserRoots]
dev = "dev"
asset_capture = "asset capture"

Looks to be an alternative to the INI file format. Unsure of it’s advantages, but figured I’d throw it in for perspective. I found it used in the Hugo static-site generator. Though considering it’s new (~2 years) and less common, I personally think any of the above are better suited even if they were to lack certain features.

Edit: Also used in Rust (the programming language) package distribution system “Crate”.

BigRoy · May 25, 2015, 2:48pm

Story

I know it’s not the biggest point of what we’re doing here, but the story seems to lack some depth. Or at least the payoff isn’t that great. What is good though is that it’s a super short storyline, which is exactly what we’re looking for. But if it’s also to have a short ad coming out of this that promotes this is made with Pyblish then it’d be great if it survives as a good/strong video on its own? If that’s the case I think the story can be developed a bit further before delving into a production.

Our studio is pretty busy at the moment but I can have a look whether two of our interns might want to delve into designing their own 10 second short story for this. They can then take it up to the point where there’s some reference moodboards, some artwork and a storyboard (similar to this). If we boot up like that they might also be interested in running along with testing this production with Pyblish. More people on board, and not just our tech-heads?

Pipeline

I’ve to admit I just keep getting stuck just designing. I know my way around with programming, but grasping the concept of a full pipeline without a head start just seems to much at the moment. There are times that I really don’t know how a normal pipeline operates and feel as if I’m only tinkering the whole time. Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.

It seems to me there are people on this forum, unlike me, who’ve been around the big boys that just know their stuff. It feels like a waste to give it a go or “wing it” from my end knowing I’ll hit the wall in the short run. Especially if there’s knowledge here about the pros and cons of already established solutions.

marcus · May 25, 2015, 3:17pm

Hey @BigRoy, I think we’re getting a bit ahead of ourselves.

I think we should focus on the plug-ins first, using the film as a means to an end so that we don’t get stuck in production for too long.

Ideally, I’d like us to spend 95% of the time developing plug-ins, and the rest developing assets and creating shots. If the film is too complicated, it might never get finished.

If you have interns to put on a better story, we should definitely have a look into that too, as a separate project. Maybe by the time we’re done with this piece, they will be done with theirs? Sounds like a good next step once Magenta is “production proven” with the most minimal project.

Trust me, normal is undefined here. The pipeline you’ve built at Colorbleed is without a doubt as normal as any other pipeline out there.

And by the time 2 or more studios use Magenta, it will be the most normal pipeline in the world.

marcus · May 25, 2015, 3:56pm

Here’s a potential asset list for The Deal.

The Deal Assets

▸ characters/
  ▸ ben/
    ▸ model
    ▸ rig
  ▸ jerry/
    ▸ model
    ▸ rig
  ▸ hand/
    ▸ model
    ▸ rig

▸ setdressing
  ▸ table
  ▸ cup
  ▸ bangGraphic
  ▸ speecBubble/
    ▸ model
    ▸ rig
  ▸ gunA/
    ▸ model
    ▸ rig
  ▸ gunB/
    ▸ model
    ▸ rig
  ▸ window/
    ▸ frame
    ▸ exterior

tokejepsen · May 25, 2015, 4:15pm

Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.

I’m still in the mind set that one pipeline might not be the best place to start, but rather look at the more common issues we all have with pipelines and tackle them in smaller packages. Examples are the Deadline and Ftrack packages, and the discussion of folder structure.
Off the top of my head I can think of these for packages:

playblasting/viewport capturing
point caching

It would be up to the workflow packages like magenta to pick and mix smaller packages together.