Pyblish Magenta

Pipeline Development
#5

Great. Thanks for adding to the discussion @marcus.

Personally I really love how the yaml looks and feels. What I think is easier with the XML version is if we need to embed extra data as a tag to the folder, but maybe I’m expecting overcomplications too much. :wink:

I’ve pushed an update to my repository with a test for loading from a yaml version.

This sounds pretty accurate, let’s do this!

I’m having a hard time visualising the next steps forward.
Anyone some ideas on first drafts regarding the ‘pipeline’?

#6

This is where an example project can really shine!

I would suggest mocking up a simplistic story, produce a few assets, stuff them into a few shots and make a film. Cubes and Maya spot-lights will do, so long as it makes good use out of the plug-ins.

From there, it will become obvious to us all what works and what can be improved.

I’d be happy to spend time on it, I could contribute storyboards and animatic? What do you think, thriller? Sci-fi?

#7

Ftrack does not have any folder structure customisation. Shotgun with its Pipeline Toolkit is a strong contender, but personally I wouldn’t worry about that.

Do you have some more information about what it is you’re looking for?

Its mostly an easy way of customising the output folder structure. Think @marcus has outlined a good example of token based customising. As for other examples I would have a look at https://github.com/csaez/naming http://www.cesarsaez.me/2013/01/riglab-naming-convention.html

Personally I really love how the yaml looks and feels.

Yaml defo is the best for human readability. Json next with the benefit of being built-in, and lastly XML.

Folder Structure Resolver
#8

This is indeed a unique strength of XML, to be able to tag individual words within a sentence with metadata.

In this case however, this level of flexibility might be overkill, and a similar flexibility can be achieved in JSON and YAML as well by just replacing the string-value with another dictionary.

folders:
  asset.capture:
    path: "{project_root}/asset/r.."
    priority: 5
    age: 12.4
  asset.camera:
    path: "{project_root}/asset/r.."
    priority: 3
    age: 118.0

In our case, I wouldn’t let this bother us. We can distirbute YAML with the repo, like Pyblish is already doing, and Magenta too I think.

from pyblish.vendor import yaml
1 Like
#9

I’ll stick with YAML for now. Works as expected and I think we’re all on the same page about it being the most-readable format. Also checked the different formats with some non-technical Artists here and they all pointed to the YAML variant to be making the most sense.

I’ll have a go somewhere today or tomorrow.

#10

I’m working on a captivating story as we speak. :slight_smile:

  • Thriller
  • 10 seconds
  • 5-6 shots
  • 1 environment
  • 2 characters
  • 3 animated props
  • 3 static props

Will be back with a storyboard proposal.

1 Like
#11

Is it a whodunit about who crashed the pipeline?

#12

It’s basically Avatar, but with a lot more VFX.

#13

This might be old information, but ftrack actually has extensive folder structure customisation as part of its locations system (you can have different structures per location, different storage engine structures etc.). Some relevant docs: http://doc.ftrack.com/using/locations/overview.html and http://ftrack.rtd.ftrack.com/en/latest/developing/locations/example/structure_plugin.html

Note: I am CTO at ftrack.

1 Like
#14

I believe @tokejepsen meant, ‘easily customizable’ folder structure . As much as locations are a brilliant concept. It is currently quite involved process which still requires you to build to whole logic of matching entities to paths, for which you need to use something like lucidity anyways, and that’s what the discussion was based around. End users (especially the ones without full time developer) need something simple and quick to use. I’m sure locations will eventually get there, but from what I’ve seen and tried, it’s not there yet.

#15

The Deal

Here’s the thriller I mentioned.

magenta_storyboards.png2000×2000 374 KB

Story

Two men meet at a diner to come to an agreement and make a deal. Tension is high, some things are said. The weaker of the two finally stands up prepared to accept when bam! he’s shot down.

Mostly static, with very few objects in a single environment with no dependency on dialogue nor sound. I’m estimating it’d take around 1 full week for a single person to develop, any less and we might miss out on subtleties to cover in the plug-ins, but any more and we might be better off with a real project. Note that the speech bubbles are literally filled with straight lines, no text is involved.

The benefit of a shared, open project, is that we can openly discuss and dissect it to find the most efficient manner in which it can be delivered. The goal would be to utilise and refine the current plug-ins.

I’d be happy to contribute characters and animation, we’d need someone to develop environment, along with rendering and compositing.

Thoughts?

#16

Discovered another plausible contender to a schema file-format; TOML.

[folders]
asset_capture = "{project_root}/asset/renders/...{playblast/{asset_name}</folder>"
asset_camera = "{project_root}/asset/shots/...{asset_name}/camera</folder>"

[browserRoots]
dev = "dev"
asset_capture = "asset capture"

Looks to be an alternative to the INI file format. Unsure of it’s advantages, but figured I’d throw it in for perspective. I found it used in the Hugo static-site generator. Though considering it’s new (~2 years) and less common, I personally think any of the above are better suited even if they were to lack certain features.

Edit: Also used in Rust (the programming language) package distribution system “Crate”.

#17

Story

I know it’s not the biggest point of what we’re doing here, but the story seems to lack some depth. Or at least the payoff isn’t that great. What is good though is that it’s a super short storyline, which is exactly what we’re looking for. But if it’s also to have a short ad coming out of this that promotes this is made with Pyblish then it’d be great if it survives as a good/strong video on its own? If that’s the case I think the story can be developed a bit further before delving into a production.

Our studio is pretty busy at the moment but I can have a look whether two of our interns might want to delve into designing their own 10 second short story for this. They can then take it up to the point where there’s some reference moodboards, some artwork and a storyboard (similar to this). If we boot up like that they might also be interested in running along with testing this production with Pyblish. More people on board, and not just our tech-heads?

Pipeline

I’ve to admit I just keep getting stuck just designing. I know my way around with programming, but grasping the concept of a full pipeline without a head start just seems to much at the moment. There are times that I really don’t know how a normal pipeline operates and feel as if I’m only tinkering the whole time. Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.

It seems to me there are people on this forum, unlike me, who’ve been around the big boys that just know their stuff. It feels like a waste to give it a go or “wing it” from my end knowing I’ll hit the wall in the short run. :wink: Especially if there’s knowledge here about the pros and cons of already established solutions.

#18

Hey @BigRoy, I think we’re getting a bit ahead of ourselves.

I think we should focus on the plug-ins first, using the film as a means to an end so that we don’t get stuck in production for too long.

Ideally, I’d like us to spend 95% of the time developing plug-ins, and the rest developing assets and creating shots. If the film is too complicated, it might never get finished.

If you have interns to put on a better story, we should definitely have a look into that too, as a separate project. Maybe by the time we’re done with this piece, they will be done with theirs? Sounds like a good next step once Magenta is “production proven” with the most minimal project.

Trust me, normal is undefined here. The pipeline you’ve built at Colorbleed is without a doubt as normal as any other pipeline out there.

And by the time 2 or more studios use Magenta, it will be the most normal pipeline in the world.

#19

Here’s a potential asset list for The Deal.

The Deal Assets

▸ characters/
  ▸ ben/
    ▸ model
    ▸ rig
  ▸ jerry/
    ▸ model
    ▸ rig
  ▸ hand/
    ▸ model
    ▸ rig

▸ setdressing
  ▸ table
  ▸ cup
  ▸ bangGraphic
  ▸ speecBubble/
    ▸ model
    ▸ rig
  ▸ gunA/
    ▸ model
    ▸ rig
  ▸ gunB/
    ▸ model
    ▸ rig
  ▸ window/
    ▸ frame
    ▸ exterior
#20

Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.

I’m still in the mind set that one pipeline might not be the best place to start, but rather look at the more common issues we all have with pipelines and tackle them in smaller packages. Examples are the Deadline and Ftrack packages, and the discussion of folder structure.
Off the top of my head I can think of these for packages:

  • playblasting/viewport capturing
  • point caching

It would be up to the workflow packages like magenta to pick and mix smaller packages together.

1 Like
#21

I’m totally on board with this. I’ll be implementing pyblish into another studio which asked me for consultation and they as many others are stubborn with certain approaches they like and won’t be willing to change. Dropping in pyblish with a few plugins however shows them the potential power on tasks that they need done. but don’t know how to deal with. Once they get used to the fact that it works, it will be very easy to just keep plugging new extensions and slowly transition to cleaner workflow. Smaller packages that I can potentially distribute to other clients are definitely when my time will be spent most of the time.

#22

Yeah, loving it. This is the way I expect the majority studios to use Pyblish today. Magenta on the other hand, and feel free to correct me on this one @BigRoy, is for new studios, new productions.

New projects are started every day, some launch a studio specifically for a given project, and it’s here where Magenta and Napoleon is meant to kickstart things.

Down the line, once a pipeline definition like Magenta has reached a mature age and is well enough used, tested and documented, I think it will leave little room for doubt whether to go with an ad-hoc, in-house, hack-and-slash pipeline or Magenta.

In the future, I’d like for there to be many pipeline definitions, each benefiting various types of productions, like full cg, vfx, arch-vis. This is where you guys come in. You can codify what you know and like, document it, and share it.

Picture a place like this in which ever entry is a pipeline definition, kit or extension to Pyblish.

  • “What sort of project is this?”
  • “It’s an ad, around 30 seconds, heavily character based”
  • “Ok, since we’re Maya- and Fusion-based, let’s give Pyblish Magenta a go”

I expect pipeline definitions to have their own website, tutorials and community.

#23

My main concern is the fact on how to deal with Selection and Conforming. How do you know what to extract and where the extracted data will need to end? Especially if you want to supply that in a manner that is usable for everyone to pickup.

The most of the system to learn of Pyblish (I think) for small studios is in the Selection and Conforming department. I think (and I hope) many small studios already have their own set of small utilities/ways to check their data is up to par with their requirements, but is just not as streamlined/customizable as a toolset like Pyblish. The validators that are currently in Pyblish Magenta took me around a day (or maybe two) to get in there.

To be honest I feel that to write a Validator you won’t need to know that much about Pyblish, but about how your host works and the data you want to check. Within the Extraction I also consider it to be exporting to an arbitrary format to an arbitrary location (if Conformer is used) using tech and commands that are unrelated to Pyblish. (Of course you need the Selector to select the nodes you want to export, but still the issue is mostly in the Selector.)

Selection and Conforming means knowing what data you need to select and where it needs to go is the number one issue where things will get hardcoded. If Pyblish is so much about introducing a good workflow/standard, why skip this step?

I think it might that pipeline might be misinterpreted here. I’m not setting out to get to the new ftrack or shotgun. Preferrably build even something that is so barebones that it could work with ftrack and shotgun. The issue at the moment though is that the issues mentioned above are a 100% present in every small studio, especially if you consider they are not hooked up to tools like ftrack and shotgun.

I think this is a very interesting topic that should both get their own thread. Especially since they are on your mind feel free to start a discussion with a brief description of the goals you would like to reach. I would be happy to contribute in that area!

I do have to say that one could have Pyblish without playblasting, but not have it running optimally without a way of knowing some ins- and outs about selecting and conforming data.

Exactly this! Pipelines will evolve or even get build up over time anyway.

But we need to have some barebones foundation for “hey this comes from here and goes to there!” so we can deliver such behaviour. (So I feel.) Again, I think Validators (once Selection is done correctly) can currently be easily swapped around, but that does mean there is some fundamentals to be build for Selection?

#24

Good point. In the meantime I’ll have a look at getting some less tech-savvy people around for a possible test-case.

Sounds good. Let’s have Pyblish put something solid in place for the future.