I believe @tokejepsen meant, ‘easily customizable’ folder structure . As much as locations are a brilliant concept. It is currently quite involved process which still requires you to build to whole logic of matching entities to paths, for which you need to use something like lucidity anyways, and that’s what the discussion was based around. End users (especially the ones without full time developer) need something simple and quick to use. I’m sure locations will eventually get there, but from what I’ve seen and tried, it’s not there yet.
The Deal
Here’s the thriller I mentioned.
Story
Two men meet at a diner to come to an agreement and make a deal. Tension is high, some things are said. The weaker of the two finally stands up prepared to accept when bam! he’s shot down.
Mostly static, with very few objects in a single environment with no dependency on dialogue nor sound. I’m estimating it’d take around 1 full week for a single person to develop, any less and we might miss out on subtleties to cover in the plug-ins, but any more and we might be better off with a real project. Note that the speech bubbles are literally filled with straight lines, no text is involved.
The benefit of a shared, open project, is that we can openly discuss and dissect it to find the most efficient manner in which it can be delivered. The goal would be to utilise and refine the current plug-ins.
I’d be happy to contribute characters and animation, we’d need someone to develop environment, along with rendering and compositing.
Thoughts?
Discovered another plausible contender to a schema file-format; TOML.
[folders]
asset_capture = "{project_root}/asset/renders/...{playblast/{asset_name}</folder>"
asset_camera = "{project_root}/asset/shots/...{asset_name}/camera</folder>"
[browserRoots]
dev = "dev"
asset_capture = "asset capture"
Looks to be an alternative to the INI file format. Unsure of it’s advantages, but figured I’d throw it in for perspective. I found it used in the Hugo static-site generator. Though considering it’s new (~2 years) and less common, I personally think any of the above are better suited even if they were to lack certain features.
Edit: Also used in Rust (the programming language) package distribution system “Crate”.
Story
I know it’s not the biggest point of what we’re doing here, but the story seems to lack some depth. Or at least the payoff isn’t that great. What is good though is that it’s a super short storyline, which is exactly what we’re looking for. But if it’s also to have a short ad coming out of this that promotes this is made with Pyblish then it’d be great if it survives as a good/strong video on its own? If that’s the case I think the story can be developed a bit further before delving into a production.
Our studio is pretty busy at the moment but I can have a look whether two of our interns might want to delve into designing their own 10 second short story for this. They can then take it up to the point where there’s some reference moodboards, some artwork and a storyboard (similar to this). If we boot up like that they might also be interested in running along with testing this production with Pyblish. More people on board, and not just our tech-heads?
Pipeline
I’ve to admit I just keep getting stuck just designing. I know my way around with programming, but grasping the concept of a full pipeline without a head start just seems to much at the moment. There are times that I really don’t know how a normal pipeline operates and feel as if I’m only tinkering the whole time. Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.
It seems to me there are people on this forum, unlike me, who’ve been around the big boys that just know their stuff. It feels like a waste to give it a go or “wing it” from my end knowing I’ll hit the wall in the short run. 
Especially if there’s knowledge here about the pros and cons of already established solutions.
Hey @BigRoy, I think we’re getting a bit ahead of ourselves.
I think we should focus on the plug-ins first, using the film as a means to an end so that we don’t get stuck in production for too long.
Ideally, I’d like us to spend 95% of the time developing plug-ins, and the rest developing assets and creating shots. If the film is too complicated, it might never get finished.
If you have interns to put on a better story, we should definitely have a look into that too, as a separate project. Maybe by the time we’re done with this piece, they will be done with theirs? Sounds like a good next step once Magenta is “production proven” with the most minimal project.
Trust me, normal is undefined here. The pipeline you’ve built at Colorbleed is without a doubt as normal as any other pipeline out there.
And by the time 2 or more studios use Magenta, it will be the most normal pipeline in the world.
Here’s a potential asset list for The Deal.
The Deal Assets
▸ characters/
▸ ben/
▸ model
▸ rig
▸ jerry/
▸ model
▸ rig
▸ hand/
▸ model
▸ rig
▸ setdressing
▸ table
▸ cup
▸ bangGraphic
▸ speecBubble/
▸ model
▸ rig
▸ gunA/
▸ model
▸ rig
▸ gunB/
▸ model
▸ rig
▸ window/
▸ frame
▸ exterior
Mostly I feel as if I’m thinking to far ahead and with that never really tackle the issue I started thinking about.
I’m still in the mind set that one pipeline might not be the best place to start, but rather look at the more common issues we all have with pipelines and tackle them in smaller packages. Examples are the Deadline and Ftrack packages, and the discussion of folder structure.
Off the top of my head I can think of these for packages:
- playblasting/viewport capturing
- point caching
It would be up to the workflow packages like magenta to pick and mix smaller packages together.
1 Like
I’m totally on board with this. I’ll be implementing pyblish into another studio which asked me for consultation and they as many others are stubborn with certain approaches they like and won’t be willing to change. Dropping in pyblish with a few plugins however shows them the potential power on tasks that they need done. but don’t know how to deal with. Once they get used to the fact that it works, it will be very easy to just keep plugging new extensions and slowly transition to cleaner workflow. Smaller packages that I can potentially distribute to other clients are definitely when my time will be spent most of the time.
Yeah, loving it. This is the way I expect the majority studios to use Pyblish today. Magenta on the other hand, and feel free to correct me on this one @BigRoy, is for new studios, new productions.
New projects are started every day, some launch a studio specifically for a given project, and it’s here where Magenta and Napoleon is meant to kickstart things.
Down the line, once a pipeline definition like Magenta has reached a mature age and is well enough used, tested and documented, I think it will leave little room for doubt whether to go with an ad-hoc, in-house, hack-and-slash pipeline or Magenta.
In the future, I’d like for there to be many pipeline definitions, each benefiting various types of productions, like full cg, vfx, arch-vis. This is where you guys come in. You can codify what you know and like, document it, and share it.
Picture a place like this in which ever entry is a pipeline definition, kit or extension to Pyblish.
- “What sort of project is this?”
- “It’s an ad, around 30 seconds, heavily character based”
- “Ok, since we’re Maya- and Fusion-based, let’s give Pyblish Magenta a go”
I expect pipeline definitions to have their own website, tutorials and community.
My main concern is the fact on how to deal with Selection and Conforming. How do you know what to extract and where the extracted data will need to end? Especially if you want to supply that in a manner that is usable for everyone to pickup.
The most of the system to learn of Pyblish (I think) for small studios is in the Selection and Conforming department. I think (and I hope) many small studios already have their own set of small utilities/ways to check their data is up to par with their requirements, but is just not as streamlined/customizable as a toolset like Pyblish. The validators that are currently in Pyblish Magenta took me around a day (or maybe two) to get in there.
To be honest I feel that to write a Validator you won’t need to know that much about Pyblish, but about how your host works and the data you want to check. Within the Extraction I also consider it to be exporting to an arbitrary format to an arbitrary location (if Conformer is used) using tech and commands that are unrelated to Pyblish. (Of course you need the Selector to select the nodes you want to export, but still the issue is mostly in the Selector.)
Selection and Conforming means knowing what data you need to select and where it needs to go is the number one issue where things will get hardcoded. If Pyblish is so much about introducing a good workflow/standard, why skip this step?
I think it might that pipeline might be misinterpreted here. I’m not setting out to get to the new ftrack or shotgun. Preferrably build even something that is so barebones that it could work with ftrack and shotgun. The issue at the moment though is that the issues mentioned above are a 100% present in every small studio, especially if you consider they are not hooked up to tools like ftrack and shotgun.
I think this is a very interesting topic that should both get their own thread. Especially since they are on your mind feel free to start a discussion with a brief description of the goals you would like to reach. I would be happy to contribute in that area!
I do have to say that one could have Pyblish without playblasting, but not have it running optimally without a way of knowing some ins- and outs about selecting and conforming data.
Exactly this! Pipelines will evolve or even get build up over time anyway.
But we need to have some barebones foundation for “hey this comes from here and goes to there!” so we can deliver such behaviour. (So I feel.) Again, I think Validators (once Selection is done correctly) can currently be easily swapped around, but that does mean there is some fundamentals to be build for Selection?
Good point. In the meantime I’ll have a look at getting some less tech-savvy people around for a possible test-case.
Sounds good. Let’s have Pyblish put something solid in place for the future.
This got me thinking.
Maybe there could be a Selection-only extension to Pyblish? Something that governs how you format the data within the host so as to make it validatable, extractable and conformable by subsequent extensions or your own custom ones.
For example, one extension could provide the means to generate Instances based on putting Maya nodes into an objectSet, whereas another extension could do the same but base it on some naming convention, like a _RIG prefix and anything within for rigs.
If the instances are made encapsulated enough - as in they contain all relevant data like names of nodes and relevant properties and attributes - then to subsequent plug-ins it makes no difference from where they were generated. All they see is an Instance packed with information to be validated.
In a sense, Selectors could pick up as much as possible from any host, getting close to duplicating the whole scene, just so that subsequent plug-ins have an identical set of data to work on, regardless of which extension generated the instances (or even which host they came from?)
Down the line, one may even introduce some form of compatibility between extensions. Such that Validator, Extractor and Conform “packs” could be made to support one or more very specific Selector-extensions.
Shared Project Directory
I created a Dropbox account for Pyblish, and created a The Deal project directory in it, what do you think about working on it in here? (The files in there currently I copied from the Magenta Foobar example).
At first I considered using GitHub, but considering the size, and amount of binary information and how important it is that we stay in sync, I thought that maybe Dropbox is a more friendly place.
The link above is for glancing into the directory from anywhere, and I’ve sent you a link to sync with, and anyone else who’s interested are welcome in too. You are also able to invite people, @BigRoy.
Absolutely, there is more work to do to make locations simpler for end users, but I don’t think any kind of config/templating is the solution there. Most end users won’t manage those anyway - they want either a GUI or no config at all (“it just works”).
GUI would be the holy grail of this, that is very true. To be honest I saw that as a ‘very far ahead’ solution which would be lovely though.
I’d be very careful with ‘it just works’ approach. For something like that to be really liked by studios I’d say it would have to be very clear about how to change it if studio doesn’t like the default behavior. For example the client I mentioned in my precious post here told as one of the first things, that they didn’t even consider using ftrack_connect because when they tried (with the default config of course), they didn’t like how it organised data. Didn’t even think that it’s customisable, because they assumed it ‘just works’ as it does on demo videos. Dumb but common.
Yeah, hard to please everyone! In that case it was probably more a failure of the docs (or even getting them to the docs) rather than the system though.
In contrast, I’m definitely seeing an increasing trend that folks don’t really want to have to worry about all this stuff - they just want something sensible for the type of project. This is especially true in more collaborative workflows where there cannot be one true structure (think of how a client’s delivery structure differs from internal structure and multiply by 20). Of course the door always has to be left open for those that do care.
I believe Shotgun are discussing working on a GUI for their YAML syntax now which might be good reference if you want to go that way.
Hey @BigRoy, you ready?
I’m thinking we spend between 30 mins to an hour a day, 3-4 days a week developing The Deal. At a total estimated time of 40h (1 week), means it would take around 6-7 weeks to complete. (30 mins * 2 persons * 3 days = 6 hours a week, 40h/6hpw = 6.7w).
@tokejepsen, @mkolar, @davidmartinezanim are you in? With you guys we could pull it off much quicker.
The goal is simple, we’ll follow the conventions of Magenta as though it were an external client and we were all freelancers. As we go, things will pop up that needs plug-ins developed, and existing plug-ins will be tested and refined. By the time we’re finished, Magenta will be production ready.
Go bananas, go?
1 Like
Unfortunately I don’t have 30 mins each day at the moment, or not something that I can put in my agenda as a regular thing. But I can definitely try to push in some things asap. I’m definitely in to help out as much as I can.
Anyway, what’s the first stage?
The boards are there and we know roughly the amount of assets.
- Get at least 1 way in Pyblish Magenta to identify what you’re publishing and where to.
- Once that’s in we can start using Pyblish.
- Design a YAML schema. (preferred over XML?)
- Where doe assets/shots go? Where do we work? And where do we publish to?
- Where does possible artwork/storyboards/references go?
- How do we start with a new asset? (What’s the workflow there?)
Anything that can get us prototyping as fast as possible?
Today and tomorrow will be super busy for me so if you could give it another kick start (maybe in the form of one model that’s able to publish) then everyone knows what we’re up to.
That’s very good enough, I just figured I’d give out an estimate of what I was expecting personally, there’s no schedule.
First stage to me would be to figure out the lengths and exact angles of the camera in each shot; in the form of an animatic.
Would you like Magenta to cover this stage too? It’s more or less pre-production.
In that case, what I’ve seen people struggle with before is where it’s assets go. Some studios do this.
▾ projects/
▾ Spiderman/
▸ model
▾ Spiderman_previs/
▸ model
Keeping the entire project separate, whereas others do this.
▾ projects/
▾ Spiderman/
▸ model
▸ model_previs
And then the same applies to all assets, and renders and compositing projects etc.
How do we start with a new asset? (What’s the workflow there?)
If we were to go with your existing conventions for reference, how would you continue from here if it was a project given to you from the outside? To me, Magenta is a replica of the Colorbleed pipeline.
Roughed out what Ben is to look like when modeled.
Published it here:
/Pyblish/thedeal/dev/conceptArt/ben
As there are no validators in place for concept art, it’s probably wrong. 
To figure out
- Should pre-production be part of Magenta?
- How do we publish images?
For images, we could implement a Selector for python that looks for files.
import os
import operator
import PIL
class SelectConceptArt(pyblish.api.Selector):
def process_context(self, context):
for fname in os.listdir(context.data("cwd")):
if fname.endswith(".png"):
name = os.path.basename(fname)
instance = context.create_instance(name)
# Capture certain characteristics for validation
instance.set_data("family", "conceptArt")
instance.set_data("path", os.path.join(context.data("cwd"), fname))
instance.set_data("bytes", os.stat(fname).st_size)
img = PIL.Image.open(fname)
instance.set_data("size", img.size())
instance.set_data("aspectRatio", operator.div(img.size()))
A validator could ensure certain things.
class ValidateConceptArt(pyblish.api.Validator):
families = ["conceptArt"]
def process_instance(self, instance):
assert instance.data("size") > (300, 300), "Image must be at least 300x300"
assert instance.data("aspectRatio") == 2, "Image must have an aspect ratio of 2.0"
assert instance.data("bytes") < 2 * 1000**2, "Image must be less than 2 mb"
Followed by a global conform for placing it in the right, published directory.
Thoughts?