I’m totally on board with this. I’ll be implementing pyblish into another studio which asked me for consultation and they as many others are stubborn with certain approaches they like and won’t be willing to change. Dropping in pyblish with a few plugins however shows them the potential power on tasks that they need done. but don’t know how to deal with. Once they get used to the fact that it works, it will be very easy to just keep plugging new extensions and slowly transition to cleaner workflow. Smaller packages that I can potentially distribute to other clients are definitely when my time will be spent most of the time.
Yeah, loving it. This is the way I expect the majority studios to use Pyblish today. Magenta on the other hand, and feel free to correct me on this one @BigRoy, is for new studios, new productions.
New projects are started every day, some launch a studio specifically for a given project, and it’s here where Magenta and Napoleon is meant to kickstart things.
Down the line, once a pipeline definition like Magenta has reached a mature age and is well enough used, tested and documented, I think it will leave little room for doubt whether to go with an ad-hoc, in-house, hack-and-slash pipeline or Magenta.
In the future, I’d like for there to be many pipeline definitions, each benefiting various types of productions, like full cg, vfx, arch-vis. This is where you guys come in. You can codify what you know and like, document it, and share it.
Picture a place like this in which ever entry is a pipeline definition, kit or extension to Pyblish.
- “What sort of project is this?”
- “It’s an ad, around 30 seconds, heavily character based”
- “Ok, since we’re Maya- and Fusion-based, let’s give Pyblish Magenta a go”
I expect pipeline definitions to have their own website, tutorials and community.
My main concern is the fact on how to deal with Selection and Conforming. How do you know what to extract and where the extracted data will need to end? Especially if you want to supply that in a manner that is usable for everyone to pickup.
The most of the system to learn of Pyblish (I think) for small studios is in the Selection and Conforming department. I think (and I hope) many small studios already have their own set of small utilities/ways to check their data is up to par with their requirements, but is just not as streamlined/customizable as a toolset like Pyblish. The validators that are currently in Pyblish Magenta took me around a day (or maybe two) to get in there.
To be honest I feel that to write a Validator you won’t need to know that much about Pyblish, but about how your host works and the data you want to check. Within the Extraction I also consider it to be exporting to an arbitrary format to an arbitrary location (if Conformer is used) using tech and commands that are unrelated to Pyblish. (Of course you need the Selector to select the nodes you want to export, but still the issue is mostly in the Selector.)
Selection and Conforming means knowing what data you need to select and where it needs to go is the number one issue where things will get hardcoded. If Pyblish is so much about introducing a good workflow/standard, why skip this step?
I think it might that pipeline might be misinterpreted here. I’m not setting out to get to the new ftrack or shotgun. Preferrably build even something that is so barebones that it could work with ftrack and shotgun. The issue at the moment though is that the issues mentioned above are a 100% present in every small studio, especially if you consider they are not hooked up to tools like ftrack and shotgun.
I think this is a very interesting topic that should both get their own thread. Especially since they are on your mind feel free to start a discussion with a brief description of the goals you would like to reach. I would be happy to contribute in that area!
I do have to say that one could have Pyblish without playblasting, but not have it running optimally without a way of knowing some ins- and outs about selecting and conforming data.
Exactly this! Pipelines will evolve or even get build up over time anyway.
But we need to have some barebones foundation for “hey this comes from here and goes to there!” so we can deliver such behaviour. (So I feel.) Again, I think Validators (once Selection is done correctly) can currently be easily swapped around, but that does mean there is some fundamentals to be build for Selection?
Good point. In the meantime I’ll have a look at getting some less tech-savvy people around for a possible test-case.
Sounds good. Let’s have Pyblish put something solid in place for the future.
This got me thinking.
Maybe there could be a Selection-only extension to Pyblish? Something that governs how you format the data within the host so as to make it validatable, extractable and conformable by subsequent extensions or your own custom ones.
For example, one extension could provide the means to generate Instances based on putting Maya nodes into an objectSet, whereas another extension could do the same but base it on some naming convention, like a _RIG prefix and anything within for rigs.
If the instances are made encapsulated enough - as in they contain all relevant data like names of nodes and relevant properties and attributes - then to subsequent plug-ins it makes no difference from where they were generated. All they see is an Instance packed with information to be validated.
In a sense, Selectors could pick up as much as possible from any host, getting close to duplicating the whole scene, just so that subsequent plug-ins have an identical set of data to work on, regardless of which extension generated the instances (or even which host they came from?)
Down the line, one may even introduce some form of compatibility between extensions. Such that Validator, Extractor and Conform “packs” could be made to support one or more very specific Selector-extensions.
Shared Project Directory
I created a Dropbox account for Pyblish, and created a The Deal project directory in it, what do you think about working on it in here? (The files in there currently I copied from the Magenta Foobar example).
At first I considered using GitHub, but considering the size, and amount of binary information and how important it is that we stay in sync, I thought that maybe Dropbox is a more friendly place.
The link above is for glancing into the directory from anywhere, and I’ve sent you a link to sync with, and anyone else who’s interested are welcome in too. You are also able to invite people, @BigRoy.
Absolutely, there is more work to do to make locations simpler for end users, but I don’t think any kind of config/templating is the solution there. Most end users won’t manage those anyway - they want either a GUI or no config at all (“it just works”).
GUI would be the holy grail of this, that is very true. To be honest I saw that as a ‘very far ahead’ solution which would be lovely though.
I’d be very careful with ‘it just works’ approach. For something like that to be really liked by studios I’d say it would have to be very clear about how to change it if studio doesn’t like the default behavior. For example the client I mentioned in my precious post here told as one of the first things, that they didn’t even consider using ftrack_connect because when they tried (with the default config of course), they didn’t like how it organised data. Didn’t even think that it’s customisable, because they assumed it ‘just works’ as it does on demo videos. Dumb but common.
Yeah, hard to please everyone! In that case it was probably more a failure of the docs (or even getting them to the docs) rather than the system though.
In contrast, I’m definitely seeing an increasing trend that folks don’t really want to have to worry about all this stuff - they just want something sensible for the type of project. This is especially true in more collaborative workflows where there cannot be one true structure (think of how a client’s delivery structure differs from internal structure and multiply by 20). Of course the door always has to be left open for those that do care.
I believe Shotgun are discussing working on a GUI for their YAML syntax now which might be good reference if you want to go that way.
Hey @BigRoy, you ready?
I’m thinking we spend between 30 mins to an hour a day, 3-4 days a week developing The Deal. At a total estimated time of 40h (1 week), means it would take around 6-7 weeks to complete. (30 mins * 2 persons * 3 days = 6 hours a week, 40h/6hpw = 6.7w).
@tokejepsen, @mkolar, @davidmartinezanim are you in? With you guys we could pull it off much quicker.
The goal is simple, we’ll follow the conventions of Magenta as though it were an external client and we were all freelancers. As we go, things will pop up that needs plug-ins developed, and existing plug-ins will be tested and refined. By the time we’re finished, Magenta will be production ready.
Go bananas, go?
1 Like
Unfortunately I don’t have 30 mins each day at the moment, or not something that I can put in my agenda as a regular thing. But I can definitely try to push in some things asap. I’m definitely in to help out as much as I can.
Anyway, what’s the first stage?
The boards are there and we know roughly the amount of assets.
- Get at least 1 way in Pyblish Magenta to identify what you’re publishing and where to.
- Once that’s in we can start using Pyblish.
- Design a YAML schema. (preferred over XML?)
- Where doe assets/shots go? Where do we work? And where do we publish to?
- Where does possible artwork/storyboards/references go?
- How do we start with a new asset? (What’s the workflow there?)
Anything that can get us prototyping as fast as possible?
Today and tomorrow will be super busy for me so if you could give it another kick start (maybe in the form of one model that’s able to publish) then everyone knows what we’re up to.
That’s very good enough, I just figured I’d give out an estimate of what I was expecting personally, there’s no schedule.
First stage to me would be to figure out the lengths and exact angles of the camera in each shot; in the form of an animatic.
Would you like Magenta to cover this stage too? It’s more or less pre-production.
In that case, what I’ve seen people struggle with before is where it’s assets go. Some studios do this.
▾ projects/
▾ Spiderman/
▸ model
▾ Spiderman_previs/
▸ model
Keeping the entire project separate, whereas others do this.
▾ projects/
▾ Spiderman/
▸ model
▸ model_previs
And then the same applies to all assets, and renders and compositing projects etc.
How do we start with a new asset? (What’s the workflow there?)
If we were to go with your existing conventions for reference, how would you continue from here if it was a project given to you from the outside? To me, Magenta is a replica of the Colorbleed pipeline.
Roughed out what Ben is to look like when modeled.
Published it here:
/Pyblish/thedeal/dev/conceptArt/ben
As there are no validators in place for concept art, it’s probably wrong. 
To figure out
- Should pre-production be part of Magenta?
- How do we publish images?
For images, we could implement a Selector for python that looks for files.
import os
import operator
import PIL
class SelectConceptArt(pyblish.api.Selector):
def process_context(self, context):
for fname in os.listdir(context.data("cwd")):
if fname.endswith(".png"):
name = os.path.basename(fname)
instance = context.create_instance(name)
# Capture certain characteristics for validation
instance.set_data("family", "conceptArt")
instance.set_data("path", os.path.join(context.data("cwd"), fname))
instance.set_data("bytes", os.stat(fname).st_size)
img = PIL.Image.open(fname)
instance.set_data("size", img.size())
instance.set_data("aspectRatio", operator.div(img.size()))
A validator could ensure certain things.
class ValidateConceptArt(pyblish.api.Validator):
families = ["conceptArt"]
def process_instance(self, instance):
assert instance.data("size") > (300, 300), "Image must be at least 300x300"
assert instance.data("aspectRatio") == 2, "Image must have an aspect ratio of 2.0"
assert instance.data("bytes") < 2 * 1000**2, "Image must be less than 2 mb"
Followed by a global conform for placing it in the right, published directory.
Thoughts?
We usually do animatics in the form of an animation scene.
So they are basically an animation scene. 
I’d love to know how it’s dealt with in other places?
The same with models. The previs ones are considered just early versions of the ones that are used later on.
What I do like though (I heard this from one of our last interns) is putting a possible sandbox area per asset where you can perform tests or draft prototyping without being restrained too much to rules. Haven’t put that in production and am still looking where it would be a nice fit.
–
Seems like Ben is quite the guy.
Pre-production (concepting) is an important stage, especially to keep it organized because it’s referred to pretty often. But at Colorbleed we’ve found it a stage that’s very hard to define, or put a set of rules on. Sometimes you jump into Photoshop to sketch, or scan some images you drew. Other times you leap straight into zbrush and start prototyping shapes. Then there are times clients comes over with some draft concepts. Characters/props get ditched as quickly as they are added. So it has always been kind of a sandbox folder.
At this moment I don’t see a (programmable) solution that doesn’t hold back building concepts more than it gains for the artist. What do you think? Any ideas?
We have a tool that uses the Schemas/Template to create the folders based on the asset information you provide. So basically we create new assets with that. Again would love to hear ideas on what you think is suited best here.
I hope to build Magenta as a draft for our 2.0 design of the pipeline, or maybe even become it!
If the goal is to figure out lengths and angles of shots, then there are lots of ways. You could make a 3d animation, stick together some scanned images in Premiere, put sound on it, not put sound on it, do a 2d animation or just a Google spreadsheet with start and end frames and use your imagination. Each has it’s own pros/cons.
Some studios also make a separation between what is an animatic, and what is a “previs”. The previs more closely resembling the final version - sometimes with animated characters, original music, voices and lighting, but in a crude manner.
Was there anything specific you wanted to know more about?
Yes, this is referred to differently by different houses, but the underlying concept is that convention is irrelevant up until an asset is actually published. Meaning an artist could work on the desktop for all anyone cares, so long as it passes through validation and put in the right place once he’s done.
Some studios do exactly that (yeah, work on the desktop), whereas others put a mild constraint on it for performance, security and/or disk space concerns, e.g. organise it by user, asset or department.
Considering the small size of the project, how about we chuck everything into one giant sandbox folder and have a closer look at why that is a good or bad idea?
/Pyblish/thedeal/sandbox
Sure. Considering I don’t know the pipeline and have finished a piece of art that I need to put somewhere, we will at the very least need something to put the art into the correct folder, a Conformer. And for that we need a Selector. We can skip validation until it becomes more obvious what it is we need from the art.
[quote=“BigRoy, post:34, topic:79”]
We have a tool that uses the Schemas/Template to create the folders based on the asset information you provide. So basically we create new assets with that.[/quote]
I say let’s bring it in.
I’ve got art ready right now, and it’s just sitting around. The sooner we get it conformed the quicker we can move on with more stuff.
Let’s start simple and publish from the command-line, from within the folder in which the art is located.
$ cd /Pyblish/thedeal/sandbox
$ pyblish publish
We can use the Selector from above to fetch it, but what does the Conformer look like?
And while we’re at it, how does one install Magenta?
What would you think is a good and easy way for users to get started? Especially those who want to hop on in with the project?
I guess core requirements would be having the repository on your python path plus the plug-ins loaded with pyblish. Then next up would be getting a project up and running, which is what the example project will take us through as a test case.
I hope to set it up with the latest Lucidity repository soon! Might have some time tomorrow night or the day after.
Allrighty, then this is how you install.
Install Pyblish Magenta on Windows for Maya
Here’s how you can get set up with Magenta from the command-line. For a persistent install, either add the contents of \integration\maya\userSetup.py to your own userSetup.py, or append the directory to your permanent PYTHONPATH.
Method 1 - Run-time
$ git clone https://github.com/pyblish/pyblish-magenta.git
$ cd pyblish-magenta
$ maya
>>> import pyblish_magenta
>>> pyblish_magenta.setup()
Method 2 - Environment
$ git clone https://github.com/pyblish/pyblish-magenta.git
$ cd pyblish-magenta
$ set PYTHONPATH=%cd%\integration\maya
$ maya
To get access to The Deal project directory, ask here.
Things to note:
- The instructions only apply to Windows
- The plug-ins only apply to Maya
To keep Magenta open for extension to other hosts, I added a /integration/maya directory, with the expectation that other hosts reside there as well. However, this doesn’t stop Maya-only plug-ins from being discovered by other hosts, resulting in tons of warnings of No module names maya.
How about refactoring Maya-only plug-ins into it’s own subdirectory?
▸ plugins
▸ maya
select_maya_instance.py
validate_any_instance.py
Cool. Until then, to keep things going, where are you planning on programming Lucidity to put concept art?
I added a selector and conformer for concept art.
Behaviour
FROM: thedeal/dev/conceptArt/characters/ben/ben_model.png
TO: thedeal/asset/model/characters/ben/conceptArt/ben_model.png
Usage
$ cd thedeal/dev/conceptArt/characters/ben
$ pyblish --verbose publish
INFO - Selecting concept art..
INFO - Creating instance: ben_model
INFO - Conforming ben_model to ...set\model\characters\ben\conceptArt
INFO - Successfully conformed ben_model!
Things to note:
- There are no versions,
ben's concept art is being overwritten each publish. - Input directory depth is assumed to be 5 levels deep
- Output directory is the responsibility of the conformer
I’ve created a draft schema for the project, it’s inside the project in the database folder, there you’ll find schema.yaml.
Currently it’s using lucidity to format/parse paths and my forked repository of lucidity to use a Schema class (collection of templates) and perform the yaml parsing. You can find that repository here: https://github.com/BigRoy/lucidity
This is the current YAML file:
# Ensure the default settings for the templates
defaults:
anchor: 'start'
mode: 'relaxed'
# Set some references that we don't want to parse on its own.
# It's solely used to define a reference for a template, not a template.
# This is used to simplify the patterns in the templates.
references:
root: '{root:(^[\w]*:*[\/]?(?:(?:(?<=[\/])[^\/]+[\/]?)*))}'
shot_asset: '{@root}/asset/shots/{container}/{asset}'
model_asset: '{@root}/asset/model/{container}/{asset}'
# The paths of our project
paths:
# Model/Assets
# --------------
# Concept Art
conceptArt.dev:
pattern: '{@root}/dev/conceptArt/{container}/{asset}'
conceptArt.asset:
pattern: '{@model_asset}/conceptArt'
# Modeling
model.dev:
pattern: '{@root}/dev/modeling/{container}/{asset}'
model.asset:
pattern: '{@model_asset}/geo'
# Rigging
rig.dev:
pattern: '{@root}/dev/rigging/{container}/{asset}'
rig.asset:
pattern: '{@model_asset}/rig'
# LookDev
lookdev.dev:
pattern: '{@root}/dev/lookdev/{container}/{asset}'
lookdev.asset:
pattern: '{@model_asset}/lookdev'
# Shots
# ------
# Animation
animation.dev:
pattern: '{@root}/dev/animation/{container}/{asset}'
animation.asset:
pattern: '{@shot_asset}/anim'
animation.playblast:
pattern: '{@shot_asset}/renders/playblast'
# Lighting
lighting.dev:
pattern: '{@root}/dev/lighting/{container}/{asset}'
lighting.asset:
pattern: '{@shot_asset}/renders/3D'
# Compositing
comp.dev:
pattern: '{@root}/dev/comp/{container}/{asset}'
comp.asset:
pattern: '{@shot_asset}/renders/comp'
# FX
comp.dev:
pattern: '{@root}/dev/fx/{container}/{asset}'
comp.asset:
pattern: '{@shot_asset}/{asset}/fx'
This is what I currently tested it with:
import lucidity
import lucidity.error
root_path = 'E:/Dropbox/pyblish/thedeal'
yaml_path = os.path.join(root_path, 'database', 'schema.yaml')
# Retrieve the schema
schema = lucidity.Schema.from_yaml(yaml)
# Parse a path
path = "Dropbox/pyblish/thedeal/dev/conceptArt/characters/ben/ben_model.PNG"
data, template = schema.parse(path)
print data
# {'root': 'Dropbox/pyblish/thedeal', 'container': 'characters', 'asset': 'ben'}
print template
# Template(name='conceptArt.dev', pattern='{@root}/dev/conceptArt/{container}/{asset}')
if template.name.startswith("conceptArt"):
new_name = template.name.replace('dev', 'asset')
asset_template = schema.get_template(new_name)
output_path = asset_template.format(data)
print output_path
# Dropbox/pyblish/thedeal/asset/model/characters/ben/conceptArt
Excellent, thanks.
About the filenaming convention, is ben_model.PNG a good one? Is there plug-in to handle this? Otherwise we should make one. A Validator I presume, warning about a misnamed file, else a Selector that simply can’t see misnamed files. What do you think?
It depends on what model in the name stands for.
Is it about the contents of the image?
Does it refer to “art for the model for Ben”?
Also what I’d like to do is use versioning in this project. So the published files can have multiple versions as well. Something like ben_model_v01.png
What do you recommend as a setup?
What do you think is convenient for a user to decide when to version up with Pyblish? (eg. Shotgun Toolkit does it inside of the publishing solution)