Pyblish Magenta

tokejepsen · January 11, 2016, 5:32pm

Can’t remember exactly, but I think we transferred the animation from the cache to the shaded mesh by connecting attributes together.

marcus · January 11, 2016, 5:39pm

Ah, yeah @BigRoy had a solid, albeit ad-hoc, solution to this as well. Seeing as you’re all using it, maybe make it a meal and start a GitHub repo about it?

tokejepsen · January 11, 2016, 5:41pm

Could be the start of pyblish-alembic?

marcus · January 11, 2016, 5:51pm

I would make it a standalone package altogether; doesn’t seem like it needs anything from Pyblish in order to work. Then we could have a look at integrating it with Pyblish. Kind of like capture.py; useful on it’s own, and useful with Pyblish.

mkolar · January 11, 2016, 6:04pm

for that. Working name maya-look-hookup ?

it is indeed

github.com

shotgunsoftware/tk-framework-simple/blob/master/hooks/maya/loader_actions_abc.py#L282


        maya_ver = cmds.about(version=True)
        # handle a couple of different formats: 'Maya XXXX' & 'XXXX':
        if maya_ver.startswith("Maya "):
            maya_ver = maya_ver[5:]
        # strip of any extra stuff including decimals:
        major_version_number_str = maya_ver.split(" ")[0].split(".")[0]
        if major_version_number_str and major_version_number_str.isdigit():
            self._maya_major_version = int(major_version_number_str)
    return self._maya_major_version
    
def _hookup_shaders(reference_node):


hookup_prefix = "SHADER_HOOKUP_"
shader_hookups = {}
for node in cmds.ls(type="script"):
    if not node.startswith(hookup_prefix):
        continue
    obj_pattern = node.replace(hookup_prefix, "") + "\d*"
    obj_pattern = "^" + obj_pattern + "$"
    shader = cmds.scriptNode(node, query=True, beforeScript=True)
    shader_hookups[obj_pattern] = shader

mkolar · January 11, 2016, 6:30pm

I know I’ll be needing it in the upcoming project so I went ahead a started one to make sure I actually dig into it

We can discuss functionality and all ins and outs there in the issues I suppose so we don’t pollute pyblish forums with technically unrelated things.

marcus · January 11, 2016, 6:54pm

Loving it, haha!

BigRoy · January 17, 2016, 12:21am

Finally had some time to hop into the forums again and reply to this. To start discussion I’ve set up an issue on the repository, see Transfering look method

I remember a draft version of method 1 being implemented in Magenta already.

marcus · February 27, 2016, 11:36am

Some updates.

To install

Clone
Add to PYTHONPATH
Run pyblish_magenta.setup()
Boot up the example scene, and publish away.

At this point, you have access to all of Magenta’s plug-ins, and can start publishing. The example scene provides a good starting point.

Todo

First thing I wanted to do was to try a plug-in or two in isolation. To learn about it before trusting it with my data. The problem was, a single family was associated with 20+ plug-ins, and I had no choice but to run either all of them or none.

Distribute plug-ins amongst many families.

The could potentially retain their all-or-nothing status; no harm in appending families rather than replacing them. They do form a coherent whole.

Document which combinations of families must be used together; like an extractor specifically tailored to a collector. These should be few, but deliberate.
Document which combinations of families are useful together, like an extra gif output alongside geometry publishes. The should be the majority, to enable isolated use.
A tutorial on how to publish the most simple of assets, using a contrived example
A tutorial on how to publish a real asset, under real circumstance, based on real experience

Lastly, though things are being validated and extracted, it isn’t actually being saved anywhere. There needs to be an integrator, no matter how trivial, just so people can take Magenta for a spin.

Questions

Distribution of plug-ins amongst families are using the multiple family feature of Pyblish. So, the question is, how many plug-ins should each family be associated with?

instance = context.create_instance("MyModel")
instance.data["families"] = ["magenta.uv.rigging",
                             "magenta.uv.texturing",
                             "magenta.mesh.animation",
                             "magenta.mesh.cfx"]

In this example, a model is being published for both rigging, animation and cloth simulation. The UVs for this model are meant for use in rigging and animation, but not relevant for cfx, or other departments, such as fx and fluid simulations. If it were, those might have put additional constraints on what those UVs could look like, perhaps they could not be overlapping due to some limitation of an in-house plug-in or external vendor. But we do know it’s being used for texturing, so it’s likely they need to be uniform and not stretch too much. Also rigging, which suggests that the animation rig will likely have a preview texture based on the same UVs.

Technically, magenta.uv.rigging might be associated with these plug-ins.

- validate_mesh_has_uvs.py
- validate_mesh_single_uvset.py
- validate_mesh_no_overlapping_uvs.py

Or, there could be an additional family for more detailed plug-ins. Such as magenta.uv.strictRigging.

Or the other way around, there could be less families where each family encompasses more plug-ins.

The benefit of more families is flexibility, whereas the disadvantage is that the user will be managing more families in their plug-ins. Sometimes, a coherent working whole is desirable, especially when just starting out. Other times, absolute control and flexibility is needed.

Maybe a plug-in could support both the coherent whole, and the detail?

class ValidateUvs(pyblish.api.InstancePlugin):
  order = pyblish.api.ValidatorOrder
  families = ["magenta.model", "magenta.rig", "magenta.uv.rigging"]

class ExtractGif(pyblish.api.InstancePlugin):
  order = pyblish.api.ExtractorOrder
  families = ["magenta.model", "magenta.rig", "magenta.review.gif"]

In this example, any instance of magenta.model would implicitly be associated with both of these, but if the developer would rather only get a gif extracted, he could instead apply magenta.review.gif in which case ValidateUVs will never get run.

Best of both worlds?

marcus · February 27, 2016, 3:36pm

Family organisation suggestion.

# These are always run
collect_comment.py: "*"
collect_maya_units.py: "*"
collect_metadata.py: "*"
collect_metadata_maya.py: "*"
collect_scene_modified.py: "*"
cleanup_comment.py: "*"
cleanup_tempdir.py: "*"

# magenta.bake
extract_alembic.py: "magenta.bake.alembic"
validate_references_only.py: "magenta.bake.alembic"
validate_zero_transform.py: "magenta.bake.alembic"

validate_units_angular.py: "magenta.bake, magenta.pipeline.units"
validate_units_fps.py: "magenta.bake, magenta.pipeline.units"
validate_units_linear.py: "magenta.bake, magenta.pipeline.units"

# magenta.review
extract_quicktime.py: "magenta.asset.model, magenta.model.rig, magenta.review, magenta.review.qt"
extract_gif.py: "magenta.asset.model, magenta.model.rig, magenta.review, magenta.review.gif"

# magenta.asset - general operations
extract_origin.py: "magenta.asset, magenta.asset.metadata"
extract_maya_ascii.py: "magenta.asset"
log_history.py: "magenta.asset.metadata"

# etiquette - cosmetics and good behavior
validate_display_layer_empty.py: "magenta.rig.etiquette"
validate_joints_hidden.py: "magenta.rig.etiquette"
validate_no_unknown_nodes.py: "magenta.etiquette"
validate_references_no_failed_edits.py: "magenta.etiquette"
validate_shape_default_names.py: "magenta.etiquette"
validate_shape_render_stats.py: "magenta.etiquette"
validate_transform_naming_suffix.py: "magenta.etiquette"

# uv
validate_mesh_has_uv.py: "magenta.uv, magenta.uv.texturing, magenta.mesh.rigging"
validate_mesh_single_uv_set.py: "magenta.uv.texturing"

# meshes
validate_mesh_lamina_faces.py: "magenta.mesh, magenta.mesh.correctness, magenta.mesh.rigging, magenta.mesh.cfx"
validate_mesh_non_manifold.py: "magenta.mesh, magenta.mesh.correctness, magenta.mesh.rigging, magenta.mesh.cfx"
validate_mesh_non_zero_vertices.py: "magenta.mesh, magenta.mesh.correctness"
validate_mesh_normals_unlocked.py: "magenta.mesh, magenta.mesh.animation"
validate_mesh_no_negative_scale.py: "magenta.mesh, magenta.mesh.correctness, magenta.mesh.animation"
validate_namespace_empty.py: "magenta.mesh.rigging"
validate_no_namespace.py: "magenta.mesh.rigging"
validate_no_null_transforms.py: "magenta.mesh.rigging, magenta.mesh.correctness"
validate_no_transform_zero_scale.py: "magenta.mesh.rigging"

validate_rig_contents.py: "magenta.asset.rig, magenta.rig.layout"
validate_model_content.py: "magenta.asset.model"

validate_scene_set_workspace.py: "magenta.pipeline"
validate_single_assembly.py: "magenta.pipeline"
extract_comment.py: "magenta.pipeline"

# question marks
validate_node_no_ghosting.py: "magenta.?"
validate_no_default_camera.py: "magenta.?"
validate_no_rename.py: "magenta.?"
validate_no_intermediate_objects.py: "magenta.?"
extract_renders.py: "magenta.?" # What does this produce?
validate_scene_dimensions.py: "magenta.?" # When does this matter?

These are all families used.

magenta
    .mesh
        .animation
        .correctness
    .animation
        .alembic
    .review
        .gif
        .qt
    .asset
        .metadata
        .model
        .rig
    .uv
        .animation
        .texturing
    .rig
        .etiquette
    .pipeline
        .units
    .etiquette

All of which deserves a description and tutorial on use, but for now all of this is a rough sketch to get your minds started.

Oh, and I’ve put question marks on plug-ins I don’t understand.

Things to note

Notice how you can specify magenta.asset.model to get Quicktime and gif output, but if you’d rather only have gif, you specify just magenta.review.gif.

That way, magenta.asset.model works the way model is currently, whilst at the same time opening up for individual use.

BigRoy · February 27, 2016, 3:39pm

Whilst @marcus was typing the above I was also listing my own categorization. Great write-up btw. This is how far I came:

Roll your own pipeline

Magenta at this moment houses a very nice suite of Plug-ins of which many have been used in production at Colorbleed. With its current state I believe it’s time to clean the extension up in such a way that it becomes easy for many to clone it and cherry-pick features that they would benefit from in their production. As such Magenta provides two sides:

A mini pipeline out of the box
A suite of Plug-ins that are built to be generic constraints which can be applied to any production.

The idea is now to allow you to “tag on” a subset of its Plug-ins to use in your own productions even if you’d rather stay away from a “pipeline out of the box”. So aside from being a mini-pipeline it’s much more, it’s a toolkit to do more.

Current powers

Right now the power of Magenta is housed mostly in its wide range of constraints that it allows you to put on your pipeline. For example it contains the plug-ins to easily validate the scene units, like cm, 25 fps, degrees. And want to expand this much further so it fits your productions.

The idea is that if you have a constraint in the pipeline you can add it. Have a constraint on the scene units? Add that family magenta.units.

Categorize

It’s very hard to categorize many plug-ins, but for now let me try to do so for the current Validators:

### mesh: Plug-ins useful for models.

mesh.contents
- Validate it's under a single top group (no multiple hierarchies)
- Validate shape naming conventions (shape is "{transform}Shape")
- Validate transform naming suffix (mesh transform has "_GEO")

mesh.frozen
- Validate transforms have frozen transforms
- Validate transforms have non-zero scale

mesh.normals
- Validate meshes have unlocked normals

mesh.uvs
- Validate meshes have (at least any) UVs
- Validate meshes have at most 1 uvSet

mesh.clean
- Validate meshes have no non-manifold edges/vertices
- Validate meshes have no lamina faces (faces sharing all edges)
- Validate meshes have "frozen points" (check for non-zero points)
- Validate no intermediate meshes
- Validate no null transforms (transforms without shape childrens)
- Validate meshes aren't extremely big (`1e5` from origin)
- Validate shapes have default render stats (primaryVisibility, etc.)
- Validate nodes don't have "ghosting enabled"

### rig: Plug-ins useful for a rig.

rig.contents
- Validate it's under a single top group (no multiple hierarchies)
- Validate a rig specifies a "controls_SET" and "pointcache_SET"

rig.clean
- Validate a Mesh' transform has no negative scale
- Validate a rig has no visible joints
- Validate a rig's controls are all local identity matrix (TODO)
- Validate nodes don't have "ghosting enabled"

### layout: Plug-ins useful for a collection of references

layout.content
- Validate the instance only has references

layout.clean
- Validate there are no "failed reference edits"

### Scene: Generic scene checks to ensure clean workflows

scene.units
- Validate the scene has the correct FPS (`25 FPS`)
- Validate the scene has the correct angular units (`degrees`)
- Validate the scene has the correct linear units (`cm`)

scene.workspace
- Validate the Maya workspace is set correctly for the current scene 

scene.clean
- Validate there are no empty display layers

As such your pipeline requires UVs on a mesh, frozen transforms and is very strict about how clean it should be you could do: families = ["mesh.uvs", "mesh.frozen", "mesh.clean"]

if additionally you want to stick to Magenta’s “structure” of a model. It has to be a single hierarchy in the outliner (in one group) and stick to its naming conventions you can append the mesh.contents family.

I still feel they are somewhat “wide” on what they cover per family. But going this way it would mean that you decide what should happen with your types, like a mesh in your pipeline.

marcus · February 28, 2016, 10:13am

Cool, that’s great Roy.

I think the most interesting thing about this is what you said in the chat right after seeing my post. Hope you don’t mind me sharing it here!

It’s funny that when reading yours I have no idea where I’d start actually using something like that… For example why would I care alembic could only come from something that was references only?

To me, it perfectly highlights the two extremes at which we can categorise plug-ins in this way.

On one end, categories are general down to the exact operation being performed, such as mesh.frozen. Anyone familiar with Maya and possibly 3d overall could understand what that does, without a description and without reading the source. Let’s call these families “unbiased”.

On the other end you have magenta.mesh.correctness. What is that?

To me, Magenta has a flavour. A personality. A direction. It encapsulates a particular style of working. It’s your style; the working style of Colorbleed. What I was thinking when assigning magenta.mesh.correctness was for a mesh to be correct according to what is correct at Colorbleed. There is policy involved, decisions.

The same goes for uv, rigs, etiquette and pipeline. Let’s call these families “biased”.

Personally, I think unbiased families are great for a mish mash of plug-ins. Where plug-ins don’t relate to each other, but are instead useful on their own. Like a pyblish-basics or pyblish-common.

Biased families on the other hand I think are better suited for a coherent library where each family cooperates to form a whole.

In practice, you couldn’t know what magenta.mesh.correctness does without consulting the docs or knowing something about the author. It would involve a choice similar to…

“Do I want to stick with the same strict convention as Digital Doman and use dd.mesh.correctness?”
or “Do I want my playblasts to be extracted and formatted like they do over at Colorbleed with magenta.review?”

From general knowledge and showreel of a studio, users would know which is best suited to them and could trust that, when a choice is made that they are either unfamiliar with, or would strictly oppose (such as having references only in an alembic), it might be a good idea to at least give it a try because it has already been proven to work.