Separate pyblish process from "Host" processes


#1

Hi All, I am new to Pyblish and i’ve been reading through the tutorials following them quite simply. I am wondering if there is any way to create the pyblish process separate from say, Maya , and allow Maya to communicate with that UI through a unix socket perhaps? I would be glad if there is something i missed and if not maybe if someone could point me in the right direction.

The aim of this is to launch a maya session and when a user wants to publish, maya will send a message to a listening process that will spawn Pyblish outside the maya process so in the event of a crash or a hang on pyblish’s end, Maya will not be affected.

I am using PyblishQML for the GUI. Anything at all would be greatly appreciated, thank you!


#2

Hi @jamalm and welcome to Pyblish!

Yes, that should be trivial. Simply make a Python script and call it from mayapy

publish_from_mayapy.py

from maya import standalone
standalone.initialize()

import pyblish_qml
pyblish_qml.show()

Then from Maya, you can call it as such.

From Maya

import subprocess
subprocess.Popen([
   "/absolute/path/to/mayapy",
   "/absolute/path/to/publish_from_mayapy.py"
])

This should open a new console window, where Pyblish (mayapy) will start outputting messages, like you would normally see in Maya Script Editor when running from the same process.

Once you’ve got that running, you might want to pass the current scene file to it.

publish_from_mayapy.py

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("scenefile")

args = parser.parse_args()

from maya import standalone, cmds
standalone.initialize()

cmds.file(args.scenefile, open=True, force=True)

import pyblish_qml
pyblish_qml.show()

To call this, you’d probably save your scene, and pass the path to it as an argument.

From Maya

import subprocess
from maya import cmds

cmds.file(save=True)
scenefile = cmds.file(sceneName=True, query=True)

subprocess.Popen([
   "/absolute/path/to/mayapy",
   "/absolute/path/to/publish_from_mayapy.py",
   scenefile
])

I just typed this up as I went and haven’t actually run it (so beware), but also let me know if something doesn’t work as I’d described, and we’ll get to the bottom of it. Short answer at least is that yes, it’s possible!


#3

Thank you @marcus for the greeting and the quick response!
If i’m reading it correctly, Maya is creating the GUI application? or at least forking to create it( using subprocess ).

I’m worried about subprocess as it forks from Maya, i’m trying to avoid forking from Maya’s process. Is there a way that maya can talk to pyblish over a UNIX socket connection, or to a service that relays the message to pyblish.

Forgive me if this is outside the scope of pyblish or if i am looking at this at the wrong angle. Thank you again for the response.


#4

Curious, what makes you worried?

Currently, Pyblish QML is already running as a separate process to Maya and communicates via STDIN/STDOUT; if Pyblish crashes, it won’t take Maya down with it.

If you want the publishing process to run separately from Maya, but communicate with the original Maya, then you’ll need another Maya which is what the above example does; not the GUI of Maya, but rather just the “engine”, if you will. The only GUI you’ll see when running this, is Pyblish QML, publishing asynchronously from the actively running Maya from which Pyblish was called.


#5

Well from my knowledge, forking processes is generally not good if you depend on the main application to be reactive/stable, and as you stated, pyblish crashing was my concern.

I see now that you are creating the maya engine to perform the publish, this is good i believe! thank you so much for clearing this up! probably the nicest forum i’ve been on! I’ll test it out and let you know how it goes! Thanks again @marcus!


#6

It depends on if you mean the literal meaning fork, or whether you mean starting a new process from another process; i.e. “subprocess”.

If it’s the latter, then there is no risk of stability problems; the subprocess has no connection to the original process, other than letting the original process have have a handle to is (that Pyblish QML uses to send and receive data via stdin/stdout).

You can run any number of subprocesses from Maya, they each crash independently too, without impacting Maya.

Either way, you’re safe now, happy to help! :slight_smile:


#7

Hi @marcus,
I’ve run into an issue when setting up the standalone maya, pyblish_qml throws an error in host.py, it seems like it fails to “Get top window in host” as the comment in the source states. all i’ve done is initialise a standalone maya and called pyblish_qml.show() aka the first part of your solution for me.

The traceback is here:

Setting up Pyblish QML in Maya
Traceback (most recent call last):
  File "/scratch/ssd/jamal/dev/local/pyblish_test_env/teststuff/publish_from_maya.py", line 8, in 
    pyblish_qml.show()
  File "/super/secret/location/venv/lib/python3.6/site-packages/pyblish_qml/__init__.py", line 13, in show
    return host.show(parent, targets, modal)
  File "/super/secret/location/venv/lib/python3.6/site-packages/pyblish_qml/host.py", line 83, in show
    install(modal)
  File "/super/secret/location/venv/lib/python3.6/site-packages/pyblish_qml/host.py", line 58, in install
    install_host(modal)
  File "/super/secret/location/venv/lib/python3.6/site-packages/pyblish_qml/host.py", line 229, in install_host
    install(modal)
  File "/super/secret/location/venv/lib/python3.6/site-packages/pyblish_qml/host.py", line 347, in _install_maya
    _connect_host_event(app)
  File "/super/secret/location/venv/lib/python3.6/site-packages/pyblish_qml/host.py", line 320, in _connect_host_event
    parent_window = app_top_window.parent()
AttributeError: 'NoneType' object has no attribute 'parent'

My first guess is that its trying to spawn the GUI but can’t find Maya’s window?

The host.py code where it hits the error is here: (taken from the end of the _connect_host_event() method)

# Get top window in host
    app_top_window = app.activeWindow()
    while True:
        parent_window = app_top_window.parent()
        if parent_window:
            app_top_window = parent_window
        else:
            break
    # install event filter
    try:
        host_event_filter = HostEventFilter(app_top_window)
        app_top_window.installEventFilter(host_event_filter)
    except Exception:
        pass

Do you have any ideas what could be causing this?


#8

Yes, recently we added the ability for QML to respond to changes in the main Maya window; looks like we haven’t yet tested this standalone. Sorry about that!

It’s safe to not run _connect_host_event(), try that and let me know how it goes.


#9

Okay now there’s another issue ( thats almost expected when doing such great things :stuck_out_tongue: )
The standalone crashes (strengthening my trust in subprocess seeing as it doesnt affect the main maya process) with a stack trace, it seems mostly unreadable so i’ll minimise the gory details, the error outputted is as follows:

Setting up Pyblish QML in Maya
QWidget: Cannot create a QWidget when no GUI is being used
[details=Gory Deets]
Stack trace:
  /lib64/libc.so.6(+0x35270) [0x2aae76190270]
  gsignal
  abort
  qt_message_output(QtMsgType, char const*)
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/../lib/libQtCore.so.4(+0x86c22) [0x2aae8da7dc22]
  qFatal(char const*, ...)
  QWidgetPrivate::init(QWidget*, QFlags)
  QWidget::QWidget(QWidget*, QFlags)
  /apps/autodesk/maya2016.0.0.0/maya2016/lib/python2.7/site-packages/PySide/QtGui.so(+0x78a0c5) [0x2aaece48d0c5]
  /apps/autodesk/maya2016.0.0.0/maya2016/lib/python2.7/site-packages/PySide/QtGui.so(+0x78a515) [0x2aaece48d515]
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/../lib/libpython2.7.so.1.0(+0xa17ec) [0x2aae755097ec]
  PyObject_Call
  PyEval_EvalFrameEx
  PyEval_EvalCodeEx
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/../lib/libpython2.7.so.1.0(+0x741a2) [0x2aae754dc1a2]
  PyObject_Call
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/../lib/libpython2.7.so.1.0(+0x56e8f) [0x2aae754bee8f]
  PyObject_Call
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/../lib/libpython2.7.so.1.0(+0xa85ac) [0x2aae755105ac]
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/../lib/libpython2.7.so.1.0(+0xa24e8) [0x2aae7550a4e8]
  PyObject_Call
  PyEval_EvalFrameEx
  PyEval_EvalCodeEx
  PyEval_EvalFrameEx
  PyEval_EvalCodeEx
  PyEval_EvalFrameEx
  PyEval_EvalCodeEx
  PyEval_EvalCode
  PyRun_FileExFlags
  PyRun_SimpleFileExFlags
  Py_Main
  __libc_start_main
  /apps/autodesk/maya2016.0.0.0/maya2016/bin/python-bin() [0x400619][/details]

EDIT: I should say that the only change was commenting out _connect_host_event() in the _install_maya() method


#10

Ah, yes this is great. :slight_smile: It shows how little we’ve tested the standalone functionality recently.

This is likely due to the Splash screen normally being shown whilst awaiting for QML to finish booting up. You won’t need that, and it doesn’t affect the functionality of QML. It should really be smart enough to not show when it’s unable to.

Comment out this line and let me know how it goes.


#11

And we have liftoff!!! It successfully started up the QML server. Guess i’ve almost helped in the way of testing this stuff out :stuck_out_tongue:

Setting up Pyblish QML in Maya
Using Python @ '/super/secret/location/venv/bin/python'
Using PyQt5 @ 'None'
Targets: default
Success. QML server available as pyblish_qml.api.current_server()

Now a more trivial follow up, should qml’s GUI spawn or do i have to do that myself? Thank you again @marcus!


#12

Hm, you should be seeing a window when calling pyblish_qml.show().

If not, then you can call it manually. See this. You can find the server at pyblish_qml.api.current_server().


#13

Sorry for the late reply, i’ve been out of office for a week, I see now that when i run pyblish_qml.show(), python3.6 gets SIGABRT’ed , it doesnt report it on the commandline so there is no indication of this happening, there is only a crash report signalled by the operating system once the “Success. QML server available as pyblish_qml.api.current_server()” props onto the cmdline.

Honestly i’m not sure where to continue here as there is no helpful output. The only lines i’ve changed is commenting out splash = Splash() and splash.show() from the host.py file.

Thanks again @marcus


#14

Could you show me the steps leading up to the crash? Like, are you on the command-line? Or from within Maya etc.


#15

Sure!

  1. I start up maya through my terminal so i can view the output.
  2. I open the Script Editor inside the Maya GUI and i run subprocess.popen to execute the publish_from_maya.py script
  3. This starts up the standalone maya and i can see the startup output on the commandline.
  4. This also starts up the QML server, which also appears on the commandline.
Maya cmdline output

user@host $ maya16

#Startup Maya

Initialized VP2.0 renderer {
Version : 6.3.1.0. Feature Level 4.
Adapter : Quadro K4000/PCIe/SSE2
Vendor ID: 4318. Device ID :
Driver : 4.5.0 NVIDIA 384.59.
API : OpenGL V.4.5.
Max texture size : 16384 * 16384.
Max tex coords : 8
Shader versions supported (Vertex: 4, Geometry: 4, Pixel 4).
Shader compiler profile : (Best card profile)
Active stereo support available : 0
GPU Memory Limit : 3072 MB.
CPU Memory Limit: 60980.9 MB.
}
OpenCL evaluator is attempting to initialize OpenCL.
Detected 1 OpenCL Platforms:
0: NVIDIA Corporation. NVIDIA CUDA. OpenCL 1.2 CUDA 9.0.130.
Supported extensions: cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_khr_gl_event cl_nv_create_buffer
OpenCL evaluator choosing OpenCL platform NVIDIA Corporation.
Choosing OpenCL Device Quadro K4000. Device Type: GPU Device is available.


#Standalone Startup

Warning: file: /apps/autodesk/maya2016.0.0.0/maya/scripts/startup/initialStartup.mel line 195: Y-axis is already the Up-axis
[Redshift] Redshift for Maya 2016
[Redshift] Version 2.0.82, Jan 25 2017
Installing Redshift MEL overrides…
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/MLdeleteUnused.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/connectNodeToAttrOverride.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/createMayaSoftwareCommonGlobalsTab.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/createRenderNode.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/mayaBatchRender.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/relationshipEditor.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/renderWindowPanel.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/renderWithCurrentRenderer.mel
Sourcing /apps/redshift/redshift4maya/common/scripts/override/2016/unifiedRenderGlobalsWindow.mel
Running submission script “//mnt/Deadline8-Repo/submission/Maya/Main/SubmitMayaToDeadline.mel”
Evotis(): Evotis for Maya 1.2c

#QML.show()

Setting up Pyblish QML in Maya
Using Python @ ‘/scratch/ssd/venv/bin/python’
Using PyQt5 @ ‘/scratch/ssd/venv/lib/python3.6/site-packages/PyQt5’
Targets: default
Success. QML server available as pyblish_qml.api.current_server()

Thats all that is seen from cmdline. No windows appear and i receive an OS crash report about python3.6 crashing.

EDIT: i added comments so you know what sections of the output are from what operation.


#16

Thanks for that, one more question, are you able to run Pyblish QML from command-line with that Python 3.6 you’ve got going?

$ python -m pyblish_qml --demo

#17

Yeah that runs fine! Starts up the GUI n all!


#18

I open the Script Editor inside the Maya GUI and i run subprocess.popen to execute the publish_from_maya.py script

What script was that exactly? If you mean publish_from_mayapy.py from above, then that’s supposed to be run from mayapy, not Maya.

From Maya, the script below that one should be run.

As a simpler test, try this from within the Maya Script Editor.

subprocess.Popen(["/absolute/path/to/python3", "-m", "pyblish_qml"])

That’s effectively what should happen, but now we’re doing it manually and explicitly. It won’t be enough to give you the Maya session, but it’ll be a first step to launching QML from Maya.


#19

This is what i run from the Script Editor in Maya, i assumed using the mayapy path was what you meant?

import subprocess

subprocess.Popen([
“/path/to/maya/bin/mayapy”,
“/path/to/script/publish_from_maya.py”
])


I also did as you suggested in the Script Editor…

Running the popen you suggested does open the qml GUI cleanly, no issues there!


#20

Ok, then we’re close! I have to run, but will take another stab at it tomorrow morning.