Pyblish and pyblish-win renovation

Pyblish Development
#1

Hi all,

Just to let you know, pyblish and [pyblish-win]((https://github.com/pyblish/pyblish-win) are going through some changes on GitHub at the moment in an effort to rid them both from submodules and simplify updates from one version to the next.

The goal is to merge all submodules into one and distribute them both as plain old git repos. This has already been done successfully for pyblish but isn’t quite there yet for pyblish-win.

Ideally I would like to rid the world of pyblish-win completely and release distributions via the releases portion of the main pyblish repository, for not only Windows, but other platforms as well, and to have a single merged repository, as opposed to several.

Because of this, the latest changes to Pyblish will take a little longer to hit the pyblish-win distribution. If you are in a rush, you can either (1) stick with the current release, (2) install from source or…

(3) help out

I need a script, Python and/or bash, capable of running via a terminal to download and bundle dependencies and Pyblish together for release. The contents of the result are exactly what is currently present in pyblish-win, primarily Python 2.7 with custom packages installed and a unique qt.conf along with PyQt5 compiled for this version of Python.

It’s no cakewalk, but it’s what we need for the continued health of the pyblish-win distribution.

Let me know if you’re hungry for the most challenging and boring, yet necessary, part of software development!

Pyblish Shell on OSX
#2

So after having meditated on this, I took a step back to consider “what does a user actually need?”. And I got this idea.

pyblish-shell

Instead of pyblish-win, where everything is bundled together - binaries and code - we’ll split it up.

Where you can choose to update one or the other.

At the moment, updating pyblish-win is kind of a pain, and what’s more, it includes a duplicate copy of pyblish which means I’ll update both whenever one changes. This is bad, and lends to the situation we’re currently in, where pyblish-win lags behind the master release.

pyblish-shell would then be an installer or standalone executable, much like pyblish-win is today, but you would rarely need to update it. So long as Pyblish still runs on Python 2.7.10 and PyQt5.4. On the flipside, you could update pyblish at any time, as pyblish-shell would simply be a Python interpreter with access to your PYTHONPATH and use whatever libraries you have got there.

I ran a small test this morning and it seems that technically this is possible.

It also means deployment will be super simplified, because at the moment, I’m recompiling the pyblish-win installer for every release, even though only a few, if any, has ever changed its dependencies. We’d also be getting a unique .exe/executable, which means no more anonymous “python” processes, but instead a single “pyblish-shell.exe” binary. Which apart from being useful, is awesome too. :slight_smile:

#3

I’ve got an initial working version of pyblish-shell for Windows and I’m now looking for a brave volunteer to give it a whirl.

Any takers?

Usage

First off, I gave this a try on a clean install of Windows 7 with success, but it’s likely there will be other problems for you. That’s why I need you.

Here’s what you gotta do.

  1. Download and unpack pyblish-shell.rar
  2. A) Put your existing Pyblish libraries on your PYTHONPATH
  3. B) Or download pyblish.zip and put the pythonpath directory on your PYTHONPATH.
  4. Run pyblish-shell\pyblish-qml.exe --debug

Description

pyblish_qml.exe is a standalone Python interpreter that has been hard-coded to import and run pyblish_qml, it therefore assumes you have it on your PYTHONPATH. And that’s it. That’s the only requirement. Pure Python libraries on your PYTHONPATH.

It includes a Python distribution and PyQt5 compiled for that distribution.

If that fails for whatever reason, give this a try instead.

$ cd pyblish-shell
$ pyblish_shell.exe -m pyblish_qml --debug

The really good part of all of this, is that you can now have a completely cross-platform installation of Pyblish somewhere on your server, and one or more copies of pyblish-shell for each platform you use, and it will use whichever libraries you have on your PYTHONPATH. It could be the master pyblish project, or separately installed repositories so you can make modifications and contribute, it doesn’t matter. Not to mention that the binaries now take 29 mb, as opposed to 1 gb+ because of python-qt5 and Git history. (!)

This is huge! Let me know how it goes!

#4

Sounds great!

I’m away this week, but when I get back. I’ll give it a shot.

#5

Thanks Milan!

#6

Windows and Debian releases available!

This has been a great success. The same script used to bundle pyblish-shell on Windows now works on Linux as well, which means we’ll be able to distribute it as installers and packages on many if not all relevant platforms, automatically.

The next adventure lies in bundling it into native Linux distributions, such as .deb and .rpm at which point it can be hosted such that you can simply say:

$ apt-get install pyblish-shell # Debian-based systems
$ yum install pyblish-shell     # Redhat-based systems
#7

Both the pyblish-shell\pyblish-qml.exe --debug and pyblish_shell.exe -m pyblish_qml --debug run fine with the pyblish.zip path’s on my PYTHONPATH.

I’m especially pleased with how smooth this goes and how much smaller the filesizes are! :smiley:

Can I already use this with pyblish_maya and alike to take this into production?

Update

Ok. Some strange things are happening. Not sure if related. Running the pyblish-qml.exe --debug a couple of times sometimes show me the lines:

<Unknown File>: QML VisualDataModel: Error creating delegate
file:///C:/Users/Roy/Desktop/pyblish-master/pyblish-master/pythonpath/pyblish_qml/qml/delegates/RecordDelegate.qml: Object destroyed during incubation
file:///C:/Users/Roy/Desktop/pyblish-master/pyblish-master/pythonpath/pyblish_qml/qml/delegates/RecordDelegate.qml: Object destroyed during incubation

This is in the console (command prompt). Note that the Pyblish Window shows up like normal and doesn’t seem to be affected. Might just be a warning.

Also I got this once when running Maya just after testing with the shell stuff and starting Maya with the old pyblish-win:

Failed to execute userSetup.py
Traceback (most recent call last):
  File "P:\pipeline\dev\git\pyblish-win\lib\pyblish\modules\pyblish-maya\pyblish_maya\pythonpath\userSetup.py", line 3, in <module>
    __import__("pyblish_maya")
  File "P:\pipeline\dev\git\pyblish-win\pythonpath\pyblish_maya\__init__.py", line 5, in <module>
    util.wrap_module(__name__.replace("_", "-"))
  File "P:\pipeline\dev\git\pyblish-win\pythonpath\pyblish_maya\__init__.pyc\..\..\__pyblish_util.py", line 30, in wrap_module
    reload(mod)
  File "P:\pipeline\dev\git\pyblish-win\lib\pyblish\modules\pyblish-maya\pyblish_maya\__init__.py", line 3, in <module>
    from lib import (
  File "P:\pipeline\dev\git\pyblish-win\lib\pyblish\modules\pyblish-maya\pyblish_maya\lib.py", line 10, in <module>
    import pyblish_integration
  File "P:\pipeline\dev\git\pyblish-win\pythonpath\pyblish_integration\__init__.py", line 5, in <module>
    util.wrap_module(__name__.replace("_", "-"))
  File "P:\pipeline\dev\git\pyblish-win\pythonpath\pyblish_integration\__init__.pyc\..\..\__pyblish_util.py", line 30, in wrap_module
    reload(mod)
  File "P:\pipeline\dev\git\pyblish-win\lib\pyblish\modules\pyblish-integration\pyblish_integration\__init__.py", line 2, in <module>
    from .lib import (
  File "P:\pipeline\dev\git\pyblish-win\lib\pyblish\modules\pyblish-integration\pyblish_integration\lib.py", line 22, in <module>
    import pyblish_rpc.server
  File "P:\pipeline\dev\git\pyblish-win\lib\pyblish\modules\pyblish-rpc\pyblish_rpc\server.py", line 14, in <module>
    from SimpleXMLRPCServer import (
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\SimpleXMLRPCServer.py", line 102, in <module>
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\xmlrpclib.py", line 144, in <module>
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\httplib.py", line 79, in <module>
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\mimetools.py", line 6, in <module>
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\tempfile.py", line 35, in <module>
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\random.py", line 880, in <module>
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\random.py", line 97, in __init__
  File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\random.py", line 111, in seed
WindowsError: [Error -2146893801] Provider type not defined

Yet this only happened once and I haven’t been able to reproduce this again afterwards. This might have just been Windows/Maya acting up.

#8

Hey Roy,

Glad to hear you’re trying things out!

Yes, this should definitely already work as-is. But probably not in conjuction with pyblish-win, as it modifies the libraries and path to use the supplied binaries, which might cause some hassle.

What I would suggest, is that you clone pyblish/pyblish and put it’s /pythonpath folder on your PYTHONPATH prior to launching Maya.

At that point, with pyblish_qml.exe running, you should be good to go.

QML VisualDataModel: Error creating delegate

That’s “normal”. It’s a side-effect of asynchronously creating widgets in QML, sometimes another thread comes in and swoops some away. It’s something that could be fixed, but it is really just a warning.

Oh, and a slight note about --debug, it has the exact same effect as launching without the flag, except that when you close the GUI, it kills the process. So you can try things out with it, the GUI will still switch to host environments and things, but don’t make it a habit.

with the pyblish.zip path’s on my PYTHONPATH.

About this, those shouldn’t need to be on your PYTHONPATH. All you should need, is an empty PYTHONPATH, apart from the pyblish/pythonpath directory.

Let me know if that works for you.

#9

Thanks for clarifying!

Then I think it was actually working, because I clicked the pyblish_qml.exe once and as such ran it without --debug. This would mean it would stay open and then pyblish-win interfered with it.

Sweet!

Actually those paths I referred to came from that directory! So I think I was actually doing exactly that.

Do I need to run pyblish_qml.exe before Maya? Or does it know to run that, and how does it know? I’m assuming shell needs to be on my path somewhere? Or it should’ve been triggered beforehand, almost like tray?

#10

pyblish-qml should run at start-up of your machine. Once it’s running, you can forget about it. In this case, you’ll get a terminal window open, and you’ll need to leave it open.

Once things have settled, we can have a look at how to get rid of that. There are probably things you can do locally as well, I would start having a look at running an .exe as a Windows “service”.

#11

Here’s an example of running Pyblish Shell at startup, without a lingering console window.

no_console.py

# Redirect arguments to a standalone subprocess
# The parent Python process then finishes, leaving the child intact.
import sys
import subprocess
CREATE_NO_WINDOW = 0x08000000
subprocess.Popen(sys.argv[1:],
                 creationflags=CREATE_NO_WINDOW,
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT)

Usage

This will launch Pyblish QML from a terminal. You’ll notice that it doesn’t block the terminal. You’ll also notice that you can kill the console without affecting the child.

$ python no_console.py c:\path\to\pyblish_qml.exe --debug

To do this automatically, you can put the command in something like a .bat script and run it like so.

pyblish_qml.bat

set PYTHONPATH=c:\your\pyblish\libraries
python no_console.py c:\path\to\pyblish_qml.exe
exit
#12

This seems to work nicely! Is there a way to combine this with the pyblish tray functionality?

Basically so that the running pyblish_qml process acts like the tray icon (something useful for debugging or an artist knowing if its running)? Or would that not be recommended?

#13

That’s a good idea. Let’s add an executable for that to pyblish-tray.

Could you make a feature issue for it?

#14

Thanks for the issue, and here’s the result.

Pyblish Shell 1.1

Added support for Pyblish Tray, unbuffered and background mode.

Pyblish Tray

Pyblish Tray can now be started natively using the supplied executable.

$ pyblish_tray

On Windows, this will automatically run without an associated command prompt.

Unbuffered

Unbuffered mode means STDOUT is sent to the parent process immediately, which means it can be used with things like subprocess. Pyblish Tray relies on this functionality to monitor the child Pyblish QML process.

$ # Trigger unbuffered mode with the -u flag
$ pyblish_shell -u -m my_module

Background

On Windows, you can mute the native command prompt by passing --background.

$ pyblish_shell --background -m pyblish_tray

Implementation Detail

Program executables have been converted to simple scripts to simplify both maintenance and your understanding of what is actually going on under the hood.

Have a look inside the pyblish_qml.bat (Windows) or pyblish_qml (Unix) for a peek.



Potential trouble

Had some trouble running tray in background mode on Windows 7, but it still works in non background mode. Let me know how this works out for you.

$ pyblish_shell -m pyblish_tray
1 Like
#15

Thanks. Seems like you’re sometimes calling the binaries python-shell and other times pyblish-shell. Typo?

Anyway, had a quick test run and it seems to work fine. Using the pyblish_tray.bat didn’t seem to work for me (Windows 7) but running it through the no_console.py described earlier seems to work as expected.

Will have a more in-depth look soon.

#16

Where? :open_mouth:

#17

I meant the Github releases.

#18
  • pyblish-shell-1.1.0-build67-x64-setup.exe
  • python-shell-1.1.0-build67-win32-x64.zip

Ah! That’s a typo. Thanks for pointing that out.

#19

Ok did some more testing. On Windows 10 the background (and no_console.py) method seems to work fine. On Windows 7 on the other hand I can’t get tray to work correctly with --background or using no_console.py. Only when I have a remaining open command prompt window does it work correctly.

In this case working incorrectly means it launches the tray icon, but never establishes any connection. So clicking “Show” in the tray UI will not show any Pyblish QML window. The tray log only shows:

Launching Pyblish QML..

And basically stops there.
If I run it without --background I get:

Launching Pyblish QML..
Finished
Listening for output..
Starting Pyblish..
Spent 161.00 ms creating the application
Entering state: "hidden"
Entering state: "ready"
Entering state: "clean"
Entering state: "alive"
Launching virtual host..
Virtual server listening on127.0.0.1:9001
Listening on 127.0.0.1:9090
Finding available port..
Distributing new port 9001
#20

That’s the same results I am getting. Quite odd.

This works though.

$ pyblish_shell --background -m pyblish_qml