Maya2020 python-2 pyblish running python-3 pyblish-qml gui env problem

enno.s · September 4, 2020, 3:53pm

Hi,

we have hit a minor issue with using pyblish in maya-2020 and a external python-2 process running pyblish-qml.

Background

We wanted to make the pyblish-qml ui work in maya instead of pyblish-lite with out having to build PyQt5 for python-2.

Problem

The problem is that the python-3 pyblish-qml process starts with a ‘not-so-clean’ PYTHONPATH that is still coming from the maya python-2 pyblish-qml,

that confuses the python-3 pyblish-qml.

Here is such an error.


Could not import runpy module

ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

raise ImportError('This package should not be accessible on Python 3. '

File ".../python-2.7.17/site-packages/reprlib/__init__.py", line 7, in <module>

from reprlib import recursive_repr as _recursive_repr

File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/collections/__init__.py", line 27, in <module>

from collections import deque

File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/contextlib.py", line 5, in <module>

from contextlib import contextmanager

File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/importlib/util.py", line 14, in <module>

import importlib.util

File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/runpy.py", line 15, in <module>

What we did as a workaround was to replace the python-3 execuable with a bootstrap.sh that cleans the PYTHONPATH inside and points to the correct python-3 version of the pyblish-qml only.

before wihout cleaning PYTHONPATH (without a bootstrap.sh):


'.../pyqt/5.15.0/python-3.8.1/site-packages',

'.../python-3.8.1/site-packages', # pip install of pyblish

'.../python-2.7.17/site-packages',# pip install of pyblish

#...some_maya_script_dirs ...

'.../pyqt/5.15.0/python-3.8.1/site-packages/PyQt5'

# hard setting PYTHONPATH (with a bootstrap.sh)


'.../pyqt/5.15.0/python-3.8.1/site-packages'

'.../python-3.8.1/site-packages'

Most of the code that seems to be responsible for this seems to be in

https://github.com/pyblish/pyblish-qml/blob/master/pyblish_qml/ipc/server.py

pretty much here where the python-2 environment get inherited to the python-3 process

github.com

pyblish/pyblish-qml/blob/3c293468cdc6c390d29f956e473fd55a44b95c5d/pyblish_qml/ipc/server.py#L209




    if IS_WIN32 and is_embedded:
        # This will prevent an embedded Python
        # from opening an external terminal window.
        kwargs["creationflags"] = CREATE_NO_WINDOW


    print("Targets: {0}".format(", ".join(targets)))
    kwargs["args"].append("--targets")
    kwargs["args"].extend(targets)


    self.popen = subprocess.Popen(**kwargs)


def stop(self):
    try:
        return self.popen.kill()
    except OSError as e:
        # Assume process is already dead
        print(e)


def wait(self):
    return self.popen.wait()

Would be most interested if anybody else was running into this ?

Would that classify as a bug or is this somehow desired behaviour ?

Kind Regards

Enno

marcus · September 4, 2020, 4:55pm

Hello and welcome to the forums @enno.s!

Let’s walk through a few troubleshooting steps.

1. Vanilla Maya

Launch vanilla Maya, from the Autodesk installation directory, with nothing but pyblish_qml and pyblish_base on your PYTHONPATH.

At this point pyblish_qml.show() should complain that it cannot start without Qt.

import pyblish_qml
pyblish_qml.show()
# Error?

What happens?

2. Explicit QML variables

Rather than having PyQt5 on your global PYTHONPATH, or have Maya aware of anything related to Python 3, you can tell Pyblish QML explicitly about their location and leave Maya out of it entirely.

Launch the same Maya, this time with 2 new environment variables, see below.

export PYBLISH_QML_PYTHON_EXECUTABLE=/path/to/python3
export PYBLISH_QML_PYQT5=/path/to/pyqt5
/opt/autodesk/maya2020/bin/maya # Or wherever it was installed

There should be a PyQt5/ Python package at the PYBLISH_QML_PYQT5 path
E.g. /path/to/pyqt5/PyQt5/__init__.py

import pyblish_qml
pyblish_qml.show()
# Success?

What happens?

Based on this, it should hopefully become clear what the error was, else we can go from there.

enno.s · September 7, 2020, 8:38am

Hi Marcus,

thanks for the prompt reply.

exactly. I get this error:

import pyblish_qml
pyblish_qml.show()

import pyblish_qml
pyblish_qml.show()


>>>Already installed, uninstalling..
>>>Pyblish QML shutdown successful.
>>>Setting up Pyblish QML in Maya
>>>Setting up Pyblish QML in Maya
>>>uninstalling..
>>>Installing..
>>># pyblish_qml.host : Headless host # 
>>>Using Python @ '/usr/bin/python'
>>>Using PyQt5 @ 'None'
>>>Targets: default
>>># pyblish_qml.host : Success. QML server available as pyblish_qml.api.current_server() # 
>>>File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
>>>Traceback (most recent call last):
>>>ImportError: No module named PyQt5
>>>from PyQt5 import (
>>>File "/mnt/ps-storage01/appdepot/dev_packages/enno.schluender/tmp/pyblish_qml/vendor/Qt5.py", line 42, in <module>
>>>from .vendor.Qt5 import QtCore
>>>File "/mnt/ps-storage01/appdepot/dev_packages/enno.schluender/tmp/pyblish_qml/util.py", line 10, in <module>
>>>from . import util, compat, control, settings, ipc
>>>File "/mnt/ps-storage01/appdepot/dev_packages/enno.schluender/tmp/pyblish_qml/app.py", line 12, in <module>
>>>from . import app
>>>File "/mnt/ps-storage01/appdepot/dev_packages/enno.schluender/tmp/pyblish_qml/__main__.py", line 6, in <module>
>>>exec code in run_globals
>>>File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
>>>"__main__", fname, loader, pkg_name)

Pyblish_qml launches fine with only those variables set

# path to pyblish, and pyblish_qml python-2 site-packages
export PYTHONPATH=/mnt/ps-storage01/appdepot/dev_packages/enno.schluender/tmp
export PYBLISH_QML_PYTHON_EXECUTABLE=/mnt/ps-storage01/appdepot/packages/python/3.8.1/python-3.8.1/bin/python
export PYBLISH_QML_PYQT5=/mnt/ps-storage01/appdepot/dev_packages/enno.schluender/pyqt/5.15.0/python-3.8.1/site-packages/PyQt5

However once we start using PYTHONPATH along side with PYBLISH_QML_PYTHON_EXECUTABLE and PYBLISH_QML_PYQT5
or even the pyblish_qml.api.register_python_executable pyblish.api.register_gui it inherits the PYTHONPATH env to the python-3 executable and python-3 gets confused about the import paths (which also contain python-2 packages from pyblish and other packages…)

This is why the only workaround seems to be cleaning the PYTHONPATH before launching the external python-3 pyblish_qml dialoge ?!

marcus · September 7, 2020, 8:40am

What happens specifically? I’d like to try and reproduce it on my end, could you walk me through it? Starting from where it works with only the PYBLISH_* variables set.

enno.s · September 7, 2020, 9:10am

hey sure,

let me try to make something re-creatable myself.

Also I have to correct my previous post.
The example you asked for also includes PYTHONPATH next to PYBLISH_QML_PYTHON_EXECUTABLE and PYBLISH_QML_PYQT5 of course

enno.s · September 7, 2020, 10:06am

Hey so from this original error:

enno.s:

Could not import runpy module ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted. raise ImportError('This package should not be accessible on Python 3. ' File ".../python-2.7.17/site-packages/reprlib/__init__.py", line 7, in <module> from reprlib import recursive_repr as _recursive_repr File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/collections/__init__.py", line 27, in <module> from collections import deque File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/contextlib.py", line 5, in <module> from contextlib import contextmanager File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/importlib/util.py", line 14, in <module> import importlib.util File "/app/packages/python/3.8.1/python-3.8.1/lib/python3.8/runpy.py", line 15, in <module

it appears that the pip python-2 environment that we used to install pyblish and pyblish_qml into is already a bit convoluted. It includes the reprlib that is causing the error. So the problem seems more to be on our side.

I can still imagine how a similar problem could arise, if somebody pip installs pyblish into a broader python-2 site-packages env.

So, I would still suggest it could be useful to optionally clean the PYTHONPATH before this line perhaps

github.com

pyblish/pyblish-qml/blob/3c293468cdc6c390d29f956e473fd55a44b95c5d/pyblish_qml/ipc/server.py#L209




    if IS_WIN32 and is_embedded:
        # This will prevent an embedded Python
        # from opening an external terminal window.
        kwargs["creationflags"] = CREATE_NO_WINDOW


    print("Targets: {0}".format(", ".join(targets)))
    kwargs["args"].append("--targets")
    kwargs["args"].extend(targets)


    self.popen = subprocess.Popen(**kwargs)


def stop(self):
    try:
        return self.popen.kill()
    except OSError as e:
        # Assume process is already dead
        print(e)


def wait(self):
    return self.popen.wait()

what do you think ?