Pipeline Development Environment

Pipeline Development
#33

So my goal is to encapsulate an entire pipeline in an easy to install/update package.

The end users would be individuals or studios that would like to use the pipeline, and be able to update it without any coding knowledge.

The problem with using just git, is that I quickly run into complexity with non-pip packages like ffmpeg to install. I think conda has a solution to that, but maybe not entirely fully realised. So I think its more to do with using an existing solution instead of trying to re-invent.

During my exploration of conda, their environment handling I think is quite interesting. Currently Iā€™m playing around with that you could have environment for different situation like: pyblish-qml, pyblish-maya. With the conda environments not being python specific you can have an entire pipeline described, and with that you could have other pipelines described.

#34

I think itā€™s important to realise that this is really deep waters.

You should definitely get familiar with Conda; the developers will likely have encountered many of the issues you would face if you were going to approach this yourself. You can learn from that.

At some point however I think youā€™ll come to realise that this is an unsolved problem. That there is no universal solution. Those that have tried encapsulating it into a holistic and reusable solution end up with something like Puppet. Or Salt. People build careers with that.

What Iā€™m saying is that - rather than trying to delegate an understanding of this problem to an outside product, I would suggest you narrow down your problem, target audience and scope of your solution to the smallest possible and solve that. Because otherwise I think youā€™ll spend more time learning about something else, like a management system, then about what you are actually trying to solve.

#35

Thanks Iā€™ll keep that in mind. Iā€™m very aware of the scope of the problem, and even more aware of how much time Iā€™m spending on it :smile:

#36

Ok then. :slight_smile:

#37

Iā€™ve now done a couple of takes on using conda with a pipeline, and I think I have the beginning of a structure that can be used by other people.

In the end all pipelines that want to use conda as their package manager, should really be looking at making conda packages, but this is not an easy task and it doesnā€™t work well with remote git repositories.

The aim of the project would be to make conda easily accessible for people to setup a known environment for their pipeline. Thus this project would be used for isolating the environment and resolving dependencies.

The idea is to have a single entry point through batch and shell script, that installs Miniconda and runs any python scripts from a configuration. This seems to be the simplest and most flexible.
Another idea is to be more based in git. Here the configuration would be pointing to a git repsitory (remote or local), which would be pulled/updated and a python script from the repository would be executed. Here the projects responsibilities would to extended to keep git repositories updated, and execute a python script in a known environment.

Iā€™m very open to ideas or pros/cons, as Iā€™m still toying with the main concept.

#38

Cool! How about typing up a getting-started tutorial, so we can give it a whirl?

1 Like
#39

Got a working prototype for this; https://github.com/tokejepsen/conda-git-deployment

Its a bit rough, but should work for the example any a bit more.

Let me know what you think :slight_smile:

#40

Ok, here we go.

Setup on Ubuntu 16.04

$ apt-get update && apt-get install -y git python nano wget
$ wget -c http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh
$ chmod +x Miniconda-latest-Linux-x86_64.sh
$ ./Miniconda-latest-Linux-x86_64.sh
# ... interactive installation process ...
$ export PATH=/root/miniconda2/bin:$PATH
$ export PYTHONPATH=/root/miniconda2/lib/python2.7/site-packages

Launch

$ git clone https://github.com/tokejepsen/conda-git-deployment.git
$ cd conda-git-deployment
$ cp environment.yml.example environment.yml
$ python conda_git_deployment/update.py
# conda-git-deployment
# Already up-to-date.
# Traceback (most recent call last):
#   File "conda_git_deployment/update.py", line 21, in <module>
#     conf = utils.get_configuration()
#   File "/root/conda-git-deployment/conda_git_deployment/utils.py", line 55, in get_configuration
#     import conda.common.yaml
# ImportError: No module named common.yaml

I know you said you only had looked Windows so far, but I felt more comfortable experimenting on a machine that wouldnā€™t infect my personal setup, and Linux was the quickest thing to boot up.

Any idea where it could have gone wrong?

#41

Good point, although it all installs locally and only modifies the sessionā€™s environment instead of system.

Iā€™ll have a look at a linux shell script.

#42

Got the linux setup ready now. Please try with a fresh install.

git clone http://github.com/tokejepsen/conda-git-deployment.git
cp conda-git-deployment/environment.yml.example conda-git-deployment/environment.yml
sh conda-git-deployment/startup.sh
#43

Ok, update.

Setup on Ubuntu 16.04

apt-get update && apt-get install -y git wget bzip2
git clone https://github.com/tokejepsen/conda-git-deployment.git
cd conda-git-deployment
cp environment.yml.example environment.yml
sh startup.sh

Some notes.

  1. I first had trouble with the install, it didnā€™t finish due to some missing dependencies (in this case bzip2). It might be worth checking the install during your script and make sure it finished successfully, before moving on to the next phase.
  2. I saw you just added the environment.yml line to your previous post; this also caused a problem.

About the missing environment.yml error message; at the moment itā€™s a bare IOError, but seeing as itā€™s something related to the workflow of this project, it might be worth wrapping that up into a nicer message so that the user knows that it isnā€™t a problem with your software, but in how he uses it and that he can fix it.

Apart from this, it eventually started running, and hereā€™s where I got.

root@2658659d4132:~# sh conda-git-deployment/startup.sht.yml.exa
OS is Linux
conda-git-deployment
Already up-to-date.
Cloning into 'conda-git-example'...
remote: Counting objects: 21, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 21 (delta 3), reused 17 (delta 2), pack-reused 0
Unpacking objects: 100% (21/21), done.
Checking connectivity... done.
conda-git-example
Already up-to-date.
Executing: python /root/conda-git-deployment/repositories/conda-git-example/startup.py
root@2658659d4132:~# Fetching package metadata: ......
Solving package specifications: .........

Package plan for installation in environment /root/conda-git-deployment/linux/miniconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-env-2.6.0            |                0          502 B
    expat-2.1.0                |                0         365 KB
    icu-54.1                   |                0        11.3 MB
    jpeg-8d                    |                2         806 KB
    libffi-3.2.1               |                0          36 KB
    libgcc-5.2.0               |                0         1.1 MB
    libxcb-1.12                |                1         1.5 MB
    sqlite-3.13.0              |                0         4.0 MB
    dbus-1.10.10               |                0         2.4 MB
    glib-2.43.0                |                1         5.4 MB
    libpng-1.6.22              |                0         214 KB
    libxml2-2.9.2              |                0         4.2 MB
    python-2.7.12              |                1        12.1 MB
    enum34-1.1.6               |           py27_0          53 KB
    freetype-2.5.5             |                1         2.5 MB
    gstreamer-1.8.0            |                0         2.6 MB
    ruamel_yaml-0.11.14        |           py27_0         352 KB
    conda-4.2.12               |           py27_0         373 KB
    fontconfig-2.11.1          |                6         405 KB
    gst-plugins-base-1.8.0     |                0         3.1 MB
    qt-5.6.0                   |                1        43.9 MB
    shiboken-1.2.1             |           py27_0         883 KB
    pyside-1.2.1               |           py27_1         5.7 MB
    ------------------------------------------------------------
                                           Total:       103.1 MB

The following NEW packages will be INSTALLED:

    dbus:             1.10.10-0
    enum34:           1.1.6-py27_0
    expat:            2.1.0-0
    fontconfig:       2.11.1-6
    freetype:         2.5.5-1
    glib:             2.43.0-1
    gst-plugins-base: 1.8.0-0
    gstreamer:        1.8.0-0
    icu:              54.1-0
    jpeg:             8d-2
    libffi:           3.2.1-0
    libgcc:           5.2.0-0
    libpng:           1.6.22-0
    libxcb:           1.12-1
    libxml2:          2.9.2-0
    pyside:           1.2.1-py27_1
    qt:               5.6.0-1
    ruamel_yaml:      0.11.14-py27_0
    shiboken:         1.2.1-py27_0

The following packages will be UPDATED:

    conda:            4.0.5-py27_0 --> 4.2.12-py27_0
    conda-env:        2.4.5-py27_0 --> 2.6.0-0
    python:           2.7.11-0     --> 2.7.12-1
    sqlite:           3.9.2-0      --> 3.13.0-0

Fetching packages ...
conda-env-2.6. 100% |##################################################| Time: 0:00:00 600.38 kB/s
expat-2.1.0-0. 100% |##################################################| Time: 0:00:00 714.64 kB/s
icu-54.1-0.tar 100% |##################################################| Time: 0:00:02   4.64 MB/s
jpeg-8d-2.tar. 100% |##################################################| Time: 0:00:00   1.17 MB/s
libffi-3.2.1-0 100% |##################################################| Time: 0:00:00 187.69 kB/s
libgcc-5.2.0-0 100% |##################################################| Time: 0:00:00   1.44 MB/s
libxcb-1.12-1. 100% |##################################################| Time: 0:00:00   1.85 MB/s
sqlite-3.13.0- 100% |##################################################| Time: 0:00:01   2.62 MB/s
dbus-1.10.10-0 100% |##################################################| Time: 0:00:01   2.28 MB/s
glib-2.43.0-1. 100% |##################################################| Time: 0:00:03   1.79 MB/s
libpng-1.6.22- 100% |##################################################| Time: 0:00:00 482.80 kB/s
libxml2-2.9.2- 100% |##################################################| Time: 0:00:01   2.36 MB/s
python-2.7.12- 100% |##################################################| Time: 0:00:02   4.58 MB/s
enum34-1.1.6-p 100% |##################################################| Time: 0:00:00 316.83 kB/s
freetype-2.5.5 100% |##################################################| Time: 0:00:01   2.62 MB/s
gstreamer-1.8. 100% |##################################################| Time: 0:00:01   2.65 MB/s
ruamel_yaml-0. 100% |##################################################| Time: 0:00:00 691.20 kB/s
conda-4.2.12-p 100% |##################################################| Time: 0:00:00   5.71 MB/s
fontconfig-2.1 100% |##################################################| Time: 0:00:00 775.68 kB/s
gst-plugins-ba 100% |##################################################| Time: 0:00:01   2.93 MB/s
qt-5.6.0-1.tar 100% |##################################################| Time: 0:00:07   6.27 MB/s
shiboken-1.2.1 100% |##################################################| Time: 0:00:00   1.26 MB/s
pyside-1.2.1-p 100% |##################################################| Time: 0:00:02   2.02 MB/s
Extracting packages ...
[      COMPLETE      ]|#####################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|#####################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#####################################################################| 100%
Traceback (most recent call last):
  File "/root/conda-git-deployment/repositories/conda-git-example/startup.py", line 6, in <module>
    from PySide import QtGui
ImportError: libQtGui.so.4: cannot open shared object file: No such file or directory

What are your thoughts on the error at the bottom?

Aside from that, what is the next step? What have I accomplished by running startup.sh? Am I in an isolated environment now? If so, it might be worth having a look at modifying the PS1 environment variable to give the user some hint of being in a new environment; similar to how virtualenv does it.

$ export PS1="(awesome) $ "
(awesome) $
#44

Thank you for testing this, and giving feedback:-)

Interesting problems you encounter. What distribution of Linux are you using and how are you using it? Virtualbox or Docker?

Good point about the error message, will fix that.

Iā€™ll try and replicate with your setup. Sounds odd since the pyside package is installed.

Yeah, you are in an isolated environment with your git repo python script executed. The example just shows a dialogue with ā€œHello Worldā€.
We could definitely present a nice terminal to the user, although I was thinking that the developer would take over here and present dialogue and terminals to the user.

#45

No problem, itā€™s an interesting problem youā€™re solving and Iā€™ve seen many ways of solving it, but not with Conda so Iā€™m curious to narrow down both where it shines but also where it falls short.

I was running Ubuntu 16.04 via Docker, so any GUIs appearing would have been difficult. Iā€™d preferably experiment with it via the terminal and ideally see or write some unit tests for it under various circumstances. To test for (1) accuracy, (2) isolation and (3) performance.

Accuracy in terms of conflict resolution; something Iā€™d imagine Conda to handle implicitly in this case, in which case how does it relay errors/warnings to the user and what can he do?

Isolation in terms of, what else on my system influences the isolated environment? E.g. is my previous PYTHONPATH preserved? Why, why not.

Note these would be questions Iā€™d be looking to have answered via tests, not necessarily for you to answer here, although youā€™d be welcome to. :slight_smile:

And finally performance, is there a delay between requesting an environment and being in one. Some systems Iā€™ve seen are instant, whereas others install the environment dynamically each time, causing seconds if not minutes of delay. That particular system was using virtualenv, pip and requirements.txt to set, track and install dependencies, and Iā€™m curious to find out how conda differs.

#46

Since one of the end goals is to be able to send a zipped folder with an environment.yml to someone, it might be worth including these dependencies in the startup script.

I tried to use it myself for a project, and I quickly ran into issues where I didnā€™t nessecaryly wanted any packages to be installed, but just have an environment where I could execute terminal commands.
Next iteration the environment.yml file can be empty when it comes to git repositories, and still create an environment where pip and conda is available.

Iā€™m currently in the process of some restructuring so then environment.yml file is more in line with the standard conda environment files. Will report back when I have something workable.

#47

In this case I would make a note of what exactly your solution does better than a plain virtualenv, which does just that and include libraries that were installed previously.

$ virtualenv my_env
$ my_env\scripts\activate
(my_env) $ pip install this
(my_env) $ pip install that
(my_env) $ deactivate

Iā€™ve seen a company use this for their environment management and it could work; though it is limited to resolving dependencies for Python packages only

In fact it might be a good idea to contrast your solution to ā€œmanualā€ environment management, virtualenv, Rez and Ecosystem to make sure (to yourself and others) that you really are providing unique value.

#48

Good point:-)

I havenā€™t used virtualenv, EcoSystem or Rez only read about them, so I canā€™t say Iā€™ve done extensive research. Here is my breakdown of why I chose Conda over these other solutions:

Virtualenv

Cons:

  • Only python packages.
  • Need pip or python installed.

Rez

Pros:

  • Purely python

Cons:

  • Package are assumed to be accessible from local sources.

Ecosystem

Pros:

  • Simplish setup

Cons:

  • Small user base.

Conda

Pros:

  • User base.
    • Conda serves the web and scientific community as well, so there is a larger community.
  • pip included
  • Binary packages.
    • All batteries can be included once deployment is setup.
    • Actually any package you can build from batch and shell scripts.

Cons:

  • Missing easy deployment.
    • Users have to install conda and then exeute some commands.
    • This should be dead simple when sending to artists to use.

I may have misunderstood that other packages, but this was why I started investigating conda for deployment. Iā€™ve done a successful implementation at a studio where we used Conda for deployment but the setup was hardcoded for that studio, hence now I would like to try and make a deployment that can be used by other studios.

#49

Iā€™ve finally gotten around to get a working version for linux now.

To get conda installed and available;

apt-get update && apt-get install -y git wget bzip2
git clone https://github.com/tokejepsen/conda-git-deployment.git
cd conda-git-deployment
source startup.sh

From here you can use conda to create environments etc. But the interesting part comes when you try to use git and clone repositories to use in the environment. Here is the example;

apt-get update && apt-get install -y git wget bzip2
git clone https://github.com/tokejepsen/conda-git-deployment.git
cd conda-git-deployment
cp environment.conf.example environment.conf
source startup.sh

I have been using it on the Windows side for a while now, and its working quite nicely. You build your environment locally and you can give people an environment file to replicate the environment. A more advanced example can be found here; https://github.com/bumpybox/bumpybox-environment/blob/master/environment.yml

#50

Trying this out, got what looks like an error message, but canā€™t be sure.

$ apt-get update && apt-get install -y git wget bzip2
$ git clone https://github.com/tokejepsen/conda-git-deployment.git
$ cd conda-git-deployment
$ source startup.sh
installation finished.
...
Could not find the "environment.conf" file in "/conda-git-deployment".
Please create an environment pointer file and save it as "/conda-git-deployment/environment.conf".
You can also modify the included example "/conda-git-deployment/environment.conf.example", and rename to "/conda-git-deployment/environment.conf".

I would put either a WARNING: in front, or INFO:depending the kind of message it is. At this point Iā€™m not sure the sourcing went ok and whether I can continue.

Continuing on with your next example, I got this.

cp environment.conf.example environment.conf
source startup.sh
Executing: python /conda-git-deployment/repositories/conda-git-example/python-game/my_game.py

==================================================
Goal: Reach level 10!
You are level 0 and your health is 10.

--------------------------------------------------
Oh no! Looks like you've run into a monster. It's level is 5.
It's too strong for you, there's no way you can defeat it. Run away or fight? fight
It was a valiant effort but you were no match for this beast.
-1 level down, -4 health
You are level -1 and your health is 6.
>>>GAME OVER<<<

What is this? :slight_smile:

On workflow, how does updating of the dependencies work?

1 Like
#51

Here are a few suggestions off the top of my head.

  1. Only output things if you must. At the moment, output from each command run internally is outputted into the console, it makes it tricky for the user to know whether things are ok or erroring. Instead, you could collect output internally, and only print to the console on error. That way, you control what is printed, such as Installing.. with an animated ... and the user knows that when symbols and percentages start showing up, thereā€™s a problem.
  2. Document. The benefit of all of the alternatives listed above is that they are documented. This to me currently is a black box, and I cannot trust a black box. Especially not without a track record. Which is difficult to get without documentation.
  3. When you document, help break it down at a fundamental level. What is this project doing? To me, it looks like it is (1) cloning git repositories and (2) optionally running a command at completion. How is an environment encapsulated? Can I undo or escape an environment? What variables were set by the progam, is there any way I can inspect this? Illustrations help here, like a node graph showing where and how communication is happening.
#52

For completeness, since the initial post of this thread, Iā€™ve had some more time to think about this and have come up with a solution that worked for the particular pipeline I was developing recently.

It involvesā€¦

  1. A GitHub repository containing the pipeline.
  2. A clone on the local network from which everyone is working.
  3. A clone per development machine for making changes.

The repository is then self-contained, where using an alternative clone in-place of the global pipeline is a matter of switching exposing your clone to PYTHONPATH

Changes

One of the struggles me and Roy were having initially was, how can we keep a global pipeline versioned, while at the same time work independently within our own clones? As written in the initial post, we tried making a server location into a ā€œbareā€ repository, like GitHub was doing, in hope that this would somehow enable us to use the files while at the same time push to it.

In retrospect, we were overthinking it. What we should have done, and @BigRoy if youā€™re listening and havenā€™t changed how we did things then this may apply to you as well, was to have a regular clone exposed globally and git pull into this whenever the GitHub/GitLab repository was updated.

In this way I also managed to make a ā€œdevelopmentā€ fork of the pipeline, for testing of new features and fixes before it got pulled into the main repository.

In this particular case, using the global versus development pipeline works like this.

$ mb  # Enter global pipeline
$ mb-dev # Enter development pipeline

Where the fork is https://github.com/LegacyID1991/core and global is https://github.com/mindbender-studio/core.

Distribution

For distributing this pipeline to another freelancer or studio, I chose to bundle dependencies into a separate repository called setup. The difference here is that, like your project @tokejepsen, external Git repositories of particular versions are included in the project, and installing it is a matter of simply pulling this repo.

$ git clone --recursive https://github.com/mindbender-studio/setup mindbender-setup
$ cd mindbender-setup
$ mb

The equivalent to your environement.yml file is here, and install instructions and documentation is here.

The project is entirely standalone and should work on any machine with Windows and Maya 2016 if you want to test things out for yourself.