Good point, although it all installs locally and only modifies the session’s environment instead of system.
I’ll have a look at a linux shell script.
Got the linux setup ready now. Please try with a fresh install.
git clone http://github.com/tokejepsen/conda-git-deployment.git
cp conda-git-deployment/environment.yml.example conda-git-deployment/environment.yml
sh conda-git-deployment/startup.sh
Ok, update.
Setup on Ubuntu 16.04
apt-get update && apt-get install -y git wget bzip2
git clone https://github.com/tokejepsen/conda-git-deployment.git
cd conda-git-deployment
cp environment.yml.example environment.yml
sh startup.sh
Some notes.
- I first had trouble with the install, it didn’t finish due to some missing dependencies (in this case
bzip2). It might be worth checking the install during your script and make sure it finished successfully, before moving on to the next phase. - I saw you just added the
environment.ymlline to your previous post; this also caused a problem.
About the missing environment.yml error message; at the moment it’s a bare IOError, but seeing as it’s something related to the workflow of this project, it might be worth wrapping that up into a nicer message so that the user knows that it isn’t a problem with your software, but in how he uses it and that he can fix it.
Apart from this, it eventually started running, and here’s where I got.
root@2658659d4132:~# sh conda-git-deployment/startup.sht.yml.exa
OS is Linux
conda-git-deployment
Already up-to-date.
Cloning into 'conda-git-example'...
remote: Counting objects: 21, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 21 (delta 3), reused 17 (delta 2), pack-reused 0
Unpacking objects: 100% (21/21), done.
Checking connectivity... done.
conda-git-example
Already up-to-date.
Executing: python /root/conda-git-deployment/repositories/conda-git-example/startup.py
root@2658659d4132:~# Fetching package metadata: ......
Solving package specifications: .........
Package plan for installation in environment /root/conda-git-deployment/linux/miniconda:
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-env-2.6.0 | 0 502 B
expat-2.1.0 | 0 365 KB
icu-54.1 | 0 11.3 MB
jpeg-8d | 2 806 KB
libffi-3.2.1 | 0 36 KB
libgcc-5.2.0 | 0 1.1 MB
libxcb-1.12 | 1 1.5 MB
sqlite-3.13.0 | 0 4.0 MB
dbus-1.10.10 | 0 2.4 MB
glib-2.43.0 | 1 5.4 MB
libpng-1.6.22 | 0 214 KB
libxml2-2.9.2 | 0 4.2 MB
python-2.7.12 | 1 12.1 MB
enum34-1.1.6 | py27_0 53 KB
freetype-2.5.5 | 1 2.5 MB
gstreamer-1.8.0 | 0 2.6 MB
ruamel_yaml-0.11.14 | py27_0 352 KB
conda-4.2.12 | py27_0 373 KB
fontconfig-2.11.1 | 6 405 KB
gst-plugins-base-1.8.0 | 0 3.1 MB
qt-5.6.0 | 1 43.9 MB
shiboken-1.2.1 | py27_0 883 KB
pyside-1.2.1 | py27_1 5.7 MB
------------------------------------------------------------
Total: 103.1 MB
The following NEW packages will be INSTALLED:
dbus: 1.10.10-0
enum34: 1.1.6-py27_0
expat: 2.1.0-0
fontconfig: 2.11.1-6
freetype: 2.5.5-1
glib: 2.43.0-1
gst-plugins-base: 1.8.0-0
gstreamer: 1.8.0-0
icu: 54.1-0
jpeg: 8d-2
libffi: 3.2.1-0
libgcc: 5.2.0-0
libpng: 1.6.22-0
libxcb: 1.12-1
libxml2: 2.9.2-0
pyside: 1.2.1-py27_1
qt: 5.6.0-1
ruamel_yaml: 0.11.14-py27_0
shiboken: 1.2.1-py27_0
The following packages will be UPDATED:
conda: 4.0.5-py27_0 --> 4.2.12-py27_0
conda-env: 2.4.5-py27_0 --> 2.6.0-0
python: 2.7.11-0 --> 2.7.12-1
sqlite: 3.9.2-0 --> 3.13.0-0
Fetching packages ...
conda-env-2.6. 100% |##################################################| Time: 0:00:00 600.38 kB/s
expat-2.1.0-0. 100% |##################################################| Time: 0:00:00 714.64 kB/s
icu-54.1-0.tar 100% |##################################################| Time: 0:00:02 4.64 MB/s
jpeg-8d-2.tar. 100% |##################################################| Time: 0:00:00 1.17 MB/s
libffi-3.2.1-0 100% |##################################################| Time: 0:00:00 187.69 kB/s
libgcc-5.2.0-0 100% |##################################################| Time: 0:00:00 1.44 MB/s
libxcb-1.12-1. 100% |##################################################| Time: 0:00:00 1.85 MB/s
sqlite-3.13.0- 100% |##################################################| Time: 0:00:01 2.62 MB/s
dbus-1.10.10-0 100% |##################################################| Time: 0:00:01 2.28 MB/s
glib-2.43.0-1. 100% |##################################################| Time: 0:00:03 1.79 MB/s
libpng-1.6.22- 100% |##################################################| Time: 0:00:00 482.80 kB/s
libxml2-2.9.2- 100% |##################################################| Time: 0:00:01 2.36 MB/s
python-2.7.12- 100% |##################################################| Time: 0:00:02 4.58 MB/s
enum34-1.1.6-p 100% |##################################################| Time: 0:00:00 316.83 kB/s
freetype-2.5.5 100% |##################################################| Time: 0:00:01 2.62 MB/s
gstreamer-1.8. 100% |##################################################| Time: 0:00:01 2.65 MB/s
ruamel_yaml-0. 100% |##################################################| Time: 0:00:00 691.20 kB/s
conda-4.2.12-p 100% |##################################################| Time: 0:00:00 5.71 MB/s
fontconfig-2.1 100% |##################################################| Time: 0:00:00 775.68 kB/s
gst-plugins-ba 100% |##################################################| Time: 0:00:01 2.93 MB/s
qt-5.6.0-1.tar 100% |##################################################| Time: 0:00:07 6.27 MB/s
shiboken-1.2.1 100% |##################################################| Time: 0:00:00 1.26 MB/s
pyside-1.2.1-p 100% |##################################################| Time: 0:00:02 2.02 MB/s
Extracting packages ...
[ COMPLETE ]|#####################################################################| 100%
Unlinking packages ...
[ COMPLETE ]|#####################################################################| 100%
Linking packages ...
[ COMPLETE ]|#####################################################################| 100%
Traceback (most recent call last):
File "/root/conda-git-deployment/repositories/conda-git-example/startup.py", line 6, in <module>
from PySide import QtGui
ImportError: libQtGui.so.4: cannot open shared object file: No such file or directory
What are your thoughts on the error at the bottom?
Aside from that, what is the next step? What have I accomplished by running startup.sh? Am I in an isolated environment now? If so, it might be worth having a look at modifying the PS1 environment variable to give the user some hint of being in a new environment; similar to how virtualenv does it.
$ export PS1="(awesome) $ "
(awesome) $
Thank you for testing this, and giving feedback:-)
Interesting problems you encounter. What distribution of Linux are you using and how are you using it? Virtualbox or Docker?
Good point about the error message, will fix that.
I’ll try and replicate with your setup. Sounds odd since the pyside package is installed.
Yeah, you are in an isolated environment with your git repo python script executed. The example just shows a dialogue with “Hello World”.
We could definitely present a nice terminal to the user, although I was thinking that the developer would take over here and present dialogue and terminals to the user.
No problem, it’s an interesting problem you’re solving and I’ve seen many ways of solving it, but not with Conda so I’m curious to narrow down both where it shines but also where it falls short.
I was running Ubuntu 16.04 via Docker, so any GUIs appearing would have been difficult. I’d preferably experiment with it via the terminal and ideally see or write some unit tests for it under various circumstances. To test for (1) accuracy, (2) isolation and (3) performance.
Accuracy in terms of conflict resolution; something I’d imagine Conda to handle implicitly in this case, in which case how does it relay errors/warnings to the user and what can he do?
Isolation in terms of, what else on my system influences the isolated environment? E.g. is my previous PYTHONPATH preserved? Why, why not.
Note these would be questions I’d be looking to have answered via tests, not necessarily for you to answer here, although you’d be welcome to. 
And finally performance, is there a delay between requesting an environment and being in one. Some systems I’ve seen are instant, whereas others install the environment dynamically each time, causing seconds if not minutes of delay. That particular system was using virtualenv, pip and requirements.txt to set, track and install dependencies, and I’m curious to find out how conda differs.
Since one of the end goals is to be able to send a zipped folder with an environment.yml to someone, it might be worth including these dependencies in the startup script.
I tried to use it myself for a project, and I quickly ran into issues where I didn’t nessecaryly wanted any packages to be installed, but just have an environment where I could execute terminal commands.
Next iteration the environment.yml file can be empty when it comes to git repositories, and still create an environment where pip and conda is available.
I’m currently in the process of some restructuring so then environment.yml file is more in line with the standard conda environment files. Will report back when I have something workable.
In this case I would make a note of what exactly your solution does better than a plain virtualenv, which does just that and include libraries that were installed previously.
$ virtualenv my_env
$ my_env\scripts\activate
(my_env) $ pip install this
(my_env) $ pip install that
(my_env) $ deactivate
I’ve seen a company use this for their environment management and it could work; though it is limited to resolving dependencies for Python packages only
In fact it might be a good idea to contrast your solution to “manual” environment management, virtualenv, Rez and Ecosystem to make sure (to yourself and others) that you really are providing unique value.
Good point:-)
I haven’t used virtualenv, EcoSystem or Rez only read about them, so I can’t say I’ve done extensive research. Here is my breakdown of why I chose Conda over these other solutions:
Virtualenv
Cons:
- Only python packages.
- Need pip or python installed.
Rez
Pros:
- Purely python
Cons:
- Package are assumed to be accessible from local sources.
Ecosystem
Pros:
- Simplish setup
Cons:
- Small user base.
Conda
Pros:
- User base.
- Conda serves the web and scientific community as well, so there is a larger community.
- pip included
- Binary packages.
- All batteries can be included once deployment is setup.
- Actually any package you can build from batch and shell scripts.
Cons:
- Missing easy deployment.
- Users have to install conda and then exeute some commands.
- This should be dead simple when sending to artists to use.
I may have misunderstood that other packages, but this was why I started investigating conda for deployment. I’ve done a successful implementation at a studio where we used Conda for deployment but the setup was hardcoded for that studio, hence now I would like to try and make a deployment that can be used by other studios.
I’ve finally gotten around to get a working version for linux now.
To get conda installed and available;
apt-get update && apt-get install -y git wget bzip2
git clone https://github.com/tokejepsen/conda-git-deployment.git
cd conda-git-deployment
source startup.sh
From here you can use conda to create environments etc. But the interesting part comes when you try to use git and clone repositories to use in the environment. Here is the example;
apt-get update && apt-get install -y git wget bzip2
git clone https://github.com/tokejepsen/conda-git-deployment.git
cd conda-git-deployment
cp environment.conf.example environment.conf
source startup.sh
I have been using it on the Windows side for a while now, and its working quite nicely. You build your environment locally and you can give people an environment file to replicate the environment. A more advanced example can be found here; https://github.com/bumpybox/bumpybox-environment/blob/master/environment.yml
Trying this out, got what looks like an error message, but can’t be sure.
$ apt-get update && apt-get install -y git wget bzip2
$ git clone https://github.com/tokejepsen/conda-git-deployment.git
$ cd conda-git-deployment
$ source startup.sh
installation finished.
...
Could not find the "environment.conf" file in "/conda-git-deployment".
Please create an environment pointer file and save it as "/conda-git-deployment/environment.conf".
You can also modify the included example "/conda-git-deployment/environment.conf.example", and rename to "/conda-git-deployment/environment.conf".
I would put either a WARNING: in front, or INFO:depending the kind of message it is. At this point I’m not sure the sourcing went ok and whether I can continue.
Continuing on with your next example, I got this.
cp environment.conf.example environment.conf
source startup.sh
Executing: python /conda-git-deployment/repositories/conda-git-example/python-game/my_game.py
==================================================
Goal: Reach level 10!
You are level 0 and your health is 10.
--------------------------------------------------
Oh no! Looks like you've run into a monster. It's level is 5.
It's too strong for you, there's no way you can defeat it. Run away or fight? fight
It was a valiant effort but you were no match for this beast.
-1 level down, -4 health
You are level -1 and your health is 6.
>>>GAME OVER<<<
What is this?
On workflow, how does updating of the dependencies work?
1 Like
Here are a few suggestions off the top of my head.
- Only output things if you must. At the moment, output from each command run internally is outputted into the console, it makes it tricky for the user to know whether things are ok or erroring. Instead, you could collect output internally, and only print to the console on error. That way, you control what is printed, such as
Installing..with an animated...and the user knows that when symbols and percentages start showing up, there’s a problem. - Document. The benefit of all of the alternatives listed above is that they are documented. This to me currently is a black box, and I cannot trust a black box. Especially not without a track record. Which is difficult to get without documentation.
- When you document, help break it down at a fundamental level. What is this project doing? To me, it looks like it is (1) cloning git repositories and (2) optionally running a command at completion. How is an environment encapsulated? Can I undo or escape an environment? What variables were set by the progam, is there any way I can inspect this? Illustrations help here, like a node graph showing where and how communication is happening.
For completeness, since the initial post of this thread, I’ve had some more time to think about this and have come up with a solution that worked for the particular pipeline I was developing recently.
It involves…
- A GitHub repository containing the pipeline.
- A clone on the local network from which everyone is working.
- A clone per development machine for making changes.
The repository is then self-contained, where using an alternative clone in-place of the global pipeline is a matter of switching exposing your clone to PYTHONPATH
Changes
One of the struggles me and Roy were having initially was, how can we keep a global pipeline versioned, while at the same time work independently within our own clones? As written in the initial post, we tried making a server location into a “bare” repository, like GitHub was doing, in hope that this would somehow enable us to use the files while at the same time push to it.
In retrospect, we were overthinking it. What we should have done, and @BigRoy if you’re listening and haven’t changed how we did things then this may apply to you as well, was to have a regular clone exposed globally and git pull into this whenever the GitHub/GitLab repository was updated.
In this way I also managed to make a “development” fork of the pipeline, for testing of new features and fixes before it got pulled into the main repository.
In this particular case, using the global versus development pipeline works like this.
$ mb # Enter global pipeline
$ mb-dev # Enter development pipeline
Where the fork is https://github.com/LegacyID1991/core and global is https://github.com/mindbender-studio/core.
Distribution
For distributing this pipeline to another freelancer or studio, I chose to bundle dependencies into a separate repository called setup. The difference here is that, like your project @tokejepsen, external Git repositories of particular versions are included in the project, and installing it is a matter of simply pulling this repo.
$ git clone --recursive https://github.com/mindbender-studio/setup mindbender-setup
$ cd mindbender-setup
$ mb
The equivalent to your environement.yml file is here, and install instructions and documentation is here.
The project is entirely standalone and should work on any machine with Windows and Maya 2016 if you want to test things out for yourself.