Me and @BigRoy have recently been looking into setting up a shared pipeline environment suitable for internal development and version control. But we ran into issues, so the below is a summary of how far we got.
Intro
Work in progress
From a 10,000 meter perspective, we’re asking ourselves.
- How do others do it?
- Is Git a good candidate, or is something like Subversion better suited?
- Should we version control external libraries?
- What does the general workflow look like when deploying new features to production, in a safe manner?
1. Setting the bar
For this to work, we figure it be best if there was a “testing ground”-type of area into which the development of new features or bug fixes were put in preparation for deployment into a live production environment.
We’ll call these two areas “development” and “production” environments, where there can be many development environments, but only one for production.
Once up and running, the goal would be to provide an environment within which software could be launched in either “production” or “development” mode.
2. Organisation
Similar to how Pyblish is being developed and hosted on GitHub, there would be one “master” repository used for production, and various forks for development.
-
studio/Production(master)-
studio/Development(fork) -
marcus/Development(fork) -
roy/Development(fork)
-
Production and Development are always available internally and assumed to exist as far as pipeline tools are concerned. The production environment contains code that has first been tested in the Development environment, whereas other forks represents an individuals “sandbox” environment, some of which may or may not eventually get merged into Development.
On disk, it may look something along these lines.
$ cd z:\pipeline
$ tree
development
└── python
├── core
│ ├── __init__.py
│ └── model.py
├── maya
└── tools
production
└── python
├── core
├── maya
└── tools
Through the power of Git, the state of each environment may be “tagged” with a version, each version encapsulating a particular set of requirements for the next release until eventually merged with the Production repository where changes become live.
In practice, this means that whenever a new release is made, the previous one is still there and can be reverted back to. It also means there is room for scheduled releases with a fixed set of expected changes for inclusion in each release.
3. External software
We’ll also want to utilise external libraries and frameworks, such as Pyblish. This is where things got difficult for us, so I’ll talk about the long way around here in an attempt at possibly optimising it once a greater understanding has been reached.
As we can’t expect too much from externals, such as whether it utilises submodules with sane remotes, we’ll need to keep them as far away as possible from internals.
On disk, it may look something along these lines.
$ cd z:\pipeline
$ tree
development
production
externals
production
development
└── python
├── core
│ ├── __init__.py
│ └── model.py
├── maya
└── tools
Where the externals are kept outside of the version controlled internal pipeline, but retain similarities in terms of how it laid out on disk; in development and production directories respectively.
During production, external libraries are taken from the /production directory whereas the equivalent development directory is used during testing of new releases, similar to the development environment used internally.
The difference here is how libraries are updated.
The non-external libraries are cloned, pulled and merged internally whereas external libraries are merely deleted and copied over as-is. History is left to their official development environments, such as GitHub.
Any internal development of an external library then takes place either completely outside of the internal pipeline or within the development area where changes may get pushed to an external source such as GitHub and later merged.
External production libraries can then either pull individually from the official source, or from the development area.
4. Practical examples
Let’s consider a few practical scenarios, from an artists perspective.
4.1 Running Software
For example, running Autodesk Maya under Windows.
set_production_environment.bat
maya_2016.bat
Under the hood, a series of environment variables are set, most importantly the the PYTHONPATH from which Python packages are to be loaded at the launch of any tool within Maya.
For example, the userSetup.py of Maya might look something along these lines.
from pipeline.maya.tools import instance_creator
instance_creator.install()
In this case, the instance_creator of the production environment is automatically picked up due to it being the first item within the PYTHONPATH.
Running the development environment could look like this.
$ set_development_environment.bat
$ maya_2016.bat
4.2 Feature requests, bug fixes
There comes a time when an artist requests a new feature or a fix for a recurring problem, the ideal workflow might be something like this.
- A ticket/an issue is submitted
- Developer clones production and implements solution
- Developer merges potential solution into the development area
- The artist tries solution via the development area
- If the solution is accepted, it is merged with production
- Otherwise, repeat 2-4 until accepted.
This workflow implies a tight connection between development and production; the two should generally be equal, apart from temporal situations like the one above.
The development area is also where beta-testing could occur, possibly in dedicated branches of development such that an immediate fix could be merged with production as soon as possible.
Perhaps this is where something like Git Flow comes in?
4.3 Personal sandbox
In addition to running within either Production or Development, it may sometimes be useful to run within your own personal sandbox. An environment only you have access to and can both break without affecting others and maintain code only relevant to you anyway, such as personal scripts or software configurations.
- Clone production
- Implement personal solutions
- Run software with your environment
$ set_custom_environment.bat c:\users\marcus\git\pipeline
Validating custom pipeline..
Pipeline valid
$ maya_2016.bat
This setup assumes that your custom pipeline is based on (“subclassed” from?) the production pipeline such that the surrounding infrastructure can make assumptions about it.
5. Hosting
Ideally, none of this should ever leak outside of the internals of a company. But certain outside vendors provide services that can be hard to reproduce internally, such as GitHub, and sometimes having remote access is a good thing.
For the above workflow to function, there needs to be a server into which code may be pushed. In fact there needs to be several servers, one per environment, but primarily two; production and development.
There needs to be a concept of “forking” a repository along with “cloning”. Both of which are native to Git, but less intuitive from the command-line when compared with GitHub where these things are visual.
We’ll need “issue tracking” and “tagging”. Issues are where artists submits a detailed report of either a feature of bug such that it may at one point be implemented, whereas tagging is how known-to-work editions of a repository is marked.
5.1 Options
- Manual
- GitHub
- BitBucket
- GitLab
