Publish separately VS pull newest version

No problem, glad to help.

To map our terminology towards each other, it sounds like development from the illustrations above is your /version directory, and published is your /hero. Does that sound about right?

In your case, where Pyblish comes in is right at the point where a file is moved from /version to /hero. An implementation of that might look something like this.

import shutil
import pyblish.api as pyblish

class ConformShot(pyblish.Conformer):
  def process_instance(self, instance):

    # Get current filename
    source = instance.data("path")  # /version/ep08_seq01_shot0010_block_v002.ma
    base = os.path.basename(source)  # ep08_seq01_shot0010_block_v002.ma
    name, ext = os.path.splitext(base)

    # Compute new filename
    name = name.rsplit("_", 2)[0]  # ep08_seq01_shot001
    base = name + ext  # ep08_seq01_shot001.ma

    # Produce output destination
    dest = os.path.join(source, "..", "..", "hero", base)
    dest = os.path.realpath(dest)

    # Copy it
    shutil.copy(source, dest)

Which is, as you say, a push method as it overwrites the same output each time.

For a pull version, you could add a version number each time the file is about to be written. Building on the example above.

# Compute new filename
name = name.rsplit("_", 2)[0] + "v001"  # ep08_seq01_shot001_v001
base = name + ext  # ep08_seq01_shot001.ma

In which case you increment the v001 each time.

In a pull system, you could simply assert that the latest version available is always ready to be referenced. That way, there wouldn’t be a need for any external influences. Or how come you feel the need for a database?

You can validate either way. Whenever a file is about to be shared, you validate it. Whether it ends up overwriting or incrementing a version happens after it has been deemed valid. The line inbetween development and published in the illustrations above is meant to represent the step at which validation takes place. If validation fails, the file never reaches the other side.

A version control system (VSC) is useful in “push” systems. With pulling, the version control is in your naming convention - such as v001 - and you wouldn’t normally need both.

When pushing, a VCS can act as your versioning in that a user “pushes” a new hero file, overwriting anything that already exists as usual, but in this case, the previous version of the file is stored internally within the VCS and can be reverted back to if needed.

As a side-note, VCS’s like Subversion and Perforce differ from Git in that they both push towards a central repository that everyone references against. With a decentralised VCS like Git, you could potentially benefit from overwriting a hero file where the artist is pulling the latest version on his own behalf, effectively having as much control over versions as in a pulling system. At the cost of having to store an entire project on your local hard drive.

Pushing with VCS is very common in the game development industry, but less so in the commercial and film markets. There are giants who use this approach, ILM and Pixar come to mind. They use it because there are practical benefits such as improved disk space use - as a VCS can do smart things like deduplication - which is important when a production produces massive amounts of data each day. But they still have to work around the fact, as @mkolar put it, that the next time you open your file you might not be getting what you left it as because files may have been updated without you knowing about it.

Pinging @davidmartinezanim as he has more experience with it than I.

At the end of the day, it’s a balancing act and you choose the method that fits your production the best. Some love pulling, whereas others love pushing.

2 Likes

Wow, thank you for your indept explanation Marcus.

Yes development and published sounds right. Are those the words used in larger studio too? I’d love to adapt myself to it.

In a pull system, you could simply assert that the latest version
available is always ready to be referenced. That way, there wouldn’t be a
need for any external influences. Or how come you feel the need for a
database?

You can validate either way. Whenever a file is about to be shared, you
validate it. Whether it ends up overwriting or incrementing a version
happens after it has been deemed valid. The line inbetween development and published
in the illustrations above is meant to represent the step at which
validation takes place. If validation fails, the file never reaches the
other side.

Currently I haven’t implemented any check-out system. So the artists work and save their scenes directly on the server. This means that the newest version may still be WIP and not ready to be referenced. That why I came up with the idea of artist flagging the ones that they consider ready. So they could flag v003, v012, v024 for example while their current version may already be v027. Other artist will know as well which version they can fall back to in case the newest one has problem.

About check-out system, I have an issue I don’t understand. From what I gathered it seems when artist pull files from repository, they are creating a local copy on their own drive right? They keep working and saving locally, until it’s time they publish it back to the server as a new version. What I’m worried is the path conversion. Since all textures and etc will point to the artist’s local drive while the file is checked out, how do you deal with this? Textures may be easy enough to deal with, but I can’t imagine converting paths pf FX, cache, Alembic and all the crazy stuff.

Again thank you for your great insight @marcus

I don’t think it’s so much about a local drive, but more about a separation of location. That is possibly even within the project. For example one could publish one folder up:

# working folder (development)
project/assets/character/hero/work/hero_v001.ma
project/assets/character/hero/work/hero_v002.ma
project/assets/character/hero/work/hero_v003.ma
project/assets/character/hero/work/hero_v004.ma
project/assets/character/hero/work/hero_v005_mayaCrashedAndNowINeedToFixEverything.ma
project/assets/character/hero/work/hero_v006.ma
project/assets/character/hero/work/hero_v007.ma
# published folder
project/assets/character/hero/hero_v001.ma
project/assets/character/hero/hero_v007.ma

Of course the difference between where the development file resides and the published one can be much bigger than only going one folder up.

# working folder (development)
project/development/character/hero/hero_v001.ma
project/development/character/hero/hero_v002.ma
# published folder
project/assets/character/hero/hero_v002.ma

But as long as they remain within the project boundaries you should never have to convert paths right?

Does that clarify things?

1 Like

Not sure I follow you here, didn’t you already flag which files were ready by putting them in your /hero directory?

When does a file go from /version to /hero, could you take me through the baby-steps?

Yes, you’re right, files can be copied locally (though that’s not always the case) in which case paths can potentially be changed around.

//server/project/mytexture.png
c:\local\mytexture.png

What some people do is map a common root drive.

//server/project/mytexture.png
x:\mytexture.png

For example, everyone maps X:/ to a local folder on their drive, and check files out onto it. That way, everyone references files on X:/, but everone’s X:/ is different. When it comes time to render, the render node can then checkout everything it needs to render, and it does so to X:/.

But you should know that this is relatively complicated compared with traditional versioning, which is more common and works equally well with both push and pull methods and has got less moving parts. Versioning is hard either way, unfortunately.

They might, but they don’t typically expose hard drive contents to artists, but rather wrap it up in an asset management software of sorts in which case all they see are the assets and not their parent directory. And at the end of the day, it doesn’t really matter what you call it, so long as you stay consistent and understand what it means.

I’d say the key thing that separates the two is that one is shared (public), and the other one is not (private). Shared data is typically immutable; i.e. it isn’t changed once written.

2 Likes

@panupat I was wondering whether you’ve been able to take the information provided here and make some decisions on your own.

Would love to see this topic evolve into more useful snippets of information from everyone since it’s a good topic with some many different solutions!

I can add something about what I’ve learned since the last post, about paths.

Within a working file, you can reference a file like this.

//server/project/mytexture.png
c:\local\mytexture.png

Which is an absolute path, making things immobile.

But in Maya, and hopefully others, you can also reference it like this.

$PROJECT/mytexture.png

Which then resolves to an absolute path, based on this environment variable. The great thing about this is that anyone launching Maya can set this variable beforehand to his/her local absolute path to the project, and the definition can remain the same in the Maya scene.

Machine A

$ set PROJECT=c:\hulk
$ maya

Machine B

$ export PROJECT=/projects/hulk
$ maya

For example, in The Deal, we’re all working through Dropbox which has a different root on each of our machines, e.g. c:\users\marcus\Dropbox. The environment variable PROJECTROOT is automatically set when entering the project, and referenced files then look like this.

$PROJECTROOT/assets/ben/modeling/publish/v001/model.ma

Meaning that wherever we open this file, the path will automatically resolve to wherever out Dropboxes are, never breaking any internal references.

This is absolutely essential in my eyes to any flexible and effective setup. We’ve been doing it for long time and it works great. It can get a little tricky to keep an eye on all the paths to make sure they are not absolute, but it’s definitely worth it.

Well I wouldn’t call it essential, I’ve been in plenty of successful productions without it, and the above methods are of course also equally valid.

But it’s good to know you’ve got good experiences with it. Do you know if other software also offer this?

Spontaneously I was thinking of commandeering the save function, and resolving absolute paths to environment variables where possible.

variables = {
  "\\server\projects\hulk": "$PROJECT",
  "\\server\projects\hulk\assets\Bruce": "$ASSET"
}

# psuedo
# Turn \\server\projects\hulk\mytexture.png
# Into $PROJECT\mytexture.png
for node in pm.ls(type="file"):
  for path in variables:
    node.path = node.path.replace(path, variables[path])
    

What do you think about something like that?

Not sure if you could always replace it within every save functionality. What happens upon export? Or if they use a custom exporter?

I guess it’s a perfect place for a Validator. This way the Artist learns about how the relative thing works and it can come with its own stable repair!

I think Houdini works with this functionality as well. Not sure about Fusion (did a quick test in 6.4, didn’t seem to work), but I do know Fusion has relative paths to the comp. This is similar to how Maya has relative paths from the workspace root directory (which works perfectly as well!) plus Maya makes paths automatically relative to workspace’s project root.

Houdini works out of the box like this, Nuke works as well, however it is tiny bit more involved (super easy to make paths relative to workspace, but needs a python syntax in the path to find envvar)

I’m with BigRoy. Validation might be a better option. There might of course be situations where absolute paths are desirable and tweaking the scene paths without telling the artist while saving might be a bit intrusive.

2 Likes

In our current setup we are doing a dual tracking system where there is both a Push (a published file without version, similar to Hero) and Pull (file name is always have version number), and the pushed file is always the same as the newest pulled file. Artist usually just reference the Push file, however, there is always the options to use version numbered file as well (if the pushed file cause them problem). One issue with this current setup is that there is currently no good way to store metadata (other than using a scripted node) in Maya scene file. So there is no telling of what the good version number file they last used.

I could, in theory, write a callback to update and store such data as separate file and use script node, but it seems overtly heavy handed.

Thanks for sharing!

How often would you say the pulled files, with versions, are used relative the pushed ones without version? It sounds like there is rarely any reason to go with versions when you have a choice, as you would have to handle updating your file whenever an update came out.

Most artist would go with Push file (haha, human nature I guess). However, the Pull file is there as a backup in case they need to revert back to an earlier version of the file. That happens sometimes but it is indeed pretty rare.

We don’t keep Pull file for certain type of data, though. For example, animated Alembic archives, openVDB or Render sequences. For these type of file it is usually pretty binary (they are either Approved or Retake) so there are not much ambiguity.

Some comper will use rendered elements from different takes which we really discourage. We mostly let them handle that manually as it is too difficult handle these type of edge case in the proper pipeline code.

Ah, that makes sense.

Have you considered a compromise between push and pull as described above, something like SVN/Perforce?

With SVN, you could have a versionless pushed file, but like Git, you could update it from a versioned repository. That way, updates can happen globally or locally, and a file would “automatically update” where they were referenced, like in Maya.

That’s what I had in mind as well. However, I need to think about how to track the good version of the published file for that particular working file the artist had used.

This way the artist only need to use the command like “Revert Back To Last Good Published Reference” to get back in good shape.

However this can get complex fast when the reference chain is long. To me this sounds like a job for a good dependency resolution system. (Pyblish-DAG maybe?)

Yes, quite complex.

One system I’ve seen handling such things is the “approved” versus “latest” variants of a version. In a nutshell, each version published would increment the “latest” status of a particular asset, whereas “approved” would only get updated by manually dialling in that this asset is approved.

Me and @BigRoyhave been working on such a system for a few months and I’d be happy to share it with you. In it’s current form, we have the foundation for how it could work along with a prototype in the Magenta project.

Here is the worksheet, it’s quite in-depth and potentially cryptic. It builds on previous versions that you can also find linked in there.

The idea is to bundle it up and make the proper introductions here on the forums when the time is right (which is why it’s “private” currently), but if you are interested we might be able to do that sooner.

You guys should also checkout Alembic with Git:

This works mainly with Alembic currently. Maybe your system can use this to checkout different Alembic branch where one is the Approved branch while the other one is the latest branch

Yeah that project is pretty interesting, I probably wouldn’t rely on it for pipeline work yet however since it’s so new.

If you are set on Git, then it did recently gain more support for large files which is based on an older and seemingly stable basis called “git annex”.

But I’m sceptical as to whether Git is the right approach here, since the Git methodology is “clone everything” which can be difficult when your asset library is in the terabytes range. SVN, Perforce and the like are on the opposite side of the fence, “clone what you need”, which I think might be better suited for out kind of work. But I don’t know of anyone who has actually tried and I’d be interested in seeing what benefits lie therein, as I’m sure there are many (maybe more than there are downsides!)

Very very nicely presented. I’ve been toying with versions of similar ideas (just way more scattered around projects), for a while now, but you guys are putting it into a very nice and understandable package. I took a few months break from following magenta development (no dev time on my side currently), but I’ll try to get involved more considering am very much on the same page with you with most of the concepts presented there. I do have a few points for discussion but won’t be spamming this thread with those at this point.

We’ll be trying the same system here, however I want to employ logic where the ‘pushed’ file is always the same as the ‘approved’ file, which is not necessarily the latest. It happens fairly regularly that director eventually decided to unapprove v05 and wants everyone to go back to v03. Code wise I’d run automatic action that copies file that get’s approved and overwrites the ‘pushed’ file with that. Publishing wise on artist side we then only need to deal with creating new versions of assets. The ‘push’ (or what I call ‘master’ version) is then created automatically when supervisor approves some version number. If he accidentally approves 2 of them (totally happens), then the latest one approved becomes the master by definition.

[quote=“mkolar, post:23, topic:97”]
but I’ll try to get involved more considering am very much on the same page with you with most of the concepts presented there. I do have a few points for discussion but won’t be spamming this thread with those at this point.[/quote]

You’d be welcome to, @mkolar. At the moment, I think both me and @BigRoy feel we’re onto something that works; the current status is that he’s set to run it through a couple of projects within the coming days at Colorbleed to work out any kinks.

As soon as he’s back from holidays, we should catch up on where things are at in the Magenta thread.