Jade Source Control using Git

[JohnBeaufoy]

Following the Jade Source Control Seminar in early 2016, I left with an impression that several Dev Teams were on the lookout for better tooling.

One approach to address this is to use industry standard software to do some of the heavy lifting, so I’ve recently had a go at getting Jade to play nicely with Git.

Below is a youtube demo on how far I’ve taken it. Hopefully there’s enough here for Developers to gauge whether something like this would be a meaningful improvement to how they currently work.

Feedback and interest will play a big part in determining whether this idea is worth pursuing, most likely as an open source project.

http://tiny.cc/jade_git_concept_video

Comments please!

[murray]

Hi John,
Git integration with Jade is definitely a worthy cause and something that must happen for Jade to keep current as a toolset.
I will find some time soon, John, to check out your work. I am curious to see how you have approached it, as Jade does not lend itself easily to Git integration due to using a database repository instead of a filesystem based repository.
We use Git for all our other toolsets (Java, .NET, Swift, etc.) and it really hurts to not have the capability in Jade.
Any Jade Git integration, however, must be robust and handle full merging of schema changes, not just code.

[allistar]

Hi John,
At first I was sceptical on how appropriate a tool like git would be at tracking changes to a JADE environment but after watching your video I am left very impressed and intrigued at the possibilities. Well done! Changing to such a system would be a paradigm shift for the development team I work with where we rely on Jade deltas in a single master development environment plus bespoke tools of our own to manage sequential releases.

The benefits I see are:
- git is awesome! We use is heavily for javascript development and features like branching and tagging are very powerful.
- we would be able to move from a single multi-user development environment to many single user systems. The advantage of this is less disruption by other developers. The disadvantage of this is that we're not running code that is testing things like contention. Of course, we'd have to work out how such a move would effect our Jade licence costs
- moving to a particular branch point becomes easier. As in your video we could "checkout" a tag from the last release.

I see quite a few dangers too, some of which will be difficult to solve:
- we already have a stable and robust deployment strategy, changing it would be resource expensive, we'd need a compelling reason to do so.
- merging or checking out other branches would inevitably result in reorgs having to be done. It would be quite common for these to fail as the checkout system is only dealing with the code and not the data. How would this be handled? Quite often structural changes are either preceded or followed by data massage or initialisation scripts. How would that work? Checking out a branch that removes a property would see all data for that property being lost.
- having a single development environment makes it easier for a larger team to work in unison, especially when you have many developers working on the same or related projects.
- it would need to handle Jade interfaces and packages, and my experience of these is only pain, especially when it comes to changing them.

The way we manage source amongst a team of developers is one piece in the development lifecycle, all of the pieces fit together. This would need to be considered in conjunction with development, peer review, testing and most importantly deployment,

I will discuss this with the rest of my team when we can find a breathing point in our current workload to see if there's value here. It's definitely an interesting proposal.

Thanks very much!
Regards,
Allistar.

[BeeJay]

John,

Quite a slick looking integration with the changes made in the IDE being simultaneously reflected in the file system. As you say in the video, there's plenty of potential gotchas and more tricky scenarios to flesh out but it's certainly an impressive looking start. Out of interest, are you using the development security dll hooks, utilising transaction tracing, or using some other mechanism to keep the file system version of the class/method definitions updated as/when you make the changes using the Jade IDE?

Cheers,
BeeJay.

[JohnBeaufoy]

@Murray
I’ll look forward to any further comments you have, in particular, whether what you use today is “good enough”.
I haven’t used Git for long, and never used it commercially, so experiences from those in the know would certainly be appreciated!

[JohnBeaufoy]

@BeeJay
Re: how are the files being updated.

Currently it uses the security dll hook, but my hack is currently neither atomic nor performant at this point... So work to do there, but one of the easier problems to crack.

There’s actually two alternate approaches to writing to files that I've come across, and cohorts of mine have prototypes to prove it:

Storing the git info as objects in Jade rather than files, by way of a pluggable backend. Means you'd never get out of sync with your Jade system.

Using a virtual file system. This seemingly innocuous post is actually quite compelling for the few that have seen it in action!

[JohnBeaufoy]

@Allistar

Re: Whether your team would use it
Indeed our current tooling and processes are mature/ingrained now too, so it would need to be compelling. Possibly an easier business case for those who haven’t built as much of their own tooling.

Re: Single environment/unison
Pros and cons, reorgs in dev environments are a killer too.
As it stands the approach wouldn’t prevent several users from coding in the same environment, you’d likely just have to forgo some features and work more regimented eg. committing in unison, no accuracy in who changed what.


Re: Difficult points to solve

Inbuilt vs Existing Deployment functionality
Maybe a combination, so native support up to a point, so being able to run a mix of jcfs, scm loads and scripts to allow for loading into test and managed live environments. But additionally to provide the work files/API for some other smarter & more specialised tools, such as your own & JCM to take over?

Data migrations
You’re right, this will be tough to crack, there’s many scenarios to consider.
I have this theory to try though, store specifics (so anything that cant be inferred) as part of the commit info. Lets say moveFwd and moveBack migration scripts, so when you change branches, you moveBack down to the common ancestor, then moveFwd to the new point.

Support for all types of Jade constructs
You mention interfaces & packages, agreed.
Definitely need to fail gracefully if the load fails, because we all know they do, and not always Jades fault either!
As for supporting all types of Jade constructs, that seems like a great area to put out to the wider community to help with, ie. we all pick 2
Also have an eye on the JadePlant for this, they’re looking at round tripping a type of extract, so we might be able to tap into that.

Jade Forms
This wasn’t mentioned, but I expect will be one of the most complex areas to crack. You’ve seen the ddbs, how we merge those… much to think about on this one!

Git or Mercurial
Git seems the more popular, but have recently been reading about the latter. Git infers what has happened, where (as I understand it) you can tell Hg you’ve done a rename vs an add/remove. For a code file you could live your tool not getting this right, but for a jade data class that could be a disaster!

Thanks.

[Jason]

Great effort so far John. Not being able to integrate with version control systems has always been a disadvantage of Jade, so any efforts in this area are great.

Some initial thoughts I have are:
[*]Git is a source control for text (i.e. code). One of the interesting things about Jade is that it integrates code and database. I think version controlling the database changes will be the hardest problem, but there are techniques for doing this for databases such as SQL Server. For example, the posts linked to in https://blog.codinghorror.com/get-your- ... n-control/ explain one way of doing it. Perhaps we could leverage some of these ideas?
[*]Do the files contain enough information for a full deployment? I.e. could you integrate the Git repository with a CI system that could deploy the changes to a Jade database and run unit tests on every commit?

Great job, this looks really promising!

[JohnBeaufoy]

@Jason
Thanks for the link, I've given that a read and it looks helpful, so will certainly refer to if we decide to progress! I suspect a big part of the dev/test cycle with this would revolve around curtailing ways deployments break based on advanced IDE changes (add,amend,rename,move,delete combinations)

We haven’t given CI a lot of thought yet to be honest, but the general idea of sucking in a set of marked changes into an environment [say all features ready for test] seems like a common requirement, and not too difficult once fundamentals were sorted. I'd suspect some supporting info would need to be carried at a commit level as far as special deploy instructions goes, which may be a vehicle for other file info too?

[murray]

Hi John,
I finally took the time to give your presentation some serious consideration. Bravo! You have really hit the nails on the head, point by point. You have taken this much further than my expectation. I'm impressed. It is a well considered approach to a difficult problem.

Jade is the only tool our team is using now that does not have git integration. We are using the "git-flow" branching model and it works very well. I thought I might have more to add and advice to give, but you have comprehensively covered all the main points. I really see the database reorgs as one of the main areas of difficulty.

Leaving the underlying implementation for now (being a prototype), the concept you have presented is exactly what is needed. I can see that you have already put a big effort in to get it this far. A robust implementation would be a major benefit. It's a real shame that the Jade plant has not seen the need to provide such functionality.

Our team is committed to using best-in-class tools and we can justify paying for such (e.g. we have licenses for Resharper Ultimate for c#). This would fall into the same category for us (i.e. essential add-on productivity tool). However, I understand your concern that the Jade community is a small one which makes it hard to have a profitable business case to develop such a tool. Please keep us updated on any progress.