Diffability

[locutisParticipant]

Hey there. Great tool.

I was just wondering if it is possible to have unity write the asset file (I see that it is YAML) so that the JSON isn’t compressed?  This would make it much better for diff tools and version control.  I tried forcing ‘pretty’ to true as a test, and it look like unity escaped the entire string and not NodeCanvas itself.

Another approach could be perhaps having the asset load from json instead of unity asset files when they are not bound to a gameobject?  I’m not sure if this works within the contraints of unity.  I am happy to do the work myself, just point me in the right direction and I’ll take a whack at it

Thanks!

Colin

[locutisParticipant]

I was able to get this working with some slightly hacky techniques, see pictured.  Works well for our team (this is a diff which is easy to read)

 

[sugoidevParticipant]

What did you do?

This is a huge pain point for us too.

 

I previously made a system to convert the json inside Unity’s yaml back and forth between a diff-able format and Unity’s format. I ended up giving up because parsing yaml is demonic by itself, dealing with json inside of it back and forth was a recipe for doom.

I’ve seen talks in this forum about having an external json file with the serialized data that Unity would consume. This would help because it can be properly pretty-printed (unity has fixed lengths for yaml and will break the string at any point it sees fit, so pretty-printed json inside yaml is terrible). But that would introduce a lot of maintenance issues with having to keep the file in sync with its parent, so I’m not sure it had any progress.

[GavalakisKeymaster]

Hello there,

I am glad you like NodeCanvas and welcome!
I have tried “hacking” this a lot of times and many people have requested this, but to no success.
I would be very interested to know how you did this if you don’t care sharing with us! 🙂

Thank you in advance!

[locutisParticipant]

here you go

https://gitlab.com/gopokyo_colinn/doodads/blob/master/GraphPostprocessor.cs

[GavalakisKeymaster]

Thank you.
I will take a look at this and try to find what’s the trick and if possible implement in the JsonPrinter.cs directly.

[locutisParticipant]

Hey Gavalakis!

I’ve adjusted the JSON serialization for diffability.  I noticed you turned it off a while ago and I have another hacky way around it.  It might be able to be cleaned up and released if its reasonably stable.

It works by writing to a JSON file when a graph asset is imported (it uses a hacky path strategy).  That gets the JSON out of the asset file.  It also monitors TextAssets that use the same hacky path strategy.  When those change the graph serializedGraph field is updated to reflect the JSON file.

—————————————————–

#if UNITY_EDITOR

using System.Collections.Generic;
using UnityEditor;
using NodeCanvas.Framework;
using ParadoxNotion.Design;
using System.IO;
using System.Text.RegularExpressions;
using UnityEngine;

namespace NodeCanvas.Editor
{
///Handles post processing of graph assets
public class GraphAssetPostProcessor
{

[InitializeOnLoadMethod]
static void PreInit() {
EditorApplication.delayCall -= Init;
EditorApplication.delayCall += Init;
}

static void Init(){
#if UNITY_2019_3_OR_NEWER
ParadoxNotion.Design.AssetTracker.onAssetsImported -= OnAssetsImported;
ParadoxNotion.Design.AssetTracker.onAssetsImported += OnAssetsImported;
#endif
//we track graph assets so that we can access them on a diff thread
AssetTracker.BeginTrackingAssetsOfType(typeof(Graph));
AssetTracker.BeginTrackingAssetsOfType(typeof(TextAsset));
}

private const string SERIALIZATION_START = “#—GRAPH_START—“;
private const string SERIALIZATION_END = “#—GRAPH_END—“;
private const string IS_YAML = “%YAML”;

//Asset Tracker callback
static void OnAssetsImported(string[] paths) {
var willRefresh = false;
foreach ( var path in paths ) {
var asset = AssetDatabase.LoadAssetAtPath(path, typeof(Graph));
if ( asset is Graph graphAsset)
{
// If the graph has changed we should update the json text asset
var localPath = AssetDatabase.GetAssetPath(graphAsset) + “.json”;
var systemPath = EditorUtils.AssetToSystemPath(localPath);
var pretyJson = ParadoxNotion.Serialization.JSONSerializer.PrettifyJson(graphAsset.GetSerializedJsonData());

// has the graph actually changed?
var writeJsonFile = File.Exists(systemPath) == false || File.ReadAllText(systemPath) != pretyJson;
if (writeJsonFile)
{
willRefresh = true;
File.WriteAllText(systemPath, pretyJson);
}
}

asset = AssetDatabase.LoadAssetAtPath(path, typeof(TextAsset));
if ( asset is TextAsset textAsset) {

// If the json has changed we should update the graph asset
var localPath = AssetDatabase.GetAssetPath(textAsset);
var match = Regex.Match(localPath, @”(?.*.asset).json$”);
if (match.Success)
{
var graphPath = match.Groups[“graph”].Value;
var graph = (Graph)AssetDatabase.LoadAssetAtPath(graphPath, typeof(Graph));
if (graph != null)
{
willRefresh = true;
graph.OverwriteSerializedGraphString(textAsset.text);
EditorUtility.SetDirty(graph);
AssetDatabase.ImportAsset(graphPath); // Do I need to do this? I have no idea!
}
}
}
}
if ( willRefresh ) { EditorApplication.delayCall += () => AssetDatabase.Refresh(); }
}

///Append prety json to yaml file as comments for version control diff purposes
static void AppendPrettyJSONComments(Graph graph, string path) {
var systemPath = EditorUtils.AssetToSystemPath(path);
var lines = File.ReadAllLines(systemPath);

//not a yaml? bail out.
if ( lines.Length == 0 || !lines[0].StartsWith(IS_YAML) ) { return; }

var result = new List(lines.Length);

//clear previous. Unity actually does not keep any changes made to the file, but I don’t trust this will always be the case.
var skip = false;
for ( var i = 0; i < lines.Length; i++ ) {
var line = lines[i];
if ( line.StartsWith(SERIALIZATION_START) ) { skip = true; }
if ( skip ) { continue; }
if ( line.StartsWith(SERIALIZATION_END) ) {
skip = false;
continue;
}
result.Add(line);
}

//add new
result.Add(SERIALIZATION_START);
result.Add(“#The pretty formatted json serialization bellow is only a reference to help in version control diff. Other than that it is not used at all.”);
var pretyJson = ParadoxNotion.Serialization.JSONSerializer.PrettifyJson(graph.GetSerializedJsonData());
var split = pretyJson.Split(new string[] { System.Environment.NewLine }, System.StringSplitOptions.None);
for ( var i = 0; i < split.Length; i++ ) {
var newLine = ‘#’ + split[i];
result.Add(newLine);
}
result.Add(SERIALIZATION_END);

File.WriteAllLines(systemPath, result);
}

}
}

#endif

This allows changes to the JSON file that serialize into the main asset … or you can just ignore it because the original graph asset is the single source of truth in non-editor mode.  It seems to generally work but I’m not a very experienced Editor scripter.

Anyways, I thought I would share.

[GavalakisKeymaster]

Hey,

Thanks for sharing. How does it differ from the previous (now disabled) way of writing the json to the asset? Please note that I disabled that feature because it was risky to fiddle with the asset file directly like this and it was creating some serialization bugs. I remember the most important one was that if the same graph asset was to be used in the same graph (eg BT with 2 sub-graphs), then the subgraphs were getting cleaned up. So I suggest to test that before deciding to use it in your project to avoid unhappy accidents 🙂

Let me know. Thanks!

[locutisParticipant]

Hello,

I think the main difference is that the code doesn’t modify the yaml file directly and instead reads in and out json and lets the serialization methods you provide do the work.  I don’t 100% follow you warning regarding 2 sub-graphs – do you have an image of the situation you could share and I could test it out?

[GavalakisKeymaster]

Hello again,
Regarding the error with 2 subgraphs, hmm. I re-enabled that old feature of writing json to the asset file and tried to replicate the problem but I wasn’t able to. I don’t remember exactly the reproduction steps, but I wasn’t able to make something bad happen now. If I remember correctly, it had to do with using the same graph as sub-graph simultaneously (in the same parent graph), but I might be wrong :/ The unhappy accident was that the graph contents would simply be cleared and lost (obviously that is very bad). This is why I removed that feature back then, but I can’t really replicate anything bad right now with it enabled. I don’t think it matters to your solution since you do not change the YAML at all though.

In any case and regarding your solution; it is nice and reminds me of this -> https://nodecanvas.paradoxnotion.com/forums/topic/file-format/#post-14363

So basically the “true” json lives in the TextAsset and when that changes it updates the graph asset serialization, but the graph asset serialization is never the “true” serialization. Correct?

Let me know.
Thank you!

[locutisParticipant]

Hey there – I will certainly keep your explanation of the ‘unhappy accident’ in mind as we work with the experimental approach we have.

The graph asset actually holds the true serialization – its just that editing the json text file causes unity to parse it instead, update the graph, and reserialze it.  If you never edit the Json text file this will never occur.  I made a diagram 🙂  This is from memory so might be 100% accurate 😀

[locutisParticipant]

[attachment file=”15727″]

[Author]

Posts